SIRIS-Lab/Toscana-TechDomain-classifier

Technical Domain Classifier

This model is a fine-tuned version of allenai/specter2_base for multilabel scientific domain classification aligned with ERC panel taxonomy.
It achieves the following results on the held-out test set:

• Best validation loss: 0.0372
• Micro F1: 0.74

---

Model description

This model is a fine-tuned variant of SPECTER2 (allenai/specter2_base) adapted for multilabel classification of scientific documents into S3 technical domains.

The model takes as input the title and abstract of a scientific project or publication and predicts one or more domains.
Since scientific outputs may legitimately span multiple domains, the model is trained using sigmoid activation with binary cross-entropy loss, allowing independent assignment of multiple labels.

Key characteristics

• Base model: allenai/specter2_base
• Task: multilabel document classification
• Labels: 28 S3 technical domains
• Activation: sigmoid (independent scores per label)
• Loss: BCEWithLogitsLoss
• Output: list of predicted panels with associated probabilities
• Decision threshold: 0.5 (tunable)

This model enables automatic tagging aligned with the S3 technical domains structure.

---

Intended uses & limitations

Intended uses

This model is designed for:

• Automatic assignment of S3 technical domains

The model supports:

• title only
• abstract only
• title + abstract (recommended)

Limitations

• Labels are derived from automatically annotated data
• Class imbalance may affect recall for underrepresented panels
• The model does not encode explicit hierarchical relationships between panels

---

How to use

Training and evaluation data

Training data

• Scientific documents with S3 technical domains annotations
• Inputs:
  • title
  • abstract
• Task type: multilabel classification

Dataset characteristics

Property	Value
Documents	~16k
Labels	28 panels
Input fields	Title, Abstract
Task type	Multilabel
License	Dataset-dependent

---

Training procedure

Preprocessing

• Input text constructed as:

  title + ". " + abstract

• Tokenization using the SPECTER2 tokenizer

• Maximum sequence length: 512 tokens

Model

• Base model: allenai/specter2_base
• Classification head: linear → sigmoid
• Loss function: BCEWithLogitsLoss
• Predictions: independent probability per label

Training hyperparameters

Hyperparameter	Value
Learning rate	5e-5
Train batch size	4
Eval batch size	8
Epochs	5
Weight decay	0.01
Optimizer	AdamW
Metric for best model	Micro F1

---

Training results

Epoch	Training Loss	Duration seconds
1	0.1618	234.58
2	0.0857	234.35
3	0.0575	234.92
4	0.0365	235.46
5	0.0217	236.22

Evaluation results (multilabel test set)

Panel	Precision	Recall	F1-score	Support
2D materials	0.90	0.65	0.75	71
3D printing materials	0.89	0.50	0.64	48
Additive manufacturing	0.69	0.75	0.72	79
Artificial Intelligence and machine learning	0.55	0.90	0.69	176
Big-data and analytics	0.66	0.67	0.66	218
Biomaterials, biosensors, bioactuators and bioactivators	0.76	0.78	0.77	117
Biomedical, pharmaceutical, pharmacological technologies	0.90	0.78	0.84	278
Biorobotics	0.92	0.75	0.82	59
Biotechnology for health	0.55	0.93	0.69	142
Business models and digital creativity	0.60	0.74	0.66	323
Cyber connectivity and 5G	0.77	0.79	0.78	119
Cyber-security, blockchain	0.88	0.92	0.90	120
Development of energy-saving solutions	0.63	0.76	0.69	143
Development of production and organisational processes based on intelligent sensors/devices	0.68	0.49	0.57	126
Environmental resilience and pollution	0.71	0.76	0.74	167
Exploitation of renewable energy sources	0.83	0.80	0.81	113
Genomics	0.90	0.86	0.88	42
Green propulsion technologies	0.95	0.52	0.67	69
High-performance and sustainable composite materials	0.77	0.64	0.70	181
Industrial biotechnology	0.52	0.88	0.65	58
Internet of the things and services	0.68	0.78	0.73	228
Nanomaterials and nanotechnology	0.82	0.89	0.85	266
Neuroscience	0.72	0.93	0.81	67
Nutraceutics	0.36	0.94	0.52	17
Photonic, micro and nanoelectronic solutions	0.79	0.88	0.84	259
Robotic, mechatronic and automation solutions	0.42	0.94	0.58	144
Technologies for decarbonisation	0.74	0.86	0.80	225
Technologies for the circular economy and bioeconomy	0.87	0.76	0.81	123

Overall performance

	Precision	Recall	F1-score	Support
Micro avg	0.70	0.79	0.74	2077
Macro avg	0.73	0.78	0.73	2077

---

Framework versions

• Transformers: 4.57.x
• PyTorch: 2.8.0
• Datasets: 3.x
• Tokenizers: 0.22.x