Ronysalem/Bert-Multiclass-Classification

bert-finetuned-sem_eval-english

This model is a fine-tuned version of bert-base-uncased onMulti-Label Classification Dataset . It achieves the following results on the evaluation set:

• Loss: 0.1673
• F1: 0.8389
• Roc Auc: 0.8999
• Accuracy: 0.7046

Model description

This model is a BERT base uncased model fine-tuned for multi-label classification of research papers into 6 categories: Computer Science, Physics, Mathematics, Statistics, Quantitative Biology, and Quantitative Finance. It classifies papers based on their title and abstract text.

Intended uses & limitations

This model can be used to automatically tag research papers with relevant categories based on the paper's title and abstract. It works best on academic papers in quantitative research fields. Performance may be lower on papers from other domains or with very short abstracts.

Training and evaluation data

The model was trained on a dataset of ~15,000 research paper abstracts labeled with one or more of 6 category tags:

• Computer Science
• Physics
• Mathematics
• Statistics
• Quantitative Biology
• Quantitative Finance
• The training data includes papers from arXiv and peer-reviewed journals.

The model was evaluated on a held-out test set of ~3,000 labeled research paper abstracts drawn from the same distribution as the training data.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	F1	Roc Auc	Accuracy
0.1857	1.0	2098	0.1924	0.8143	0.8825	0.6760
0.1586	2.0	4196	0.1673	0.8389	0.8999	0.7046
0.1194	3.0	6294	0.1777	0.8361	0.8982	0.6989
0.0975	4.0	8392	0.1958	0.8312	0.8932	0.6946
0.0695	5.0	10490	0.2113	0.8315	0.8957	0.6918

Framework versions

• Transformers 4.35.2
• Pytorch 2.1.0+cu118
• Tokenizers 0.15.0