roberta-large-plm-nsp-100000-1e06

This model is a fine-tuned version of roberta-large on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.2181

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
0.2638	1.0	3125	0.1423
0.1316	2.0	6250	0.1276
0.1044	3.0	9375	0.1188
0.0873	4.0	12500	0.1247
0.0751	5.0	15625	0.1357
0.0661	6.0	18750	0.1383
0.0579	7.0	21875	0.1458
0.0522	8.0	25000	0.1474
0.0482	9.0	28125	0.1673
0.0456	10.0	31250	0.1820
0.0412	11.0	34375	0.1799
0.0393	12.0	37500	0.1920
0.0361	13.0	40625	0.1941
0.0335	14.0	43750	0.2066
0.0343	15.0	46875	0.2038
0.0327	16.0	50000	0.2029
0.0307	17.0	53125	0.2096
0.0298	18.0	56250	0.2130
0.0295	19.0	59375	0.2167
0.0289	20.0	62500	0.2181

Framework versions

• Transformers 4.40.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1