nlp_te_mlm_scibert

This model is a fine-tuned version of allenai/scibert_scivocab_uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1478

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 16
• eval_batch_size: 8
• seed: 5678
• gradient_accumulation_steps: 16
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 500
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.3828	0.9963	152	1.2566
1.3087	1.9992	305	1.2295
1.289	2.9955	457	1.2237
1.262	3.9984	610	1.2054
1.2516	4.9947	762	1.1999
1.229	5.9975	915	1.1944
1.2272	6.9939	1067	1.1880
1.2066	7.9967	1220	1.1879
1.1991	8.9996	1373	1.1807
1.1978	9.9959	1525	1.1760
1.1803	10.9988	1678	1.1724
1.1819	11.9951	1830	1.1716
1.1659	12.9980	1983	1.1731
1.1658	13.9943	2135	1.1673
1.1524	14.9971	2288	1.1669
1.1481	16.0	2441	1.1590
1.1468	16.9963	2593	1.1626
1.1361	17.9992	2746	1.1623
1.1371	18.9955	2898	1.1582
1.125	19.9984	3051	1.1540
1.1276	20.9947	3203	1.1551
1.1143	21.9975	3356	1.1518
1.118	22.9939	3508	1.1550
1.104	23.9967	3661	1.1525
1.1011	24.9996	3814	1.1483
1.1061	25.9959	3966	1.1533
1.0941	26.9988	4119	1.1473
1.0951	27.9951	4271	1.1444
1.0866	28.9980	4424	1.1462
1.089	29.9943	4576	1.1453
1.0768	30.9971	4729	1.1496
1.0744	32.0	4882	1.1493
1.0773	32.9963	5034	1.1478

Framework versions

• Transformers 4.41.2
• Pytorch 2.2.1
• Datasets 2.19.2
• Tokenizers 0.19.1