art-des-bert-large-cased

This model is a fine-tuned version of bert-large-cased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2776

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 100

Training results

Training Loss	Epoch	Step	Validation Loss
1.8053	5.19	100	1.4663
1.3212	10.39	200	1.3795
1.0223	15.58	300	1.3545
0.8991	20.78	400	1.3239
0.7579	25.97	500	1.3276
0.6554	31.17	600	1.3435
0.5786	36.36	700	1.2276
0.5386	41.56	800	1.1930
0.479	46.75	900	1.2091
0.4336	51.95	1000	1.0554
0.3776	57.14	1100	1.4044
0.3582	62.34	1200	1.1651
0.3343	67.53	1300	1.2394
0.3093	72.73	1400	1.1313
0.2952	77.92	1500	1.2107
0.2845	83.12	1600	1.2804
0.2585	88.31	1700	1.1700
0.2548	93.51	1800	1.2391
0.2581	98.7	1900	1.2776

Framework versions

• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.19.0
• Tokenizers 0.15.2