DistriBert_TPF_v9

This model is a fine-tuned version of distilbert/distilbert-base-uncased on the None dataset. It achieves the following results on the evaluation set:

• Accuracy@en: 0.8349
• F1@en: 0.8353
• Precision@en: 0.8406
• Recall@en: 0.8358
• Loss@en: 0.5675
• Loss: 0.5675

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Accuracy@en	F1@en	Precision@en	Recall@en	Loss@en	Validation Loss
3.3814	1.0	276	0.1672	0.1000	0.1207	0.1699	2.9259	2.9259
2.705	2.0	552	0.2122	0.1458	0.1464	0.2121	2.4904	2.4904
2.4079	3.0	828	0.2664	0.2101	0.2919	0.2674	2.2948	2.2948
2.1179	4.0	1104	0.3952	0.3393	0.3872	0.3941	1.9555	1.9555
1.7552	5.0	1380	0.4966	0.4568	0.4748	0.4960	1.5776	1.5776
1.4478	6.0	1656	0.5635	0.5259	0.5638	0.5640	1.3563	1.3563
1.2016	7.0	1932	0.6286	0.5985	0.6485	0.6334	1.1235	1.1235
1.0053	8.0	2208	0.6971	0.6751	0.7205	0.6954	0.9861	0.9861
0.8457	9.0	2484	0.7537	0.7466	0.7582	0.7534	0.8329	0.8329
0.7163	10.0	2760	0.7831	0.7780	0.7957	0.7832	0.7397	0.7397
0.6167	11.0	3036	0.7992	0.7962	0.8098	0.7995	0.6955	0.6955
0.5421	12.0	3312	0.7995	0.7923	0.8148	0.7996	0.6953	0.6953
0.4798	13.0	3588	0.8212	0.8209	0.8311	0.8218	0.6242	0.6242
0.4415	14.0	3864	0.8278	0.8273	0.8370	0.8284	0.6013	0.6013
0.3899	15.0	4140	0.8325	0.8331	0.8439	0.8329	0.5969	0.5969
0.3671	16.0	4416	0.8349	0.8353	0.8406	0.8358	0.5675	0.5675
0.3278	17.0	4692	0.8392	0.8399	0.8459	0.8401	0.5784	0.5784
0.3041	18.0	4968	0.8392	0.8390	0.8451	0.8399	0.5765	0.5765

Framework versions

• Transformers 4.44.2
• Pytorch 2.5.0+cu121
• Datasets 3.0.2
• Tokenizers 0.19.1