distilbert-base-uncased

This model is a fine-tuned version of distilbert/distilbert-base-uncased on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0002

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 15

Training results

Training Loss	Epoch	Step	Validation Loss
0.0273	0.2907	500	0.0062
0.0049	0.5814	1000	0.0024
0.0033	0.8721	1500	0.0020
0.0014	1.1628	2000	0.0009
0.001	1.4535	2500	0.0007
0.0008	1.7442	3000	0.0009
0.0011	2.0349	3500	0.0011
0.0003	2.3256	4000	0.0012
0.0008	2.6163	4500	0.0008
0.0006	2.9070	5000	0.0010
0.0006	3.1977	5500	0.0009
0.0002	3.4884	6000	0.0008
0.0005	3.7791	6500	0.0005
0.0003	4.0698	7000	0.0005
0.0002	4.3605	7500	0.0003
0.0004	4.6512	8000	0.0015
0.0004	4.9419	8500	0.0008
0.0002	5.2326	9000	0.0002
0.0003	5.5233	9500	0.0003
0.0002	5.8140	10000	0.0002
0.0002	6.1047	10500	0.0003
0.0001	6.3953	11000	0.0002
0.0001	6.6860	11500	0.0002
0.0001	6.9767	12000	0.0003
0.0	7.2674	12500	0.0002
0.0	7.5581	13000	0.0009
0.0001	7.8488	13500	0.0005
0.0002	8.1395	14000	0.0007
0.0001	8.4302	14500	0.0007
0.0001	8.7209	15000	0.0006
0.0001	9.0116	15500	0.0005
0.0	9.3023	16000	0.0007
0.0001	9.5930	16500	0.0005
0.0003	9.8837	17000	0.0004
0.0	10.1744	17500	0.0004
0.0002	10.4651	18000	0.0003
0.0	10.7558	18500	0.0003
0.0	11.0465	19000	0.0004
0.0	11.3372	19500	0.0004
0.0	11.6279	20000	0.0002
0.0	11.9186	20500	0.0002
0.0	12.2093	21000	0.0003
0.0	12.5	21500	0.0003
0.0	12.7907	22000	0.0004
0.0	13.0814	22500	0.0002
0.0001	13.3721	23000	0.0002
0.0001	13.6628	23500	0.0002
0.0	13.9535	24000	0.0002
0.0	14.2442	24500	0.0002
0.0	14.5349	25000	0.0002
0.0	14.8256	25500	0.0002

Framework versions

• Transformers 4.41.2
• Pytorch 2.3.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1