BERTModified-rawbert-finetuned-wikitext-test

This model is a fine-tuned version of distilbert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 20.9598
• Precision: 0.0387
• Recall: 0.0387
• F1: 0.0387
• Accuracy: 0.0387

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Precision	Recall	F1	Accuracy
21.2559	1.0	25	21.1094	0.0121	0.0121	0.0121	0.0121
18.2119	2.0	50	20.8262	0.0145	0.0145	0.0145	0.0145
15.7553	3.0	75	20.6929	0.0169	0.0169	0.0169	0.0169
13.4714	4.0	100	20.6579	0.0266	0.0266	0.0266	0.0266
11.7546	5.0	125	20.6640	0.0266	0.0266	0.0266	0.0266
10.1859	6.0	150	20.6235	0.0206	0.0206	0.0206	0.0206
8.9318	7.0	175	20.6836	0.0266	0.0266	0.0266	0.0266
7.8875	8.0	200	20.6899	0.0278	0.0278	0.0278	0.0278
6.9846	9.0	225	20.7376	0.0314	0.0314	0.0314	0.0314
6.2595	10.0	250	20.8058	0.0339	0.0339	0.0339	0.0339
5.6063	11.0	275	20.8430	0.0326	0.0326	0.0326	0.0326
5.1992	12.0	300	20.8778	0.0339	0.0339	0.0339	0.0339
4.72	13.0	325	20.9012	0.0326	0.0326	0.0326	0.0326
4.405	14.0	350	20.9282	0.0363	0.0363	0.0363	0.0363
4.1	15.0	375	20.9423	0.0351	0.0351	0.0351	0.0351
3.8665	16.0	400	20.9187	0.0363	0.0363	0.0363	0.0363
3.7095	17.0	425	20.9525	0.0351	0.0351	0.0351	0.0351
3.509	18.0	450	20.9430	0.0399	0.0399	0.0399	0.0399
3.5125	19.0	475	20.9929	0.0326	0.0326	0.0326	0.0326
3.4485	20.0	500	20.9752	0.0411	0.0411	0.0411	0.0411

Framework versions

• Transformers 4.39.2
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2