polynomial_1450_7e-4_16b_w0.05

This model is a fine-tuned version of gpt2 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 3.0237

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0007
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 10
• total_train_batch_size: 160
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: polynomial
• lr_scheduler_warmup_steps: 250
• training_steps: 1450

Training results

Training Loss	Epoch	Step	Validation Loss
9.0635	0.1029	50	7.2771
6.7176	0.2058	100	6.2551
6.0127	0.3088	150	5.7232
5.5517	0.4117	200	5.3470
5.2297	0.5146	250	5.0446
4.9361	0.6175	300	4.7729
4.6976	0.7205	350	4.5588
4.497	0.8234	400	4.3733
4.3221	0.9263	450	4.1939
4.1357	1.0292	500	4.0081
3.892	1.1322	550	3.8139
3.7559	1.2351	600	3.6703
3.6297	1.3380	650	3.5671
3.5399	1.4409	700	3.4772
3.4656	1.5438	750	3.4074
3.3949	1.6468	800	3.3532
3.3297	1.7497	850	3.3031
3.2878	1.8526	900	3.2604
3.254	1.9555	950	3.2267
3.1231	2.0585	1000	3.1899
3.0568	2.1614	1050	3.1603
3.0347	2.2643	1100	3.1349
3.0197	2.3672	1150	3.1148
2.9893	2.4702	1200	3.0940
2.9801	2.5731	1250	3.0725
2.951	2.6760	1300	3.0551
2.9265	2.7789	1350	3.0397
2.9438	2.8818	1400	3.0299
2.9292	2.9848	1450	3.0237

Framework versions

• Transformers 4.40.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1