pretraining4

This model is a fine-tuned version of gpt2 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.8431

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0006
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 10
• total_train_batch_size: 320
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 250
• training_steps: 3000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
9.1583	0.1830	50	7.4069
6.7299	0.3660	100	6.2135
5.9685	0.5490	150	5.6480
5.4899	0.7321	200	5.2615
5.1274	0.9151	250	4.9402
4.8039	1.0981	300	4.6237
4.4931	1.2811	350	4.3674
4.2228	1.4641	400	4.0680
3.9722	1.6471	450	3.7854
3.7567	1.8302	500	3.6085
3.5888	2.0132	550	3.4916
3.3834	2.1962	600	3.4104
3.3441	2.3792	650	3.3378
3.2791	2.5622	700	3.2787
3.2254	2.7452	750	3.2305
3.1704	2.9283	800	3.1834
3.0657	3.1113	850	3.1470
2.9908	3.2943	900	3.1220
2.9647	3.4773	950	3.0917
2.956	3.6603	1000	3.0673
2.9532	3.8433	1050	3.0451
2.9104	4.0264	1100	3.0164
2.7555	4.2094	1150	3.0038
2.761	4.3924	1200	2.9899
2.7569	4.5754	1250	2.9708
2.7543	4.7584	1300	2.9556
2.7482	4.9414	1350	2.9404
2.6217	5.1245	1400	2.9319
2.5826	5.3075	1450	2.9246
2.5911	5.4905	1500	2.9123
2.5896	5.6735	1550	2.9030
2.5835	5.8565	1600	2.8907
2.5468	6.0395	1650	2.8852
2.4098	6.2225	1700	2.8864
2.4415	6.4056	1750	2.8788
2.4343	6.5886	1800	2.8705
2.4396	6.7716	1850	2.8587
2.4435	6.9546	1900	2.8499
2.3213	7.1376	1950	2.8586
2.2896	7.3206	2000	2.8576
2.3	7.5037	2050	2.8532
2.308	7.6867	2100	2.8479
2.3162	7.8697	2150	2.8413
2.2637	8.0527	2200	2.8455
2.1797	8.2357	2250	2.8478
2.1874	8.4187	2300	2.8480
2.1949	8.6018	2350	2.8448
2.1943	8.7848	2400	2.8389
2.1926	8.9678	2450	2.8383
2.1142	9.1508	2500	2.8436
2.1062	9.3338	2550	2.8442
2.1165	9.5168	2600	2.8435
2.1119	9.6999	2650	2.8413
2.1163	9.8829	2700	2.8412
2.0926	10.0659	2750	2.8425
2.0754	10.2489	2800	2.8437
2.0622	10.4319	2850	2.8435
2.0666	10.6149	2900	2.8433
2.0754	10.7980	2950	2.8431
2.0774	10.9810	3000	2.8431

Framework versions

• Transformers 4.40.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.0
• Tokenizers 0.19.1