ElegAI-GPT2

This model is trained on a custom dataset containing knowledge of everything I have learned in the field of generative AI up to now. It is based on the GPT-2 architecture. It achieves the following results on the evaluation set:

• Loss: 6.0431

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 16

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	115	7.7538
No log	2.0	230	6.7783
No log	3.0	345	6.4315
No log	4.0	460	6.2719
7.2074	5.0	575	6.1580
7.2074	6.0	690	6.1347
7.2074	7.0	805	6.0930
7.2074	8.0	920	6.0586
5.7064	9.0	1035	6.0425
5.7064	10.0	1150	6.0486
5.7064	11.0	1265	6.0434
5.7064	12.0	1380	6.0360
5.7064	13.0	1495	6.0272
5.2244	14.0	1610	6.0327
5.2244	15.0	1725	6.0443
5.2244	16.0	1840	6.0431

Framework versions

• Transformers 4.30.0
• Pytorch 2.2.1+cu121
• Datasets 2.19.0
• Tokenizers 0.13.3