Llama-31-8B_task-2_180-samples_config-4

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-2, the GaetanMichelet/chat-120_ft_task-2 and the GaetanMichelet/chat-180_ft_task-2 datasets. It achieves the following results on the evaluation set:

• Loss: 0.7156

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 150

Training results

Training Loss	Epoch	Step	Validation Loss
1.0917	0.9412	8	1.1326
1.1392	2.0	17	1.1271
1.1572	2.9412	25	1.1161
1.0856	4.0	34	1.0970
1.0789	4.9412	42	1.0719
1.0546	6.0	51	1.0337
0.9806	6.9412	59	0.9923
0.9617	8.0	68	0.9472
0.933	8.9412	76	0.9129
0.9005	10.0	85	0.8825
0.9183	10.9412	93	0.8578
0.7739	12.0	102	0.8335
0.7649	12.9412	110	0.8165
0.8197	14.0	119	0.8009
0.7488	14.9412	127	0.7889
0.7651	16.0	136	0.7770
0.6992	16.9412	144	0.7679
0.7609	18.0	153	0.7580
0.6868	18.9412	161	0.7510
0.7077	20.0	170	0.7437
0.6862	20.9412	178	0.7379
0.6939	22.0	187	0.7319
0.6564	22.9412	195	0.7275
0.6446	24.0	204	0.7236
0.6304	24.9412	212	0.7200
0.6583	26.0	221	0.7178
0.5974	26.9412	229	0.7159
0.6266	28.0	238	0.7159
0.6081	28.9412	246	0.7156
0.5853	30.0	255	0.7165
0.54	30.9412	263	0.7200
0.5438	32.0	272	0.7231
0.5163	32.9412	280	0.7268
0.5302	34.0	289	0.7345
0.4865	34.9412	297	0.7542
0.4774	36.0	306	0.7611

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.1.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1