Llama-31-8B_task-3_60-samples_config-4

This model is a fine-tuned version of meta-llama/Meta-Llama-3.1-8B-Instruct on the GaetanMichelet/chat-60_ft_task-3 dataset. It achieves the following results on the evaluation set:

• Loss: 0.4733

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 16
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 150

Training results

Training Loss	Epoch	Step	Validation Loss
2.4485	0.6957	2	2.4786
2.4413	1.7391	5	2.4667
2.6263	2.7826	8	2.4443
2.1438	3.8261	11	2.4112
2.3995	4.8696	14	2.3654
2.2475	5.9130	17	2.3021
2.2194	6.9565	20	2.2092
2.2976	8.0	23	2.0988
2.0386	8.6957	25	2.0090
1.8757	9.7391	28	1.8478
1.753	10.7826	31	1.6617
1.5394	11.8261	34	1.4736
1.4055	12.8696	37	1.2968
1.1544	13.9130	40	1.1374
1.0965	14.9565	43	0.9952
0.8824	16.0	46	0.8632
0.8412	16.6957	48	0.7849
0.7232	17.7391	51	0.7002
0.6275	18.7826	54	0.6459
0.6078	19.8261	57	0.6102
0.5477	20.8696	60	0.5828
0.4369	21.9130	63	0.5587
0.5025	22.9565	66	0.5396
0.5043	24.0	69	0.5226
0.3742	24.6957	71	0.5101
0.449	25.7391	74	0.5003
0.3276	26.7826	77	0.4925
0.4754	27.8261	80	0.4932
0.3724	28.8696	83	0.4876
0.4679	29.9130	86	0.4861
0.3245	30.9565	89	0.4884
0.3613	32.0	92	0.4922
0.3511	32.6957	94	0.4899
0.5275	33.7391	97	0.4931
0.3403	34.7826	100	0.4883
0.4209	35.8261	103	0.4815
0.3543	36.8696	106	0.4805
0.4115	37.9130	109	0.4767
0.3902	38.9565	112	0.4794
0.3735	40.0	115	0.4776
0.3227	40.6957	117	0.4733
0.2983	41.7391	120	0.4797
0.4421	42.7826	123	0.4791
0.3819	43.8261	126	0.4739
0.2965	44.8696	129	0.4764
0.2661	45.9130	132	0.4765
0.3827	46.9565	135	0.4778
0.3144	48.0	138	0.4797

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.1.2+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1