llama3-chat_50000_500

This model is a fine-tuned version of unsloth/llama-3-8b-Instruct-bnb-4bit on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.7378

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 16
• eval_batch_size: 16
• seed: 3407
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 5
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss
1.7748	0.128	100	1.4928
1.5097	0.256	200	1.4743
1.5122	0.384	300	1.4623
1.4924	0.512	400	1.4609
1.485	0.64	500	1.4526
1.4779	0.768	600	1.4511
1.4728	0.896	700	1.4446
1.4476	1.024	800	1.4512
1.3725	1.152	900	1.4558
1.3747	1.28	1000	1.4560
1.3735	1.408	1100	1.4548
1.3717	1.536	1200	1.4499
1.3694	1.6640	1300	1.4526
1.3698	1.792	1400	1.4542
1.3701	1.92	1500	1.4512
1.3004	2.048	1600	1.4977
1.1904	2.176	1700	1.5075
1.1977	2.304	1800	1.5041
1.1888	2.432	1900	1.5094
1.1885	2.56	2000	1.5024
1.1989	2.6880	2100	1.5039
1.1905	2.816	2200	1.5046
1.1914	2.944	2300	1.5077
1.0764	3.072	2400	1.6027
0.9757	3.2	2500	1.6227
0.9768	3.328	2600	1.6228
0.9795	3.456	2700	1.6225
0.9775	3.584	2800	1.6190
0.9781	3.7120	2900	1.6164
0.981	3.84	3000	1.6199
0.9812	3.968	3100	1.6254
0.8731	4.096	3200	1.7307
0.8376	4.224	3300	1.7343
0.8352	4.352	3400	1.7398
0.8429	4.48	3500	1.7357
0.8431	4.608	3600	1.7386
0.8383	4.736	3700	1.7380
0.8375	4.864	3800	1.7376
0.842	4.992	3900	1.7378

Framework versions

• PEFT 0.10.0
• Transformers 4.40.2
• Pytorch 2.3.0
• Datasets 2.19.1
• Tokenizers 0.19.1