llama381binstruct_summarize_short

This model is a fine-tuned version of NousResearch/Meta-Llama-3.1-8B-Instruct on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 2.5714

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 30
• training_steps: 500

Training results

Training Loss	Epoch	Step	Validation Loss
1.7592	1.3889	25	1.5098
0.7167	2.7778	50	1.5089
0.4127	4.1667	75	1.7655
0.0811	5.5556	100	2.0770
0.0501	6.9444	125	2.2019
0.0184	8.3333	150	2.2421
0.0268	9.7222	175	2.1572
0.0206	11.1111	200	2.2244
0.0085	12.5	225	2.3922
0.0056	13.8889	250	2.3233
0.0032	15.2778	275	2.4017
0.0021	16.6667	300	2.4708
0.0021	18.0556	325	2.5064
0.0014	19.4444	350	2.5270
0.0016	20.8333	375	2.5422
0.0014	22.2222	400	2.5534
0.002	23.6111	425	2.5619
0.0015	25.0	450	2.5669
0.0013	26.3889	475	2.5706
0.0012	27.7778	500	2.5714

Framework versions

• PEFT 0.12.0
• Transformers 4.44.2
• Pytorch 2.4.0+cu121
• Datasets 3.0.0
• Tokenizers 0.19.1