llama_finetune_race_20

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.4534

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 5
• num_epochs: 20
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.2478	1.0	150	1.4180
0.7183	2.0	300	1.4867
0.5553	3.0	450	1.6282
0.6139	4.0	600	1.8190
0.2089	5.0	750	1.9858
0.1597	6.0	900	2.1239
0.1706	7.0	1050	2.2716
0.1075	8.0	1200	2.4046
0.0903	9.0	1350	2.5014
0.0755	10.0	1500	2.5739
0.0686	11.0	1650	2.7048
0.0547	12.0	1800	2.6282
0.0495	13.0	1950	2.8231
0.0455	14.0	2100	3.0115
0.0415	15.0	2250	3.0576
0.0436	16.0	2400	3.2100
0.048	17.0	2550	3.3084
0.0396	18.0	2700	3.3839
0.0402	19.0	2850	3.4183
0.036	20.0	3000	3.4534

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.0+cu121
• Datasets 2.13.1
• Tokenizers 0.14.1