MSc_llama2_finetuned_model_secondData5

This model is a fine-tuned version of meta-llama/Llama-2-7b-chat-hf on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7187

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• _load_in_8bit: False
• _load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: bfloat16
• load_in_4bit: True
• load_in_8bit: False

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• training_steps: 250

Training results

Training Loss	Epoch	Step	Validation Loss
3.9919	1.36	10	3.6771
3.369	2.71	20	2.9923
2.6302	4.07	30	2.2344
1.9467	5.42	40	1.7496
1.5893	6.78	50	1.5028
1.2919	8.14	60	1.1706
0.9447	9.49	70	0.8988
0.8096	10.85	80	0.8443
0.745	12.2	90	0.8025
0.6904	13.56	100	0.7733
0.6546	14.92	110	0.7539
0.6267	16.27	120	0.7387
0.5954	17.63	130	0.7316
0.5799	18.98	140	0.7256
0.5596	20.34	150	0.7228
0.5432	21.69	160	0.7215
0.5389	23.05	170	0.7176
0.5234	24.41	180	0.7175
0.518	25.76	190	0.7189
0.5122	27.12	200	0.7177
0.5036	28.47	210	0.7185
0.5049	29.83	220	0.7191
0.5041	31.19	230	0.7195
0.5028	32.54	240	0.7188
0.4973	33.9	250	0.7187

Framework versions

• PEFT 0.4.0
• Transformers 4.38.2
• Pytorch 2.3.1+cu121
• Datasets 2.13.1
• Tokenizers 0.15.2