MSc_llama3_finetuned_model_secondData

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7622

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• _load_in_8bit: False
• _load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: bfloat16
• load_in_4bit: True
• load_in_8bit: False

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• training_steps: 250

Training results

Training Loss	Epoch	Step	Validation Loss
3.7647	1.36	10	3.3602
2.8	2.71	20	2.0480
1.5819	4.07	30	1.2852
1.1832	5.42	40	1.1025
1.0318	6.78	50	1.0150
0.9674	8.14	60	0.9718
0.8975	9.49	70	0.9348
0.8375	10.85	80	0.8912
0.7851	12.2	90	0.8685
0.728	13.56	100	0.8443
0.6804	14.92	110	0.8038
0.6123	16.27	120	0.7684
0.5536	17.63	130	0.7314
0.4922	18.98	140	0.6943
0.4738	20.34	150	0.7095
0.4467	21.69	160	0.7344
0.4452	23.05	170	0.7397
0.4258	24.41	180	0.7332
0.4179	25.76	190	0.7436
0.4105	27.12	200	0.7373
0.4081	28.47	210	0.7596
0.4005	29.83	220	0.7552
0.4001	31.19	230	0.7652
0.393	32.54	240	0.7612
0.4016	33.9	250	0.7622

Framework versions

• PEFT 0.4.0
• Transformers 4.38.2
• Pytorch 2.3.1+cu121
• Datasets 2.13.1
• Tokenizers 0.15.2