Llama-2-7b-chat-hf-finetune-SWE_90_10

This model is a fine-tuned version of meta-llama/Llama-2-7b-chat-hf on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.8121

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 3
• eval_batch_size: 3
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: reduce_lr_on_plateau
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.8511	0.9992	1231	0.9063
0.836	1.9984	2462	0.8752
0.4611	2.9976	3693	0.9398
0.3698	3.9968	4924	1.0870
0.2231	4.9959	6155	1.1993
0.2689	5.9951	7386	1.2892
0.1975	6.9943	8617	1.3711
0.1591	7.9935	9848	1.4205
0.1284	8.9927	11079	1.4248
0.1301	9.9919	12310	1.4770
0.1298	10.9911	13541	1.5282
0.1526	11.9903	14772	1.5175
0.1313	12.9894	16003	1.5433
0.0817	13.9886	17234	1.5538
0.0869	14.9878	18465	1.6030
0.0788	15.9870	19696	1.6366
0.0783	16.9862	20927	1.6603
0.0768	17.9854	22158	1.6776
0.0751	18.9846	23389	1.6930
0.0727	19.9838	24620	1.7093
0.0769	20.9830	25851	1.7225
0.0692	21.9821	27082	1.7329
0.0713	22.9813	28313	1.7415
0.0737	23.9805	29544	1.7592
0.0628	24.9797	30775	1.7804
0.0688	25.9789	32006	1.7868
0.0648	26.9781	33237	1.7914
0.0622	27.9773	34468	1.7932
0.0608	28.9765	35699	1.7962
0.0595	29.9756	36930	1.7978
0.0575	30.9748	38161	1.8001
0.1117	31.9740	39392	1.8011
0.1038	32.9732	40623	1.8036
0.0987	33.9724	41854	1.8038
0.0922	34.9716	43085	1.8065
0.0912	35.9708	44316	1.8072
0.0901	36.9700	45547	1.8079
0.0913	37.9692	46778	1.8084
0.0821	38.9683	48009	1.8091
0.0868	39.9675	49240	1.8096
0.0823	40.9667	50471	1.8101
0.0806	41.9659	51702	1.8104
0.0813	42.9651	52933	1.8108
0.0795	43.9643	54164	1.8112
0.08	44.9635	55395	1.8116
0.0783	45.9627	56626	1.8119
0.0767	46.9619	57857	1.8120
0.0778	47.9610	59088	1.8120
0.0765	48.9602	60319	1.8121
0.0753	49.9594	61550	1.8121

Framework versions

• PEFT 0.10.0
• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1