NumTrainEpochs10_SaveStrategiesno_reward_modeling_anthropic_hh

This model is a fine-tuned version of facebook/opt-1.3b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 3.2160
• Accuracy: 0.6289
• Train Rewards/chosen: 13.3266
• Train Rewards/rejected: -10.7412
• Train Rewards/accuracies: 0.9925
• Train Rewards/margins: 24.0678
• Train Nll Loss: 1.9271
• Train Logit Total Loss: 0.0395
• Train Logit Loss: 0.0204
• Rewards/chosen: 4.7138
• Rewards/rejected: -1.7686
• Rewards/accuracies: 0.6145
• Rewards/margins: 6.4823
• Nll Loss: 2.0087
• Logit Total Loss: 3.2131
• Logit Loss: 3.2252

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1.41e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Nll Loss	Logit Total Loss	Logit Loss
0.7879	0.11	100	0.7507	0.4845	-0.1740	-0.1876	0.4714	0.0135	6.2659	0.7498	0.6941
0.7291	0.23	200	0.7331	0.5773	-0.2697	-0.3880	0.5645	0.1184	6.0096	0.7310	0.6777
0.6843	0.34	300	0.7057	0.5876	0.2058	-0.0389	0.5754	0.2448	3.9577	0.7039	0.6710
0.6773	0.46	400	0.6950	0.5918	-0.0097	-0.2138	0.5774	0.2041	4.2789	0.6968	0.6607
0.7071	0.57	500	0.7107	0.5918	0.7447	0.5198	0.5815	0.2249	4.4170	0.7087	0.6712
0.6881	0.69	600	0.6687	0.6186	-0.8010	-1.0541	0.6028	0.2531	3.2753	0.6671	0.6408
0.6871	0.8	700	0.6847	0.5753	-1.7064	-1.9330	0.5601	0.2266	3.7264	0.6839	0.6532
0.7125	0.91	800	0.6885	0.5814	-1.4574	-1.6521	0.5734	0.1947	4.3386	0.6851	0.6482
0.62	1.03	900	0.6955	0.6103	-1.4133	-1.8571	0.5964	0.4438	3.1332	0.6958	0.6712
0.5929	1.14	1000	0.6537	0.6371	-1.9413	-2.5254	0.6226	0.5841	2.9107	0.6524	0.6296
0.5825	1.26	1100	0.6749	0.6515	0.4669	-0.0787	0.6367	0.5455	2.9364	0.6732	0.6503
0.614	1.37	1200	0.6697	0.6351	0.1785	-0.2933	0.6258	0.4718	2.9997	0.6659	0.6423
0.5528	1.49	1300	0.6553	0.6268	-1.0780	-1.6306	0.6177	0.5526	2.9051	0.6504	0.6276
0.6501	1.6	1400	0.6379	0.6412	-1.6259	-2.1203	0.6306	0.4944	2.9085	0.6351	0.6121
0.545	1.71	1500	0.6640	0.6660	-0.3375	-1.1855	0.6560	0.8480	3.0934	0.6573	0.6327
0.6653	1.83	1600	0.6379	0.6639	-1.0663	-1.6961	0.6528	0.6298	2.8475	0.6376	0.6153
0.5792	1.94	1700	0.6447	0.6577	-0.0283	-0.6093	0.6480	0.5810	3.0457	0.6420	0.6178
0.2858	2.06	1800	0.9327	0.6330	1.7576	0.2131	0.6226	1.5445	2.8731	0.9216	0.9019
0.3404	2.17	1900	0.8438	0.6144	0.9326	-0.2443	0.6024	1.1769	2.7925	0.8418	0.8221
0.2734	2.29	2000	0.9082	0.6227	1.6779	0.4268	0.6125	1.2511	2.7991	0.8986	0.8794
0.2562	2.4	2100	0.9566	0.6371	2.2122	0.5184	0.6266	1.6937	2.7729	0.9522	0.9338
0.3796	2.51	2200	0.8839	0.6351	0.7900	-0.5311	0.6218	1.3211	2.7689	0.8720	0.8528
0.2316	2.63	2300	0.8741	0.6454	2.0133	0.5784	0.6359	1.4349	2.7465	0.8633	0.8443
0.3679	2.74	2400	0.8584	0.6515	-0.8628	-2.1801	0.6379	1.3173	2.7134	0.8483	0.8294
0.3384	2.86	2500	0.9165	0.6412	-0.9835	-2.3685	0.6294	1.3850	2.7084	0.9087	0.8905
0.3595	2.97	2600	0.9173	0.6454	0.3307	-1.0129	0.6347	1.3436	2.7089	0.9049	0.8867
0.1331	3.09	2700	1.4595	0.6557	0.6119	-2.1780	0.6468	2.7900	2.6967	1.4381	1.4254
0.1464	3.2	2800	1.4234	0.6351	5.4974	2.9945	0.6258	2.5029	2.6392	1.3999	1.3874
0.137	3.31	2900	1.4612	0.6474	3.1356	0.4400	0.6363	2.6956	2.6002	1.4435	1.4318
0.1593	3.43	3000	1.7826	0.6433	3.8280	0.7687	0.6282	3.0593	2.6206	1.7676	1.7590
0.0834	3.54	3100	1.5493	0.6474	2.4447	-0.2971	0.6355	2.7418	2.6296	1.5386	1.5275
0.136	3.66	3200	1.5847	0.6495	2.6691	-0.1416	0.6375	2.8108	2.6007	1.5701	1.5597
0.0859	3.77	3300	1.7114	0.6227	0.8690	-1.9033	0.6093	2.7723	2.5630	1.6942	1.6854
0.1705	3.89	3400	1.7792	0.6268	-1.4030	-4.0698	0.6121	2.6669	2.5917	1.7786	1.7704
0.1675	4.0	3500	2.1762	0.6268	-1.5886	-5.0180	0.6133	3.4294	2.5716	2.1579	2.1537
0.0589	4.11	3600	2.3409	0.6309	1.1330	-2.8993	0.6173	4.0323	2.4949	2.3055	2.3036
0.1014	4.23	3700	2.3221	0.6268	2.6255	-1.3486	0.6125	3.9741	2.4617	2.2985	2.2969
0.0697	4.34	3800	2.4256	0.6351	2.8885	-1.2680	0.6194	4.1565	2.4613	2.4010	2.4004
0.1687	4.46	3900	2.1905	0.6433	3.3404	-1.0572	0.6347	4.3976	2.4074	2.1582	2.1556
0.0315	4.57	4000	2.3170	0.6619	2.0050	-2.4036	0.6480	4.4086	2.4112	2.2812	2.2799
0.1071	4.69	4100	2.2205	0.6454	0.9399	-3.4755	0.6379	4.4154	2.3561	2.1998	2.1983
0.1342	4.8	4200	2.2640	0.6557	10.1640	5.7216	0.6419	4.4424	2.3536	2.2410	2.2399
0.0793	4.91	4300	2.0629	0.6495	-0.6830	-4.8288	0.6327	4.1458	2.3658	2.0407	2.0374
0.0587	5.03	4400	2.3862	0.6371	3.2076	-1.4161	0.6258	4.6238	2.3529	2.3625	2.3626
0.0433	5.14	4500	2.5409	0.6454	4.9940	0.1253	0.6286	4.8687	2.3166	2.5250	2.5271
0.0506	5.26	4600	2.5949	0.6557	6.7660	1.6624	0.6395	5.1035	2.2864	2.5983	2.6014
0.0506	5.37	4700	2.7389	0.6351	7.2608	2.0917	0.6226	5.1690	2.2691	2.7197	2.7243
0.0644	5.49	4800	2.8523	0.6309	2.3756	-2.9285	0.6173	5.3041	2.2594	2.8574	2.8634
0.0714	5.6	4900	2.5013	0.6309	2.5445	-2.3571	0.6206	4.9016	2.2422	2.5045	2.5072
0.1087	5.71	5000	2.6378	0.6227	-0.0320	-5.0243	0.6113	4.9923	2.2318	2.6303	2.6344
0.0874	5.83	5100	2.8088	0.6412	5.9816	0.6049	0.6278	5.3767	2.2257	2.7811	2.7867
0.0871	5.94	5200	2.4819	0.6433	7.2347	2.1895	0.6306	5.0451	2.2034	2.4679	2.4706
0.0331	6.06	5300	2.8775	0.6268	9.8380	4.4195	0.6145	5.4184	2.1978	2.8663	2.8730
0.024	6.17	5400	2.8923	0.6433	5.1441	-0.5990	0.6306	5.7431	2.1912	2.8713	2.8781
0.0354	6.29	5500	2.7626	0.6433	-1.4206	-6.9376	0.6315	5.5170	2.1826	2.7519	2.7577
0.0289	6.4	5600	2.8423	0.6371	7.1683	1.7904	0.6246	5.3779	2.1707	2.8182	2.8248
0.0389	6.51	5700	2.9096	0.6412	2.0666	-3.5386	0.6234	5.6052	2.1672	2.9140	2.9215
0.0245	6.63	5800	2.8677	0.6495	4.5194	-1.1798	0.6347	5.6992	2.1466	2.8461	2.8532
0.0804	6.74	5900	2.9668	0.6371	5.6766	-0.3308	0.6226	6.0074	2.1468	2.9437	2.9518
0.029	6.86	6000	3.0269	0.6371	3.9285	-2.2229	0.6226	6.1514	2.1305	2.9998	3.0086
0.0438	6.97	6100	2.8192	0.6639	2.2607	-4.3102	0.6476	6.5708	2.1277	2.8101	2.8170
0.0451	7.09	6200	2.8547	0.6577	2.5219	-3.4933	0.6395	6.0152	2.1111	2.8383	2.8456
0.0761	7.2	6300	2.9610	0.6536	4.7705	-1.5571	0.6435	6.3275	2.1023	2.9370	2.9454
0.0477	7.31	6400	2.8708	0.6619	2.7809	-3.7082	0.6488	6.4891	2.0958	2.8410	2.8485
0.0449	7.43	6500	3.0901	0.6619	5.8808	-0.8822	0.6496	6.7630	2.0873	3.0685	3.0784
0.0418	7.54	6600	2.9687	0.6371	2.2079	-4.1264	0.6206	6.3343	2.0853	2.9514	2.9602
0.0473	7.66	6700	2.9895	0.6351	2.4455	-3.8039	0.6206	6.2494	2.0790	2.9705	2.9795
0.0459	7.77	6800	3.0660	0.6392	4.6892	-1.6980	0.6254	6.3872	2.0757	3.0540	3.0638
0.045	7.89	6900	3.0811	0.6474	2.9687	-3.4595	0.6347	6.4282	2.0697	3.0561	3.0661
0.0493	8.0	7000	2.9549	0.6330	3.3733	-2.8947	0.6214	6.2680	2.0679	2.9435	2.9523
0.031	8.11	7100	2.9964	0.6330	4.2065	-2.1412	0.6206	6.3477	2.0650	2.9810	2.9903
0.0196	8.23	7200	3.0962	0.6371	4.8289	-1.6916	0.6246	6.5204	2.0550	3.0800	3.0904
0.0223	8.34	7300	3.0038	0.6392	2.7990	-3.5327	0.6246	6.3317	2.0497	2.9870	2.9965
0.0629	8.46	7400	3.0349	0.6351	5.2916	-0.8920	0.6206	6.1836	2.0453	3.0076	3.0173
0.0922	8.57	7500	3.0735	0.6227	1.5229	-4.6388	0.6105	6.1617	2.0409	3.0489	3.0591
0.0302	8.69	7600	3.1279	0.6289	1.4324	-4.7615	0.6185	6.1939	2.0355	3.1060	3.1168
0.0589	8.8	7700	3.1274	0.6412	4.6809	-1.6469	0.6306	6.3279	2.0298	3.1051	3.1159
0.0389	8.91	7800	3.0308	0.6330	4.8002	-1.3492	0.6206	6.1494	2.0277	3.0129	3.0229
0.0252	9.03	7900	3.0680	0.6330	5.0212	-1.1246	0.6165	6.1458	2.0236	3.0565	3.0670
0.0652	9.14	8000	3.1190	0.6351	4.3150	-1.9926	0.6165	6.3076	2.0196	3.1234	3.1345
0.0201	9.26	8100	3.1413	0.6289	4.7573	-1.5726	0.6165	6.3299	2.0164	3.1389	3.1503
0.0443	9.37	8200	3.1135	0.6247	4.3945	-1.9119	0.6125	6.3065	2.0140	3.1029	3.1139
0.0186	9.49	8300	3.1597	0.6289	3.7131	-2.6943	0.6165	6.4074	2.0114	3.1487	3.1602
0.0352	9.6	8400	3.1513	0.6247	3.9594	-2.4902	0.6085	6.4496	2.0100	3.1409	3.1523
0.0225	9.71	8500	3.1966	0.6227	4.9750	-1.5016	0.6125	6.4766	2.0095	3.1854	3.1973
0.0385	9.83	8600	3.2165	0.6268	4.9076	-1.6079	0.6125	6.5155	2.0094	3.2082	3.2203
0.0266	9.94	8700	3.2160	0.6289	4.7138	-1.7686	0.6145	6.4823	2.0087	3.2131	3.2252

Framework versions

• Transformers 4.37.2
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.15.2