rumodernbert-nli

This model is a fine-tuned version of deepvk/RuModernBERT-base on the cointegrated/nli-rus-translated-v2021 dataset. It achieves the following results on the evaluation set:

• Loss: 0.5055
• Accuracy: 0.8143
• Macro F1: 0.8045
• Mean Roc Auc: 0.9351
• Roc Auc Entailment: 0.9496
• Roc Auc Contradiction: 0.9487
• Roc Auc Neutral: 0.9071

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 16
• eval_batch_size: 32
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.06
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Macro F1	Mean Roc Auc	Roc Auc Entailment	Roc Auc Contradiction	Roc Auc Neutral
2.2104	0.0123	500	1.0678	0.4379	0.3603	0.5997	0.6448	0.5677	0.5868
1.5496	0.0245	1000	0.7990	0.6590	0.6391	0.8146	0.8588	0.8359	0.7492
1.3165	0.0368	1500	0.7175	0.7100	0.6866	0.8577	0.8929	0.8815	0.7985
1.2338	0.0491	2000	0.6983	0.7160	0.6927	0.8672	0.9033	0.8898	0.8084
1.1754	0.0613	2500	0.7087	0.7241	0.7076	0.8708	0.8986	0.9002	0.8135
1.1110	0.0736	3000	0.6735	0.7313	0.7159	0.8796	0.9114	0.9033	0.8242
1.1363	0.0859	3500	0.6352	0.7442	0.7293	0.8881	0.9147	0.9095	0.8402
1.1236	0.0981	4000	0.6343	0.7435	0.7261	0.8868	0.9141	0.9071	0.8393
1.0685	0.1104	4500	0.6201	0.7544	0.7383	0.8960	0.9220	0.9155	0.8505
1.1451	0.1227	5000	0.6254	0.7515	0.7284	0.8950	0.9249	0.9150	0.8452
1.0986	0.1349	5500	0.5993	0.7610	0.7512	0.9012	0.9275	0.9163	0.8598
1.0415	0.1472	6000	0.6263	0.7549	0.7313	0.8992	0.9249	0.9181	0.8548
1.0306	0.1595	6500	0.6624	0.7402	0.7137	0.8984	0.9240	0.9190	0.8522
1.0294	0.1717	7000	0.6017	0.7693	0.7534	0.9077	0.9289	0.9240	0.8704
1.0892	0.1840	7500	0.6379	0.7562	0.7328	0.9042	0.9264	0.9213	0.8648
1.0483	0.1963	8000	0.5756	0.7693	0.7520	0.9091	0.9298	0.9266	0.8709
1.0151	0.2085	8500	0.5726	0.7742	0.7633	0.9107	0.9313	0.9269	0.8739
0.9694	0.2208	9000	0.6317	0.7681	0.7525	0.9087	0.9312	0.9243	0.8706
1.0105	0.2331	9500	0.5735	0.7751	0.7643	0.9107	0.9312	0.9246	0.8764
0.9759	0.2454	10000	0.5718	0.7778	0.7693	0.9116	0.9352	0.9240	0.8756
1.0269	0.2576	10500	0.5584	0.7820	0.7691	0.9146	0.9355	0.9303	0.8782
1.0052	0.2699	11000	0.5409	0.7868	0.7762	0.9183	0.9380	0.9336	0.8833
0.9736	0.2822	11500	0.5483	0.7864	0.7707	0.9185	0.9377	0.9343	0.8835
0.9673	0.2944	12000	0.5449	0.7869	0.7768	0.9189	0.9393	0.9335	0.8840
0.9870	0.3067	12500	0.5487	0.7857	0.7737	0.9184	0.9366	0.9345	0.8841
0.9597	0.3190	13000	0.5490	0.7883	0.7795	0.9196	0.9370	0.9342	0.8876
0.9885	0.3312	13500	0.5532	0.7864	0.7764	0.9203	0.9392	0.9352	0.8865
0.9791	0.3435	14000	0.5332	0.7940	0.7853	0.9224	0.9400	0.9377	0.8894
0.9304	0.3558	14500	0.5508	0.7930	0.7790	0.9231	0.9400	0.9374	0.8919
0.9576	0.3680	15000	0.5403	0.7954	0.7842	0.9241	0.9408	0.9375	0.8940
0.9085	0.3803	15500	0.5327	0.7943	0.7860	0.9233	0.9396	0.9396	0.8907
0.9422	0.3926	16000	0.5296	0.7986	0.7900	0.9257	0.9432	0.9394	0.8944
0.9444	0.4048	16500	0.5108	0.8016	0.7905	0.9265	0.9445	0.9394	0.8957
0.8995	0.4171	17000	0.5236	0.8014	0.7932	0.9253	0.9420	0.9412	0.8928
0.9130	0.4294	17500	0.5190	0.8010	0.7907	0.9265	0.9427	0.9408	0.8960
0.9136	0.4416	18000	0.5214	0.8033	0.7945	0.9277	0.9442	0.9418	0.8971
0.8579	0.4539	18500	0.5147	0.8039	0.7940	0.9286	0.9449	0.9414	0.8996
0.8914	0.4662	19000	0.5307	0.7969	0.7912	0.9276	0.9431	0.9408	0.8988
0.8881	0.4784	19500	0.5401	0.7962	0.7895	0.9236	0.9440	0.9386	0.8883
0.8726	0.4907	20000	0.5130	0.8023	0.7956	0.9289	0.9459	0.9410	0.8998
0.8909	0.5030	20500	0.5075	0.8038	0.7955	0.9291	0.9454	0.9431	0.8989
0.8748	0.5152	21000	0.5080	0.8073	0.7970	0.9301	0.9468	0.9435	0.9000
0.8879	0.5275	21500	0.5117	0.8029	0.7952	0.9281	0.9429	0.9426	0.8988
0.8578	0.5398	22000	0.5001	0.8081	0.7983	0.9308	0.9469	0.9457	0.8998
0.8965	0.5520	22500	0.4930	0.8095	0.7991	0.9315	0.9478	0.9443	0.9023
0.8542	0.5643	23000	0.5220	0.8059	0.7968	0.9297	0.9449	0.9428	0.9014
0.8808	0.5766	23500	0.5017	0.8060	0.7960	0.9313	0.9468	0.9440	0.9030
0.8487	0.5888	24000	0.5146	0.8100	0.8011	0.9317	0.9458	0.9460	0.9035
0.8643	0.6011	24500	0.5091	0.8073	0.7981	0.9300	0.9464	0.9420	0.9017
0.8851	0.6134	25000	0.4869	0.8136	0.8052	0.9334	0.9483	0.9468	0.9050
0.9011	0.6256	25500	0.4874	0.8141	0.8061	0.9343	0.9491	0.9465	0.9072
0.8509	0.6379	26000	0.4911	0.8114	0.7984	0.9342	0.9500	0.9474	0.9051
0.8263	0.6502	26500	0.4925	0.8150	0.8069	0.9353	0.9502	0.9480	0.9078
0.8017	0.6624	27000	0.5023	0.8118	0.8015	0.9331	0.9484	0.9465	0.9043
0.8267	0.6747	27500	0.5126	0.8109	0.8040	0.9336	0.9475	0.9472	0.9062
0.8766	0.6870	28000	0.4923	0.8141	0.8061	0.9351	0.9494	0.9473	0.9086
0.8348	0.6992	28500	0.5225	0.8099	0.8019	0.9295	0.9456	0.9471	0.8957
0.8355	0.7115	29000	0.5055	0.8143	0.8045	0.9351	0.9496	0.9487	0.9071

Framework versions

• Transformers 5.8.1
• Pytorch 2.11.0+cu130
• Datasets 5.0.0
• Tokenizers 0.22.2