modernbert-CGEdit-AAE_sv3cg_final

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.8995

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.1
• num_epochs: 40

Training results

Training Loss	Epoch	Step	Validation Loss
3.6176	1.0	231	0.9080
3.6313	2.0	462	0.9049
3.6073	3.0	693	0.9038
3.6226	4.0	924	0.9023
3.6097	5.0	1155	0.9021
3.5765	6.0	1386	0.9020
3.5584	7.0	1617	0.9011
3.5297	8.0	1848	0.9007
3.5115	9.0	2079	0.9017
3.4401	10.0	2310	0.9008
3.5154	11.0	2541	0.9000
3.5245	12.0	2772	0.9008
3.5288	13.0	3003	0.9005
3.5021	14.0	3234	0.8999
3.5240	15.0	3465	0.9003
3.5122	16.0	3696	0.8999
3.5032	17.0	3927	0.8997
3.5365	18.0	4158	0.8995
3.5470	19.0	4389	0.8997
3.4434	20.0	4620	0.8996
3.5303	21.0	4851	0.8994
3.5203	22.0	5082	0.8997
3.5465	23.0	5313	0.8994
3.5335	24.0	5544	0.8996
3.5333	25.0	5775	0.8994
3.5339	26.0	6006	0.8995
3.5429	27.0	6237	0.8995
3.5261	28.0	6468	0.8995

Framework versions

• Transformers 5.0.0
• Pytorch 2.5.1+cu121
• Tokenizers 0.22.1