text2odrl-llama32-3b-lora-v0

This model is a fine-tuned version of meta-llama/Llama-3.2-3B on the text_to_odrl_train dataset. It achieves the following results on the evaluation set:

• Loss: 0.1083

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 8
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use paged_adamw_32bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 3.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.1857	0.2155	50	0.1730
0.1262	0.4310	100	0.1315
0.1218	0.6466	150	0.1240
0.1198	0.8621	200	0.1193
0.1148	1.0776	250	0.1161
0.1172	1.2931	300	0.1147
0.1117	1.5086	350	0.1132
0.1081	1.7241	400	0.1120
0.1058	1.9397	450	0.1106
0.1074	2.1552	500	0.1094
0.1099	2.3707	550	0.1087
0.1054	2.5862	600	0.1083
0.1017	2.8017	650	0.1080

Framework versions

• PEFT 0.15.2
• Transformers 4.50.1
• Pytorch 2.7.1+cu126
• Datasets 3.6.0
• Tokenizers 0.21.1