roberta-large-finetuned-lora-captures

This model is a fine-tuned version of roberta-large on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.4657
• Accuracy: 0.9264

Model description

Captures classification including all keyword list categories except hacking/cybercrime, sexual assault, and counter-radicalisation. This is a LoRA model based on Roberta-large.

Intended uses & limitations

For Safeguarding & Wellbeing. There was insufficient data for self-harm, so this was resampled. Adult Content and Bullying was labelled using Llama Guard, but the prompt may be more permissive than allowed by teachers.

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 4
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.2608	0.9994	772	0.3877	0.8888
0.3173	2.0	1545	0.3443	0.8932
0.2885	2.9994	2317	0.2995	0.9161
0.2566	4.0	3090	0.2884	0.9163
0.1908	4.9994	3862	0.3115	0.9140
0.1973	6.0	4635	0.2891	0.9186
0.1071	6.9994	5407	0.2913	0.9218
0.1177	8.0	6180	0.3057	0.9212
0.1775	8.9994	6952	0.3390	0.9184
0.0994	10.0	7725	0.3260	0.9218
0.08	10.9994	8497	0.3303	0.9264
0.1041	12.0	9270	0.3738	0.9209
0.0633	12.9994	10042	0.3629	0.9271
0.0253	14.0	10815	0.3967	0.9239
0.0625	14.9994	11587	0.4285	0.9246
0.0627	16.0	12360	0.4360	0.9244
0.0551	16.9994	13132	0.4430	0.9267
0.0545	18.0	13905	0.4695	0.9251
0.0434	18.9994	14677	0.4622	0.9271
0.021	19.9871	15440	0.4657	0.9264

Framework versions

• PEFT 0.11.1
• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.0