Holmeister/Gemma_AAID_mixed_train_old

Gemma_AAID_mixed_train

This model is a fine-tuned version of google/gemma-7b on the AAID_mixed dataset. It achieves the following results on the evaluation set:

• Loss: 0.0002

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 256
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.6198	0.2023	200	0.4104
0.3557	0.4047	400	0.2979
0.235	0.6070	600	0.1811
0.1356	0.8093	800	0.0971
0.0762	1.0116	1000	0.0588
0.04	1.2140	1200	0.0382
0.0228	1.4163	1400	0.0160
0.0119	1.6186	1600	0.0078
0.0053	1.8209	1800	0.0029
0.0023	2.0233	2000	0.0015
0.0008	2.2256	2200	0.0007
0.0005	2.4279	2400	0.0003
0.0002	2.6302	2600	0.0002
0.0002	2.8326	2800	0.0002

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1