G0514BMA5b

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.5764

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 80
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
-0.1378	0.09	10	-0.4063
-0.7754	0.18	20	-1.3256
-1.8887	0.27	30	-2.7074
-3.4996	0.36	40	-4.6101
-5.6334	0.45	50	-7.0234
-8.1253	0.54	60	-9.6448
-10.8034	0.63	70	-12.2239
-13.2442	0.73	80	-14.4267
-15.137	0.82	90	-15.8906
-16.2451	0.91	100	-16.5978
-16.7446	1.0	110	-16.9186
-17.0061	1.09	120	-17.0987
-17.1516	1.18	130	-17.2214
-17.2598	1.27	140	-17.3048
-17.3215	1.36	150	-17.3631
-17.3805	1.45	160	-17.4073
-17.4204	1.54	170	-17.4427
-17.455	1.63	180	-17.4700
-17.4781	1.72	190	-17.4923
-17.501	1.81	200	-17.5113
-17.5135	1.9	210	-17.5260
-17.5294	1.99	220	-17.5379
-17.5391	2.08	230	-17.5475
-17.5526	2.18	240	-17.5551
-17.5589	2.27	250	-17.5616
-17.5631	2.36	260	-17.5665
-17.5685	2.45	270	-17.5701
-17.5727	2.54	280	-17.5724
-17.5698	2.63	290	-17.5743
-17.5709	2.72	300	-17.5755
-17.5732	2.81	310	-17.5761
-17.5761	2.9	320	-17.5764
-17.5794	2.99	330	-17.5764

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0