G0514HMA5H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.7865

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 80
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.9281	0.09	10	0.0810
-0.8728	0.18	20	-2.3616
-3.7424	0.27	30	-5.5992
-6.9773	0.36	40	-8.8629
-10.1201	0.45	50	-11.8272
-12.8541	0.54	60	-14.2293
-15.0144	0.63	70	-15.8856
-16.3327	0.73	80	-16.8287
-17.0246	0.82	90	-17.2467
-17.335	0.91	100	-17.4367
-17.4797	1.0	110	-17.5384
-17.5709	1.09	120	-17.6024
-17.6217	1.18	130	-17.6413
-17.6522	1.27	140	-17.6697
-17.6777	1.36	150	-17.6893
-17.6963	1.45	160	-17.7051
-17.7096	1.54	170	-17.7187
-17.7252	1.63	180	-17.7321
-17.7353	1.72	190	-17.7430
-17.7471	1.81	200	-17.7499
-17.751	1.9	210	-17.7561
-17.7563	1.99	220	-17.7617
-17.7638	2.08	230	-17.7659
-17.7726	2.18	240	-17.7701
-17.7714	2.27	250	-17.7736
-17.7766	2.36	260	-17.7772
-17.7823	2.45	270	-17.7800
-17.7809	2.54	280	-17.7827
-17.7872	2.63	290	-17.7841
-17.7876	2.72	300	-17.7856
-17.7846	2.81	310	-17.7863
-17.7907	2.9	320	-17.7865
-17.7901	2.99	330	-17.7865

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0