G0514HMA6H

This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -17.8450

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 80
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.885	0.09	10	-0.0170
-1.0117	0.18	20	-2.5324
-3.9238	0.27	30	-5.8063
-7.2226	0.36	40	-9.1305
-10.3592	0.45	50	-12.0206
-13.0512	0.54	60	-14.4433
-15.2088	0.63	70	-16.0571
-16.4849	0.73	80	-16.9607
-17.1295	0.82	90	-17.3344
-17.4173	0.91	100	-17.5098
-17.5475	1.0	110	-17.5970
-17.6219	1.09	120	-17.6470
-17.6632	1.18	130	-17.6764
-17.6872	1.27	140	-17.6993
-17.7061	1.36	150	-17.7169
-17.7227	1.45	160	-17.7282
-17.7359	1.54	170	-17.7437
-17.751	1.63	180	-17.7575
-17.7607	1.72	190	-17.7661
-17.7725	1.81	200	-17.7749
-17.7774	1.9	210	-17.7813
-17.7869	1.99	220	-17.7911
-17.798	2.08	230	-17.8003
-17.8094	2.18	240	-17.8108
-17.8151	2.27	250	-17.8191
-17.826	2.36	260	-17.8291
-17.8364	2.45	270	-17.8346
-17.8373	2.54	280	-17.8387
-17.8439	2.63	290	-17.8419
-17.8433	2.72	300	-17.8437
-17.8438	2.81	310	-17.8445
-17.8504	2.9	320	-17.8449
-17.8486	2.99	330	-17.8450

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0