collapse_gemma-2-27b_hs2_replace_iter2_sftsd2

This model is a fine-tuned version of google/gemma-2-27b on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.2781
• Num Input Tokens Seen: 3603012

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-06
• train_batch_size: 4
• eval_batch_size: 16
• seed: 2
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant_with_warmup
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
No log	0	0	1.1282	0
2.7288	0.0560	5	1.0428	210528
2.2631	0.1121	10	1.0614	409500
1.7035	0.1681	15	1.0857	615964
1.0224	0.2242	20	1.1604	820452
0.7612	0.2802	25	1.1895	1018848
0.5972	0.3363	30	1.1930	1215560
0.6178	0.3923	35	1.1725	1409568
0.4143	0.4483	40	1.1575	1608480
0.5734	0.5044	45	1.1651	1812632
0.4687	0.5604	50	1.1621	2009960
0.6309	0.6165	55	1.1799	2223316
0.5393	0.6725	60	1.1957	2428084
0.271	0.7285	65	1.2141	2637248
0.4383	0.7846	70	1.2308	2837024
0.2703	0.8406	75	1.2080	3032684
0.3999	0.8967	80	1.2440	3240344
0.3003	0.9527	85	1.2357	3443968

Framework versions

• Transformers 4.44.0
• Pytorch 2.4.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1