O0506TEST3

This model is a fine-tuned version of allenai/OLMo-1B on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: -4.4239

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine_with_restarts
• lr_scheduler_warmup_steps: 100
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.1005	0.09	10	-1.5637
-1.7829	0.18	20	-1.8563
-1.8732	0.27	30	-1.8844
-1.9118	0.36	40	-1.9458
-1.9739	0.45	50	-2.0139
-2.0453	0.54	60	-2.1005
-2.1436	0.63	70	-2.2061
-2.2555	0.73	80	-2.3258
-2.4047	0.82	90	-2.5182
-2.5613	0.91	100	-2.6960
-2.7428	1.0	110	-2.8828
-2.4287	1.09	120	-3.0421
-2.9323	1.18	130	-3.1984
-2.9297	1.27	140	-3.1196
-2.9262	1.36	150	1.6830
-1.7801	1.45	160	-1.1498
-2.8852	1.54	170	-3.4537
-3.2749	1.63	180	-3.6075
-3.6454	1.72	190	-3.7508
-3.813	1.81	200	-3.8830
-3.9317	1.9	210	-3.9917
-4.0157	1.99	220	-4.0920
-4.1285	2.08	230	-4.1672
-4.1735	2.18	240	-4.2356
-4.1966	2.27	250	-4.2894
-4.2903	2.36	260	-4.3295
-4.3409	2.45	270	-4.3633
-4.3074	2.54	280	-4.3890
-4.3943	2.63	290	-4.4015
-4.3886	2.72	300	-4.4165
-4.3791	2.81	310	-4.4221
-4.3777	2.9	320	-4.4237
-4.4116	2.99	330	-4.4239

Framework versions

• Transformers 4.36.0.dev0
• Pytorch 2.1.2+cu121
• Datasets 2.14.6
• Tokenizers 0.14.0