Edit model card

fresh-2-layer-arc2000-distill-of-fresh-2-layer-mmlu_EVAL_mmlu

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 232.3996
  • Accuracy: 0.3238

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0005
  • train_batch_size: 32
  • eval_batch_size: 32
  • seed: 321
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 500
  • training_steps: 5000

Training results

Training Loss Epoch Step Validation Loss Accuracy
No log 1.59 100 200.6664 0.258
No log 3.17 200 214.1454 0.284
No log 4.76 300 217.4835 0.298
No log 6.35 400 230.0120 0.354
109.2674 7.94 500 246.2496 0.332
109.2674 9.52 600 229.0414 0.34
109.2674 11.11 700 225.5762 0.338
109.2674 12.7 800 235.3921 0.324
109.2674 14.29 900 229.8624 0.346

Framework versions

  • Transformers 4.34.0.dev0
  • Pytorch 2.0.1+cu117
  • Datasets 2.14.5
  • Tokenizers 0.14.0
Downloads last month
3
Inference API
Inference API (serverless) does not yet support transformers models for this pipeline type.