Hugging Face's logo

Utshav
/
Llama3-70b-Instruct-extractor-adaptor

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files Community
Llama3-70b-Instruct-extractor-adaptor / README.md
Utshav's picture
Utshav
Utshav/Llama2-70b-Instruct-finetuned
71ab30d verified
preview code
|
raw
history blame contribute delete
No virus
4.03 kB
metadata
license: other
library_name: peft
tags:
  - trl
  - sft
  - generated_from_trainer
base_model: meta-llama/Meta-Llama-3-70B-Instruct
datasets:
  - generator
model-index:
  - name: Llama2-70b-Instruct-finetuned
    results: []

Llama2-70b-Instruct-finetuned

This model is a fine-tuned version of meta-llama/Meta-Llama-3-70B-Instruct on the generator dataset. It achieves the following results on the evaluation set:

  • Loss: 0.6518

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2.5e-05
  • train_batch_size: 1
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 0.03
  • training_steps: 500

Training results

Training Loss Epoch Step Validation Loss
1.4651 0.0305 10 1.4614
1.2857 0.0610 20 1.3825
1.3614 0.0915 30 1.3106
1.1613 0.1220 40 1.2354
1.1483 0.1524 50 1.1536
1.1123 0.1829 60 1.0586
1.0873 0.2134 70 0.9631
0.8711 0.2439 80 0.8951
0.8709 0.2744 90 0.8485
0.8406 0.3049 100 0.8152
0.8181 0.3354 110 0.7901
0.8111 0.3659 120 0.7741
0.7284 0.3963 130 0.7599
0.7409 0.4268 140 0.7474
0.6901 0.4573 150 0.7374
0.6622 0.4878 160 0.7287
0.6912 0.5183 170 0.7212
0.7425 0.5488 180 0.7156
0.7135 0.5793 190 0.7095
0.7233 0.6098 200 0.7040
0.7314 0.6402 210 0.6985
0.659 0.6707 220 0.6916
0.6884 0.7012 230 0.6870
0.6891 0.7317 240 0.6845
0.6736 0.7622 250 0.6813
0.6487 0.7927 260 0.6784
0.5908 0.8232 270 0.6755
0.6864 0.8537 280 0.6728
0.6581 0.8841 290 0.6688
0.6816 0.9146 300 0.6667
0.6503 0.9451 310 0.6648
0.6625 0.9756 320 0.6626
0.6392 1.0061 330 0.6616
0.6319 1.0366 340 0.6613
0.6228 1.0671 350 0.6613
0.5918 1.0976 360 0.6606
0.6028 1.1280 370 0.6589
0.6563 1.1585 380 0.6569
0.6154 1.1890 390 0.6556
0.5797 1.2195 400 0.6545
0.6137 1.25 410 0.6538
0.6174 1.2805 420 0.6533
0.5981 1.3110 430 0.6528
0.5793 1.3415 440 0.6526
0.5626 1.3720 450 0.6523
0.5864 1.4024 460 0.6520
0.5874 1.4329 470 0.6519
0.6221 1.4634 480 0.6518
0.5727 1.4939 490 0.6517
0.5776 1.5244 500 0.6518

Framework versions

  • PEFT 0.10.1.dev0
  • Transformers 4.41.0.dev0
  • Pytorch 2.1.1+cu121
  • Datasets 2.19.0
  • Tokenizers 0.19.1