falcon-7b-alpaca / README.md
amztheory's picture
Model save
862dcf0 verified
metadata
license: apache-2.0
library_name: peft
tags:
  - trl
  - sft
  - generated_from_trainer
base_model: tiiuae/falcon-7b-instruct
model-index:
  - name: falcon-7b-alpaca
    results: []

falcon-7b-alpaca

This model is a fine-tuned version of tiiuae/falcon-7b-instruct on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.9657

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 6
  • eval_batch_size: 6
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 24
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss
1.3865 0.0631 50 1.4032
1.2768 0.1262 100 1.2262
1.021 0.1893 150 1.0728
1.0843 0.2524 200 1.0175
0.9927 0.3155 250 0.9966
0.9746 0.3785 300 0.9836
1.02 0.4416 350 0.9767
0.9806 0.5047 400 0.9722
1.0579 0.5678 450 0.9684
1.0075 0.6309 500 0.9670
0.9909 0.6940 550 0.9661
0.9792 0.7571 600 0.9661
0.9449 0.8202 650 0.9654
0.9268 0.8833 700 0.9656
1.0441 0.9464 750 0.9657

Framework versions

  • PEFT 0.11.1
  • Transformers 4.41.0
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1