See axolotl config

axolotl version: 0.6.0

adapter: lora
base_model: peft-internal-testing/tiny-dummy-qwen2
batch_size: 64
bf16: true
chat_template: tokenizer_default_fallback_alpaca
datasets:
- format: custom
  path: argilla/databricks-dolly-15k-curated-en
  type:
    field_input: original-instruction
    field_instruction: original-instruction
    field_output: original-response
    format: '{instruction} {input}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
device_map: auto
eval_sample_packing: false
eval_steps: 0.1
flash_attention: true
gradient_checkpointing: true
group_by_length: true
hub_model_id: SystemAdmin123/test-repo
hub_strategy: checkpoint
learning_rate: 0.0001
logging_steps: 10
lora_alpha: 256
lora_dropout: 0.1
lora_r: 128
lora_target_linear: true
lr_scheduler: cosine
max_steps: 160.0
micro_batch_size: 7
model_type: AutoModelForCausalLM
num_epochs: 10000
optimizer: adamw_bnb_8bit
output_dir: /root/.sn56/axolotl/tmp/test-repo
pad_to_sequence_len: true
resize_token_embeddings_to_32x: false
sample_packing: true
save_steps: 40
save_total_limit: 1
sequence_len: 2048
tokenizer_type: Qwen2TokenizerFast
torch_dtype: bf16
training_args_kwargs:
  disable_tqdm: true
  hub_private_repo: true
  save_only_model: true
trust_remote_code: true
val_set_size: 0.01
wandb_entity: ''
wandb_mode: online
wandb_name: peft-internal-testing/tiny-dummy-qwen2-argilla/databricks-dolly-15k-curated-en
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: default
warmup_ratio: 0.05

test-repo

This model is a fine-tuned version of peft-internal-testing/tiny-dummy-qwen2 on the argilla/databricks-dolly-15k-curated-en dataset. It achieves the following results on the evaluation set:

• Loss: 11.9145

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 7
• eval_batch_size: 7
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• total_train_batch_size: 28
• total_eval_batch_size: 28
• optimizer: Use adamw_bnb_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 8
• training_steps: 160

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0345	1	11.9318
11.9309	0.5517	16	11.9279
11.9236	1.1034	32	11.9212
11.9214	1.6552	48	11.9193
11.9198	2.2069	64	11.9178
11.9188	2.7586	80	11.9161
11.9181	3.3103	96	11.9151
11.9175	3.8621	112	11.9147
11.9174	4.4138	128	11.9143
11.917	4.9655	144	11.9140
11.9169	5.5172	160	11.9145

Framework versions

• PEFT 0.14.0
• Transformers 4.47.1
• Pytorch 2.3.1+cu121
• Datasets 3.1.0
• Tokenizers 0.21.0