See axolotl config

axolotl version: 0.4.1

base_model: allganize/Llama-3-Alpha-Ko-8B-Evo
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

trust_remote_code: true
# is_mistral_derived_model: true
# trust_remote_code: true

load_in_8bit: false
load_in_4bit: true

datasets:
  - path: allganize/finance-multiple-choice-ko-240605-processed
    type: sharegpt
    conversation: llama3

dataset_prepared_path: ./data/prepared_dataset
val_set_size: 0.002
output_dir: ./data/models/llama3-alpha-ko-fmmlu-exp2
chat_template: llama3

sequence_len: 4096  # supports up to 8192
sample_packing: false
eval_sample_packing: false
pad_to_sequence_len: true

adapter: qlora
lora_model_dir:
lora_r: 256
lora_alpha: 128
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
  - gate_proj
  - down_proj
  - up_proj
  - q_proj
  - v_proj
  - k_proj
  - o_proj
lora_modules_to_save:
  - embed_tokens
  - lm_head

wandb_project: llama3
wandb_entity:
wandb_watch:
wandb_name: llama3-alpha-ko-fmmlu-exp2
wandb_log_model:

max_grad_norm: 1.0
gradient_accumulation_steps: 16
micro_batch_size: 4
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: constant
learning_rate: 1e-5

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3

debug: true

warmup_steps: 100
eval_steps: 0.10
eval_table_size:
eval_table_max_new_tokens: 128
save_steps: 200 # 전체 스텝이 13725라 0.1 같이 퍼센트로 바꿔도 될듯.
save_total_limit: 4
weight_decay: 0.01
deepspeed:
fsdp:
fsdp_config:
special_tokens:
  eos_token: "<|eot_id|>"
  pad_token: "<|eot_id|>"

data/models/llama3-alpha-ko-fmmlu-exp2

This model is a fine-tuned version of allganize/Llama-3-Alpha-Ko-8B-Evo on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.3217

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• distributed_type: multi-GPU
• num_devices: 2
• gradient_accumulation_steps: 16
• total_train_batch_size: 128
• total_eval_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_steps: 100
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
1.0954	0.0104	1	1.0094
0.7671	0.1036	10	0.7111
0.6585	0.2071	20	0.6337
0.6094	0.3107	30	0.5744
0.518	0.4142	40	0.5192
0.5139	0.5178	50	0.4666
0.3988	0.6214	60	0.4250
0.4307	0.7249	70	0.3842
0.4507	0.8285	80	0.3551
0.4253	0.9320	90	0.3217

Framework versions

• PEFT 0.11.1
• Transformers 4.41.1
• Pytorch 2.1.2+cu118
• Datasets 2.19.1
• Tokenizers 0.19.1