Workermen/qwenpyx

See axolotl config

axolotl version: 0.8.0.dev0

base_model: "Workermen/ru_qwen"
model_type: Qwen2ForCausalLM
tokenizer_type: AutoTokenizer

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_glu_activation: true
liger_fused_linear_cross_entropy: true

load_in_8bit: false
load_in_4bit: false
strict: false

chat_template: chatml

datasets:
  - path: "/workspace/data/axolotl/examples/qwen2/train"
    data_files: "./train/train_01.jsonl"
    type: chat_template
    field_messages: conversations

  - path: "/workspace/data/axolotl/examples/qwen2/train"
    data_files: "./train/train_02.jsonl"
    type: chat_template
    field_messages: conversations
    
  - path: "/workspace/data/axolotl/examples/qwen2/train"
    data_files: "./train/train_03.jsonl"
    type: chat_template
    field_messages: conversations
    
  - path: "/workspace/data/axolotl/examples/qwen2/train"
    data_files: "./train/train_04.jsonl"
    type: chat_template
    field_messages: conversations
    
  - path: "/workspace/data/axolotl/examples/qwen2/train"
    data_files: "./train/validation.jsonl"
    type: chat_template
    field_messages: conversations


dataset_prepared_path: "/workspace/data/axolotl/examples/qwen2"
val_set_size: 0.05
output_dir: "/workspace/data/pyx_03_15_1_tochno"

sequence_len: 5300
sample_packing: false
pad_to_sequence_len: true

#mlflow_tracking_uri: "http://localhost:8080"
#mlflow_experiment_name: "my_axolotl_experiment"
#mlflow_run_name: "run-1"
#hf_mlflow_log_artifacts: true

wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

gradient_accumulation_steps: 32
micro_batch_size: 4
num_epochs: 2

optimizer: paged_adamw_8bit
lr_scheduler: cosine
cosine_min_lr_ratio: 0.5
cosine_constant_lr_ratio: 0.5
learning_rate: 0.00004

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
max_grad_norm: 2

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

save_safetensors: true

warmup_steps: 10
evals_per_epoch: 4
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed: #deepspeed_configs/zero3_bf16_cpuoffload_all.json
weight_decay: 0.0
fsdp:
#  - full_shard
#  - auto_wrap
fsdp_config:
#  fsdp_limit_all_gathers: true
#  fsdp_sync_module_states: true
#  fsdp_offload_params: true
#  fsdp_use_orig_params: false
#  fsdp_cpu_ram_efficient_loading: true
#  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
#  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
#  fsdp_state_dict_type: FULL_STATE_DICT
#  fsdp_sharding_strategy: FULL_SHARD
#  fsdp_backward_prefetch: BACKWARD_PRE
special_tokens:

workspace/data/pyx_03_15_1_tochno

This model is a fine-tuned version of Workermen/ru_qwen on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7111

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 4e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 32
• total_train_batch_size: 128
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 2.0

Training results

Training Loss	Epoch	Step	Validation Loss
1.1643	0.0010	1	1.1436
0.8729	0.2506	244	0.8838
0.8384	0.5012	488	0.8257
0.6237	0.7517	732	0.7867
0.5029	1.0021	976	0.7567
0.5637	1.2526	1220	0.7540
0.5629	1.5032	1464	0.7309
0.4682	1.7538	1708	0.7111

Framework versions

• Transformers 4.49.0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0