Model save

README.md

@@ -0,0 +1,246 @@
+---
+library_name: transformers
+base_model: Dans-DiscountModels/Mistral-Nemo-Base-2407-ChatML-Mod
+tags:
+- axolotl
+- generated_from_trainer
+datasets:
+- PocketDoc/Dans-Reasoningmaxx-NaturalReasoning
+- PocketDoc/Dans-Reasoningmaxx-WebInstruct
+- PocketDoc/Dans-Reasoningmaxx-GeneralReasoning
+- PocketDoc/Dans-Benchmaxx-COT
+- PocketDoc/Dans-Logicmaxx-SAT-AP
+- PocketDoc/Dans-Assistantmaxx-Opus-Merge
+model-index:
+- name: 12b-mn-dans-reasoning-test-5
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+[<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
+<details><summary>See axolotl config</summary>
+
+axolotl version: `0.8.0.dev0`
+```yaml
+base_model: Dans-DiscountModels/Mistral-Nemo-Base-2407-ChatML-Mod
+model_type: AutoModelForCausalLM
+tokenizer_type: AutoTokenizer
+
+trust_remote_code:
+
+# wandb configuration
+wandb_project: 12b-mn-dans-reasoning-test
+wandb_watch:
+
+wandb_run_id: V0.0.4-1-2 # V{Version}-{Run Number}-{Attempt Number}
+wandb_log_model:
+
+# push checkpoints to hub
+hub_model_id: Dans-DiscountModels/12b-mn-dans-reasoning-test-5
+# how to push checkpoints to hub
+# https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.hub_strategy
+hub_strategy: "every_save"
+# Whether to use hf `use_auth_token` for loading datasets. Useful for fetching private datasets
+# Required to be true when used in combination with `push_dataset_to_hub`
+hf_use_auth_token: true
+
+# where to save the finished model to
+output_dir: ./12b-mn-dans-reasoning-test
+
+save_safetensors: true
+
+# dataset settings (local or huggingface repo)
+datasets:
+  - path: PocketDoc/Dans-Reasoningmaxx-NaturalReasoning
+    type: dan-chat-advanced
+  - path: PocketDoc/Dans-Reasoningmaxx-WebInstruct
+    type: dan-chat-advanced
+  - path: PocketDoc/Dans-Reasoningmaxx-GeneralReasoning
+    type: dan-chat-advanced
+  - path: PocketDoc/Dans-Benchmaxx-COT
+    type: dan-chat-advanced
+  - path: PocketDoc/Dans-Logicmaxx-SAT-AP
+    type: dan-chat-advanced
+  - path: PocketDoc/Dans-Assistantmaxx-Opus-Merge
+    type: dan-chat-advanced
+  # - path: PocketDoc/Dans-Assistantmaxx-sonnetorca-subset
+  #   type: dan-chat-advanced
+
+plugins:
+  - axolotl.integrations.liger.LigerPlugin
+  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin
+liger_rope: true
+liger_rms_norm: true
+liger_layer_norm: true
+liger_glu_activation: true
+liger_fused_linear_cross_entropy: false
+cut_cross_entropy: true
+
+load_in_8bit: false
+load_in_4bit: false
+strict: false
+
+adapter:
+lora_model_dir:
+
+lora_r: 128
+lora_alpha: 128
+lora_dropout: 0.1
+lora_target_linear: True
+lora_target_modules:
+lora_modules_to_save:
+  - embed_tokens
+  - lm_head
+lora_fan_in_fan_out:
+peft_use_rslora: true
+
+dataset_prepared_path: ./12b-mn-dans-reasoning-test-data
+val_set_size: 0.005
+
+sequence_len: 8192
+
+sample_packing: true
+eval_sample_packing: true
+
+pad_to_sequence_len: true
+
+gradient_checkpointing: true
+# gradient_checkpointing_kwargs:
+# use_reentrant: false
+
+gradient_accumulation_steps: 1
+micro_batch_size: 4
+
+num_epochs: 2
+
+optimizer: came_pytorch
+
+lr_scheduler: rex
+learning_rate: 0.0000015
+cosine_min_lr_ratio: 0.1
+
+weight_decay: 0.1
+
+max_grad_norm: 0.1
+
+train_on_inputs: false
+group_by_length: true
+
+bf16: true
+fp16: false
+tf32: false
+
+early_stopping_patience:
+
+resume_from_checkpoint:
+auto_resume_from_checkpoints: true
+
+local_rank:
+logging_steps: 1
+xformers_attention:
+flash_attention: true
+
+warmup_ratio: 0.05
+
+evals_per_epoch: 16
+eval_table_size:
+eval_max_new_tokens:
+
+saves_per_epoch: 4
+save_total_limit: 1
+
+debug: false
+
+deepspeed: deepspeed_configs/zero3_bf16.json
+
+fsdp:
+fsdp_config:
+
+special_tokens:
+
+```
+
+</details><br>
+
+# 12b-mn-dans-reasoning-test-5
+
+This model is a fine-tuned version of [Dans-DiscountModels/Mistral-Nemo-Base-2407-ChatML-Mod](https://huggingface.co/Dans-DiscountModels/Mistral-Nemo-Base-2407-ChatML-Mod) on the PocketDoc/Dans-Reasoningmaxx-NaturalReasoning, the PocketDoc/Dans-Reasoningmaxx-WebInstruct, the PocketDoc/Dans-Reasoningmaxx-GeneralReasoning, the PocketDoc/Dans-Benchmaxx-COT, the PocketDoc/Dans-Logicmaxx-SAT-AP and the PocketDoc/Dans-Assistantmaxx-Opus-Merge datasets.
+It achieves the following results on the evaluation set:
+- Loss: 0.5607
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 1.5e-06
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- total_train_batch_size: 16
+- total_eval_batch_size: 16
+- optimizer: Use OptimizerNames.ADAMW_HF with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 72
+- num_epochs: 2.0
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| 0.7429        | 0.0014 | 1    | 0.7557          |
+| 0.6704        | 0.0634 | 46   | 0.6654          |
+| 0.632         | 0.1269 | 92   | 0.6255          |
+| 0.6256        | 0.1903 | 138  | 0.6108          |
+| 0.5898        | 0.2538 | 184  | 0.6022          |
+| 0.6078        | 0.3172 | 230  | 0.5967          |
+| 0.5585        | 0.3807 | 276  | 0.5925          |
+| 0.609         | 0.4441 | 322  | 0.5887          |
+| 0.6203        | 0.5076 | 368  | 0.5854          |
+| 0.5651        | 0.5710 | 414  | 0.5825          |
+| 0.5886        | 0.6345 | 460  | 0.5802          |
+| 0.5479        | 0.6979 | 506  | 0.5783          |
+| 0.5924        | 0.7614 | 552  | 0.5764          |
+| 0.5515        | 0.8248 | 598  | 0.5745          |
+| 0.5891        | 0.8883 | 644  | 0.5727          |
+| 0.5757        | 0.9517 | 690  | 0.5712          |
+| 0.5241        | 1.0152 | 736  | 0.5718          |
+| 0.5303        | 1.0786 | 782  | 0.5723          |
+| 0.5586        | 1.1421 | 828  | 0.5713          |
+| 0.5087        | 1.2055 | 874  | 0.5702          |
+| 0.5251        | 1.2690 | 920  | 0.5694          |
+| 0.5336        | 1.3324 | 966  | 0.5689          |
+| 0.4921        | 1.3959 | 1012 | 0.5689          |
+| 0.5164        | 1.4593 | 1058 | 0.5670          |
+| 0.5546        | 1.5228 | 1104 | 0.5665          |
+| 0.49          | 1.5862 | 1150 | 0.5649          |
+| 0.4836        | 1.6497 | 1196 | 0.5647          |
+| 0.4768        | 1.7131 | 1242 | 0.5642          |
+| 0.5207        | 1.7766 | 1288 | 0.5630          |
+| 0.5127        | 1.8400 | 1334 | 0.5624          |
+| 0.5111        | 1.9034 | 1380 | 0.5615          |
+| 0.4927        | 1.9669 | 1426 | 0.5607          |
+
+
+### Framework versions
+
+- Transformers 4.49.0
+- Pytorch 2.4.1+cu124
+- Datasets 3.2.0
+- Tokenizers 0.21.1

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": 2,
+  "transformers_version": "4.49.0"
+}