Upload folder using huggingface_hub

README.md

@@ -1,92 +1,16 @@
 ---
-language:
-- en
-license: mit
+base_model:
+- LeroyDyer/Mixtral_AI_CyberBrain_3_0
+- ezelikman/quietstar-8-ahead
 library_name: transformers
 tags:
 - mergekit
 - merge
-- unsloth
-base_model:
-- LeroyDyer/Mixtral_AI_CyberBrain_2.0
-- ezelikman/quietstar-8-ahead
----
-
-ActulLLY ITS woRKING IT JUST NEEDS TRAINING DATA!! .... 
-
-
-This project is implemented by simply patching the base Mistral implementation in Huggingface transformers using a new modeling_mistral.py and a new configuration_mistral.py and otherwise applying standard transformers features (e.g. the default Trainer). 
-
-IE: First Clone the latest transformers 
-enter the models\mistral folder and upload the modelling_mistral.py
-then cd transformers and install frot he folder pip install ./transformers 
-
-after it can be loaded normally for training; 
-
-```
-
-from unsloth import FastLanguageModel
-import torch
-max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
-dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
-load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.
-
-# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
-fourbit_models = [
-    "unsloth/mistral-7b-bnb-4bit",
-    "unsloth/mistral-7b-instruct-v0.2-bnb-4bit",
-    "unsloth/llama-2-7b-bnb-4bit",
-    "unsloth/llama-2-13b-bnb-4bit",
-    "unsloth/codellama-34b-bnb-4bit",
-    "unsloth/tinyllama-bnb-4bit",
-    "unsloth/gemma-7b-bnb-4bit", # New Google 6 trillion tokens model 2.5x faster!
-    "unsloth/gemma-2b-bnb-4bit",
-] # More models at https://huggingface.co/unsloth
-
-model = FastLanguageModel.from_pretrained(
-    model_name = "LeroyDyer/Mixtral_AI_CyberBrain_3.0", # Choose ANY! eg teknium/OpenHermes-2.5-Mistral-7B
-    max_seq_length = 2048,
-    dtype = dtype,
-    load_in_4bit = load_in_4bit,
-#    trust_remote_code = True,
-    ignore_mismatched_sizes = True,
-    merged_talk_heads=True,
-    merged_lm_and_talk_heads=False,
-    merged_lm_and_think_heads=True,
-    use_concat_talk_head=True,
-    use_shallow_think=True,
-    use_shallow_talk=False,
-    use_complex_think_head=False,
-    use_complex_talk_head=True,
-    use_weighted_talk_head=True,
-
-    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
-)
-
-tokenizer = AutoTokenizer.from_pretrained(tokenizer_id,truncation=True,padding_side="right")
-tokenizer.pad_token_id = tokenizer.eos_token_id
-
-
-
-model.tokenizer = tokenizer
-
-model.train
-
-
-```
-
-
-right now the modelling_mistral.py s still havng problems loading remotely hence the hacky way... but after its fixed it will be fine. 
-
-
 
+---
 # merge
 
 This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
-yes multiple verions of this model was merged in attempts to grab the neccasary tensors ...
-but some how it did not build as some parameters was not loading. ie it would not load the config file! hopefully this will be rectified soon. so remote loading will be fine ... enabling for enhanced training.
-the model was trained to perfection so it still works fine! 
-the lora was made so tat later it can be loaded with the model for further training of the effected tensors... 
 
 ## Merge Details
 ### Merge Method
@@ -96,7 +20,7 @@ This model was merged using the SLERP merge method.
 ### Models Merged
 
 The following models were included in the merge:
-* [LeroyDyer/Mixtral_AI_CyberBrain_2.0](https://huggingface.co/LeroyDyer/Mixtral_AI_CyberBrain_2.0)
+* [LeroyDyer/Mixtral_AI_CyberBrain_3_0](https://huggingface.co/LeroyDyer/Mixtral_AI_CyberBrain_3_0)
 * [ezelikman/quietstar-8-ahead](https://huggingface.co/ezelikman/quietstar-8-ahead)
 
 ### Configuration
@@ -107,18 +31,14 @@ The following YAML configuration was used to produce this model:
 
 slices:
   - sources:
-      - model: LeroyDyer/Mixtral_AI_CyberBrain_2.0
+      - model: LeroyDyer/Mixtral_AI_CyberBrain_3_0
         layer_range: [0, 32]
       - model: ezelikman/quietstar-8-ahead
         layer_range: [0, 32]
 # or, the equivalent models: syntax:
 # models:
 #   - model: mistralai/Mistral-7B-Instruct-v0.2
-# LaRGER MODEL MUST BE BASE  or
-#  BASE MODEL MUST BE THE TOKENIZER YOU WISH TO ADOPT 
-# so for models with customized processes they must be the base model
-# If the base model has remote code then this must be collected and added 
-# to the repo after and the config file adusted to allow for automapping to your new repo
+# LaRGER MODEL MUST BE BASE
 #   - model: yanismiraoui/Yarn-Mistral-7b-128k-sharded
 merge_method: slerp
 base_model: ezelikman/quietstar-8-ahead
@@ -131,4 +51,4 @@ parameters:
     - value: 0.5 # fallback for rest of tensors
 dtype: float16
 
-```
+```

added_tokens.json

@@ -1,4 +1,4 @@
 {
-  "<|endthought|>": 32001,
-  "<|startthought|>": 32000
+  "<|endthought|>": 32000,
+  "<|startthought|>": 32001
 }

config.json

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "LeroyDyer/Mixtral_AI_CyberBrain_3.0",
+  "_name_or_path": "LeroyDyer/Mixtral_AI_CyberBrain_3_0",
   "architectures": [
     "MistralForCausalLM"
   ],
@@ -20,13 +20,11 @@
   "num_hidden_layers": 32,
   "num_key_value_heads": 8,
   "rms_norm_eps": 1e-05,
-  "rope_scaling": null,
   "rope_theta": 10000.0,
   "sliding_window": 4096,
   "tie_word_embeddings": false,
   "torch_dtype": "float16",
   "transformers_version": "4.40.0.dev0",
-  "unsloth_version": "2024.3",
   "use_cache": true,
   "use_complex_talk_head": true,
   "use_complex_think_head": false,

mergekit_config.yml

@@ -0,0 +1,22 @@
+
+slices:
+  - sources:
+      - model: LeroyDyer/Mixtral_AI_CyberBrain_3_0
+        layer_range: [0, 32]
+      - model: ezelikman/quietstar-8-ahead
+        layer_range: [0, 32]
+# or, the equivalent models: syntax:
+# models:
+#   - model: mistralai/Mistral-7B-Instruct-v0.2
+# LaRGER MODEL MUST BE BASE
+#   - model: yanismiraoui/Yarn-Mistral-7b-128k-sharded
+merge_method: slerp
+base_model: ezelikman/quietstar-8-ahead
+parameters:
+  t:
+    - filter: self_attn
+      value: [0.3, 0.6, 0.3786, 0.6, 0.6]
+    - filter: mlp
+      value: [0.7, 0.4, 0.6, 0.4, 0.7]
+    - value: 0.5 # fallback for rest of tensors
+dtype: float16

model-00001-of-00008.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:13f0403298b237fdef972835a728b3dd73fb62b6ff5045fbcc1a7f83b9780b3a
-size 1889603392
+oid sha256:9aa1b16f05743b2f339f6e70521c166198221a90edc9a9559a2d666416020ca6
+size 1973489936

model-00002-of-00008.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2e7673c6932fd497f568911dd8b11582c13bdbe8cf5ca785f4c7b9aaed4ec2de
-size 1946243896
+oid sha256:356d76c09d93a81c26368bb2fee8698d036ef21b6ee7007cf88e71463adc642e
+size 1979798000

model-00003-of-00008.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:98ab95d023531246ab6c0788384e4565a4023d351d63b421848613ece7a05b8b
-size 1979781392
+oid sha256:907a3d64fc26868db58b658da99d2aa49f979d8db0d459c703637416b671368c
+size 1946227312

model-00004-of-00008.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:128fc561e53f00066aa158fb1921abf73e8a2ad656c91315d7a9aa3817e81e1d
-size 1946243936
+oid sha256:68dd86eba610c49dbdf57c29721f7d7040358eaf990fc3d6b1c0ce3d6fddb153
+size 1979798016

model-00005-of-00008.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8b258a99da847688b4efb47d3bf6c2add1b4a9e3e04bcc63c37749b04d2423a1
-size 1979781416
+oid sha256:7abb6a3613474b4eae84e76fd4636149738b693d10eb12242250b37cd2af600b
+size 1946227336

model-00006-of-00008.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:151cee02c522a17a02f1ef81c26dee71df3775cd107728e5285b799270b09d35
-size 1946243936
+oid sha256:22c18413958dacaa1f724841221f89a772a9f2aca52ec92c375e78a054e30813
+size 1979798016

model-00007-of-00008.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:880b0efcadbc14cdcc85c7931ebe3bf4b07becf261c87f6a03f794dedf71d99f
-size 1979781416
+oid sha256:1725e5ba912edc98426340511e1984347d123ca4d92c66cbd6ba7e9d3ee1309a
+size 1862340784

model-00008-of-00008.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2dc3589a195594dd02eea2dc123d2857067eeeae5bbd5f0a2dc0e4a563becae2
+oid sha256:95e0b00098431ab705f87191f852793a1082b2f5e373c060f22c2ea3804ff51a
 size 815851048

model.safetensors.index.json

@@ -1,298 +1 @@
-{
-  "metadata": {
-    "total_size": 14483496960
-  },
-  "weight_map": {
-    "lm_head.weight": "model-00008-of-00008.safetensors",
-    "model.embed_tokens.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.input_layernorm.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.input_layernorm.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.10.input_layernorm.weight": "model-00003-of-00008.safetensors",
-    "model.layers.10.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.10.mlp.gate_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.10.mlp.up_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.10.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
-    "model.layers.10.self_attn.k_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.10.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.10.self_attn.q_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.10.self_attn.v_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.input_layernorm.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.mlp.gate_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.mlp.up_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.self_attn.k_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.self_attn.q_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.11.self_attn.v_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.12.input_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.12.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.12.mlp.gate_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.12.mlp.up_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.12.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.12.self_attn.k_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.12.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.12.self_attn.q_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.12.self_attn.v_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.13.input_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.13.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.13.mlp.gate_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.13.mlp.up_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.13.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.13.self_attn.k_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.13.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.13.self_attn.q_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.13.self_attn.v_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.input_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.mlp.gate_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.mlp.up_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.self_attn.k_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.self_attn.q_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.14.self_attn.v_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.input_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.mlp.gate_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.mlp.up_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.self_attn.k_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.self_attn.q_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.15.self_attn.v_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.input_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.mlp.gate_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.mlp.up_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.self_attn.k_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.self_attn.q_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.16.self_attn.v_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.17.input_layernorm.weight": "model-00005-of-00008.safetensors",
-    "model.layers.17.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.17.mlp.gate_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.17.mlp.up_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.17.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
-    "model.layers.17.self_attn.k_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.17.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.17.self_attn.q_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.17.self_attn.v_proj.weight": "model-00004-of-00008.safetensors",
-    "model.layers.18.input_layernorm.weight": "model-00005-of-00008.safetensors",
-    "model.layers.18.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.18.mlp.gate_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.18.mlp.up_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.18.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
-    "model.layers.18.self_attn.k_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.18.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.18.self_attn.q_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.18.self_attn.v_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.input_layernorm.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.mlp.gate_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.mlp.up_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.self_attn.k_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.self_attn.q_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.19.self_attn.v_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.2.input_layernorm.weight": "model-00001-of-00008.safetensors",
-    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00008.safetensors",
-    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.20.input_layernorm.weight": "model-00005-of-00008.safetensors",
-    "model.layers.20.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.20.mlp.gate_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.20.mlp.up_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.20.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
-    "model.layers.20.self_attn.k_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.20.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.20.self_attn.q_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.20.self_attn.v_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.21.input_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.21.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.21.mlp.gate_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.21.mlp.up_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.21.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.21.self_attn.k_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.21.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.21.self_attn.q_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.21.self_attn.v_proj.weight": "model-00005-of-00008.safetensors",
-    "model.layers.22.input_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.22.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.22.mlp.gate_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.22.mlp.up_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.22.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.22.self_attn.k_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.22.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.22.self_attn.q_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.22.self_attn.v_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.input_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.mlp.gate_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.mlp.up_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.self_attn.k_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.self_attn.q_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.23.self_attn.v_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.input_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.mlp.gate_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.mlp.up_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.self_attn.k_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.self_attn.q_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.24.self_attn.v_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.input_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.mlp.gate_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.mlp.up_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.self_attn.k_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.self_attn.q_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.25.self_attn.v_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.26.input_layernorm.weight": "model-00007-of-00008.safetensors",
-    "model.layers.26.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.26.mlp.gate_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.26.mlp.up_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.26.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
-    "model.layers.26.self_attn.k_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.26.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.26.self_attn.q_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.26.self_attn.v_proj.weight": "model-00006-of-00008.safetensors",
-    "model.layers.27.input_layernorm.weight": "model-00007-of-00008.safetensors",
-    "model.layers.27.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.27.mlp.gate_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.27.mlp.up_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.27.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
-    "model.layers.27.self_attn.k_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.27.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.27.self_attn.q_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.27.self_attn.v_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.input_layernorm.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.mlp.gate_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.mlp.up_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.self_attn.k_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.self_attn.q_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.28.self_attn.v_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.input_layernorm.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.mlp.gate_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.mlp.up_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.self_attn.k_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.self_attn.q_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.29.self_attn.v_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.3.input_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.3.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.3.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00008.safetensors",
-    "model.layers.30.input_layernorm.weight": "model-00008-of-00008.safetensors",
-    "model.layers.30.mlp.down_proj.weight": "model-00008-of-00008.safetensors",
-    "model.layers.30.mlp.gate_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.30.mlp.up_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.30.post_attention_layernorm.weight": "model-00008-of-00008.safetensors",
-    "model.layers.30.self_attn.k_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.30.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.30.self_attn.q_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.30.self_attn.v_proj.weight": "model-00007-of-00008.safetensors",
-    "model.layers.31.input_layernorm.weight": "model-00008-of-00008.safetensors",
-    "model.layers.31.mlp.down_proj.weight": "model-00008-of-00008.safetensors",
-    "model.layers.31.mlp.gate_proj.weight": "model-00008-of-00008.safetensors",
-    "model.layers.31.mlp.up_proj.weight": "model-00008-of-00008.safetensors",
-    "model.layers.31.post_attention_layernorm.weight": "model-00008-of-00008.safetensors",
-    "model.layers.31.self_attn.k_proj.weight": "model-00008-of-00008.safetensors",
-    "model.layers.31.self_attn.o_proj.weight": "model-00008-of-00008.safetensors",
-    "model.layers.31.self_attn.q_proj.weight": "model-00008-of-00008.safetensors",
-    "model.layers.31.self_attn.v_proj.weight": "model-00008-of-00008.safetensors",
-    "model.layers.4.input_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.4.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.4.mlp.gate_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.4.mlp.up_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.4.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.4.self_attn.k_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.4.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.4.self_attn.q_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.4.self_attn.v_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.input_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.mlp.gate_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.mlp.up_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.self_attn.k_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.self_attn.q_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.5.self_attn.v_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.input_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.mlp.gate_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.mlp.up_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.self_attn.k_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.self_attn.q_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.6.self_attn.v_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.input_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.mlp.gate_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.mlp.up_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.8.input_layernorm.weight": "model-00003-of-00008.safetensors",
-    "model.layers.8.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.8.mlp.gate_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.8.mlp.up_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.8.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
-    "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00008.safetensors",
-    "model.layers.9.input_layernorm.weight": "model-00003-of-00008.safetensors",
-    "model.layers.9.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.9.mlp.gate_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.9.mlp.up_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.9.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
-    "model.layers.9.self_attn.k_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.9.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.9.self_attn.q_proj.weight": "model-00003-of-00008.safetensors",
-    "model.layers.9.self_attn.v_proj.weight": "model-00003-of-00008.safetensors",
-    "model.norm.weight": "model-00008-of-00008.safetensors"
-  }
-}
+{"metadata": {"mergekit_version": "0.0.4.2", "total_size": 14483496960}, "weight_map": {"model.layers.3.mlp.gate_proj.weight": "model-00001-of-00008.safetensors", "model.layers.3.mlp.up_proj.weight": "model-00001-of-00008.safetensors", "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00008.safetensors", "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00008.safetensors", "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00008.safetensors", "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00008.safetensors", "model.layers.2.mlp.down_proj.weight": "model-00001-of-00008.safetensors", "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00008.safetensors", "model.layers.2.mlp.up_proj.weight": "model-00001-of-00008.safetensors", "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00008.safetensors", "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00008.safetensors", "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00008.safetensors", "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00008.safetensors", "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00008.safetensors", "model.layers.2.input_layernorm.weight": "model-00001-of-00008.safetensors", "model.layers.1.mlp.down_proj.weight": "model-00001-of-00008.safetensors", "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00008.safetensors", "model.layers.1.mlp.up_proj.weight": "model-00001-of-00008.safetensors", "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00008.safetensors", "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00008.safetensors", "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00008.safetensors", "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00008.safetensors", "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00008.safetensors", "model.layers.1.input_layernorm.weight": "model-00001-of-00008.safetensors", "model.layers.0.mlp.down_proj.weight": "model-00001-of-00008.safetensors", "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00008.safetensors", "model.layers.0.mlp.up_proj.weight": "model-00001-of-00008.safetensors", "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00008.safetensors", "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00008.safetensors", "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00008.safetensors", "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00008.safetensors", "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00008.safetensors", "model.layers.0.input_layernorm.weight": "model-00001-of-00008.safetensors", "model.embed_tokens.weight": "model-00001-of-00008.safetensors", "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00008.safetensors", "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00008.safetensors", "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00008.safetensors", "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00008.safetensors", "model.layers.7.mlp.down_proj.weight": "model-00002-of-00008.safetensors", "model.layers.7.mlp.gate_proj.weight": "model-00002-of-00008.safetensors", "model.layers.7.mlp.up_proj.weight": "model-00002-of-00008.safetensors", "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00008.safetensors", "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00008.safetensors", "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00008.safetensors", "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00008.safetensors", "model.layers.7.input_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.6.mlp.down_proj.weight": "model-00002-of-00008.safetensors", "model.layers.6.mlp.gate_proj.weight": "model-00002-of-00008.safetensors", "model.layers.6.mlp.up_proj.weight": "model-00002-of-00008.safetensors", "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00008.safetensors", "model.layers.6.self_attn.v_proj.weight": "model-00002-of-00008.safetensors", "model.layers.6.self_attn.k_proj.weight": "model-00002-of-00008.safetensors", "model.layers.6.self_attn.q_proj.weight": "model-00002-of-00008.safetensors", "model.layers.6.input_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.5.mlp.down_proj.weight": "model-00002-of-00008.safetensors", "model.layers.5.mlp.gate_proj.weight": "model-00002-of-00008.safetensors", "model.layers.5.mlp.up_proj.weight": "model-00002-of-00008.safetensors", "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.5.self_attn.o_proj.weight": "model-00002-of-00008.safetensors", "model.layers.5.self_attn.v_proj.weight": "model-00002-of-00008.safetensors", "model.layers.5.self_attn.k_proj.weight": "model-00002-of-00008.safetensors", "model.layers.5.self_attn.q_proj.weight": "model-00002-of-00008.safetensors", "model.layers.5.input_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.4.mlp.down_proj.weight": "model-00002-of-00008.safetensors", "model.layers.4.mlp.gate_proj.weight": "model-00002-of-00008.safetensors", "model.layers.4.mlp.up_proj.weight": "model-00002-of-00008.safetensors", "model.layers.4.post_attention_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.4.self_attn.o_proj.weight": "model-00002-of-00008.safetensors", "model.layers.4.self_attn.v_proj.weight": "model-00002-of-00008.safetensors", "model.layers.4.self_attn.k_proj.weight": "model-00002-of-00008.safetensors", "model.layers.4.self_attn.q_proj.weight": "model-00002-of-00008.safetensors", "model.layers.4.input_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.3.mlp.down_proj.weight": "model-00002-of-00008.safetensors", "model.layers.3.post_attention_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.3.input_layernorm.weight": "model-00002-of-00008.safetensors", "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00008.safetensors", "model.layers.12.mlp.up_proj.weight": "model-00003-of-00008.safetensors", "model.layers.12.self_attn.o_proj.weight": "model-00003-of-00008.safetensors", "model.layers.12.self_attn.v_proj.weight": "model-00003-of-00008.safetensors", "model.layers.12.self_attn.k_proj.weight": "model-00003-of-00008.safetensors", "model.layers.12.self_attn.q_proj.weight": "model-00003-of-00008.safetensors", "model.layers.11.mlp.down_proj.weight": "model-00003-of-00008.safetensors", "model.layers.11.mlp.gate_proj.weight": "model-00003-of-00008.safetensors", "model.layers.11.mlp.up_proj.weight": "model-00003-of-00008.safetensors", "model.layers.11.post_attention_layernorm.weight": "model-00003-of-00008.safetensors", "model.layers.11.self_attn.o_proj.weight": "model-00003-of-00008.safetensors", "model.layers.11.self_attn.v_proj.weight": "model-00003-of-00008.safetensors", "model.layers.11.self_attn.k_proj.weight": "model-00003-of-00008.safetensors", "model.layers.11.self_attn.q_proj.weight": "model-00003-of-00008.safetensors", "model.layers.11.input_layernorm.weight": "model-00003-of-00008.safetensors", "model.layers.10.mlp.down_proj.weight": "model-00003-of-00008.safetensors", "model.layers.10.mlp.gate_proj.weight": "model-00003-of-00008.safetensors", "model.layers.10.mlp.up_proj.weight": "model-00003-of-00008.safetensors", "model.layers.10.post_attention_layernorm.weight": "model-00003-of-00008.safetensors", "model.layers.10.self_attn.o_proj.weight": "model-00003-of-00008.safetensors", "model.layers.10.self_attn.v_proj.weight": "model-00003-of-00008.safetensors", "model.layers.10.self_attn.k_proj.weight": "model-00003-of-00008.safetensors", "model.layers.10.self_attn.q_proj.weight": "model-00003-of-00008.safetensors", "model.layers.10.input_layernorm.weight": "model-00003-of-00008.safetensors", "model.layers.9.mlp.down_proj.weight": "model-00003-of-00008.safetensors", "model.layers.9.mlp.gate_proj.weight": "model-00003-of-00008.safetensors", "model.layers.9.mlp.up_proj.weight": "model-00003-of-00008.safetensors", "model.layers.9.post_attention_layernorm.weight": "model-00003-of-00008.safetensors", "model.layers.9.self_attn.o_proj.weight": "model-00003-of-00008.safetensors", "model.layers.9.self_attn.v_proj.weight": "model-00003-of-00008.safetensors", "model.layers.9.self_attn.k_proj.weight": "model-00003-of-00008.safetensors", "model.layers.9.self_attn.q_proj.weight": "model-00003-of-00008.safetensors", "model.layers.9.input_layernorm.weight": "model-00003-of-00008.safetensors", "model.layers.8.mlp.down_proj.weight": "model-00003-of-00008.safetensors", "model.layers.8.mlp.gate_proj.weight": "model-00003-of-00008.safetensors", "model.layers.8.mlp.up_proj.weight": "model-00003-of-00008.safetensors", "model.layers.8.post_attention_layernorm.weight": "model-00003-of-00008.safetensors", "model.layers.8.input_layernorm.weight": "model-00003-of-00008.safetensors", "model.layers.17.self_attn.o_proj.weight": "model-00003-of-00008.safetensors", "model.layers.17.self_attn.v_proj.weight": "model-00003-of-00008.safetensors", "model.layers.17.self_attn.k_proj.weight": "model-00003-of-00008.safetensors", "model.layers.17.self_attn.q_proj.weight": "model-00003-of-00008.safetensors", "model.layers.16.mlp.down_proj.weight": "model-00004-of-00008.safetensors", "model.layers.16.mlp.gate_proj.weight": "model-00004-of-00008.safetensors", "model.layers.16.mlp.up_proj.weight": "model-00004-of-00008.safetensors", "model.layers.16.post_attention_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.16.self_attn.o_proj.weight": "model-00004-of-00008.safetensors", "model.layers.16.self_attn.v_proj.weight": "model-00004-of-00008.safetensors", "model.layers.16.self_attn.k_proj.weight": "model-00004-of-00008.safetensors", "model.layers.16.self_attn.q_proj.weight": "model-00004-of-00008.safetensors", "model.layers.16.input_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.15.mlp.down_proj.weight": "model-00004-of-00008.safetensors", "model.layers.15.mlp.gate_proj.weight": "model-00004-of-00008.safetensors", "model.layers.15.mlp.up_proj.weight": "model-00004-of-00008.safetensors", "model.layers.15.post_attention_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.15.self_attn.o_proj.weight": "model-00004-of-00008.safetensors", "model.layers.15.self_attn.v_proj.weight": "model-00004-of-00008.safetensors", "model.layers.15.self_attn.k_proj.weight": "model-00004-of-00008.safetensors", "model.layers.15.self_attn.q_proj.weight": "model-00004-of-00008.safetensors", "model.layers.15.input_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.14.mlp.down_proj.weight": "model-00004-of-00008.safetensors", "model.layers.14.mlp.gate_proj.weight": "model-00004-of-00008.safetensors", "model.layers.14.mlp.up_proj.weight": "model-00004-of-00008.safetensors", "model.layers.14.post_attention_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.14.self_attn.o_proj.weight": "model-00004-of-00008.safetensors", "model.layers.14.self_attn.v_proj.weight": "model-00004-of-00008.safetensors", "model.layers.14.self_attn.k_proj.weight": "model-00004-of-00008.safetensors", "model.layers.14.self_attn.q_proj.weight": "model-00004-of-00008.safetensors", "model.layers.14.input_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.13.mlp.down_proj.weight": "model-00004-of-00008.safetensors", "model.layers.13.mlp.gate_proj.weight": "model-00004-of-00008.safetensors", "model.layers.13.mlp.up_proj.weight": "model-00004-of-00008.safetensors", "model.layers.13.post_attention_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.13.self_attn.o_proj.weight": "model-00004-of-00008.safetensors", "model.layers.13.self_attn.v_proj.weight": "model-00004-of-00008.safetensors", "model.layers.13.self_attn.k_proj.weight": "model-00004-of-00008.safetensors", "model.layers.13.self_attn.q_proj.weight": "model-00004-of-00008.safetensors", "model.layers.13.input_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.12.mlp.down_proj.weight": "model-00004-of-00008.safetensors", "model.layers.12.post_attention_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.12.input_layernorm.weight": "model-00004-of-00008.safetensors", "model.layers.21.mlp.gate_proj.weight": "model-00004-of-00008.safetensors", "model.layers.21.mlp.up_proj.weight": "model-00005-of-00008.safetensors", "model.layers.21.self_attn.o_proj.weight": "model-00005-of-00008.safetensors", "model.layers.21.self_attn.v_proj.weight": "model-00005-of-00008.safetensors", "model.layers.21.self_attn.k_proj.weight": "model-00005-of-00008.safetensors", "model.layers.21.self_attn.q_proj.weight": "model-00005-of-00008.safetensors", "model.layers.20.mlp.down_proj.weight": "model-00005-of-00008.safetensors", "model.layers.20.mlp.gate_proj.weight": "model-00005-of-00008.safetensors", "model.layers.20.mlp.up_proj.weight": "model-00005-of-00008.safetensors", "model.layers.20.post_attention_layernorm.weight": "model-00005-of-00008.safetensors", "model.layers.20.self_attn.o_proj.weight": "model-00005-of-00008.safetensors", "model.layers.20.self_attn.v_proj.weight": "model-00005-of-00008.safetensors", "model.layers.20.self_attn.k_proj.weight": "model-00005-of-00008.safetensors", "model.layers.20.self_attn.q_proj.weight": "model-00005-of-00008.safetensors", "model.layers.20.input_layernorm.weight": "model-00005-of-00008.safetensors", "model.layers.19.mlp.down_proj.weight": "model-00005-of-00008.safetensors", "model.layers.19.mlp.gate_proj.weight": "model-00005-of-00008.safetensors", "model.layers.19.mlp.up_proj.weight": "model-00005-of-00008.safetensors", "model.layers.19.post_attention_layernorm.weight": "model-00005-of-00008.safetensors", "model.layers.19.self_attn.o_proj.weight": "model-00005-of-00008.safetensors", "model.layers.19.self_attn.v_proj.weight": "model-00005-of-00008.safetensors", "model.layers.19.self_attn.k_proj.weight": "model-00005-of-00008.safetensors", "model.layers.19.self_attn.q_proj.weight": "model-00005-of-00008.safetensors", "model.layers.19.input_layernorm.weight": "model-00005-of-00008.safetensors", "model.layers.18.mlp.down_proj.weight": "model-00005-of-00008.safetensors", "model.layers.18.mlp.gate_proj.weight": "model-00005-of-00008.safetensors", "model.layers.18.mlp.up_proj.weight": "model-00005-of-00008.safetensors", "model.layers.18.post_attention_layernorm.weight": "model-00005-of-00008.safetensors", "model.layers.18.self_attn.o_proj.weight": "model-00005-of-00008.safetensors", "model.layers.18.self_attn.v_proj.weight": "model-00005-of-00008.safetensors", "model.layers.18.self_attn.k_proj.weight": "model-00005-of-00008.safetensors", "model.layers.18.self_attn.q_proj.weight": "model-00005-of-00008.safetensors", "model.layers.18.input_layernorm.weight": "model-00005-of-00008.safetensors", "model.layers.17.mlp.down_proj.weight": "model-00005-of-00008.safetensors", "model.layers.17.mlp.gate_proj.weight": "model-00005-of-00008.safetensors", "model.layers.17.mlp.up_proj.weight": "model-00005-of-00008.safetensors", "model.layers.17.post_attention_layernorm.weight": "model-00005-of-00008.safetensors", "model.layers.17.input_layernorm.weight": "model-00005-of-00008.safetensors", "model.layers.26.self_attn.o_proj.weight": "model-00005-of-00008.safetensors", "model.layers.26.self_attn.v_proj.weight": "model-00005-of-00008.safetensors", "model.layers.26.self_attn.k_proj.weight": "model-00005-of-00008.safetensors", "model.layers.26.self_attn.q_proj.weight": "model-00005-of-00008.safetensors", "model.layers.25.mlp.down_proj.weight": "model-00006-of-00008.safetensors", "model.layers.25.mlp.gate_proj.weight": "model-00006-of-00008.safetensors", "model.layers.25.mlp.up_proj.weight": "model-00006-of-00008.safetensors", "model.layers.25.post_attention_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.25.self_attn.o_proj.weight": "model-00006-of-00008.safetensors", "model.layers.25.self_attn.v_proj.weight": "model-00006-of-00008.safetensors", "model.layers.25.self_attn.k_proj.weight": "model-00006-of-00008.safetensors", "model.layers.25.self_attn.q_proj.weight": "model-00006-of-00008.safetensors", "model.layers.25.input_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.24.mlp.down_proj.weight": "model-00006-of-00008.safetensors", "model.layers.24.mlp.gate_proj.weight": "model-00006-of-00008.safetensors", "model.layers.24.mlp.up_proj.weight": "model-00006-of-00008.safetensors", "model.layers.24.post_attention_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.24.self_attn.o_proj.weight": "model-00006-of-00008.safetensors", "model.layers.24.self_attn.v_proj.weight": "model-00006-of-00008.safetensors", "model.layers.24.self_attn.k_proj.weight": "model-00006-of-00008.safetensors", "model.layers.24.self_attn.q_proj.weight": "model-00006-of-00008.safetensors", "model.layers.24.input_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.23.mlp.down_proj.weight": "model-00006-of-00008.safetensors", "model.layers.23.mlp.gate_proj.weight": "model-00006-of-00008.safetensors", "model.layers.23.mlp.up_proj.weight": "model-00006-of-00008.safetensors", "model.layers.23.post_attention_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.23.self_attn.o_proj.weight": "model-00006-of-00008.safetensors", "model.layers.23.self_attn.v_proj.weight": "model-00006-of-00008.safetensors", "model.layers.23.self_attn.k_proj.weight": "model-00006-of-00008.safetensors", "model.layers.23.self_attn.q_proj.weight": "model-00006-of-00008.safetensors", "model.layers.23.input_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.22.mlp.down_proj.weight": "model-00006-of-00008.safetensors", "model.layers.22.mlp.gate_proj.weight": "model-00006-of-00008.safetensors", "model.layers.22.mlp.up_proj.weight": "model-00006-of-00008.safetensors", "model.layers.22.post_attention_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.22.self_attn.o_proj.weight": "model-00006-of-00008.safetensors", "model.layers.22.self_attn.v_proj.weight": "model-00006-of-00008.safetensors", "model.layers.22.self_attn.k_proj.weight": "model-00006-of-00008.safetensors", "model.layers.22.self_attn.q_proj.weight": "model-00006-of-00008.safetensors", "model.layers.22.input_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.21.mlp.down_proj.weight": "model-00006-of-00008.safetensors", "model.layers.21.post_attention_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.21.input_layernorm.weight": "model-00006-of-00008.safetensors", "model.layers.30.mlp.gate_proj.weight": "model-00006-of-00008.safetensors", "model.layers.30.mlp.up_proj.weight": "model-00007-of-00008.safetensors", "model.layers.30.self_attn.o_proj.weight": "model-00007-of-00008.safetensors", "model.layers.30.self_attn.v_proj.weight": "model-00007-of-00008.safetensors", "model.layers.30.self_attn.k_proj.weight": "model-00007-of-00008.safetensors", "model.layers.30.self_attn.q_proj.weight": "model-00007-of-00008.safetensors", "model.layers.29.mlp.down_proj.weight": "model-00007-of-00008.safetensors", "model.layers.29.mlp.gate_proj.weight": "model-00007-of-00008.safetensors", "model.layers.29.mlp.up_proj.weight": "model-00007-of-00008.safetensors", "model.layers.29.post_attention_layernorm.weight": "model-00007-of-00008.safetensors", "model.layers.29.self_attn.o_proj.weight": "model-00007-of-00008.safetensors", "model.layers.29.self_attn.v_proj.weight": "model-00007-of-00008.safetensors", "model.layers.29.self_attn.k_proj.weight": "model-00007-of-00008.safetensors", "model.layers.29.self_attn.q_proj.weight": "model-00007-of-00008.safetensors", "model.layers.29.input_layernorm.weight": "model-00007-of-00008.safetensors", "model.layers.28.mlp.down_proj.weight": "model-00007-of-00008.safetensors", "model.layers.28.mlp.gate_proj.weight": "model-00007-of-00008.safetensors", "model.layers.28.mlp.up_proj.weight": "model-00007-of-00008.safetensors", "model.layers.28.post_attention_layernorm.weight": "model-00007-of-00008.safetensors", "model.layers.28.self_attn.o_proj.weight": "model-00007-of-00008.safetensors", "model.layers.28.self_attn.v_proj.weight": "model-00007-of-00008.safetensors", "model.layers.28.self_attn.k_proj.weight": "model-00007-of-00008.safetensors", "model.layers.28.self_attn.q_proj.weight": "model-00007-of-00008.safetensors", "model.layers.28.input_layernorm.weight": "model-00007-of-00008.safetensors", "model.layers.27.mlp.down_proj.weight": "model-00007-of-00008.safetensors", "model.layers.27.mlp.gate_proj.weight": "model-00007-of-00008.safetensors", "model.layers.27.mlp.up_proj.weight": "model-00007-of-00008.safetensors", "model.layers.27.post_attention_layernorm.weight": "model-00007-of-00008.safetensors", "model.layers.27.self_attn.o_proj.weight": "model-00007-of-00008.safetensors", "model.layers.27.self_attn.v_proj.weight": "model-00007-of-00008.safetensors", "model.layers.27.self_attn.k_proj.weight": "model-00007-of-00008.safetensors", "model.layers.27.self_attn.q_proj.weight": "model-00007-of-00008.safetensors", "model.layers.27.input_layernorm.weight": "model-00007-of-00008.safetensors", "model.layers.26.mlp.down_proj.weight": "model-00007-of-00008.safetensors", "model.layers.26.mlp.gate_proj.weight": "model-00007-of-00008.safetensors", "model.layers.26.mlp.up_proj.weight": "model-00007-of-00008.safetensors", "model.layers.26.post_attention_layernorm.weight": "model-00007-of-00008.safetensors", "model.layers.26.input_layernorm.weight": "model-00007-of-00008.safetensors", "lm_head.weight": "model-00008-of-00008.safetensors", "model.norm.weight": "model-00008-of-00008.safetensors", "model.layers.31.mlp.down_proj.weight": "model-00008-of-00008.safetensors", "model.layers.31.mlp.gate_proj.weight": "model-00008-of-00008.safetensors", "model.layers.31.mlp.up_proj.weight": "model-00008-of-00008.safetensors", "model.layers.31.post_attention_layernorm.weight": "model-00008-of-00008.safetensors", "model.layers.31.self_attn.o_proj.weight": "model-00008-of-00008.safetensors", "model.layers.31.self_attn.v_proj.weight": "model-00008-of-00008.safetensors", "model.layers.31.self_attn.k_proj.weight": "model-00008-of-00008.safetensors", "model.layers.31.self_attn.q_proj.weight": "model-00008-of-00008.safetensors", "model.layers.31.input_layernorm.weight": "model-00008-of-00008.safetensors", "model.layers.30.mlp.down_proj.weight": "model-00008-of-00008.safetensors", "model.layers.30.post_attention_layernorm.weight": "model-00008-of-00008.safetensors", "model.layers.30.input_layernorm.weight": "model-00008-of-00008.safetensors"}}

special_tokens_map.json

@@ -1,7 +1,7 @@
 {
   "additional_special_tokens": [
-    "<|startthought|>",
-    "<|endthought|>"
+    "<|endthought|>",
+    "<|startthought|>"
   ],
   "bos_token": {
     "content": "<s>",
@@ -17,7 +17,13 @@
     "rstrip": false,
     "single_word": false
   },
-  "pad_token": "</s>",
+  "pad_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
   "unk_token": {
     "content": "<unk>",
     "lstrip": false,

tokenizer.json

@@ -32,7 +32,7 @@
     },
     {
       "id": 32000,
-      "content": "<|startthought|>",
+      "content": "<|endthought|>",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
@@ -41,7 +41,7 @@
     },
     {
       "id": 32001,
-      "content": "<|endthought|>",
+      "content": "<|startthought|>",
       "single_word": false,
       "lstrip": false,
       "rstrip": false,

tokenizer_config.json

@@ -27,7 +27,7 @@
       "special": true
     },
     "32000": {
-      "content": "<|startthought|>",
+      "content": "<|endthought|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -35,7 +35,7 @@
       "special": true
     },
     "32001": {
-      "content": "<|endthought|>",
+      "content": "<|startthought|>",
       "lstrip": false,
       "normalized": false,
       "rstrip": false,
@@ -44,20 +44,18 @@
     }
   },
   "additional_special_tokens": [
-    "<|startthought|>",
-    "<|endthought|>"
+    "<|endthought|>",
+    "<|startthought|>"
   ],
   "bos_token": "<s>",
   "clean_up_tokenization_spaces": false,
   "eos_token": "</s>",
   "legacy": true,
-  "model_max_length": 32768,
+  "model_max_length": 1000000000000000019884624838656,
   "pad_token": "</s>",
-  "padding_side": "right",
   "sp_model_kwargs": {},
   "spaces_between_special_tokens": false,
   "tokenizer_class": "LlamaTokenizer",
-  "truncation": true,
   "unk_token": "<unk>",
   "use_default_system_prompt": false
 }