brucethemoose commited on
Commit
0823229
1 Parent(s): 0e3a0d4

Upload folder using huggingface_hub

Browse files
README.md ADDED
@@ -0,0 +1,182 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: yi-license
4
+ license_link: https://huggingface.co/01-ai/Yi-34B/blob/main/LICENSE
5
+ language:
6
+ - en
7
+ library_name: transformers
8
+ base_model: []
9
+ tags:
10
+ - mergekit
11
+ - merge
12
+ - Yi
13
+ ---
14
+ # Yi 34B 200K DARE Merge v8
15
+
16
+ A merge of many Yi 34B 200K models using the new DARE Ties method via mergekit. The goal is to create a merge model that excels at 32K+ context performance, without any additional finetuning.
17
+
18
+ ## Prompt template: Orca-Vicuna
19
+ ```
20
+ SYSTEM: {system_message}
21
+ USER: {prompt}
22
+ ASSISTANT:
23
+ ```
24
+ It might recognize ChatML, and possibly Alpaca-like formats. Raw prompting as described here is also effective: https://old.reddit.com/r/LocalLLaMA/comments/18zqy4s/the_secret_to_writing_quality_stories_with_llms/
25
+
26
+
27
+
28
+ ## Running
29
+ Being a Yi model, run a lower temperature with 0.05 or higher MinP, a little repetition penalty, maybe mirostat with a low tau, and no other samplers. Yi tends to run "hot" by default, and it really needs a low temperature + MinP to cull Yi's huge vocabulary. See the explanation here: https://github.com/ggerganov/llama.cpp/pull/3841
30
+
31
+ 24GB GPUs can efficiently run Yi-34B-200K models at **40K-90K context** with exllamav2, and performant UIs like [exui](https://github.com/turboderp/exui). I go into more detail in this [post](https://old.reddit.com/r/LocalLLaMA/comments/1896igc/how_i_run_34b_models_at_75k_context_on_24gb_fast/). 16GB GPUs can still run the high context with aggressive quantization.
32
+
33
+ I recommend exl2 quantizations profiled on data similar to the desired task. It is especially sensitive to the quantization data at low bpw. I've upload my own fiction-oriented quantizations here: https://huggingface.co/collections/brucethemoose/most-recent-merge-65742644ca03b6c514afa204
34
+
35
+ To load/train this in full-context backends like transformers, you *must* change `max_position_embeddings` in config.json to a lower value than 200,000, otherwise you will OOM! I do not recommend running high context without context-efficient backends like exllamav2, litellm or unsloth.
36
+
37
+
38
+ ## Testing Notes
39
+
40
+ See: https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-merge-v5#testing-notes
41
+
42
+ An intermediate merge model was created to try and extend the context of several 4k models before adding them to the main merge, as seen in the "megamerge" recipe below. I can upload this upon request
43
+
44
+ In addition, the weight gradients are biased towards Vicuna-format models in the first few layers to try and "emphasize" the Orca-Vicuna prompt template. How sucessful this is remains to be seen.
45
+
46
+
47
+
48
+ ## Merge Details
49
+ ### Merge Method
50
+
51
+ This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama as a base.
52
+
53
+ ### Models Merged
54
+
55
+ The following models were included in the merge:
56
+ * https://huggingface.co/kyujinpy/PlatYi-34B-200k-Q-FastChat
57
+ * https://huggingface.co/jondurbin/bagel-34b-v0.2
58
+ * https://huggingface.co/migtissera/Tess-M-Creative-v1.0
59
+ * https://huggingface.co/brucethemoose/SUS-Bagel-200K-DARE-Test
60
+ * https://huggingface.co/Mihaiii/Pallas-0.5
61
+ * https://huggingface.co/bhenrym14/airoboros-3_1-yi-34b-200k
62
+ * https://huggingface.co/adamo1139/Yi-34B-200K-AEZAKMI-v2
63
+ * https://huggingface.co/migtissera/Tess-34B-v1.4
64
+ * https://huggingface.co/SUSTech/SUS-Chat-34B
65
+ * https://huggingface.co/jondurbin/bagel-dpo-34b-v0.2
66
+ * https://huggingface.co/bhenrym14/platypus-yi-34b
67
+ * https://huggingface.co/Weyaxi/Nous-Hermes-2-SUS-Chat-34B-Slerp
68
+ * https://huggingface.co/TriadParty/deepsex-34b
69
+ * https://huggingface.co/TriadParty/deepmoney-34b-200k-base
70
+ * https://huggingface.co/chargoddard/Yi-34B-200K-Llama
71
+ * https://huggingface.co/chargoddard/Yi-34B-Llama
72
+
73
+ ### Configuration
74
+
75
+ The following YAML configuration was used to produce this model:
76
+
77
+ ```yaml
78
+ models:
79
+ - model: /home/alpha/Models/Raw/chargoddard_Yi-34B-Llama
80
+ # No parameters necessary for base model
81
+ - model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
82
+ #200K base to extend the context of 4K models, max density as we *want* it to 'interfere'
83
+ parameters:
84
+ weight: 0.33
85
+ density: 1
86
+ - model: /home/alpha/Models/Raw/Weyaxi_Nous-Hermes-2-SUS-Chat-34B-Slerp
87
+ parameters:
88
+ weight: 0.15
89
+ density: 0.36
90
+ - model: /home/alpha/Models/Raw/jondurbin_bagel-dpo-34b-v0.2
91
+ #Mix dpo with sft to tone down dpo
92
+ parameters:
93
+ weight: 0.06
94
+ density: 0.36
95
+ - model: /home/alpha/Models/Raw/jondurbin_bagel-34b-v0.2
96
+ parameters:
97
+ weight: 0.06
98
+ density: 0.41
99
+ - model: /home/alpha/Models/Raw/bhenrym14_platypus-yi-34b
100
+ #Vicuna format
101
+ parameters:
102
+ weight: 0.19
103
+ density: 0.41
104
+ # - model: /home/alpha/Models/Raw/01-ai_Yi-34B-Chat #+/home/alpha/Models/Raw/Doctor-Shotgun_limarpv3-yi-llama-34b-lora
105
+ # #Can't get lora OR base model to work without erroring out?
106
+ # parameters:
107
+ # weight: 0.04
108
+ # density: 0.36
109
+ - model: /home/alpha/Models/Raw/TriadParty_deepsex-34b
110
+ #Base model with no prompt
111
+ parameters:
112
+ weight: 0.21
113
+ density: 0.39
114
+ merge_method: dare_ties
115
+ tokenizer_source: union
116
+ base_model: /home/alpha/Models/Raw/chargoddard_Yi-34B-Llama
117
+ parameters:
118
+ int8_mask: true
119
+ dtype: bfloat16
120
+ name: 4kmerge-v2
121
+ ---
122
+ models:
123
+ - model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
124
+ # No parameters necessary for base model
125
+ - model: /home/alpha/Storage/Models/Raw/migtissera_Tess-34B-v1.4
126
+ #Emphasize the beginning of Vicuna format models
127
+ parameters:
128
+ weight: [0.22, 0.113, 0.113, 0.113, 0.113, 0.113]
129
+ density: 0.61
130
+ - model: /home/alpha/Models/Raw/Mihaiii_Pallas-0.5
131
+ # Vicuna format
132
+ parameters:
133
+ weight: [0.22, 0.113, 0.113, 0.113, 0.113, 0.113]
134
+ density: 0.61
135
+ - model: /home/alpha//Storage/Models/Raw/bhenrym14_airoboros-3_1-yi-34b-200k
136
+ parameters:
137
+ weight: [0.02, 0.081, 0.081, 0.081, 0.081, 0.081]
138
+ density: 0.59
139
+ - model: /home/alpha/Storage/Models/Raw/jondurbin_bagel-34b-v0.2
140
+ #Only the SFT in the main merge since the DPO version seems to have no long context ability at all, and some overfitting(?) issues
141
+ parameters:
142
+ weight: [0.02, 0.093, 0.093, 0.093, 0.093, 0.093]
143
+ density: 0.4
144
+ - model: /home/alpha/Storage/Models/Raw/kyujinpy_PlatYi-34B-200k-Q-FastChat
145
+ parameters:
146
+ weight: [0.02, 0.081, 0.081, 0.081, 0.081, 0.081]
147
+ density: 0.59
148
+ #- model: /home/alpha/Storage/Models/Raw/ehartford_dolphin-2.2-yi-34b-200k
149
+ # Dolphin 200K seems to be funky according to multiple leaderboards and perplexity tests?
150
+ # parameters:
151
+ # weight: 0.15
152
+ # density: 0.6
153
+ - model: /home/alpha/Models/Raw/adamo1139_Yi-34B-200K-AEZAKMI-v2
154
+ parameters:
155
+ weight: [0.02, 0.096, 0.096, 0.096, 0.096, 0.096]
156
+ density: 0.59
157
+ - model: /home/alpha/Storage/Models/Raw/Nous-Capybara-34B
158
+ parameters:
159
+ weight: [0.21, 0.115, 0.115, 0.115, 0.115, 0.115]
160
+ density: 0.59
161
+ - model: 4kmerge-v2
162
+ #Previous merge
163
+ parameters:
164
+ weight: [0.02, 0.115, 0.115, 0.115, 0.115, 0.115]
165
+ density: 0.4
166
+ - model: /home/alpha/Models/Raw/migtissera_Tess-M-Creative-v1.0
167
+ # Vicuna format
168
+ parameters:
169
+ weight: [0.21, 0.09, 0.09, 0.09, 0.09, 0.09]
170
+ density: 0.61
171
+ - model: /home/alpha/Models/Raw/TriadParty_deepmoney-34b-200k-base
172
+ # No prompt format, native long context full finetune
173
+ parameters:
174
+ weight: [0.04, 0.103, 0.103, 0.103, 0.103, 0.103]
175
+ density: 0.61
176
+ merge_method: dare_ties
177
+ tokenizer_source: union
178
+ base_model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
179
+ parameters:
180
+ int8_mask: true
181
+ dtype: bfloat16
182
+ ```
config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama",
3
+ "architectures": [
4
+ "LlamaForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 7168,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 20480,
13
+ "max_position_embeddings": 200000,
14
+ "model_type": "llama",
15
+ "num_attention_heads": 56,
16
+ "num_hidden_layers": 60,
17
+ "num_key_value_heads": 8,
18
+ "pad_token_id": 0,
19
+ "pretraining_tp": 1,
20
+ "rms_norm_eps": 1e-05,
21
+ "rope_scaling": null,
22
+ "rope_theta": 5000000.0,
23
+ "tie_word_embeddings": false,
24
+ "torch_dtype": "bfloat16",
25
+ "transformers_version": "4.35.2",
26
+ "use_cache": true,
27
+ "vocab_size": 64002
28
+ }
mergekit_config.yml ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ base_model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
2
+ dtype: bfloat16
3
+ merge_method: dare_ties
4
+ parameters:
5
+ int8_mask: 1.0
6
+ slices:
7
+ - sources:
8
+ - layer_range: [0, 60]
9
+ model: /home/alpha/Storage/Models/Raw/chargoddard_Yi-34B-200K-Llama
10
+ - layer_range: [0, 60]
11
+ model: /home/alpha/Storage/Models/Raw/migtissera_Tess-34B-v1.4
12
+ parameters:
13
+ density: 0.61
14
+ weight: [0.22, 0.113, 0.113, 0.113, 0.113, 0.113]
15
+ - layer_range: [0, 60]
16
+ model: /home/alpha/Models/Raw/Mihaiii_Pallas-0.5
17
+ parameters:
18
+ density: 0.61
19
+ weight: [0.22, 0.113, 0.113, 0.113, 0.113, 0.113]
20
+ - layer_range: [0, 60]
21
+ model: /home/alpha//Storage/Models/Raw/bhenrym14_airoboros-3_1-yi-34b-200k
22
+ parameters:
23
+ density: 0.59
24
+ weight: [0.02, 0.081, 0.081, 0.081, 0.081, 0.081]
25
+ - layer_range: [0, 60]
26
+ model: /home/alpha/Storage/Models/Raw/jondurbin_bagel-34b-v0.2
27
+ parameters:
28
+ density: 0.4
29
+ weight: [0.02, 0.093, 0.093, 0.093, 0.093, 0.093]
30
+ - layer_range: [0, 60]
31
+ model: /home/alpha/Storage/Models/Raw/kyujinpy_PlatYi-34B-200k-Q-FastChat
32
+ parameters:
33
+ density: 0.59
34
+ weight: [0.02, 0.081, 0.081, 0.081, 0.081, 0.081]
35
+ - layer_range: [0, 60]
36
+ model: /home/alpha/Models/Raw/adamo1139_Yi-34B-200K-AEZAKMI-v2
37
+ parameters:
38
+ density: 0.59
39
+ weight: [0.02, 0.096, 0.096, 0.096, 0.096, 0.096]
40
+ - layer_range: [0, 60]
41
+ model: /home/alpha/Storage/Models/Raw/Nous-Capybara-34B
42
+ parameters:
43
+ density: 0.59
44
+ weight: [0.21, 0.115, 0.115, 0.115, 0.115, 0.115]
45
+ - layer_range: [0, 60]
46
+ model: /home/alpha/FastModels/v8/4kmerge-v2
47
+ parameters:
48
+ density: 0.4
49
+ weight: [0.02, 0.115, 0.115, 0.115, 0.115, 0.115]
50
+ - layer_range: [0, 60]
51
+ model: /home/alpha/Models/Raw/migtissera_Tess-M-Creative-v1.0
52
+ parameters:
53
+ density: 0.61
54
+ weight: [0.21, 0.09, 0.09, 0.09, 0.09, 0.09]
55
+ - layer_range: [0, 60]
56
+ model: /home/alpha/Models/Raw/TriadParty_deepmoney-34b-200k-base
57
+ parameters:
58
+ density: 0.61
59
+ weight: [0.04, 0.103, 0.103, 0.103, 0.103, 0.103]
60
+ tokenizer_source: union
model-00001-of-00007.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed8a048937055c4ff5101a27482566e6cac5e668a06426958912d1e06826653f
3
+ size 9843263880
model-00002-of-00007.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c5cd1dfb138b3be14e21347ebb9ee234bd9a471c00b07ad14f53834ab05b4b38
3
+ size 9747800976
model-00003-of-00007.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:74c8ea3fb42fd1e93ef68a9cc4d7951e6d91c06afeb7ff4ffcced97fee1912a2
3
+ size 9806549816
model-00004-of-00007.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e0e7de2934f21eb3cde3a9bd2dedcba81580d4e71d1df72e02ce53ed4968052c
3
+ size 9982711400
model-00005-of-00007.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aaac62b4b058ef473aff0ac4d71c0eca37ce5ab57c40067630501520cecd04bc
3
+ size 9806549808
model-00006-of-00007.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4bbe9224aeb782db859317638f2dc6b75832e7029facd77c45fce69488129bc1
3
+ size 9747829904
model-00007-of-00007.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0c4927770499710e97c4603a80c94f3fa1f0e33d19936da770085670687ff84d
3
+ size 9843249472
model.safetensors.index.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"metadata": {"mergekit_version": "0.0.3.2"}, "weight_map": {"model.embed_tokens.weight": "model-00001-of-00007.safetensors", "model.layers.0.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.0.mlp.down_proj.weight": "model-00001-of-00007.safetensors", "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00007.safetensors", "model.layers.0.mlp.up_proj.weight": "model-00001-of-00007.safetensors", "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00007.safetensors", "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00007.safetensors", "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00007.safetensors", "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00007.safetensors", "model.layers.1.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.1.mlp.down_proj.weight": "model-00001-of-00007.safetensors", "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00007.safetensors", "model.layers.1.mlp.up_proj.weight": "model-00001-of-00007.safetensors", "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00007.safetensors", "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00007.safetensors", "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00007.safetensors", "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00007.safetensors", "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00007.safetensors", "model.layers.2.mlp.up_proj.weight": "model-00001-of-00007.safetensors", "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00007.safetensors", "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00007.safetensors", "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00007.safetensors", "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00007.safetensors", "model.layers.2.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.2.mlp.down_proj.weight": "model-00001-of-00007.safetensors", "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.3.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.3.mlp.down_proj.weight": "model-00001-of-00007.safetensors", "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00007.safetensors", "model.layers.3.mlp.up_proj.weight": "model-00001-of-00007.safetensors", "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00007.safetensors", "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00007.safetensors", "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00007.safetensors", "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00007.safetensors", "model.layers.4.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.4.mlp.down_proj.weight": "model-00001-of-00007.safetensors", "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00007.safetensors", "model.layers.4.mlp.up_proj.weight": "model-00001-of-00007.safetensors", "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00007.safetensors", "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00007.safetensors", "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00007.safetensors", "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00007.safetensors", "model.layers.5.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.5.mlp.down_proj.weight": "model-00001-of-00007.safetensors", "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00007.safetensors", "model.layers.5.mlp.up_proj.weight": "model-00001-of-00007.safetensors", "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00007.safetensors", "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00007.safetensors", "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00007.safetensors", "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00007.safetensors", "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00007.safetensors", "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00007.safetensors", "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00007.safetensors", "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00007.safetensors", "model.layers.6.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.6.mlp.down_proj.weight": "model-00001-of-00007.safetensors", "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00007.safetensors", "model.layers.6.mlp.up_proj.weight": "model-00001-of-00007.safetensors", "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.7.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.7.mlp.down_proj.weight": "model-00001-of-00007.safetensors", "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00007.safetensors", "model.layers.7.mlp.up_proj.weight": "model-00001-of-00007.safetensors", "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00007.safetensors", "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00007.safetensors", "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00007.safetensors", "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00007.safetensors", "model.layers.8.input_layernorm.weight": "model-00001-of-00007.safetensors", "model.layers.8.mlp.down_proj.weight": "model-00002-of-00007.safetensors", "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.8.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.9.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.10.input_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.10.mlp.down_proj.weight": "model-00002-of-00007.safetensors", "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.10.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.11.input_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.11.mlp.down_proj.weight": "model-00002-of-00007.safetensors", "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.11.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.12.input_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.12.mlp.down_proj.weight": "model-00002-of-00007.safetensors", "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.12.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.9.input_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.9.mlp.down_proj.weight": "model-00002-of-00007.safetensors", "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.13.input_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.13.mlp.down_proj.weight": "model-00002-of-00007.safetensors", "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.13.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.14.input_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.14.mlp.down_proj.weight": "model-00002-of-00007.safetensors", "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.14.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.15.input_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.15.mlp.down_proj.weight": "model-00002-of-00007.safetensors", "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.15.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00007.safetensors", "model.layers.16.mlp.up_proj.weight": "model-00002-of-00007.safetensors", "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00007.safetensors", "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00007.safetensors", "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00007.safetensors", "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00007.safetensors", "model.layers.16.input_layernorm.weight": "model-00002-of-00007.safetensors", "model.layers.16.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.16.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.17.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.17.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.17.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.17.mlp.up_proj.weight": "model-00003-of-00007.safetensors", "model.layers.17.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.17.self_attn.k_proj.weight": "model-00003-of-00007.safetensors", "model.layers.17.self_attn.o_proj.weight": "model-00003-of-00007.safetensors", "model.layers.17.self_attn.q_proj.weight": "model-00003-of-00007.safetensors", "model.layers.17.self_attn.v_proj.weight": "model-00003-of-00007.safetensors", "model.layers.18.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.18.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.18.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.18.mlp.up_proj.weight": "model-00003-of-00007.safetensors", "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.18.self_attn.k_proj.weight": "model-00003-of-00007.safetensors", "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00007.safetensors", "model.layers.18.self_attn.q_proj.weight": "model-00003-of-00007.safetensors", "model.layers.18.self_attn.v_proj.weight": "model-00003-of-00007.safetensors", "model.layers.19.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.19.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.19.mlp.up_proj.weight": "model-00003-of-00007.safetensors", "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00007.safetensors", "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00007.safetensors", "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00007.safetensors", "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00007.safetensors", "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00007.safetensors", "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00007.safetensors", "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00007.safetensors", "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00007.safetensors", "model.layers.20.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.20.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.20.mlp.up_proj.weight": "model-00003-of-00007.safetensors", "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.21.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.21.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.21.mlp.up_proj.weight": "model-00003-of-00007.safetensors", "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00007.safetensors", "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00007.safetensors", "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00007.safetensors", "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00007.safetensors", "model.layers.22.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.22.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.22.mlp.up_proj.weight": "model-00003-of-00007.safetensors", "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00007.safetensors", "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00007.safetensors", "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00007.safetensors", "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00007.safetensors", "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.23.mlp.up_proj.weight": "model-00003-of-00007.safetensors", "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00007.safetensors", "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00007.safetensors", "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00007.safetensors", "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00007.safetensors", "model.layers.23.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.23.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.24.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.24.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.24.mlp.up_proj.weight": "model-00003-of-00007.safetensors", "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00007.safetensors", "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00007.safetensors", "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00007.safetensors", "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00007.safetensors", "model.layers.25.input_layernorm.weight": "model-00003-of-00007.safetensors", "model.layers.25.mlp.down_proj.weight": "model-00003-of-00007.safetensors", "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00007.safetensors", "model.layers.25.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.25.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.25.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.25.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.25.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.25.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.26.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.26.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.26.mlp.gate_proj.weight": "model-00004-of-00007.safetensors", "model.layers.26.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.26.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.26.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.26.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.26.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.26.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.27.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.27.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.27.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.27.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.27.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.27.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.27.mlp.gate_proj.weight": "model-00004-of-00007.safetensors", "model.layers.27.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.27.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.28.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.28.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.28.mlp.gate_proj.weight": "model-00004-of-00007.safetensors", "model.layers.28.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.28.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.28.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.28.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.28.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.28.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.29.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.29.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.29.mlp.gate_proj.weight": "model-00004-of-00007.safetensors", "model.layers.29.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.29.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.29.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.29.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.29.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.29.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.30.mlp.gate_proj.weight": "model-00004-of-00007.safetensors", "model.layers.30.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.30.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.30.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.30.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.30.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.30.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.30.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.30.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.31.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.31.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.31.mlp.gate_proj.weight": "model-00004-of-00007.safetensors", "model.layers.31.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.31.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.31.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.31.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.31.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.31.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.32.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.32.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.32.mlp.gate_proj.weight": "model-00004-of-00007.safetensors", "model.layers.32.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.32.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.32.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.32.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.32.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.32.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.33.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.33.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.33.mlp.gate_proj.weight": "model-00004-of-00007.safetensors", "model.layers.33.mlp.up_proj.weight": "model-00004-of-00007.safetensors", "model.layers.33.post_attention_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.33.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.33.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.33.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.33.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.34.self_attn.k_proj.weight": "model-00004-of-00007.safetensors", "model.layers.34.self_attn.o_proj.weight": "model-00004-of-00007.safetensors", "model.layers.34.self_attn.q_proj.weight": "model-00004-of-00007.safetensors", "model.layers.34.self_attn.v_proj.weight": "model-00004-of-00007.safetensors", "model.layers.34.input_layernorm.weight": "model-00004-of-00007.safetensors", "model.layers.34.mlp.down_proj.weight": "model-00004-of-00007.safetensors", "model.layers.34.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.34.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.34.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.35.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.35.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.35.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.35.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.35.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.35.self_attn.k_proj.weight": "model-00005-of-00007.safetensors", "model.layers.35.self_attn.o_proj.weight": "model-00005-of-00007.safetensors", "model.layers.35.self_attn.q_proj.weight": "model-00005-of-00007.safetensors", "model.layers.35.self_attn.v_proj.weight": "model-00005-of-00007.safetensors", "model.layers.36.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.36.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.36.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.36.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.36.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.36.self_attn.k_proj.weight": "model-00005-of-00007.safetensors", "model.layers.36.self_attn.o_proj.weight": "model-00005-of-00007.safetensors", "model.layers.36.self_attn.q_proj.weight": "model-00005-of-00007.safetensors", "model.layers.36.self_attn.v_proj.weight": "model-00005-of-00007.safetensors", "model.layers.37.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.37.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.37.self_attn.k_proj.weight": "model-00005-of-00007.safetensors", "model.layers.37.self_attn.o_proj.weight": "model-00005-of-00007.safetensors", "model.layers.37.self_attn.q_proj.weight": "model-00005-of-00007.safetensors", "model.layers.37.self_attn.v_proj.weight": "model-00005-of-00007.safetensors", "model.layers.37.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.37.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.37.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.38.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.38.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.38.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.38.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.38.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.38.self_attn.k_proj.weight": "model-00005-of-00007.safetensors", "model.layers.38.self_attn.o_proj.weight": "model-00005-of-00007.safetensors", "model.layers.38.self_attn.q_proj.weight": "model-00005-of-00007.safetensors", "model.layers.38.self_attn.v_proj.weight": "model-00005-of-00007.safetensors", "model.layers.39.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.39.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.39.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.39.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.39.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.39.self_attn.k_proj.weight": "model-00005-of-00007.safetensors", "model.layers.39.self_attn.o_proj.weight": "model-00005-of-00007.safetensors", "model.layers.39.self_attn.q_proj.weight": "model-00005-of-00007.safetensors", "model.layers.39.self_attn.v_proj.weight": "model-00005-of-00007.safetensors", "model.layers.40.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.40.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.40.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.40.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.40.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.40.self_attn.k_proj.weight": "model-00005-of-00007.safetensors", "model.layers.40.self_attn.o_proj.weight": "model-00005-of-00007.safetensors", "model.layers.40.self_attn.q_proj.weight": "model-00005-of-00007.safetensors", "model.layers.40.self_attn.v_proj.weight": "model-00005-of-00007.safetensors", "model.layers.41.self_attn.k_proj.weight": "model-00005-of-00007.safetensors", "model.layers.41.self_attn.o_proj.weight": "model-00005-of-00007.safetensors", "model.layers.41.self_attn.q_proj.weight": "model-00005-of-00007.safetensors", "model.layers.41.self_attn.v_proj.weight": "model-00005-of-00007.safetensors", "model.layers.41.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.41.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.41.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.41.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.41.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.42.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.42.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.42.mlp.gate_proj.weight": "model-00005-of-00007.safetensors", "model.layers.42.mlp.up_proj.weight": "model-00005-of-00007.safetensors", "model.layers.42.post_attention_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.42.self_attn.k_proj.weight": "model-00005-of-00007.safetensors", "model.layers.42.self_attn.o_proj.weight": "model-00005-of-00007.safetensors", "model.layers.42.self_attn.q_proj.weight": "model-00005-of-00007.safetensors", "model.layers.42.self_attn.v_proj.weight": "model-00005-of-00007.safetensors", "model.layers.43.input_layernorm.weight": "model-00005-of-00007.safetensors", "model.layers.43.mlp.down_proj.weight": "model-00005-of-00007.safetensors", "model.layers.43.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.43.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.43.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.43.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.43.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.43.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.43.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.44.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.44.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.44.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.44.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.44.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.44.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.44.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.44.mlp.down_proj.weight": "model-00006-of-00007.safetensors", "model.layers.44.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.45.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.45.mlp.down_proj.weight": "model-00006-of-00007.safetensors", "model.layers.45.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.45.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.45.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.45.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.45.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.45.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.45.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.46.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.46.mlp.down_proj.weight": "model-00006-of-00007.safetensors", "model.layers.46.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.46.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.46.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.46.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.46.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.46.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.46.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.47.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.47.mlp.down_proj.weight": "model-00006-of-00007.safetensors", "model.layers.47.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.47.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.47.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.47.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.47.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.47.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.47.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.48.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.48.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.48.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.48.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.48.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.48.mlp.down_proj.weight": "model-00006-of-00007.safetensors", "model.layers.48.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.48.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.48.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.49.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.49.mlp.down_proj.weight": "model-00006-of-00007.safetensors", "model.layers.49.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.49.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.49.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.49.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.49.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.49.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.49.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.50.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.50.mlp.down_proj.weight": "model-00006-of-00007.safetensors", "model.layers.50.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.50.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.50.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.50.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.50.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.50.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.50.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.51.mlp.gate_proj.weight": "model-00006-of-00007.safetensors", "model.layers.51.mlp.up_proj.weight": "model-00006-of-00007.safetensors", "model.layers.51.self_attn.k_proj.weight": "model-00006-of-00007.safetensors", "model.layers.51.self_attn.o_proj.weight": "model-00006-of-00007.safetensors", "model.layers.51.self_attn.q_proj.weight": "model-00006-of-00007.safetensors", "model.layers.51.self_attn.v_proj.weight": "model-00006-of-00007.safetensors", "model.layers.51.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.51.mlp.down_proj.weight": "model-00006-of-00007.safetensors", "model.layers.51.post_attention_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.52.input_layernorm.weight": "model-00006-of-00007.safetensors", "model.layers.52.mlp.down_proj.weight": "model-00007-of-00007.safetensors", "model.layers.52.mlp.gate_proj.weight": "model-00007-of-00007.safetensors", "model.layers.52.mlp.up_proj.weight": "model-00007-of-00007.safetensors", "model.layers.52.post_attention_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.52.self_attn.k_proj.weight": "model-00007-of-00007.safetensors", "model.layers.52.self_attn.o_proj.weight": "model-00007-of-00007.safetensors", "model.layers.52.self_attn.q_proj.weight": "model-00007-of-00007.safetensors", "model.layers.52.self_attn.v_proj.weight": "model-00007-of-00007.safetensors", "model.layers.53.input_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.53.mlp.down_proj.weight": "model-00007-of-00007.safetensors", "model.layers.53.mlp.gate_proj.weight": "model-00007-of-00007.safetensors", "model.layers.53.mlp.up_proj.weight": "model-00007-of-00007.safetensors", "model.layers.53.post_attention_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.53.self_attn.k_proj.weight": "model-00007-of-00007.safetensors", "model.layers.53.self_attn.o_proj.weight": "model-00007-of-00007.safetensors", "model.layers.53.self_attn.q_proj.weight": "model-00007-of-00007.safetensors", "model.layers.53.self_attn.v_proj.weight": "model-00007-of-00007.safetensors", "model.layers.54.input_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.54.mlp.down_proj.weight": "model-00007-of-00007.safetensors", "model.layers.54.mlp.gate_proj.weight": "model-00007-of-00007.safetensors", "model.layers.54.mlp.up_proj.weight": "model-00007-of-00007.safetensors", "model.layers.54.post_attention_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.54.self_attn.k_proj.weight": "model-00007-of-00007.safetensors", "model.layers.54.self_attn.o_proj.weight": "model-00007-of-00007.safetensors", "model.layers.54.self_attn.q_proj.weight": "model-00007-of-00007.safetensors", "model.layers.54.self_attn.v_proj.weight": "model-00007-of-00007.safetensors", "model.layers.55.self_attn.k_proj.weight": "model-00007-of-00007.safetensors", "model.layers.55.self_attn.o_proj.weight": "model-00007-of-00007.safetensors", "model.layers.55.self_attn.q_proj.weight": "model-00007-of-00007.safetensors", "model.layers.55.self_attn.v_proj.weight": "model-00007-of-00007.safetensors", "model.layers.55.input_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.55.mlp.down_proj.weight": "model-00007-of-00007.safetensors", "model.layers.55.mlp.gate_proj.weight": "model-00007-of-00007.safetensors", "model.layers.55.mlp.up_proj.weight": "model-00007-of-00007.safetensors", "model.layers.55.post_attention_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.56.input_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.56.mlp.down_proj.weight": "model-00007-of-00007.safetensors", "model.layers.56.mlp.gate_proj.weight": "model-00007-of-00007.safetensors", "model.layers.56.mlp.up_proj.weight": "model-00007-of-00007.safetensors", "model.layers.56.post_attention_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.56.self_attn.k_proj.weight": "model-00007-of-00007.safetensors", "model.layers.56.self_attn.o_proj.weight": "model-00007-of-00007.safetensors", "model.layers.56.self_attn.q_proj.weight": "model-00007-of-00007.safetensors", "model.layers.56.self_attn.v_proj.weight": "model-00007-of-00007.safetensors", "model.layers.57.input_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.57.mlp.down_proj.weight": "model-00007-of-00007.safetensors", "model.layers.57.mlp.gate_proj.weight": "model-00007-of-00007.safetensors", "model.layers.57.mlp.up_proj.weight": "model-00007-of-00007.safetensors", "model.layers.57.post_attention_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.57.self_attn.k_proj.weight": "model-00007-of-00007.safetensors", "model.layers.57.self_attn.o_proj.weight": "model-00007-of-00007.safetensors", "model.layers.57.self_attn.q_proj.weight": "model-00007-of-00007.safetensors", "model.layers.57.self_attn.v_proj.weight": "model-00007-of-00007.safetensors", "model.layers.58.mlp.gate_proj.weight": "model-00007-of-00007.safetensors", "model.layers.58.mlp.up_proj.weight": "model-00007-of-00007.safetensors", "model.layers.58.self_attn.k_proj.weight": "model-00007-of-00007.safetensors", "model.layers.58.self_attn.o_proj.weight": "model-00007-of-00007.safetensors", "model.layers.58.self_attn.q_proj.weight": "model-00007-of-00007.safetensors", "model.layers.58.self_attn.v_proj.weight": "model-00007-of-00007.safetensors", "lm_head.weight": "model-00007-of-00007.safetensors", "model.layers.58.input_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.58.mlp.down_proj.weight": "model-00007-of-00007.safetensors", "model.layers.58.post_attention_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.59.input_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.59.mlp.down_proj.weight": "model-00007-of-00007.safetensors", "model.layers.59.mlp.gate_proj.weight": "model-00007-of-00007.safetensors", "model.layers.59.mlp.up_proj.weight": "model-00007-of-00007.safetensors", "model.layers.59.post_attention_layernorm.weight": "model-00007-of-00007.safetensors", "model.layers.59.self_attn.k_proj.weight": "model-00007-of-00007.safetensors", "model.layers.59.self_attn.o_proj.weight": "model-00007-of-00007.safetensors", "model.layers.59.self_attn.q_proj.weight": "model-00007-of-00007.safetensors", "model.layers.59.self_attn.v_proj.weight": "model-00007-of-00007.safetensors", "model.norm.weight": "model-00007-of-00007.safetensors"}}
special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<|startoftext|>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|endoftext|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<unk>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "unk_token": {
24
+ "content": "<unk>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<unk>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": false
10
+ },
11
+ "1": {
12
+ "content": "<|startoftext|>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": false
18
+ },
19
+ "2": {
20
+ "content": "<|endoftext|>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": false
26
+ },
27
+ "64000": {
28
+ "content": "<s>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": false
34
+ },
35
+ "64001": {
36
+ "content": "</s>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": false
42
+ }
43
+ },
44
+ "bos_token": "<|startoftext|>",
45
+ "clean_up_tokenization_spaces": false,
46
+ "eos_token": "<|endoftext|>",
47
+ "legacy": false,
48
+ "model_max_length": 200000,
49
+ "pad_token": "<unk>",
50
+ "padding_side": "right",
51
+ "sp_model_kwargs": {},
52
+ "spaces_between_special_tokens": false,
53
+ "tokenizer_class": "LlamaTokenizer",
54
+ "truncation_side": "right",
55
+ "unk_token": "<unk>",
56
+ "use_default_system_prompt": false
57
+ }