Hugging Face's logo

KaraKaraWitch
/
UnFimbulvetr-20B

Text Generation
Transformers
Safetensors
llama
mergekit
Merge
text-generation-inference
Inference Endpoints
Model card Files Community
1
KaraKaraWitch commited on
Commit
1d45d4d
•
1 Parent(s): c48d370

Upload folder using huggingface_hub

Browse files
Files changed (8) hide show
  1. README.md +8 -278
  2. config.json +1 -1
  3. mergekit_config.yml +6 -276
  4. model-00001-of-00004.safetensors +3 -0
  5. model-00002-of-00004.safetensors +3 -0
  6. model-00003-of-00004.safetensors +3 -0
  7. model-00004-of-00004.safetensors +3 -0
  8. model.safetensors.index.json +0 -0
README.md CHANGED
@@ -6,7 +6,7 @@ tags:
6
  - merge
7
 
8
  ---
9
- # UnFimbulator (1)
10
 
11
  This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
12
 
@@ -18,7 +18,7 @@ This model was merged using the passthrough merge method.
18
  ### Models Merged
19
 
20
  The following models were included in the merge:
21
- * Sao10K/Fimbulvetr-11B-v2 (FimbMagic)
22
 
23
  ### Configuration
24
 
@@ -28,292 +28,22 @@ The following YAML configuration was used to produce this model:
28
  slices:
29
  - sources:
30
  - model: FimbMagic
31
- layer_range: [0, 1]
32
  - sources:
33
  - model: FimbMagic
34
- layer_range: [0, 1]
35
  - sources:
36
  - model: FimbMagic
37
- layer_range: [1, 2]
38
  - sources:
39
  - model: FimbMagic
40
- layer_range: [1, 2]
41
  - sources:
42
  - model: FimbMagic
43
- layer_range: [2, 3]
44
  - sources:
45
  - model: FimbMagic
46
- layer_range: [2, 3]
47
- - sources:
48
- - model: FimbMagic
49
- layer_range: [3, 4]
50
- - sources:
51
- - model: FimbMagic
52
- layer_range: [3, 4]
53
- - sources:
54
- - model: FimbMagic
55
- layer_range: [4, 5]
56
- - sources:
57
- - model: FimbMagic
58
- layer_range: [4, 5]
59
- - sources:
60
- - model: FimbMagic
61
- layer_range: [5, 6]
62
- - sources:
63
- - model: FimbMagic
64
- layer_range: [5, 6]
65
- - sources:
66
- - model: FimbMagic
67
- layer_range: [6, 7]
68
- - sources:
69
- - model: FimbMagic
70
- layer_range: [6, 7]
71
- - sources:
72
- - model: FimbMagic
73
- layer_range: [7, 8]
74
- - sources:
75
- - model: FimbMagic
76
- layer_range: [7, 8]
77
- - sources:
78
- - model: FimbMagic
79
- layer_range: [8, 9]
80
- - sources:
81
- - model: FimbMagic
82
- layer_range: [8, 9]
83
- - sources:
84
- - model: FimbMagic
85
- layer_range: [9, 10]
86
- - sources:
87
- - model: FimbMagic
88
- layer_range: [9, 10]
89
- - sources:
90
- - model: FimbMagic
91
- layer_range: [10, 11]
92
- - sources:
93
- - model: FimbMagic
94
- layer_range: [10, 11]
95
- - sources:
96
- - model: FimbMagic
97
- layer_range: [11, 12]
98
- - sources:
99
- - model: FimbMagic
100
- layer_range: [11, 12]
101
- - sources:
102
- - model: FimbMagic
103
- layer_range: [12, 13]
104
- - sources:
105
- - model: FimbMagic
106
- layer_range: [12, 13]
107
- - sources:
108
- - model: FimbMagic
109
- layer_range: [13, 14]
110
- - sources:
111
- - model: FimbMagic
112
- layer_range: [13, 14]
113
- - sources:
114
- - model: FimbMagic
115
- layer_range: [14, 15]
116
- - sources:
117
- - model: FimbMagic
118
- layer_range: [14, 15]
119
- - sources:
120
- - model: FimbMagic
121
- layer_range: [15, 16]
122
- - sources:
123
- - model: FimbMagic
124
- layer_range: [15, 16]
125
- - sources:
126
- - model: FimbMagic
127
- layer_range: [16, 17]
128
- - sources:
129
- - model: FimbMagic
130
- layer_range: [16, 17]
131
- - sources:
132
- - model: FimbMagic
133
- layer_range: [17, 18]
134
- - sources:
135
- - model: FimbMagic
136
- layer_range: [17, 18]
137
- - sources:
138
- - model: FimbMagic
139
- layer_range: [18, 19]
140
- - sources:
141
- - model: FimbMagic
142
- layer_range: [18, 19]
143
- - sources:
144
- - model: FimbMagic
145
- layer_range: [19, 20]
146
- - sources:
147
- - model: FimbMagic
148
- layer_range: [19, 20]
149
- - sources:
150
- - model: FimbMagic
151
- layer_range: [20, 21]
152
- - sources:
153
- - model: FimbMagic
154
- layer_range: [20, 21]
155
- - sources:
156
- - model: FimbMagic
157
- layer_range: [21, 22]
158
- - sources:
159
- - model: FimbMagic
160
- layer_range: [21, 22]
161
- - sources:
162
- - model: FimbMagic
163
- layer_range: [22, 23]
164
- - sources:
165
- - model: FimbMagic
166
- layer_range: [22, 23]
167
- - sources:
168
- - model: FimbMagic
169
- layer_range: [23, 24]
170
- - sources:
171
- - model: FimbMagic
172
- layer_range: [23, 24]
173
- - sources:
174
- - model: FimbMagic
175
- layer_range: [24, 25]
176
- - sources:
177
- - model: FimbMagic
178
- layer_range: [24, 25]
179
- - sources:
180
- - model: FimbMagic
181
- layer_range: [25, 26]
182
- - sources:
183
- - model: FimbMagic
184
- layer_range: [25, 26]
185
- - sources:
186
- - model: FimbMagic
187
- layer_range: [26, 27]
188
- - sources:
189
- - model: FimbMagic
190
- layer_range: [26, 27]
191
- - sources:
192
- - model: FimbMagic
193
- layer_range: [27, 28]
194
- - sources:
195
- - model: FimbMagic
196
- layer_range: [27, 28]
197
- - sources:
198
- - model: FimbMagic
199
- layer_range: [28, 29]
200
- - sources:
201
- - model: FimbMagic
202
- layer_range: [28, 29]
203
- - sources:
204
- - model: FimbMagic
205
- layer_range: [29, 30]
206
- - sources:
207
- - model: FimbMagic
208
- layer_range: [29, 30]
209
- - sources:
210
- - model: FimbMagic
211
- layer_range: [30, 31]
212
- - sources:
213
- - model: FimbMagic
214
- layer_range: [30, 31]
215
- - sources:
216
- - model: FimbMagic
217
- layer_range: [31, 32]
218
- - sources:
219
- - model: FimbMagic
220
- layer_range: [31, 32]
221
- - sources:
222
- - model: FimbMagic
223
- layer_range: [32, 33]
224
- - sources:
225
- - model: FimbMagic
226
- layer_range: [32, 33]
227
- - sources:
228
- - model: FimbMagic
229
- layer_range: [33, 34]
230
- - sources:
231
- - model: FimbMagic
232
- layer_range: [33, 34]
233
- - sources:
234
- - model: FimbMagic
235
- layer_range: [34, 35]
236
- - sources:
237
- - model: FimbMagic
238
- layer_range: [34, 35]
239
- - sources:
240
- - model: FimbMagic
241
- layer_range: [35, 36]
242
- - sources:
243
- - model: FimbMagic
244
- layer_range: [35, 36]
245
- - sources:
246
- - model: FimbMagic
247
- layer_range: [36, 37]
248
- - sources:
249
- - model: FimbMagic
250
- layer_range: [36, 37]
251
- - sources:
252
- - model: FimbMagic
253
- layer_range: [37, 38]
254
- - sources:
255
- - model: FimbMagic
256
- layer_range: [37, 38]
257
- - sources:
258
- - model: FimbMagic
259
- layer_range: [38, 39]
260
- - sources:
261
- - model: FimbMagic
262
- layer_range: [38, 39]
263
- - sources:
264
- - model: FimbMagic
265
- layer_range: [39, 40]
266
- - sources:
267
- - model: FimbMagic
268
- layer_range: [39, 40]
269
- - sources:
270
- - model: FimbMagic
271
- layer_range: [40, 41]
272
- - sources:
273
- - model: FimbMagic
274
- layer_range: [40, 41]
275
- - sources:
276
- - model: FimbMagic
277
- layer_range: [41, 42]
278
- - sources:
279
- - model: FimbMagic
280
- layer_range: [41, 42]
281
- - sources:
282
- - model: FimbMagic
283
- layer_range: [42, 43]
284
- - sources:
285
- - model: FimbMagic
286
- layer_range: [42, 43]
287
- - sources:
288
- - model: FimbMagic
289
- layer_range: [43, 44]
290
- - sources:
291
- - model: FimbMagic
292
- layer_range: [43, 44]
293
- - sources:
294
- - model: FimbMagic
295
- layer_range: [44, 45]
296
- - sources:
297
- - model: FimbMagic
298
- layer_range: [44, 45]
299
- - sources:
300
- - model: FimbMagic
301
- layer_range: [45, 46]
302
- - sources:
303
- - model: FimbMagic
304
- layer_range: [45, 46]
305
- - sources:
306
- - model: FimbMagic
307
- layer_range: [46, 47]
308
- - sources:
309
- - model: FimbMagic
310
- layer_range: [46, 47]
311
- - sources:
312
- - model: FimbMagic
313
- layer_range: [47, 48]
314
- - sources:
315
- - model: FimbMagic
316
- layer_range: [47, 48]
317
  merge_method: passthrough
318
  dtype: bfloat16
319
  ```
 
6
  - merge
7
 
8
  ---
9
+ # Untitled Model (1)
10
 
11
  This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
12
 
 
18
  ### Models Merged
19
 
20
  The following models were included in the merge:
21
+ * FimbMagic
22
 
23
  ### Configuration
24
 
 
28
  slices:
29
  - sources:
30
  - model: FimbMagic
31
+ layer_range: [0, 13]
32
  - sources:
33
  - model: FimbMagic
34
+ layer_range: [8, 13]
35
  - sources:
36
  - model: FimbMagic
37
+ layer_range: [12, 36]
38
  - sources:
39
  - model: FimbMagic
40
+ layer_range: [12, 36]
41
  - sources:
42
  - model: FimbMagic
43
+ layer_range: [36, 48]
44
  - sources:
45
  - model: FimbMagic
46
+ layer_range: [36, 48]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
  merge_method: passthrough
48
  dtype: bfloat16
49
  ```
config.json CHANGED
@@ -14,7 +14,7 @@
14
  "max_position_embeddings": 4096,
15
  "model_type": "llama",
16
  "num_attention_heads": 32,
17
- "num_hidden_layers": 96,
18
  "num_key_value_heads": 8,
19
  "pretraining_tp": 1,
20
  "rms_norm_eps": 1e-05,
 
14
  "max_position_embeddings": 4096,
15
  "model_type": "llama",
16
  "num_attention_heads": 32,
17
+ "num_hidden_layers": 90,
18
  "num_key_value_heads": 8,
19
  "pretraining_tp": 1,
20
  "rms_norm_eps": 1e-05,
mergekit_config.yml CHANGED
@@ -1,291 +1,21 @@
1
  slices:
2
  - sources:
3
  - model: FimbMagic
4
- layer_range: [0, 1]
5
  - sources:
6
  - model: FimbMagic
7
- layer_range: [0, 1]
8
  - sources:
9
  - model: FimbMagic
10
- layer_range: [1, 2]
11
  - sources:
12
  - model: FimbMagic
13
- layer_range: [1, 2]
14
  - sources:
15
  - model: FimbMagic
16
- layer_range: [2, 3]
17
  - sources:
18
  - model: FimbMagic
19
- layer_range: [2, 3]
20
- - sources:
21
- - model: FimbMagic
22
- layer_range: [3, 4]
23
- - sources:
24
- - model: FimbMagic
25
- layer_range: [3, 4]
26
- - sources:
27
- - model: FimbMagic
28
- layer_range: [4, 5]
29
- - sources:
30
- - model: FimbMagic
31
- layer_range: [4, 5]
32
- - sources:
33
- - model: FimbMagic
34
- layer_range: [5, 6]
35
- - sources:
36
- - model: FimbMagic
37
- layer_range: [5, 6]
38
- - sources:
39
- - model: FimbMagic
40
- layer_range: [6, 7]
41
- - sources:
42
- - model: FimbMagic
43
- layer_range: [6, 7]
44
- - sources:
45
- - model: FimbMagic
46
- layer_range: [7, 8]
47
- - sources:
48
- - model: FimbMagic
49
- layer_range: [7, 8]
50
- - sources:
51
- - model: FimbMagic
52
- layer_range: [8, 9]
53
- - sources:
54
- - model: FimbMagic
55
- layer_range: [8, 9]
56
- - sources:
57
- - model: FimbMagic
58
- layer_range: [9, 10]
59
- - sources:
60
- - model: FimbMagic
61
- layer_range: [9, 10]
62
- - sources:
63
- - model: FimbMagic
64
- layer_range: [10, 11]
65
- - sources:
66
- - model: FimbMagic
67
- layer_range: [10, 11]
68
- - sources:
69
- - model: FimbMagic
70
- layer_range: [11, 12]
71
- - sources:
72
- - model: FimbMagic
73
- layer_range: [11, 12]
74
- - sources:
75
- - model: FimbMagic
76
- layer_range: [12, 13]
77
- - sources:
78
- - model: FimbMagic
79
- layer_range: [12, 13]
80
- - sources:
81
- - model: FimbMagic
82
- layer_range: [13, 14]
83
- - sources:
84
- - model: FimbMagic
85
- layer_range: [13, 14]
86
- - sources:
87
- - model: FimbMagic
88
- layer_range: [14, 15]
89
- - sources:
90
- - model: FimbMagic
91
- layer_range: [14, 15]
92
- - sources:
93
- - model: FimbMagic
94
- layer_range: [15, 16]
95
- - sources:
96
- - model: FimbMagic
97
- layer_range: [15, 16]
98
- - sources:
99
- - model: FimbMagic
100
- layer_range: [16, 17]
101
- - sources:
102
- - model: FimbMagic
103
- layer_range: [16, 17]
104
- - sources:
105
- - model: FimbMagic
106
- layer_range: [17, 18]
107
- - sources:
108
- - model: FimbMagic
109
- layer_range: [17, 18]
110
- - sources:
111
- - model: FimbMagic
112
- layer_range: [18, 19]
113
- - sources:
114
- - model: FimbMagic
115
- layer_range: [18, 19]
116
- - sources:
117
- - model: FimbMagic
118
- layer_range: [19, 20]
119
- - sources:
120
- - model: FimbMagic
121
- layer_range: [19, 20]
122
- - sources:
123
- - model: FimbMagic
124
- layer_range: [20, 21]
125
- - sources:
126
- - model: FimbMagic
127
- layer_range: [20, 21]
128
- - sources:
129
- - model: FimbMagic
130
- layer_range: [21, 22]
131
- - sources:
132
- - model: FimbMagic
133
- layer_range: [21, 22]
134
- - sources:
135
- - model: FimbMagic
136
- layer_range: [22, 23]
137
- - sources:
138
- - model: FimbMagic
139
- layer_range: [22, 23]
140
- - sources:
141
- - model: FimbMagic
142
- layer_range: [23, 24]
143
- - sources:
144
- - model: FimbMagic
145
- layer_range: [23, 24]
146
- - sources:
147
- - model: FimbMagic
148
- layer_range: [24, 25]
149
- - sources:
150
- - model: FimbMagic
151
- layer_range: [24, 25]
152
- - sources:
153
- - model: FimbMagic
154
- layer_range: [25, 26]
155
- - sources:
156
- - model: FimbMagic
157
- layer_range: [25, 26]
158
- - sources:
159
- - model: FimbMagic
160
- layer_range: [26, 27]
161
- - sources:
162
- - model: FimbMagic
163
- layer_range: [26, 27]
164
- - sources:
165
- - model: FimbMagic
166
- layer_range: [27, 28]
167
- - sources:
168
- - model: FimbMagic
169
- layer_range: [27, 28]
170
- - sources:
171
- - model: FimbMagic
172
- layer_range: [28, 29]
173
- - sources:
174
- - model: FimbMagic
175
- layer_range: [28, 29]
176
- - sources:
177
- - model: FimbMagic
178
- layer_range: [29, 30]
179
- - sources:
180
- - model: FimbMagic
181
- layer_range: [29, 30]
182
- - sources:
183
- - model: FimbMagic
184
- layer_range: [30, 31]
185
- - sources:
186
- - model: FimbMagic
187
- layer_range: [30, 31]
188
- - sources:
189
- - model: FimbMagic
190
- layer_range: [31, 32]
191
- - sources:
192
- - model: FimbMagic
193
- layer_range: [31, 32]
194
- - sources:
195
- - model: FimbMagic
196
- layer_range: [32, 33]
197
- - sources:
198
- - model: FimbMagic
199
- layer_range: [32, 33]
200
- - sources:
201
- - model: FimbMagic
202
- layer_range: [33, 34]
203
- - sources:
204
- - model: FimbMagic
205
- layer_range: [33, 34]
206
- - sources:
207
- - model: FimbMagic
208
- layer_range: [34, 35]
209
- - sources:
210
- - model: FimbMagic
211
- layer_range: [34, 35]
212
- - sources:
213
- - model: FimbMagic
214
- layer_range: [35, 36]
215
- - sources:
216
- - model: FimbMagic
217
- layer_range: [35, 36]
218
- - sources:
219
- - model: FimbMagic
220
- layer_range: [36, 37]
221
- - sources:
222
- - model: FimbMagic
223
- layer_range: [36, 37]
224
- - sources:
225
- - model: FimbMagic
226
- layer_range: [37, 38]
227
- - sources:
228
- - model: FimbMagic
229
- layer_range: [37, 38]
230
- - sources:
231
- - model: FimbMagic
232
- layer_range: [38, 39]
233
- - sources:
234
- - model: FimbMagic
235
- layer_range: [38, 39]
236
- - sources:
237
- - model: FimbMagic
238
- layer_range: [39, 40]
239
- - sources:
240
- - model: FimbMagic
241
- layer_range: [39, 40]
242
- - sources:
243
- - model: FimbMagic
244
- layer_range: [40, 41]
245
- - sources:
246
- - model: FimbMagic
247
- layer_range: [40, 41]
248
- - sources:
249
- - model: FimbMagic
250
- layer_range: [41, 42]
251
- - sources:
252
- - model: FimbMagic
253
- layer_range: [41, 42]
254
- - sources:
255
- - model: FimbMagic
256
- layer_range: [42, 43]
257
- - sources:
258
- - model: FimbMagic
259
- layer_range: [42, 43]
260
- - sources:
261
- - model: FimbMagic
262
- layer_range: [43, 44]
263
- - sources:
264
- - model: FimbMagic
265
- layer_range: [43, 44]
266
- - sources:
267
- - model: FimbMagic
268
- layer_range: [44, 45]
269
- - sources:
270
- - model: FimbMagic
271
- layer_range: [44, 45]
272
- - sources:
273
- - model: FimbMagic
274
- layer_range: [45, 46]
275
- - sources:
276
- - model: FimbMagic
277
- layer_range: [45, 46]
278
- - sources:
279
- - model: FimbMagic
280
- layer_range: [46, 47]
281
- - sources:
282
- - model: FimbMagic
283
- layer_range: [46, 47]
284
- - sources:
285
- - model: FimbMagic
286
- layer_range: [47, 48]
287
- - sources:
288
- - model: FimbMagic
289
- layer_range: [47, 48]
290
  merge_method: passthrough
291
  dtype: bfloat16
 
1
  slices:
2
  - sources:
3
  - model: FimbMagic
4
+ layer_range: [0, 13]
5
  - sources:
6
  - model: FimbMagic
7
+ layer_range: [8, 13]
8
  - sources:
9
  - model: FimbMagic
10
+ layer_range: [12, 36]
11
  - sources:
12
  - model: FimbMagic
13
+ layer_range: [12, 36]
14
  - sources:
15
  - model: FimbMagic
16
+ layer_range: [36, 48]
17
  - sources:
18
  - model: FimbMagic
19
+ layer_range: [36, 48]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
  merge_method: passthrough
21
  dtype: bfloat16
model-00001-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8a306b16dbc277119a01c4e34d677038b0c5a8eda2606a9158c924291e1e667a
3
+ size 9970243664
model-00002-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:949c154ff97cdb7b57f42a3837ed0f7c1551eeb38380556ac79c918b932f4278
3
+ size 9898941440
model-00003-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5352723459becdfd8fe877e16e5a584bdebecf59f70132a9c5c13b0e5e80753
3
+ size 9999638680
model-00004-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:548fb5ae6a72dfbc915c0ffb87bd17a172001ea2d33f98cfa36ab360a77442b5
3
+ size 9915727160
model.safetensors.index.json CHANGED
The diff for this file is too large to render. See raw diff