Hugging Face's logo

aixsatoshi
/
Ex-karakuri-8x12B-chat-v2

Text Generation
Transformers
Safetensors
mixtral
conversational
Inference Endpoints
text-generation-inference
Model card Files Community
Ex-karakuri-8x12B-chat-v2
File size: 2,544 Bytes 
83d71e4
 
 
fae100d
83d71e4
18d3e6c
83d71e4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
---
license: apache-2.0
---
Meta-Llama-3-70bのセルフマージにより120Bにパラメーター数を拡大したモデルの高性能化が報告されています  
今回高品質な日本語LLMである、[karakuri-ai/karakuri-lm-8x7b-chat-v0.1](https://huggingface.co/karakuri-ai/karakuri-lm-8x7b-chat-v0.1)の精度を更に高めるために、"num_hidden_layers": 32、から、56への自己拡張マージを行いました。  
マージに利用したスライスのインターバルから本モデル（Ex-karakuri-8x12B-chat-v2）が非マージ部分4層、[Ex-karakuri-8x12B-chat-v1](https://huggingface.co/aixsatoshi/Ex-karakuri-8x12B-chat-v1)は8層に設定しています



It was inspired by large merges like:
- [Meta-Llama-3-120B-Instruct](https://huggingface.co/mlabonne/Meta-Llama-3-120B-Instruct/)
- [alpindale/goliath-120b](https://huggingface.co/alpindale/goliath-120b)
- [nsfwthrowitaway69/Venus-120b-v1.0](https://huggingface.co/nsfwthrowitaway69/Venus-120b-v1.0)
- [cognitivecomputations/MegaDolphin-120b](https://huggingface.co/cognitivecomputations/MegaDolphin-120b)
- [wolfram/miquliz-120b-v2.0](https://huggingface.co/wolfram/miquliz-120b-v2.0).


```
slices:
- sources:
  - layer_range: [0, 4]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [2, 6]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [4, 8]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [6, 10]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [8, 12]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [10, 14]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [12, 16]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [14, 18]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [16, 20]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [18, 22]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [20, 24]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [22, 26]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [24, 28]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [26, 30]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
- sources:
  - layer_range: [28, 32]
    model: karakuri-ai/karakuri-lm-8x7b-chat-v0.1
merge_method: passthrough
dtype: bfloat16

```