Hugging Face's logo

RichardErkhov
/
NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf

GGUF
Inference Endpoints
conversational
Model card Files Community
RichardErkhov commited on
Commit
6a4ca3b
•
1 Parent(s): f4284f1

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +101 -0
README.md ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ Boundary-Solar-Chat-2x10.7B-MoE - GGUF
11
+ - Model creator: https://huggingface.co/NotAiLOL/
12
+ - Original model: https://huggingface.co/NotAiLOL/Boundary-Solar-Chat-2x10.7B-MoE/
13
+
14
+
15
+ | Name | Quant method | Size |
16
+ | ---- | ---- | ---- |
17
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q2_K.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q2_K.gguf) | Q2_K | 6.58GB |
18
+ | [Boundary-Solar-Chat-2x10.7B-MoE.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.IQ3_XS.gguf) | IQ3_XS | 7.34GB |
19
+ | [Boundary-Solar-Chat-2x10.7B-MoE.IQ3_S.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.IQ3_S.gguf) | IQ3_S | 7.75GB |
20
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q3_K_S.gguf) | Q3_K_S | 7.73GB |
21
+ | [Boundary-Solar-Chat-2x10.7B-MoE.IQ3_M.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.IQ3_M.gguf) | IQ3_M | 7.94GB |
22
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q3_K.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q3_K.gguf) | Q3_K | 8.59GB |
23
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q3_K_M.gguf) | Q3_K_M | 8.59GB |
24
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q3_K_L.gguf) | Q3_K_L | 9.32GB |
25
+ | [Boundary-Solar-Chat-2x10.7B-MoE.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.IQ4_XS.gguf) | IQ4_XS | 9.66GB |
26
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q4_0.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q4_0.gguf) | Q4_0 | 10.09GB |
27
+ | [Boundary-Solar-Chat-2x10.7B-MoE.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.IQ4_NL.gguf) | IQ4_NL | 10.19GB |
28
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q4_K_S.gguf) | Q4_K_S | 10.17GB |
29
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q4_K.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q4_K.gguf) | Q4_K | 10.79GB |
30
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q4_K_M.gguf) | Q4_K_M | 10.79GB |
31
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q4_1.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q4_1.gguf) | Q4_1 | 11.2GB |
32
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q5_0.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q5_0.gguf) | Q5_0 | 12.3GB |
33
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q5_K_S.gguf) | Q5_K_S | 12.3GB |
34
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q5_K.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q5_K.gguf) | Q5_K | 12.67GB |
35
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q5_K_M.gguf) | Q5_K_M | 12.67GB |
36
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q5_1.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q5_1.gguf) | Q5_1 | 13.41GB |
37
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q6_K.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q6_K.gguf) | Q6_K | 14.66GB |
38
+ | [Boundary-Solar-Chat-2x10.7B-MoE.Q8_0.gguf](https://huggingface.co/RichardErkhov/NotAiLOL_-_Boundary-Solar-Chat-2x10.7B-MoE-gguf/blob/main/Boundary-Solar-Chat-2x10.7B-MoE.Q8_0.gguf) | Q8_0 | 18.99GB |
39
+
40
+
41
+
42
+
43
+ Original model description:
44
+ ---
45
+ license: apache-2.0
46
+ tags:
47
+ - moe
48
+ - merge
49
+ - mergekit
50
+ - NousResearch/Nous-Hermes-2-SOLAR-10.7B
51
+ - upstage/SOLAR-10.7B-Instruct-v1.0
52
+ - llama
53
+ - Llama
54
+ base_model:
55
+ - NousResearch/Nous-Hermes-2-SOLAR-10.7B
56
+ - upstage/SOLAR-10.7B-Instruct-v1.0
57
+ ---
58
+
59
+ # Boundary-Solar-Chat-2x10.7B-MoE
60
+
61
+ Boundary-Solar-Chat-2x10.7B-MoE is a Mixture of Experts (MoE) made with the following models:
62
+ * [NousResearch/Nous-Hermes-2-SOLAR-10.7B](https://huggingface.co/NousResearch/Nous-Hermes-2-SOLAR-10.7B)
63
+ * [upstage/SOLAR-10.7B-Instruct-v1.0](https://huggingface.co/upstage/SOLAR-10.7B-Instruct-v1.0)
64
+
65
+ ## 🧩 Configuration
66
+
67
+ ```yaml
68
+ base_model: NousResearch/Nous-Hermes-2-SOLAR-10.7B
69
+ dtype: float16
70
+ gate_mode: cheap_embed
71
+ experts:
72
+ - source_model: NousResearch/Nous-Hermes-2-SOLAR-10.7B
73
+ positive_prompts: ["You are a helpful general assistant."]
74
+ - source_model: upstage/SOLAR-10.7B-Instruct-v1.0
75
+ positive_prompts: ["You are assistant for question and answering."]
76
+ ```
77
+
78
+ ## 💻 Usage
79
+
80
+ ```python
81
+ !pip install -qU transformers bitsandbytes accelerate
82
+
83
+ from transformers import AutoTokenizer
84
+ import transformers
85
+ import torch
86
+
87
+ model = "NotAiLOL/Boundary-Solar-Chat-2x10.7B-MoE"
88
+
89
+ tokenizer = AutoTokenizer.from_pretrained(model)
90
+ pipeline = transformers.pipeline(
91
+ "text-generation",
92
+ model=model,
93
+ model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
94
+ )
95
+
96
+ messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
97
+ prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
98
+ outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
99
+ print(outputs[0]["generated_text"])
100
+ ```
101
+