Create README.md

README.md

@@ -0,0 +1,29 @@
+---
+tags:
+- moe
+- llama
+- '3'
+- llama 3
+- 2x8b
+---
+# Llama-3-Teal-Instruct-2x8B-MoE
+This is a experimental MoE created from [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and [nvidia/Llama3-ChatQA-1.5-8B](https://huggingface.co/nvidia/Llama3-ChatQA-1.5-8B) using Mergekit.
+
+Mergekit yaml file:
+```
+base_model: Meta-Llama-3-8B-Instruct
+experts:
+  - source_model: Meta-Llama-3-8B-Instruct
+    positive_prompts:
+    - "explain"
+    - "chat"
+    - "assistant"
+  - source_model: Llama3-ChatQA-1.5-8B
+    positive_prompts:
+    - "python"
+    - "math"
+    - "solve"
+    - "code"
+gate_mode: hidden
+dtype: float16
+```