File size: 4,994 Bytes
50c6ad3 7cecd89 50c6ad3 7cecd89 50c6ad3 7cecd89 50c6ad3 7cecd89 50c6ad3 7cecd89 50c6ad3 7cecd89 50c6ad3 7cecd89 50c6ad3 7cecd89 50c6ad3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 |
---
license: apache-2.0
tags:
- moe
- frankenmoe
- merge
- mergekit
- lazymergekit
- CultriX/MonaTrix-v4
- mlabonne/OmniTruthyBeagle-7B-v0
- CultriX/MoNeuTrix-7B-v1
- paulml/OmniBeagleSquaredMBX-v3-7B
base_model:
- CultriX/MonaTrix-v4
- mlabonne/OmniTruthyBeagle-7B-v0
- CultriX/MoNeuTrix-7B-v1
- paulml/OmniBeagleSquaredMBX-v3-7B
---
# MoNeuTrix-MoE-4x7B
MoNeuTrix-MoE-4x7B is a Mixture of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
* [CultriX/MonaTrix-v4](https://huggingface.co/CultriX/MonaTrix-v4)
* [mlabonne/OmniTruthyBeagle-7B-v0](https://huggingface.co/mlabonne/OmniTruthyBeagle-7B-v0)
* [CultriX/MoNeuTrix-7B-v1](https://huggingface.co/CultriX/MoNeuTrix-7B-v1)
* [paulml/OmniBeagleSquaredMBX-v3-7B](https://huggingface.co/paulml/OmniBeagleSquaredMBX-v3-7B)
## 🧩 Configuration
```yaml
base_model: "CultriX/MonaTrix-v4"
dtype: bfloat16
gate:
type: "learned"
temperature: 0.1
scaling_factor: 10
experts:
- source_model: "CultriX/MonaTrix-v4" # Historical Analysis, Geopolitics, and Economic Evaluation
positive_prompts:
- "Historical Analysis"
- "Geopolitical Evaluation"
- "Economic Insights"
- "Policy Analysis"
- "Socio-Economic Impacts"
- "Geopolitical Analysis"
- "Cultural Commentary"
- "Analyze geopolitical"
- "Analyze historic"
- "Analyze historical"
- "Assess the political dynamics of the Cold War and its global impact."
- "Evaluate the historical significance of the Silk Road in ancient trade."
negative_prompts:
- "Technical Writing"
- "Mathematical Problem Solving"
- "Software Development"
- "Artistic Creation"
- "Machine Learning Development"
- "Storywriting"
- "Character Development"
- "Roleplaying"
- "Narrative Creation"
- source_model: "mlabonne/OmniTruthyBeagle-7B-v0" # Multilingual Communication and Cultural Insights
positive_prompts:
- "Multilingual Communication"
- "Cultural Insights"
- "Translation and Interpretation"
- "Cultural Norms Exploration"
- "Intercultural Communication Practices"
- "Describe cultural significance"
- "Narrate cultural"
- "Discuss cultural impact"
negative_prompts:
- "Scientific Analysis"
- "Creative Writing"
- "Technical Documentation"
- "Economic Modeling"
- "Historical Documentation"
- "Programming"
- "Algorithm Development"
- source_model: "CultriX/MoNeuTrix-7B-v1" # Creative Problem Solving and Innovation
positive_prompts:
- "Innovation and Design"
- "Problem Solving"
- "Creative Thinking"
- "Strategic Planning"
- "Conceptual Design"
- "Innovation and Design"
- "Problem Solving"
- "Compose narrative content or poetry."
- "Create complex puzzles and games."
- "Devise strategy"
negative_prompts:
- "Historical Analysis"
- "Linguistic Translation"
- "Economic Forecasting"
- "Geopolitical Analysis"
- "Cultural Commentary"
- "Historical Documentation"
- "Scientific Explanation"
- "Data Analysis Techniques"
- source_model: "paulml/OmniBeagleSquaredMBX-v3-7B" # Scientific and Technical Expertise
positive_prompts:
- "Scientific Explanation"
- "Technical Analysis"
- "Experimental Design"
- "Data Analysis Techniques"
- "Scientific Innovation"
- "Mathematical Problem Solving"
- "Algorithm Development"
- "Programming"
- "Analyze data"
- "Analyze statistical data on climate change trends."
- "Conduct basic data analysis or statistical evaluations."
negative_prompts:
- "Cultural Analysis"
- "Creative Arts"
- "Linguistic Challenges"
- "Political Debating"
- "Marketing Strategies"
- "Storywriting"
- "Character Development"
- "Roleplaying"
- "Narrative Creation"
```
## 💻 Usage
```python
!pip install -qU transformers bitsandbytes accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "CultriX/MoNeuTrix-MoE-4x7B"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
)
messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
``` |