File size: 3,431 Bytes
f0635bb 36fe3f1 f0635bb 8f0e28d f0635bb 36fe3f1 e6f8bd1 36fe3f1 f0635bb |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 |
---
license: apache-2.0
tags:
- moe
- merge
- mergekit
- lazymergekit
- mlabonne/NeuralBeagle14-7B
- AdaptLLM/finance-chat
- AdaptLLM/medicine-chat
- AdaptLLM/law-chat
datasets:
- Open-Orca/OpenOrca
- WizardLM/WizardLM_evol_instruct_V2_196k
- EleutherAI/pile
- GAIR/lima
pipeline_tag: text-generation
---
🌟 Buying me coffee is a direct way to show support for this project.
<a href="https://www.buymeacoffee.com/isotonic"><img src="https://www.buymeacoffee.com/assets/img/guidelines/download-assets-sm-1.svg" alt=""></a>
# AdaptLLM-4x7B-MoE
AdaptLLM-4x7B-MoE is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
* [mlabonne/NeuralBeagle14-7B](https://huggingface.co/mlabonne/NeuralBeagle14-7B)
* [AdaptLLM/finance-chat](https://huggingface.co/AdaptLLM/finance-chat)
* [AdaptLLM/medicine-chat](https://huggingface.co/AdaptLLM/medicine-chat)
* [AdaptLLM/law-chat](https://huggingface.co/AdaptLLM/law-chat)
## 💻 Usage
```python
Prompt Template:
<s>[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>
{{ user_message }} [/INST]
```
```python
!pip install -qU transformers bitsandbytes accelerate
from transformers import AutoTokenizer
import transformers
import torch
model = "Isotonic/AdaptLLM-4x7B-MoE"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
model_kwargs={
"torch_dtype": torch.float16,
"low_cpu_mem_usage": True,
"use_cache" : False,
"gradient_checkpointing" : True,
"device_map" : 'auto',
"load_in_8bit" : True
},
)
messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=512, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])
```
## 🧩 Configuration
```yaml
base_model: mlabonne/NeuralBeagle14-7B
experts:
- source_model: mlabonne/NeuralBeagle14-7B
positive_prompts:
- "chat"
- "assistant"
- "tell me"
- "explain"
- "storywriting"
- "write"
- "scene"
- "story"
- "character"
- "instruct"
- "summarize"
- "count"
- source_model: AdaptLLM/finance-chat
positive_prompts:
- "personal finance"
- "budgeting"
- "investing"
- "retirement planning"
- "debt management"
- "financial education"
- "consumer protection"
- "financial"
- "money"
- "investment"
- "banking"
- "stock"
- "bond"
- "portfolio"
- "risk"
- "return"
- source_model: AdaptLLM/medicine-chat
positive_prompts:
- "diagnose"
- "treat"
- "disease"
- "symptom"
- "medication"
- "anatomy"
- "physiology"
- "pharmacology"
- "clinical trial"
- "medical research"
- source_model: AdaptLLM/law-chat
positive_prompts:
- "law"
- "legal"
- "attorney"
- "lawyer"
- "court"
- "contract"
- "criminal"
- "evidence"
- "procedure"
- "contracts"
- "mergers & acquisitions"
- "corporate governance"
- "intellectual property"
- "employment law"
- "international trade"
- "competition law"
- "antitrust"
- "litigation"
- "arbitration"
- "mediation"
``` |