Create README.md

README.md

@@ -0,0 +1,80 @@
+---
+license: llama3
+library_name: transformers
+tags:
+- nsfw
+- not-for-all-audiences
+- llama-3
+- text-generation-inference
+---
+
+# Llama-Salad-4x8B
+This is a MoE merge of several Llama-3 models that aims to create a solid role-play model without sacrificing logic and reasoning. Meta-Llama-3-8B-Instruct was already quite good at role-play, but the overuse of `*wink*` and `*giggle*`, along with the tiptoeing around NSFW topics by spamming ellipses, really started to piss me off. L3-8B-Stheno-v3.1 is definitely the best role-play model out of all of the llama-3 fine-tunes so far; however, it's a little *too* horny for my liking, has a hard time keeping track of what's going on, like all other role-play fine-tunes, and has a serious issue with over-responding; there's a high chance you'll get a 4+ paragraph response to something basic.
+
+Combining the two models has kept both of their strengths but none of their weaknesses; it completely flattened out its over-responding issue, improved its logic and reasoning, and toned down the horny levels a bit. Don't be mistaken, though; this hasn't affected its ability to generate NSFW content; it's just no longer overwhelmingly horny. It's the first model I've used, aside from base L3-8B-Stheno-v3.1, that actually treats sexual content like it's normal instead of some esoteric and shameful thing.
+
+While role-play was the main focus of this merge, its base capabilities weren't affected at all, so swapping models for other tasks isn't needed unless you require a bigger model. Actually, with the addition of Tess-2.0-Llama-3-8B, I did find a small overall improvement. There isn't any particular reason Llama3-OpenBioLLM-8B is in the merge; I needed a 4th model for the merge, it seemed like a high-quality fine-tune, and its knowledge of anatomy might come in handy for role-play.
+
+Unfortunately, I can't compare it with 70B models because they're too slow on my machine, but this is the best sub-70B model I have used so far; I haven't felt the need to regenerate any responses, which hasn't happened with any other model so far. This is my first attempt at any kind of merge, and I want to share what I've learned, but this section is already longer than I wanted, so I've decided to place the rest at the bottom of the page.
+
+# Details
+- **License**: [llama3](https://llama.meta.com/llama3/license/)
+- **Instruct Format**: [llama-3](https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/)
+- **Context Size**: 8K
+
+## Models Used
+- [Meta-Llama-3-8B-Instruct](https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct)
+- [Tess-2.0-Llama-3-8B](https://huggingface.co/migtissera/Tess-2.0-Llama-3-8B)
+- [L3-8B-Stheno-v3.1](https://huggingface.co/Sao10K/L3-8B-Stheno-v3.1)
+- [Llama3-OpenBioLLM-8B](https://huggingface.co/aaditya/Llama3-OpenBioLLM-8B)
+
+## Merge Config
+```yaml
+base_model: NousResearch/Meta-Llama-3-8B-Instruct
+gate_mode: hidden
+dtype: bfloat16
+experts_per_token: 2
+experts:
+  - source_model: NousResearch/Meta-Llama-3-8B-Instruct
+    positive_prompts:
+    - "summarize"
+    - "explain"
+    - "define"
+    - "translate"
+    - "multilingual"
+    - "chat"
+    - "conversation"
+  - source_model: migtissera/Tess-2.0-Llama-3-8B
+    positive_prompts:
+    - "programming language"
+    - "math"
+    - "code"
+    - "step-by-step"
+    - "logic"
+  - source_model: Sao10K/L3-8B-Stheno-v3.1
+    positive_prompts:
+    - "role-play"
+    - "characters"
+    - "narration"
+    - "story writing"
+    - "scene"
+  - source_model: aaditya/Llama3-OpenBioLLM-8B
+    positive_prompts:
+    - "anatomy"
+    - "diagnosis"
+    - "symptom"
+    - "biomedical"
+    - "health"
+    - "medicine"
+    - "medication"
+    - "physiology"
+```
+
+# What I Have Learned
+After testing over a hundred different configurations, I have concluded that the name "Mixture of Experts" is misleading for merges; despite the name, using only domain-specific models leads to an overall worse model. For every single token, two models are used for prediction; if you only have domain-specific models in the merge, then you're always going to have a model of a completely unrelated domain contributing to the token prediction. When you fine-tune a model for a specific task, it degrades its quality when used for other tasks; if you only use models of a specific domain, then the degradation becomes more prominent.
+
+The solution I have found is quite simple: make sure that a general-purpose model is always chosen as one of the prediction models. I initially thought that adding a general-purpose model to the prediction pool would reduce the effectiveness of domain-specific models, but in practice, it keeps the strengths of both models but none of their weaknesses. I can't really judge if this extends to non-creative domains, such as coding, but for role-play, Llama-Salad-4x8B is proof that it works.
+
+It might be tempting to mix models that use different instruct formats, but this should be avoided as much as possible. Using the wrong instruct format degrades the output quality of the model; merging a domain-specific model that uses a different instruct format from the rest will improve output in that domain, but it will be in a degraded form. Merging models with different instruct formats is still useful if there's no other option, but for the best results, you should stick with models using the same instruct format.
+
+These are just my own observations from my experiments, so I very well could be wrong, but it's definitely worth further testing considering how well Llama-Salad-4x8B turned out.