|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
- de |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
tags: |
|
- finetune |
|
- sft |
|
- dpo |
|
- laser |
|
- augmentation |
|
- german |
|
- english |
|
- moe |
|
--- |
|
![SauerkrautLM](https://vago-solutions.de/wp-content/uploads/2024/02/Sauerkraut_Laserchat-14b-MoE.png "SauerkrautLM-14b-MoE-LaserChat") |
|
## VAGO solutions SauerkrautLM-14b-MoE-LaserChat |
|
Introducing **SauerkrautLM-14b-MoE-LaserChat** – our Sauerkraut (2x7b) 14b MoE version of the powerful [SauerkrautLM-7b-LaserChat](https://huggingface.co/VAGOsolutions/SauerkrautLM-7b-LaserChat) and [yam-peleg/Experiment26-7B](https://huggingface.co/yam-peleg/Experiment26-7B) ! |
|
|
|
By combining the two models, we were able to significantly increase both the German and English language skills. |
|
In addition, the initial SauerkrautLM-7b-LaserChat also acts as an adapter for Experiment26-7B, which means it benefits from the chat capabilities of the SauerkrautLM-7b-LaserChat. |
|
At the same time, the SauerkrautLM-7b-LaserChat benefits from the knowledge and creativity of Experiment26-7B. |
|
|
|
The model **SauerkrautLM-14b-MoE-LaserChat** is a **joint effort** between **VAGO solutions** and **Hyperspace.ai.** |
|
Much appreciation goes to the tremendous research effort of **Fernando Fernandes Neto, David Golchinfar and Eric Hartford on their laserRMT approach.** |
|
Without their independent research collaboration this model release would not have been possible. |
|
|
|
|
|
# Table of Contents |
|
1. [Overview of all SauerkrautLM-14b-MoE-LaserChat models](#all-sauerkrautlm-14b-MoE-laserchat-models) |
|
2. [Model Details](#model-details) |
|
- [Prompt template](#prompt-template) |
|
3. [Evaluation](#evaluation) |
|
5. [Disclaimer](#disclaimer) |
|
6. [Contact](#contact) |
|
7. [Collaborations](#collaborations) |
|
8. [Acknowledgement](#acknowledgement) |
|
|
|
|
|
## All SauerkrautLM-14b-MoE-LaserChat Models |
|
|
|
| Model | HF | GPTQ | GGUF | AWQ | |
|
|-------|-------|-------|-------|-------| |
|
| SauerkrautLM-14b-MoE-LaserChat | [Link](https://huggingface.co/VAGOsolutions/SauerkrautLM-14b-MoE-LaserChat) | coming soon | coming soon | coming soon | |
|
|
|
## Model Details |
|
**SauerkrautLM-14b-MoE-LaserChat** |
|
- **Model Type:** SauerkrautLM-14b-MoE-LaserChat is a MoE Model based on [SauerkrautLM-7b-LaserChat](https://huggingface.co/VAGOsolutions/SauerkrautLM-7b-LaserChat) and [yam-peleg/Experiment26-7B](https://huggingface.co/yam-peleg/Experiment26-7B) |
|
- **Language(s):** German, English |
|
- **License:** Apache 2.0 |
|
- **Contact:** [VAGO solutions](https://vago-solutions.ai), [Hyperspace.computer](https://hyperspace.computer/) |
|
|
|
|
|
We improved the German language skills on this model further. Nevertheless, certain formulations may occur that are not entirely correct. |
|
|
|
|
|
### Prompt Template: |
|
``` |
|
GPT4 Correct User: Hallo, wie geht es dir?<|end_of_turn|>GPT4 Correct Assistant: Hallo! Ich bin ein künstliches Intelligenzsystem und habe keine persönlichen Gefühle oder körperliche Zustände. Wie kann ich Ihnen helfen?<|end_of_turn|>GPT4 Correct User: Ich benötige nur einen kurzen Satz, den ich in das Prompt Template veröffentlichen kann.<|end_of_turn|>GPT4 Correct Assistant: |
|
|
|
|
|
``` |
|
|
|
|
|
``` |
|
GPT4 Correct User: Hello<|end_of_turn|>GPT4 Correct Assistant: Hello! How can I help you today? If you have any questions or need assistance, feel free to ask.<|end_of_turn|>GPT4 Correct User: I just need a short sentence to post in the prompt template.<|end_of_turn|>GPT4 Correct Assistant: |
|
|
|
``` |
|
|
|
|
|
## Evaluation |
|
|
|
**Open LLM Leaderboard:** |
|
|
|
benchmarked on lm-evaluation-harness 0.4.1 |
|
|
|
| Metric | Value | |
|
|-----------------------|---------------------------| |
|
| Avg. | 71.65 | |
|
| ARC (25-shot) | 68.09 | |
|
| HellaSwag (10-shot) | 84.78 | |
|
| MMLU (5-shot) | 63.59| |
|
| TruthfulQA (0-shot) | 58.57 | |
|
| Winogrande (5-shot) | 80.74 | |
|
| GSM8K (5-shot) | 74.15 | |
|
|
|
|
|
|
|
## Disclaimer |
|
We must inform users that despite our best efforts in data cleansing, the possibility of uncensored content slipping through cannot be entirely ruled out. |
|
However, we cannot guarantee consistently appropriate behavior. Therefore, if you encounter any issues or come across inappropriate content, we kindly request that you inform us through the contact information provided. |
|
Additionally, it is essential to understand that the licensing of these models does not constitute legal advice. We are not held responsible for the actions of third parties who utilize our models. |
|
|
|
## Contact |
|
If you are interested in customized LLMs for business applications, please get in contact with us via our websites. We are also grateful for your feedback and suggestions. |
|
|
|
## Collaborations |
|
We are also keenly seeking support and investment for our startups, VAGO solutions and Hyperspace where we continuously advance the development of robust language models designed to address a diverse range of purposes and requirements. If the prospect of collaboratively navigating future challenges excites you, we warmly invite you to reach out to us at [VAGO solutions](https://vago-solutions.de/#Kontakt), [Hyperspace.computer](https://hyperspace.computer/) |
|
|
|
## Acknowledgement |
|
Many thanks to [yam-peleg](https://huggingface.co/yam-peleg) for providing such valuable model to the Open-Source community |