README.md

---
language:
- en
- de
- it
- fr
- da
- sv
- fi
- 'no'
---
![Kraken](https://vago-solutions.de/wp-content/uploads/2024/05/Kraken_Pic-multi.png "Kraken-Multilingual")


## Overview

The Kraken-Multilingual model and Architecture **Kraken** is a **joint effort** between **Cognitive Computations**, **VAGO Solutions** and **Hyperspace.ai.** 

Created by **Fernando Fernandes Neto**, **David Golchinfar**, **Lucas Atkins** and **Eric Hartford**

The Kraken-Multilingual model supports German, English, Italian, French, Swedish, Finnish, Danish and Norwegian language.

The Kraken Architecture is a sophisticated machine learning framework designed for dynamic text generation tasks. It utilizes the Hugging Face transformers library to orchestrate multiple causal language models (CLMs) and intelligently route input through different models based on the context and content of the input text. The architecture is powered by a custom configuration class (KrakenConfig) that facilitates the integration and management of various components such as tokenizers, models, and routing mechanisms.

## Features

Dynamic Model Routing: Uses a sequence classification model to route inputs to the most suitable language model based on the input's characteristics.
Multiple Language Models: Supports integration of various pre-trained causal language models, allowing for flexible, context-appropriate responses.
Customizable Templates: Includes support for input formatting using predefined templates, enhancing the model's adaptability to different conversational contexts.
Extensible Configuration: Leverages a custom configuration setup that can be easily extended and adapted for various use cases involving causal language modeling.

## Selected Models as Experts:
```
      "German/English Expert": "VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct",
      "Function Italian Expert": "mii-community/zefiro-7b-dpo-ITA",
      "French Expert": "paulml/Hermes-2-Pro-French",
      "Scandinavian Expert": "norallm/normistral-7b-warm-instruct",
```

**How to load and call Kraken-Multilingual model :**
```
from transformers import AutoModelForCausalLM
device = "cuda:0" ## Setup "cuda:0" if NVIDIA, "mps" if on Mac

# Load the model and config:
model = AutoModelForCausalLM.from_pretrained("./kraken_model", trust_remote_code=True)
```

# Call the German expert:
```
messages = [
    {'role': 'system', 'content': 'Du bist ein freundlicher und hilfreicher deutscher KI-Assistent'},
    {'role': 'user', 'content': "Erzähle mir eine kurze Gute Nacht Geschichte in 2 Sätzen."}
    ]

tokenizer = model.tokenizer
input_text = tokenizer.apply_chat_template(messages, tokenize=False)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda:0")
output_ids = model.generate(input_ids, max_length=150)
print(model.expert_tokenizer(text=input_text).decode(output_ids[0], skip_special_tokens=True))
```



# Call the English expert:
```
messages = [
    {'role': 'system', 'content': '"You are a helpful AI Assistant'},
    {'role': 'user', 'content': "Find the mass percentage of Ba in BaO"}
    ]

tokenizer = model.tokenizer
input_text = tokenizer.apply_chat_template(messages, tokenize=False)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(device)
output_ids = model.generate(input_ids, max_length=250)
print(model.expert_tokenizer(text=input_text).decode(output_ids[0], skip_special_tokens=True))
```

# Call the Italian expert:
```
messages = [
    {'role': 'system', 'content': 'Sei un utile assistente AI.'},
    {'role': 'user', 'content': 'Hai qualche idea su cosa potrei fare a Roma?''}
    ]

tokenizer = model.tokenizer
input_text = tokenizer.apply_chat_template(messages, tokenize=False)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(device)
output_ids = model.generate(input_ids ,temperature=0.6, do_sample=True, top_p=0.9,top_k=20, max_length=500)
print(model.expert_tokenizer(text=input_text).decode(output_ids[0], skip_special_tokens=True))
```

# Call the French expert:
```
messages = [
    {'role': 'system', 'content': 'Vous êtes un assistant IA allemand sympathique et serviable'},
    {'role': 'user', 'content': 'J'aimerais faire du shopping à Paris. Que pouvez-vous recommander?'}
    ]

tokenizer = model.tokenizer
input_text = tokenizer.apply_chat_template(messages, tokenize=False)
print(input_text)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(device)
output_ids = model.generate(input_ids ,temperature=0.6, do_sample=True, top_p=0.9,top_k=20, max_length=250)
print(model.expert_tokenizer(text=input_text).decode(output_ids[0], skip_special_tokens=True))
```

# Call the Scandinavian expert:
```
messages = [
    {'role': 'system', 'content': 'Du är en hjälpsam AI-assistent'},
    {'role': 'user', 'content': 'Jag kommer från Tyskland och skulle vilja resa till Sverige. Är en färja över Danmark ett bra sätt att resa?'}
    ]

tokenizer = model.tokenizer
input_text = tokenizer.apply_chat_template(messages, tokenize=False)
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(device)
output_ids = model.generate(input_ids ,temperature=0.1, do_sample=True, top_p=0.9,top_k=20, max_length=250)
print(model.expert_tokenizer(text=input_text).decode(output_ids[0], skip_special_tokens=True))
```


# Switch expert and or quantization:
Go into the config file of the kraken_model folder
```
    "models": {
      "expert1": "VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct", # Switch to a german/english model of your choice
      "expert2": "mii-community/zefiro-7b-dpo-ITA",                # Switch to a italian model of your choice
      "expert3": "paulml/Hermes-2-Pro-French",                     # Switch to a french model of your choice
      "expert4": "norallm/normistral-7b-warm-instruct"             # Switch to a scandinavian model of your choice
    },
    # Currently supported: "4bit","8bit" and "awq"
    "quantization": {
      "expert1": null,
      "expert2": null,
      "expert3": null,
      "expert4": null
    },
    "router": "kraken_router",
    # Adjust the tokenizer to your selected model
    "tokenizers": {
      "expert1": "VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct",
      "expert2": "mii-community/zefiro-7b-dpo-ITA",
      "expert3": "paulml/Hermes-2-Pro-French",
      "expert4": "norallm/normistral-7b-warm-instruct"
    }
  },
  "model_type": "kraken",
  "torch_dtype": "float32",
  "transformers_version": "4.41.0"
}


```
## Disclaimer
We must inform users that despite our best efforts in data cleansing, the possibility of uncensored content slipping through cannot be entirely ruled out.
However, we cannot guarantee consistently appropriate behavior. Therefore, if you encounter any issues or come across inappropriate content, we kindly request that you inform us through the contact information provided.
Additionally, it is essential to understand that the licensing of these models does not constitute legal advice. We are not held responsible for the actions of third parties who utilize our models.
 
## Contact
If you are interested in customized LLMs for business applications, please get in contact with us via our websites. We are also grateful for your feedback and suggestions.
 
## Collaborations
We are also keenly seeking support and investment for our startups, VAGO solutions and Hyperspace where we continuously advance the development of robust language models designed to address a diverse range of purposes and requirements. If the prospect of collaboratively navigating future challenges excites you, we warmly invite you to reach out to us at [VAGO solutions](https://vago-solutions.ai), [Hyperspace.computer](https://hyperspace.computer/) and [Cognitive Computations](https://erichartford.com/)
## Cite As

Fernando Fernandes Neto, David Golchinfar, Lucas Atkins, Eric Hartford - [Kraken: An OpenSource Collection of Experts Model, 2024](https://github.com/cognitivecomputations/kraken)