---

base_model: utter-project/EuroLLM-9B-Instruct

---
This is a quantization of the [EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct).

The EuroLLM project has the goal of creating a suite of LLMs capable of understanding and generating text in all European Union languages as well as some additional relevant languages. EuroLLM-9B is a 9B parameter model trained on 4 trillion tokens divided across the considered languages and several data sources: Web data, parallel data (en-xx and xx-en), and high-quality datasets. EuroLLM-9B-Instruct was further instruction tuned on EuroBlocks, an instruction tuning dataset with focus on general instruction-following and machine translation.
## Evaluations
This model provides an accuracy recovery of 99.61%. 

| __English__   | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__   | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__   |
|:--------------|:--------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------|
| Avg.          | 66.35                                                                                 | 65.35                                                                                                          |
| Arc           | 63.3                                                                                  | 61.7                                                                                                           |
| Hellaswag     | 69.4                                                                                  | 69.0                                                                                                           |
|               |                                                                                       |                                                                                                                |
| __French__   | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__   | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__   |
| Avg.         | 61.67                                                                                 | 61.3                                                                                                           |
| Arc          | 58.1                                                                                  | 57.3                                                                                                           |
| Hellaswag    | 70.2                                                                                  | 70.3                                                                                                           |
| MMLU         | 56.7                                                                                  | 56.3                                                                                                           |
|              |                                                                                       |                                                                                                                |
| __German__   | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__   | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__   |
| Avg.         | 60.0                                                                                  | 60.37                                                                                                          |
| Arc          | 57.2                                                                                  | 56.7                                                                                                           |
| Hellaswag    | 66.3                                                                                  | 67.1                                                                                                           |
| MMLU         | 56.5                                                                                  | 57.3                                                                                                           |
|              |                                                                                       |                                                                                                                |
| __Italian__   | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__   | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__   |
| Avg.          | 61.8                                                                                  | 61.7                                                                                                           |
| Arc           | 58.3                                                                                  | 58.2                                                                                                           |
| Hellaswag     | 69.9                                                                                  | 69.4                                                                                                           |
| MMLU          | 57.2                                                                                  | 57.5                                                                                                           |
|               |                                                                                       |                                                                                                                |
| __Portuguese__   | __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__   | __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__   |
| Avg.             | 61.47                                                                                 | 61.37                                                                                                          |
| Arc              | 59.1                                                                                  | 59.3                                                                                                           |
| Hellaswag        | 70.3                                                                                  | 70.2                                                                                                           |
| MMLU             | 55.0                                                                                  | 54.6                                                                                                           |
|                  |                                                                                       |                                                                                                                |
| __Spanish__   |   __[EuroLLM-9B-Instruct](https://huggingface.co/utter-project/EuroLLM-9B-Instruct)__ |   __[EuroLLM-9B-Instruct-FP8-Dynamic (this)](https://huggingface.co/cortecs/EuroLLM-9B-Instruct-FP8-Dynamic)__ |
| Avg.          |                                                                                 62.03 |                                                                                                          61.53 |
| Arc           |                                                                                 59.7  |                                                                                                          59.3  |
| Hellaswag     |                                                                                 71.4  |                                                                                                          71    |
| MMLU          |                                                                                 55    |                                                                                                          54.3  |

We did not check for data contamination.
     Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) with `limit=1000`. 
    
## Usage
Install **vLLM** and 
    run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
    
```
python -m vllm.entrypoints.openai.api_server --model cortecs/EuroLLM-9B-Instruct-FP8-Dynamic --gpu-memory-util 0.95
```
Access the model:
```
curl http://localhost:8000/v1/completions     -H "Content-Type: application/json"     -d ' {
        "model": "cortecs/EuroLLM-9B-Instruct-FP8-Dynamic",
        "prompt": "San Francisco is a"
    } '
```
⚡ This model is optimized to handle heavy workloads providing a total throughput of ️**1976 tokens per second** using one NVIDIA L4 ⚡