---
datasets: wikitext
---
This is a quantized model of [Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) using GPTQ developed by [IST Austria](https://ist.ac.at/en/research/alistarh-group/)
 using the following configuration:
 - 4bit 
- Act order: True
 - Group size: 128

## Usage
Install **vLLM** and 
    run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
    
```
python -m vllm.entrypoints.openai.api_server --model cortecs/Mistral-7B-Instruct-v0.3-GPTQ-4b
```
Access the model:
```
curl http://localhost:8000/v1/completions     -H "Content-Type: application/json"     -d ' {
        "model": "cortecs/Mistral-7B-Instruct-v0.3-GPTQ-4b",
        "prompt": "San Francisco is a"
    } '
```

## Evaluations
| __English__   | __[Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-8b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-8b)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-4b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-4b)__   |
|:--------------|:--------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------|
| Avg.          | 67.65                                                                                       | 67.72                                                                                                     | 66.95                                                                                                     |
| ARC           | 64.2                                                                                        | 64.1                                                                                                      | 62.1                                                                                                      |
| Hellaswag     | 75.6                                                                                        | 75.6                                                                                                      | 76.0                                                                                                      |
| MMLU          | 63.16                                                                                       | 63.47                                                                                                     | 62.75                                                                                                     |
|               |                                                                                             |                                                                                                           |                                                                                                           |
| __French__   | __[Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-8b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-8b)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-4b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-4b)__   |
| Avg.         | 56.4                                                                                        | 56.17                                                                                                     | 54.77                                                                                                     |
| ARC_fr       | 51.9                                                                                        | 51.4                                                                                                      | 50.0                                                                                                      |
| Hellaswag_fr | 65.8                                                                                        | 65.8                                                                                                      | 63.8                                                                                                      |
| MMLU_fr      | 51.5                                                                                        | 51.3                                                                                                      | 50.5                                                                                                      |
|              |                                                                                             |                                                                                                           |                                                                                                           |
| __German__   | __[Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-8b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-8b)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-4b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-4b)__   |
| Avg.         | 51.83                                                                                       | 51.73                                                                                                     | 51.7                                                                                                      |
| ARC_de       | 47.6                                                                                        | 47.5                                                                                                      | 47.3                                                                                                      |
| Hellaswag_de | 58.9                                                                                        | 59.0                                                                                                      | 57.3                                                                                                      |
| MMLU_de      | 49.0                                                                                        | 48.7                                                                                                      | 50.5                                                                                                      |
|              |                                                                                             |                                                                                                           |                                                                                                           |
| __Italian__   | __[Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-8b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-8b)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-4b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-4b)__   |
| Avg.          | 54.93                                                                                       | 54.8                                                                                                      | 52.83                                                                                                     |
| ARC_it        | 51.6                                                                                        | 51.6                                                                                                      | 49.3                                                                                                      |
| Hellaswag_it  | 63.5                                                                                        | 63.8                                                                                                      | 61.0                                                                                                      |
| MMLU_it       | 49.7                                                                                        | 49.0                                                                                                      | 48.2                                                                                                      |
|               |                                                                                             |                                                                                                           |                                                                                                           |
| __Safety__          | __[Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-8b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-8b)__   | __[Mistral-7B-Instruct-v0.3-GPTQ-4b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-4b)__   |
| Avg.                | 60.32                                                                                       | 60.54                                                                                                     | 64.8                                                                                                      |
| RealToxicityPrompts | 89.7                                                                                        | 90.0                                                                                                      | 90.7                                                                                                      |
| TruthfulQA          | 59.71                                                                                       | 59.48                                                                                                     | 58.32                                                                                                     |
| CrowS               | 31.54                                                                                       | 32.14                                                                                                     | 45.38                                                                                                     |
|                     |                                                                                             |                                                                                                           |                                                                                                           |
| __Spanish__   |   __[Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3)__ |   __[Mistral-7B-Instruct-v0.3-GPTQ-8b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-8b)__ |   __[Mistral-7B-Instruct-v0.3-GPTQ-4b](https://huggingface.co/cortecs/Mistral-7B-Instruct-v0.3-GPTQ-4b)__ |
| Avg.          |                                                                                        57.9 |                                                                                                     57.97 |                                                                                                      56.1 |
| ARC_es        |                                                                                        53.5 |                                                                                                     53.5  |                                                                                                      51   |
| Hellaswag_es  |                                                                                        68.5 |                                                                                                     68.5  |                                                                                                      66.2 |
| MMLU_es       |                                                                                        51.7 |                                                                                                     51.9  |                                                                                                      51.1 |

We did not check for data contamination.
     Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) using `limit=1000`. 
    
## Performance
|             |   requests/s |   tokens/s |
|:------------|-------------:|-----------:|
| NVIDIA L4x1 |         3.75 |    1867.13 |
| NVIDIA L4x2 |         5.03 |    2503.83 |
| NVIDIA L4x4 |         5.86 |    2916.3  |
Performance measured on [cortecs inference](https://cortecs.ai).