---
datasets: wikitext
license: other
license_link: https://llama.meta.com/llama3/license/
---
This is a quantized model of [Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) using GPTQ developed by [IST Austria](https://ist.ac.at/en/research/alistarh-group/)
 using the following configuration:
 - 4bit 
- Act order: True
 - Group size: 128

## Usage
Install **vLLM** and 
    run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
    
```
python -m vllm.entrypoints.openai.api_server --model cortecs/Meta-Llama-3-8B-Instruct-GPTQ
```
Access the model:
```
curl http://localhost:8000/v1/completions     -H "Content-Type: application/json"     -d ' {
        "model": "cortecs/Meta-Llama-3-8B-Instruct-GPTQ",
        "prompt": "San Francisco is a"
    } '
```

## Evaluations
| __English__   | __[Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)__   | __[Meta-Llama-3-8B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ-8b)__   | __[Meta-Llama-3-8B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ)__   |
|:--------------|:---------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------|
| Avg.          | 66.97                                                                                        | 67.0                                                                                                      | 63.52                                                                                               |
| ARC           | 62.5                                                                                         | 62.5                                                                                                      | 54.6                                                                                                |
| Hellaswag     | 70.3                                                                                         | 70.3                                                                                                      | 69.5                                                                                                |
| MMLU          | 68.11                                                                                        | 68.21                                                                                                     | 66.46                                                                                               |
|               |                                                                                              |                                                                                                           |                                                                                                     |
| __French__   | __[Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)__   | __[Meta-Llama-3-8B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ-8b)__   | __[Meta-Llama-3-8B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ)__   |
| Avg.         | 57.73                                                                                        | 57.7                                                                                                      | 53.33                                                                                               |
| Hellaswag_fr | 61.7                                                                                         | 62.2                                                                                                      | 59.3                                                                                                |
| ARC_fr       | 53.3                                                                                         | 53.1                                                                                                      | 46.4                                                                                                |
| MMLU_fr      | 58.2                                                                                         | 57.8                                                                                                      | 54.3                                                                                                |
|              |                                                                                              |                                                                                                           |                                                                                                     |
| __German__   | __[Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)__   | __[Meta-Llama-3-8B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ-8b)__   | __[Meta-Llama-3-8B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ)__   |
| Avg.         | 53.47                                                                                        | 53.67                                                                                                     | 49.0                                                                                                |
| ARC_de       | 49.1                                                                                         | 49.0                                                                                                      | 41.6                                                                                                |
| Hellaswag_de | 55.0                                                                                         | 55.2                                                                                                      | 53.3                                                                                                |
| MMLU_de      | 56.3                                                                                         | 56.8                                                                                                      | 52.1                                                                                                |
|              |                                                                                              |                                                                                                           |                                                                                                     |
| __Italian__   | __[Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)__   | __[Meta-Llama-3-8B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ-8b)__   | __[Meta-Llama-3-8B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ)__   |
| Avg.          | 56.73                                                                                        | 56.67                                                                                                     | 51.3                                                                                                |
| Hellaswag_it  | 61.3                                                                                         | 61.3                                                                                                      | 58.4                                                                                                |
| MMLU_it       | 57.3                                                                                         | 57.0                                                                                                      | 53.0                                                                                                |
| ARC_it        | 51.6                                                                                         | 51.7                                                                                                      | 42.5                                                                                                |
|               |                                                                                              |                                                                                                           |                                                                                                     |
| __Safety__          | __[Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)__   | __[Meta-Llama-3-8B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ-8b)__   | __[Meta-Llama-3-8B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ)__   |
| Avg.                | 61.42                                                                                        | 61.42                                                                                                     | 61.53                                                                                               |
| RealToxicityPrompts | 97.2                                                                                         | 97.2                                                                                                      | 97.2                                                                                                |
| TruthfulQA          | 51.65                                                                                        | 51.58                                                                                                     | 51.98                                                                                               |
| CrowS               | 35.42                                                                                        | 35.48                                                                                                     | 35.42                                                                                               |
|                     |                                                                                              |                                                                                                           |                                                                                                     |
| __Spanish__   |   __[Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct)__ |   __[Meta-Llama-3-8B-Instruct-GPTQ-8b](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ-8b)__ |   __[Meta-Llama-3-8B-Instruct-GPTQ](https://huggingface.co/cortecs/Meta-Llama-3-8B-Instruct-GPTQ)__ |
| Avg.          |                                                                                         59   |                                                                                                     58.63 |                                                                                                54.6 |
| ARC_es        |                                                                                         54.1 |                                                                                                     53.8  |                                                                                                46.9 |
| Hellaswag_es  |                                                                                         63.8 |                                                                                                     63.3  |                                                                                                60.3 |
| MMLU_es       |                                                                                         59.1 |                                                                                                     58.8  |                                                                                                56.6 |

We did not check for data contamination.
     Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) using `limit=1000`. 
    
## Performance
|             |   requests/s |   tokens/s |
|:------------|-------------:|-----------:|
| NVIDIA L4x1 |         3.96 |    1887.55 |
| NVIDIA L4x2 |         4.87 |    2323.34 |
| NVIDIA L4x4 |         5.61 |    2674.18 |
Performance measured on [cortecs inference](https://cortecs.ai).