---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
inference: false
tags:
- pytorch
- mistral
- inferentia2
- neuron
---

# Neuronx model for [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2)

This repository contains [**AWS Inferentia2**](https://aws.amazon.com/ec2/instance-types/inf2/) and [`neuronx`](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/) compatible checkpoints for [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf).
You can find detailed information about the base model on its [Model Card](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2).

This model has been exported to the `neuron` format using specific `input_shapes` and `compiler` parameters detailed in the paragraphs below.

Please refer to the 🤗 `optimum-neuron` [documentation](https://huggingface.co/docs/optimum-neuron/main/en/guides/models#configuring-the-export-of-a-generative-model) for an explanation of these parameters.



## Usage with 🤗 `optimum-neuron`

```python
>>> from optimum.neuron import pipeline

>>> p = pipeline('text-generation', 'aws-neuron/Mistral-7B-Instruct-v0.2-Neuron-inf2.8xlarge')
>>> p("<s>[INST] Tell me something interesting about AWS. [/INST]", max_new_tokens=64, do_sample=True, top_k=50)
[{'generated_text': "<s>[INST] Tell me something interesting about AWS. [/INST] I'd be happy to tell you something interesting about Amazon Web Services (AWS). AWS is the world's most extensive and rapidly expanding cloud computing platform, offering over 200 fully featured services from data centers globally. It is used by millions of customers, including the largest enterprises and the h"}]
```


## Compilation of your own version

Deploy an AWS inf2.8xlarge or larger instance.  Deploy using the Hugging Face Deep Learning AMI so you have all the software installed.
This model was compiled and tested on version 20240123
Download a copy locally so that you can **edit the config.json file to set the sliding_window value to 4096 (instead of null)**

(See https://github.com/aws-neuron/transformers-neuronx/issues/71 for a reason why)

```
git clone https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2

```

Then edit config.json to change sliding_window to 4096

Then the standard compilation process will work.  You can change your arguments.

```
model_to_test = "Mistral-7B-Instruct-v0.2"

from optimum.neuron import NeuronModelForCausalLM
#num_cores should be changed based on the instance.  inf2.24xlarge has 6 neuron processors (they have two cores each) so 12 total
#larger models will need more cores.  You can make your model smaller by changing fp16 to f8.  Some models may requre num_cores to be a power of 2 
compiler_args = {"num_cores": 2, "auto_cast_type": 'fp16'}
input_shapes = {"batch_size": 1, "sequence_length": 2048}


model = NeuronModelForCausalLM.from_pretrained(model_to_test, export=True, **compiler_args, **input_shapes) 

from optimum.neuron import pipeline
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_to_test)

p = pipeline('text-generation', model, tokenizer)
p("<s>[INST] Tell me something interesting about AWS. [/INST]", max_new_tokens=64, do_sample=True, top_k=50)

model.save_pretrained("Mistral-7B-Instruct-v0.2-Neuron-inf2.8xlarge")

```


## Arguments passed during export

**input_shapes**

```json
{
  "batch_size": 1,
  "sequence_length": 2048,
}
```

**compiler_args**

```json
{
  "auto_cast_type": "bf16",
  "num_cores": 2,
}
```