TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ fail to generate with RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
Author: nilichen
Pull request: no

### System Info

- `transformers` version: 4.38.2
- Platform: Linux-6.5.0-1014-aws-x86_64-with-glibc2.35
- Python version: 3.11.8
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distribute the model among 4 GPUs on 1 instance

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Following the script in https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

model_name_or_path = "TheBloke/Mixtral-8x7B-Instruct-v0.1-AWQ"

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = AutoModelForCausalLM.from_pretrained(
    model_name_or_path,
    low_cpu_mem_usage=True,
    device_map="auto"
)

# Using the text streamer to stream output one token at a time
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

prompt = "Tell me about AI"
prompt_template=f'''
 {prompt} 

'''

# Convert prompt to tokens
tokens = tokenizer(
    prompt_template,
    return_tensors='pt'
).input_ids.cuda()

generation_params = {
    "do_sample": True,
    "temperature": 0.7,
    "top_p": 0.95,
    "top_k": 40,
    "max_new_tokens": 512,
    "repetition_penalty": 1.1
}

# Generate streamed output, visible one token at a time
generation_output = model.generate(
    tokens,
    streamer=streamer,
    **generation_params
)

# Generation without a streamer, which will include the prompt in the output
generation_output = model.generate(
    tokens,
    **generation_params
)

# Get the tokens from the output, decode them, print them
token_output = generation_output[0]
text_output = tokenizer.decode(token_output)
print("model.generate output: ", text_output)
```
Gives the following error with stack trace: 

> RuntimeError                              Traceback (most recent call last)
> Cell In [1], [line 35](vscode-notebook-cell:?execution_count=1&line=35)
>      [25](vscode-notebook-cell:?execution_count=1&line=25) generation_params = {
>      [26](vscode-notebook-cell:?execution_count=1&line=26)     "do_sample": True,
>      [27](vscode-notebook-cell:?execution_count=1&line=27)     "temperature": 0.7,
>    (...)
>      [31](vscode-notebook-cell:?execution_count=1&line=31)     "repetition_penalty": 1.1
>      [32](vscode-notebook-cell:?execution_count=1&line=32) }
>      [34](vscode-notebook-cell:?execution_count=1&line=34) # Generate streamed output, visible one token at a time
> ---> [35](vscode-notebook-cell:?execution_count=1&line=35) generation_output = model.generate(
>      [36](vscode-notebook-cell:?execution_count=1&line=36)     tokens,
>      [37](vscode-notebook-cell:?execution_count=1&line=37)     streamer=streamer,
>      [38](vscode-notebook-cell:?execution_count=1&line=38)     **generation_params
>      [39](vscode-notebook-cell:?execution_count=1&line=39) )
>      [41](vscode-notebook-cell:?execution_count=1&line=41) # Generation without a streamer, which will include the prompt in the output
>      [42](vscode-notebook-cell:?execution_count=1&line=42) generation_output = model.generate(
>      [43](vscode-notebook-cell:?execution_count=1&line=43)     tokens,
>      [44](vscode-notebook-cell:?execution_count=1&line=44)     **generation_params
>      [45](vscode-notebook-cell:?execution_count=1&line=45) )
> 
> File [~/slack-ml/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py:115](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py:115), in context_decorator.<locals>.decorate_context(*args, **kwargs)
>     [112](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py:112) @functools.wraps(func)
>     [113](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py:113) def decorate_context(*args, **kwargs):
>     [114](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py:114)     with ctx_factory():
> --> [115](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py:115)         return func(*args, **kwargs)
> 
> File [~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1592](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1592), in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
>    [1584](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1584)     input_ids, model_kwargs = self._expand_inputs_for_generation(
>    [1585](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1585)         input_ids=input_ids,
>    [1586](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1586)         expand_size=generation_config.num_return_sequences,
>    [1587](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1587)         is_encoder_decoder=self.config.is_encoder_decoder,
>    [1588](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1588)         **model_kwargs,
>    [1589](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1589)     )
>    [1591](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1591)     # 13. run sample
> -> [1592](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1592)     return self.sample(
>    [1593](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1593)         input_ids,
>    [1594](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1594)         logits_processor=prepared_logits_processor,
>    [1595](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1595)         logits_warper=logits_warper,
>    [1596](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1596)         stopping_criteria=prepared_stopping_criteria,
>    [1597](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1597)         pad_token_id=generation_config.pad_token_id,
>    [1598](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1598)         eos_token_id=generation_config.eos_token_id,
>    [1599](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1599)         output_scores=generation_config.output_scores,
>    [1600](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1600)         output_logits=generation_config.output_logits,
>    [1601](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1601)         return_dict_in_generate=generation_config.return_dict_in_generate,
>    [1602](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1602)         synced_gpus=synced_gpus,
>    [1603](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1603)         streamer=streamer,
>    [1604](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1604)         **model_kwargs,
>    [1605](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1605)     )
>    [1607](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1607) elif generation_mode == GenerationMode.BEAM_SEARCH:
>    [1608](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1608)     # 11. prepare beam search scorer
>    [1609](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1609)     beam_scorer = BeamSearchScorer(
>    [1610](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1610)         batch_size=batch_size,
>    [1611](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1611)         num_beams=generation_config.num_beams,
>    (...)
>    [1616](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1616)         max_length=generation_config.max_length,
>    [1617](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:1617)     )
> 
> File [~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2734](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2734), in GenerationMixin.sample(self, input_ids, logits_processor, stopping_criteria, logits_warper, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, output_logits, return_dict_in_generate, synced_gpus, streamer, **model_kwargs)
>    [2732](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2732) # sample
>    [2733](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2733) probs = nn.functional.softmax(next_token_scores, dim=-1)
> -> [2734](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2734) next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
>    [2736](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2736) # finished sentences should have their next token be a padding token
>    [2737](https://vscode-remote+ssh-002dremote-002bz-002dml-002dbeach-002dkni-002dpdx-002dbuffalo.vscode-resource.vscode-cdn.net/home/kni/slack-ml/~/slack-ml/.venv/lib/python3.11/site-packages/transformers/generation/utils.py:2737) if eos_token_id is not None:
> 
> RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

### Expected behavior

Should output responses to the prompt Tell me about AI.

Comments:
cc @gante 

This specific model upload is corrupted and should be taken down by TheBloke, but he has disappeared.

This one should work: https://huggingface.co/casperhansen/mixtral-instruct-awq

I see. Let me try it. Thanks @casper-hansen! 

Hi @nilichen 👋 let is know if @casper-hansen's suggestion is working!

(and thank you @casper-hansen for the suggestion 🤗 )

Oh yes. Sorry was having trouble accessing the multi-gpu instances in work.