Multi-GPU inference affects LLM's (Llama2-7b-chat-hf) generation.
Author: 3rdAT
Pull request: no
Labels: name: Generation

name: Text

### System Info

- `transformers` version: 4.40.2
- Platform: Linux-5.4.0-186-generic-x86_64-with-glibc2.31
- Python version: 3.9.19
- Huggingface_hub version: 0.23.1
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes (A100_80GB x 4)
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When I perform inference with two GPUs using the following command,

`CUDA_VISIBLE_DEVICES=0,1 nohup python ./inference.py`

the model generates answer properly.

Whereas, when I use more than two GPUs using the following command,

`CUDA_VISIBLE_DEVICES=0,1,3,4 nohup python ./inference.py`

the model starts generating **Gibberish**. Upon close introspection, the model outputs logits which are all **NaN** values.

Note: I use device_map = "auto" while loading the model.

### Expected behavior

I expect the model to generate properly.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @3rdAT, could you share the code in `inference.py`? Without it it's hard to help you out.

(I remember something about moving the cache position between devices that was setting them not 0 / moving hidden states reseting them but don't really remember...) clearly a reproducer would help! 

Hey @LysandreJik  and @ArthurZucker , Thanks for coming out to help guys.

Here is the code that I am using to do inference,

```
##Loading the model,

    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        return_dict=True,
        load_in_8bit=quantization,
        device_map="auto",
        low_cpu_mem_usage=True,
        token=',
    )
    
    model.eval()
    
    user_prompt = 'I liked "Breaking Bad" and "Band of Brothers". Do you have any recommendations of other shows I might like?\n'
    batch = tokenizer(user_prompt, padding='max_length', truncation=True, max_length=max_padding_length, return_tensors="pt")
    number_of_input_tokens = len(batch["input_ids"][0])
    batch = {k: v.to("cuda") for k, v in batch.items()}
    # start = time.perf_counter()
    with torch.no_grad():
        outputs = model.generate(
            **batch,
            max_new_tokens=max_new_tokens,
            do_sample=do_sample,
            top_p=top_p,
            temperature=temperature,
            min_length=min_length,
            use_cache=use_cache,
            top_k=top_k,
            repetition_penalty=repetition_penalty,
            length_penalty=length_penalty,
            output_hidden_states= True, return_dict_in_generate=True,
            **kwargs
        )

    output_text = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)

    print(output_text[len(user_prompt):])
```

I am just performing a normal inference. This code works fine with my other servers. But I am curious what could potentially be causing this behaviour.


cc @gante and @zucchini-nlp for a quick re-read :)

Hm I couldn't reproduce it with 4 A100 GPUs and the latest `transformers` and Llama-7b-chat model. Can you check if updating transformers helps?

@zucchini-nlp 
I personally feel it was an hardware issue. But I am not sure to pinpoint what can go wrong here.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@3rdAT if you end up finding more clues please share with us -- pinning this type of issues is hard, but often worth it 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.