Multi-GPU inference using accelerate giving inaccurate/gibberish results on RTX 4090s
Author: milsun
Pull request: no

### System Info

I am trying to use pretrained opt-6.7b model for inference, by using "device_map" as "auto", "balanced", basically scenarios where model weights are spread across both GPUs; the results produced are inaccurate and gibberish. Although if model is loaded on just one GPU, it works fine.

Tried opt-13b and similarly sized other models spread across both GPUs, as well, but all of them produce gibberish results.
Also, tried the same with different GPUs, 2x RTX 3090, 2x A5000; works fine with these.

Basically, only when a model is spread across multiple RTX 4090s, the results are gibberish.

Specs:
        python=3.9
        transformers==4.26.1
        accelerate==0.16.0
        cuda==11.6
  
Code:

    from transformers import AutoModelForCausalLM, AutoTokenizer
    import torch
    
    model = AutoModelForCausalLM.from_pretrained("facebook/opt-6.7b", torch_dtype=torch.float16, device_map="auto")
    tokenizer = AutoTokenizer.from_pretrained("facebook/opt-6.7b", use_fast=False)
    
    prompt = "What is the color of carrots?\nAnswer:"
    
    input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
    
    generated_ids = model.generate(input_ids)
    
    tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
    ['What is the color of carrots?\nAnswer: toinhoza, and the other half of']


### Who can help?

@ArthurZucker @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps:

1. Use 2 x RTX 4090 node
2. Install transformers, accelerate
3. Use above code to load and infer using opt model, load pretrained models, such as opt-6.7b/opt-13b
4. Set device_map as "auto" or "balanced" 
5. Run inference

### Expected behavior

With prompt = "What is the color of carrots?\nAnswer:"

Result should be "What is the color of carrots?\nAnswer: Orange"

Comments:
If you use `bfloat16` as a torch dtype, do you get the same results? I'm wondering if somehow the ops in float16 are badly implemented on those GPUs.

Just tried, still outputs gibberish.

prompt = "What is the color of carrots?\nAnswer:"
output (bfloat16) = 'What is the color of carrots?\nAnswer: " "'

Important thing to note here is, if model is loaded on just one gpu, it works fine.
That should help us narrow down the possible causes.

I have been reading up on this, seems to be a Nvidia driver issue, which is still unfixed. Basically, issue is with P2P memory access with the RTX 4090s. Seems like we are stuck with Nvidia to fix the issue.

Thanks for investigating. Do you have a link you could share so that others reading the issue later on can have all the info?

https://forums.developer.nvidia.com/t/standard-nvidia-cuda-tests-fail-with-dual-rtx-4090-linux-box

I tried running the same script in Windows, works fine. So, the conclusion is, we need to wait for Nvidia to update the drivers for Linux. Closing the issue. Thanks for your time @sgugger .