inconsistent execution time
Author: pure-rgb
Pull request: no
Labels: name: bug

### System Info

```
- `transformers` version: 4.28.1
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.10.6
- Huggingface_hub version: 0.13.4
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@Arther

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Basically, I have 100 rows dataframe, total 10 files. I like to send it each one by one to my 4 GPUs. I use Llama 3. I'm testing the execution or completion time. Now, either use the data-parallel or model-parallel for inference, I got the around same execution time. Let's say

```
# using 1 GPU
one dataframe - > prompt template 
response = model.generate(prompt)

execution time: 20 second.
```

```
# using 3 GPU 
# with data-parallel
one dataframe - > prompt template 
response = model.generate(prompt)

execution time: 20 second (GPU:0)
execution time: 21 second (GPU:0)
execution time: 19 second (GPU:0)
```

Model definition

```python
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

torch_dtype = torch.float16
attn_implementation = "eager"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch_dtype,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map={"": torch.cuda.current_device()},
    attn_implementation=attn_implementation
)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        'up_proj', 'down_proj', 'gate_proj', 
        'k_proj', 'q_proj', 'v_proj', 'o_proj'
    ]
)
model = get_peft_model(model, peft_config)

for df_file in tqdm(xcel_list):
    df = pd.read_excel(df_file)
    messages = prepare_prompt_from_excel(df)

    prompt = tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
    )

    inputs = tokenizer(
        prompt, 
        return_tensors='pt', 
        padding=True, 
        truncation=True
    ).to("cuda")

    start = time.time()
    outputs = model.generate(
        **inputs, 
        max_length=2048, 
        num_return_sequences=1
    )
    exe = time.time() - start

...
```

### Expected behavior

If I use data-parallel on multiple GPU, a replicate of model will be placed to all GPUs and data will be splitted across GPUs.  Or, if I use model-parallel (`device_map=auto'), layers of that model will be distributed across GPUs. Apart from this, if no data or model parallel, just single GPU inference, I was expecting to get longer time for inference, and less time or faster inference if I use multi-gpu, either data or model parallel. But single GPU inference time and multi-gpu inference times are almost comparable. My another concern is that, while using data-parallel, as I am sending one data-frame / prompt template to all GPUs - does this single prompt template gets splitted into many chunks? I doubt that. Each GPUs still see the full data in data-parallel setup here, and that's the cause to get similar execution time, no!

Comments:
I think this is really an `accelerate` question, so pinging @muellerzr @SunMarc 

> My another concern is that, while using data-parallel, as I am sending one data-frame / prompt template to all GPUs - does this single prompt template gets splitted into many chunks? I doubt that. Each GPUs still see the full data in data-parallel setup here, and that's the cause to get similar execution time, no!

You need to send chunks of a batch automatically to each loaded model. Please read the following documentation : https://huggingface.co/docs/accelerate/en/usage_guides/distributed_inference#distributed-inference

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.