Whisper generate does not operate as expected with DDP
Author: AvivSham
Pull request: no
Labels: name: bug

name: Audio

### System Info

Hi all,
Based on [the conversation](https://github.com/huggingface/transformers/issues/24272#issuecomment-1729975005), @sanchit-gandhi has requested me to open a new issue.
As mentioned before we are trying to fine-tune Whisper with prompts on `4 V100 GPU` machine. Following [your suggestion](https://github.com/huggingface/transformers/issues/23651) we replaced `Adam` optimizer with `Adafactor` which reduced the memory footprint a lot (thanks for the tip!). However, now we are facing another issue, when training whisper with prompt during the evaluation (that uses `generate` method) the prompts are passed as a list of independent prompts which is not supported (as far as I know).  
**_Note that this behavior is specifically related to DDP training and occurs during evaluation, if we train using single GPU / CPU this error is not raised._**

The config we use:
```python
training_args = Seq2SeqTrainingArguments(
    output_dir="./outputs/foo_foo",  # change to a repo name of your choice
    per_device_train_batch_size=4,
    gradient_accumulation_steps=1,  # increase by 2x for every 2x decrease in batch size
    learning_rate=5e-6,
    warmup_steps=0,
    # max_steps=4000,
    max_steps=10000,
    gradient_checkpointing=False,
    fp16=True,
    evaluation_strategy="steps",
    per_device_eval_batch_size=1,
    predict_with_generate=True,
    generation_max_length=225,
    save_steps=10000,
    eval_steps=10000,
    logging_steps=1,
    report_to=["none"],
    load_best_model_at_end=True,
    metric_for_best_model="eval_validation_wer",
    greater_is_better=False,
    push_to_hub=False,
    remove_unused_columns=False,
    dataloader_num_workers=0,
    optim="adafactor"
)

```

The raised error:
```
decoder_input_ids_start = torch.ones((batch_size, 1), dtype=torch.long, device=device) * decoder_start_token_id
TypeError: only integer tensors of a single element can be converted to an index
```

The reason for this error is [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py#L1719), since `prompt_ids` are list of prompts with length equal to `per_device_eval_batch_size * NUM_DEVICES` (which equals `4` in our case). After unpacking `decoder_start_token_id` is list type instead of int.

Let me provide more details about the learning setup and compare the expected behavior with the current behavior. Our dataset produces threesomes of (`audio_features`, `labels`, `prompt_ids`). Note that `prompt_ids` have identical length to `audio_features` and `labels`. This is the case for both training and evaluation scenarios. 
The expected behavior:  if passing `per_device_eval_batch_size=1` each device should be assigned a single prompt + speech features.
The current behavior: prompts are passed as batch of `per_device_eval_batch_size * number_of_available_devices` which is not supported and raises an error.

Can you please help resolve this issue?
Thanks!


### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

See above

### Expected behavior

The expected behavior:  if passing `per_device_eval_batch_size=1` in the training args each device should be assigned a single prompt + speech features.

Comments:
Thanks for opening this issue @AvivSham! Essentially, Whisper prompting at inference time assumes a batch size of 1. This is quite limiting, so we should extend it to handling batched inputs. I'll open a PR to fix this!

@sanchit-gandhi  any update on this one? the auto-bot marked it as stale for some reason. I'm waiting for your response.

Super sorry about the delay here @AvivSham - it's still on my list to do! cc'ing @ylacombe in case you have the bandwidth to look into this? Otherwise, if anyone in the community would like to try their hand at this PR, feel free to go ahead already and myself and @ylacombe can do a review!

@sanchit-gandhi @ylacombe 
can you please change the settings for this issue specifically so it won't be closed auto every time? 



Hi @AvivSham I am working on the issues related to fine-tuning with prompts as well. Would you like to collaborate and join forces on this?



Hi @samuelazran, sorry for the late reply, this is a very hectic period of time for me.
Unfortunately, I don't have the bandwidth to help solving this issue.


@amyeroberts this is sill an opened bug.

Hey @AvivSham - super sorry for the late reply here. Could you confirm that the error still persists with the latest version of Transformers? (`pip install --upgrade transformers`) We did a refactor of the prompt ids, and so would like to double check that the error still persists before proposing new changes. Apologies again for the delay - we'll try and make the process of getting a fix as quick as possible.

Hi @sanchit-gandhi  how are you?
we decided to ditch this project, therefore it won't be easy for me to reproduce this issue.
Maybe @samuelazran can help :)



@amyeroberts I'm not sure that this bug is resolved.

@AvivSham OK, are you still seeing issues? If you can share something which would enable us to reproduce, then we can reopen and work on it

> Hi @sanchit-gandhi how are you? we decided to ditch this project, therefore it won't be easy for me to reproduce this issue. Maybe @samuelazran can help :)



@AvivSham Yes, I saw the previous message. This was why it was allowed to go stale and close. I'd assumed from your follow-up comment that you were encountering the issue again. Unfortunately, without a reproducer there's not much we can do. If you or someone else is able to share one, then we can start to look at the issue again.