[SpeechT5] Attention mask not changed according to decoder inputs
Author: Joao-Maria-Janeiro
Pull request: no
Labels: name: Audio

### System Info

- `transformers` version: 4.33.3
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The decoder inputs are changed to be shifted right by one, and interleaved by the reduction factor.
However, the attention mask to the decoder remains the same, which if we use a reduction_factor != 1 will result in a shape missmatch.

You can check the line I am referring to here:
https://github.com/huggingface/transformers/blob/2f3ea08a077ba3133fa8a604b22436cad250b055/src/transformers/models/speecht5/modeling_speecht5.py#L2733

### Expected behavior

The attention mask should have the same changes applied as the decoder input, resulting in the same shape, I believe.

Comments:
cc @ylacombe could you take a look when you get the chance? You know SpeechT5 pretty well by now!

Hey, thanks for opening this issue!
I will take a look in the next few days, in the meantime, do you have a script to reproduce the mismatch @Joao-Maria-Janeiro ?

Hey @Joao-Maria-Janeiro , any update on a reproducing script?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@ylacombe  - +1 This is still an issue. It's very easy to reproduce:

```python
from transformers import SpeechT5Processor, SpeechT5ForSpeechToSpeech
import numpy as np

processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_vc")
model = SpeechT5ForSpeechToSpeech.from_pretrained("microsoft/speecht5_vc")

features = processor(
    audio=[np.random.random(size=(2048,)) for waveform in range(3)],
    audio_target=[np.random.random(size=(2048,)) for waveform in range(3)], 
    return_tensors="pt",
    padding=True,
    sampling_rate=16000,
)
outputs = model(**features, return_dict=True)
```

Produces:

```
Traceback (most recent call last):
  File "[REDACTED]/reproduce.py", line 8, in <module>
    outputs = model(**features, return_dict=True)
  File "/[REDACTED]/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/[REDACTED]/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/[REDACTED]/transformers/models/speecht5/modeling_speecht5.py", line 2953, in forward
    outputs = self.speecht5(
  File "/[REDACTED]/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/[REDACTED]/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/[REDACTED]/transformers/models/speecht5/modeling_speecht5.py", line 2211, in forward
    decoder_outputs = self.decoder(
  File "/[REDACTED]/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/[REDACTED]/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/[REDACTED]/transformers/models/speecht5/modeling_speecht5.py", line 1734, in forward
    outputs = self.wrapped_decoder(
  File "/[REDACTED]/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/[REDACTED]/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/[REDACTED]/transformers/models/speecht5/modeling_speecht5.py", line 1594, in forward
    attention_mask = _prepare_4d_causal_attention_mask(
  File "/[REDACTED]/transformers/modeling_attn_mask_utils.py", line 195, in _prepare_4d_causal_attention_mask
    attention_mask = attn_mask_converter.to_4d(
  File "/[REDACTED]/transformers/modeling_attn_mask_utils.py", line 117, in to_4d
    expanded_4d_mask = expanded_attn_mask if causal_4d_mask is None else expanded_attn_mask + causal_4d_mask
RuntimeError: The size of tensor a (9) must match the size of tensor b (4) at non-singleton dimension 3
```

Hey @DavidMChan, thanks for the script, #28071 should fix this!

Thanks for following up on this!


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale :) This still requires a fix.

Continues to be useful. @ylacombe  - what needs to be done to get #28071 merged? Is there anything that would be helpful for me to take a look at? 

Hey @DavidMChan, sorry for the late response, there was an issue with the model slow tests that I have yet to find the time to resolve, but getting back on this as soon as possible this week

Gentle ping @ylacombe 

Another ping @ylacombe 

Another ping @ylacombe