SpeechEncoderDecoderModel does not return a loss regardless of labels
Author: OllieBroadhurst
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.11.0.dev0
- Platform: Ubuntu
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.1+cu102 (GPU)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

@patrickvonplaten, @patil-suraj

Examples:

Model I am using (Bert, XLNet ...):
`SpeechEncoderDecoderModel` from `facebook/s2t-wav2vec2-large-en-de`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

Tested on the examples in the docs.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I used my own 5-second audio file for this example. Nothing fancy about it.

## To reproduce

The text encoder decoder example in the docs [here](https://huggingface.co/transformers/master/model_doc/encoderdecoder.html#encoderdecodermodel):

```python
from transformers import EncoderDecoderModel, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)

outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, labels=input_ids)
loss, logits = outputs.loss, outputs.logits
```
Produces a loss of `11.2099` as intended when provided with labels.

However, the speech encoder decoder models provide no loss regardless of the labels provided. From a slightly adapted version of the example [here](https://huggingface.co/transformers/master/model_doc/speechencoderdecoder.html#speechencoderdecodermodel):

```python
from transformers import SpeechEncoderDecoderModel, Speech2Text2Processor

processor = Speech2Text2Processor.from_pretrained('facebook/s2t-wav2vec2-large-en-de')
model = SpeechEncoderDecoderModel.from_pretrained('facebook/s2t-wav2vec2-large-en-de')

features, _ = librosa.load("my_audio.wav", sr=16_000)

input_values = processor(features, sampling_rate=16_000, return_tensors="pt").input_values
decoder_input_ids = torch.tensor([[model.config.decoder.decoder_start_token_id]])
outputs = model(input_values=input_values, decoder_input_ids=decoder_input_ids, labels="PUT_LITERALLY_ANYTHING_HERE")
```
```python
output.keys()
```
Returns `odict_keys(['logits', 'past_key_values', 'encoder_last_hidden_state'])`.


## Expected behavior

The above example should throw an error (the label param is just a raw string). If given a valid set of labels then the `loss` key and values should be in the model output dict (according to the `forward` docs).


Comments:
Hmm not sure why `labels` are accepted in the forward pass, but not send to the decoder. Will open a PR to fix this. Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.