EncoderDecoderModel incorrectly establishing decoder input ids and decoder labels
Author: mitchelldehaven
Pull request: no
Labels: name: bug

### System Info

### Who can help?

Tagging based on both gitblame and pretrained models on Hub associated with this:
@patrickvonplaten 
@NielsRogge 

### Information

I was recently using the EncoderDecoderModel setup for training a Seq2Seq model. I noticed a few oddities during training, in particular I was trying to measure the token accuracy and found that the accuracy was decreasing while the loss was also decreasing (see accuracy calculation below, which is mostly correct, doesn't properly deal with EOS in padded instances):

```
        labels = batch["labels"][:,1:]
        non_padded_idx = labels != -100
        preds = torch.argmax(output.logits, dim=2)[:,:-1]
        accuracy = (labels == preds).sum() / non_padded_idx.sum()
```

I started looking at the EncoderDecoderModel class and found that there seems to be an issue with the [shift_tokens_right](https://github.com/huggingface/transformers/blob/v4.43.3/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L150) function:

```
def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start_token_id: int):
    """
    Shift input ids one token to the right.
    """
    shifted_input_ids = input_ids.new_zeros(input_ids.shape)
    shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()
    if decoder_start_token_id is None:
        raise ValueError("Make sure to set the decoder_start_token_id attribute of the model's configuration.")
    shifted_input_ids[:, 0] = decoder_start_token_id

    if pad_token_id is None:
        raise ValueError("Make sure to set the pad_token_id attribute of the model's configuration.")
    # replace possible -100 values in labels by `pad_token_id`
    shifted_input_ids.masked_fill_(shifted_input_ids == -100, pad_token_id)

    return shifted_input_ids
```

For context, this is taking the input labels and shifting them right for the decoder inputs. There are a few problems here:

```
    shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()
```

This seems to be trying to clip the EOS token off from the decoder inputs (which seems standard), however this doesn't always work if the input sequence is padded.

```
    shifted_input_ids[:, 0] = decoder_start_token_id
```

Since we already have special tokens in the input (the examples on the class page in the documentation do not use `skip_special_tokens` and I'm fairly sure the training setup assumes they are already in there), this results in the `sos_token_id` occupying the first two positions due to `shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()` in conjunction with this. 

Final problem is that as far as I can tell, the `sos_token_id` is never actually removed the labels like it should be.
I suspect this hasn't been caught due to the training process still be mostly correct, its still offset by 1 token and the additional sos_token in the output is filtered during `tokenizer.decode`.


If all the above makes sense, I can make a PR with the fixes I made to get my training performing as expected. If I'm misunderstanding something here, can someone please point out what I'm missing.

### Tasks

### Reproduction

Using a print statement within the EncoderDecoder model of the `decoder_input_ids` prior to decoder input indicates that there are 2 `sos_token_ids` in the input, when there should only be one.

### Expected behavior

There should only be 1 sos token in the input

Comments:
This has been up for a couple weeks with no follow up, maybe I tagged the wrong people? 

@ArthurZucker  

Hey! sorry for the late reply! 
Feel free to open a PR for a fix, it's been a long time so not sure if this is right but: 
- we need to make sure your PR introduces a better behavior -> training loss / accuracy before after is a good validation of you hypothethis
- we'll ask you to add a small test to make sure we now properly shift the inputs when there is padding or edge case! 
🤗 

Alright, will get a PR up over the weekend (hopefully).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.