BartForConditionalGeneration: `decoder_input_ids` should not be computed if `decoder_inputs_embeds` is set
Author: silviu-oprea
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.8.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.10
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten, @patil-suraj

## Information

- Model I am using: Bart. The specific class is BartForConditionalGeneration.
- Summary: Cannot provide `decoder_inputs_embeds` to `BartForConditionalGeneration.forward` if `labels` are provided.

In `BartForConditionalGeneration.forward`, there is the following code on line 1285 of modelling_bart.py in the current version (4.8.2):

    if labels is not None:
        if decoder_input_ids is None:
            decoder_input_ids = shift_tokens_right(
                labels, self.config.pad_token_id, self.config.decoder_start_token_id
            )

So, if `labels` are provided, `decoder_input_ids` are set to the `labels` shifted to the right. This is problematic: if `decoder_inputs_embeds` is also set, the call to `self.model`, which eventually gets to `BartDecoder.forward`, will raise an error. In particular, the snipped that does this in `BartDecoder.forward` is (line 967 of modelling_bart.py in version 4.8.2):

        if input_ids is not None and inputs_embeds is not None:
            raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time")

The fix is quite simple, similar to what is there already in `BartModel.forward` (line 1152 in version 4.8.2). Mainly, we should not compute `decoder_input_ids` if `decoder_inputs_embeds` is provided. That is, the first snippet above in In `BartForConditionalGeneration.forward` should be:

    if labels is not None:
        if decoder_input_ids is None and decoder_inputs_embeds is None: # <- this line changed
            decoder_input_ids = shift_tokens_right(
                labels, self.config.pad_token_id, self.config.decoder_start_token_id
            )

## To reproduce

Steps to reproduce the behavior:

    model = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
    input_ids = torch.tensor([[1, 2, 3]])
    labels = torch.tensor([[1, 2, 3]])
    decoder_input_ids = torch.tensor([[1, 2, 3]])
    decoder_inputs_embeds = model.get_input_embeddings()(decoder_input_ids)
    model(input_ids, decoder_inputs_embeds=decoder_inputs_embeds, labels=labels)    

## Expected behavior
`decoder_input_ids` should not be computed if `decoder_inputs_embeds` and `labels` are provided.


Comments:
same problem here, my solution is removing argument  `labels`, but I have to calculate loss outside the forward function.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.