BartModel decodes sequence of incorrect length when decoder_input_ids is specified / Output shape mismatch due to when `use_cache` True/False
Author: xhluca
Pull request: no

From the [Bart docs](https://huggingface.co/transformers/model_doc/bart.html#bartmodel), the `decoder_input_ids` attribute should be a tensor of shape `(batch_size, target_sequence_length)`. If we call a `BartModel` without specifying `decoder_input_ids`, the decoded sequence length correctly matches that of `input_ids`. When it is specified, the output sequence is not of shape `target_sequence_length`.

## Environment

Name: torch
Version: 1.6.0+cu101

Name: transformers
Version: 3.0.2

Name: tokenizers
Version: 0.8.1rc1

The error can be reproduced in Colab or Kaggle. See [this notebook ](https://colab.research.google.com/gist/xhlulu/dd989fc7f96b777c01c083762375dfbe/bart-sequence-problems.ipynb)for example.

## Example

```python
import transformers as tfm

model = tfm.BartModel.from_pretrained('facebook/bart-base')
tokenizer = tfm.BartTokenizer.from_pretrained('facebook/bart-base')

input_seq = [
    "What's the capital of Canada?",
    "What's the capital of USA?"    
]

output_seq = [
    "It's Ottawa",
    "It's Washington"
]

input_tokens = tokenizer.batch_encode_plus(input_seq, return_tensors='pt', padding=True)
input_ids = input_tokens['input_ids']

output_tokens = tokenizer.batch_encode_plus(output_seq, return_tensors='pt', padding=True)
output_ids = output_tokens['input_ids']

print(input_ids.size(), output_ids.size())  # Returns torch.Size([2, 9]) torch.Size([2, 5])

# Okay
outputs = model.forward(input_ids)
outputs[0].size()  # Returns `torch.Size([2, 9, 768])`

# Incorrect
outputs = model.forward(input_ids, decoder_input_ids=output_ids)
outputs[0].size()  # Returns torch.Size([2, 1, 768])
```

Comments:
@patrickvonplaten Actually, going over [the source code ](https://huggingface.co/transformers/_modules/transformers/modeling_bart.html#BartModel), I found that the exact line in the definition of `class BartModel(PretrainedBartModel)` that was causing this problem is:
```python
use_cache = use_cache if use_cache is not None else self.config.use_cache
```

In the `forward` method. Since use_cache is set to `False` when `decoder_input_ids` is `None`, this line forces `use_cache` value to always be True if `decoder_input_ids` is a tensor. 

I posted my [experiments here](https://www.kaggle.com/xhlulu/bart-experiments) in case they are useful.

I realized that this is actually wrong:
> In the forward method. Since use_cache is set to False when decoder_input_ids is None, this line forces use_cache value to always be True if decoder_input_ids is a tensor.

Re-reading the code made me realize that my problem could be solved by explicitly specifying `use_cache=False` when calling `model.forward`. This is likely because when the `use_cache`  attribute in `model.forward` is `None`, it falls back to `model.config.use_cache`, which is set to True by default.

I'm not sure whether what we have here is the intended behavior for BART, so I'll let @sshleifer @patrickvonplaten make the decision to close this :)

This seems to be related to https://github.com/huggingface/transformers/issues/6348. @sshleifer do you want to take a look at this?

@sshleifer I think this is a problem because in the first pass when the `cache` is still empty, `use_cache=True` and `decoder_input_ids` is of length 9 then the `last_hidden_state` should also be of size 9 **and** the cache should be returned. I can take a look this week if you are very busy - let me know!

Yes that would be helpful @patrickvonplaten !

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@patrickvonplaten  Make sure to remove `use_cache` in this PR to solve problem.