Model fail to revert to generation_mode=False after generation
Author: AOZMH
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BART

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name): CNN/DM
* [ ] my own task or dataset: (give details below)

## To reproduce

Hi @sshleifer,

Thanks for the amazing model!
I found a bug when alternatively trying to use **forward** to train the BartForConditionalGeneration and use **generate** to inference and evaluate the trained model. As shown in the code below, if I first use generate function then call forward function, the generation_mode attribute of decoder is set to True, and the shape of the decoder_output seems incorrect.


```
model = BartForConditionalGeneration.from_pretrained('bart-large-cnn')
tokenizer = BartTokenizer.from_pretrained('bart-large-cnn')

# input sequence
input_seq = "Bart is a deep-learning pretrained model implemented in pytorch. It can smoothly handle summarization. It is a big model pretrained for generation tasks, especially summarizaition."
input_ids = torch.LongTensor([tokenizer.encode(input_seq)])

# expected output sequence
decoder_input_seq = "Bart is a big pretrained deep model in pytorch for summarization."
decoder_input_ids = torch.LongTensor([tokenizer.encode(decoder_input_seq)])

# using generate method to inference
with torch.no_grad():
    result = model.generate(input_ids=input_ids, eos_token_ids=tokenizer.eos_token_id, num_beams=4, max_length=20)
print(tokenizer.decode(result[0]))
# 'B. It is a big model pretrained for generation tasks, especially summarizaition. It'

# NOW use forward to train
result = model(input_ids, decoder_input_ids=decoder_input_ids)
# the shape of decoder_output and encoder_output
# what expected is: <1, 18, 50264> and <40, 1, 1024>
# but actual output is:  torch.Size([1, 1, 50264]) torch.Size([40, 1, 1024])
print(result[0].shape, result[2].shape)

```

Such issue can **seemingly** be addressed by mannually setting the generation_mode.

```
# mannually set the generation mode to False **seemingly** fix the issue
model.model.decoder.generation_mode=False
result = model(input_ids, decoder_input_ids=decoder_input_ids)
print(result[0].shape, result[2].shape)
# output is: torch.Size([1, 18, 50264]) torch.Size([40, 1, 1024]) and make sense
```

However, now the output of the forward function doesn't make sense, as shown below.

```
with torch.no_grad():
    result = model.generate(input_ids=input_ids, eos_token_ids=tokenizer.eos_token_id, num_beams=4, max_length=20)

model.model.decoder.generation_mode=False
new_result = model(input_ids, decoder_input_ids=result)

print(tokenizer.decode(torch.argmax(new_result[0][0], dim=1)))
# ' and and and and and and and and and and and and and and and and and and and'
```
In expectation, when we feed the generated decoder sequence to the decoder and remain the input of the encoder unchanged, the output of the decoder would at least resemble the decoder_input. However, the actual output of the model is `' and and and and and and and and and and and and and and and and and and and'`, which is definately not a reasonable output of a trained decoder.

To go deeper, I tested different samples, all feeding the generated output of a encoder_input to the decoder and inspect the decoder_output (which "should" resemble decoder_input, i.e. generated sequence), the same happens everytime for me (most output is the duplication of a single token for many times, like "the the the ...", "and and and...", ". . . . ").

Given all these results, I think there is a bug in either the implementation of **generate**  or that of **forward**, the revert of generation_mode is simple, but I don't think the output of forward is a reasonable result currently. Could you please look into the issue?

 @sshleifer If I'm not using those functions in a correct manner, any advise or instruction is welcomed! Many thanks for the help!


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->



## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: master-branch
- Platform: windows 10
- Python version: 3.7.0
- PyTorch version (GPU?):  1.4.0
- Tensorflow version (GPU?): /
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
I don't follow your workflow. Why do you want to run forward after you generate using the generated_ids as decoder_input_ids?

It would be easier to follow if you:
- checked whether the behavior you expect is achieved by the authors' implementation in fairseq
- made the example smaller

Thanks for contributing!


@sshleifer Thanks for the reply! 
Your latest PR already fixed the generation_mode, thanks! The rest of the problem (that you probably failed to follow) is like a sanity check, but the result failed to meet the expectation for me.

Basically, if I feed sequence s1 to the encoder to let the model to generate, suppose sequence s2 is generated, then if I directly feed <encoder_input_ids=s1, decoder_input_ids=s2> to the model, the output of the decoder would be something that resembles s2. However, as shown in the code below, the actual output of the decoder given input <s1,s2> is a sequence "and and ... and", which is largly different from s2, and that's where I'm confused.

```
with torch.no_grad():
    result = model.generate(input_ids=input_ids, eos_token_ids=tokenizer.eos_token_id, num_beams=4, max_length=20)

model.model.decoder.generation_mode=False
new_result = model(input_ids, decoder_input_ids=result)

print(tokenizer.decode(torch.argmax(new_result[0][0], dim=1)))
# ' and and and and and and and and and and and and and and and and and and and'
```

Of course we won't use this code in practice (since it's just a sanity check), but I post this because I'm wondering if it's my incorrect way of using `forward` function when training that caused this confusion. As for me, if I want to train the summarization model using <paragraph=s1, summary=s2> pairs, I formerly feed <encoder_input=s1, decoder_input=< bos >+s2> and train the model with decoder_output=s2+< eos >. May I ask if it is the correct way?

Thanks again for the kind help!

For summarization finetuning, I'd recommend:
- prepend a space to s1 and s2,
- then use `tokenizer.batch_encode_plus(s1, max_length=1024)` for `input_ids` and `attention_mask`
- then `tokenizer.batch_encode_plus(s2, max_length=1024)['input_ids']` to get `decoder_input_ids`


@sshleifer Thanks for the update! I'll try out you method and report the result I get for the problem above tomorrow.

The underlying issue here is fixed, closing!