Can't get BART to generate EOS token.
Author: marcotcr
Pull request: no

# 🐛 Bug
I finetuned Bart on a few seq2seq tasks. It seems to learn the right thing, but it never seems to stop generating text unless I set `max_length`, i.e. it never generates the EOS token on its own. This seems to be the case for the pretrained model as well: if I run the example [here](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration), it always produces very long summaries even if the input text is quite small.

To make the bug easy to reproduce, I set up a toy task below where I am finetuning Bart to just repeat the input.

## Information

Model I am using (Bert, XLNet ...): Bart

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a dataset for seq2seq training where the target is a copy of the source, e.g. below I create a dataset of sentences with `4,000` examples in the training set.
```python
import nlp
d = nlp.load_dataset('snli')
d = list(set(d['train']['premise'][:20000]))
import os
folder = '/tmp/bartz2'
if os.path.exists(folder):
    os.system('rm -rf %s' % folder)
os.mkdir(folder)
N = 4000
f = open(os.path.join(folder, 'train.source'), 'w')
f.write('\n'.join(d[:N]))
f = open(os.path.join(folder, 'val.source'), 'w')
f.write('\n'.join(d[N:]))
f.close()
f = open(os.path.join(folder, 'test.source'), 'w')
f.write('\n'.join(d[N:]))
f.close()
f = open(os.path.join(folder, 'train.target'), 'w')
f.write('\n'.join(d[:N]))
f.close()
f = open(os.path.join(folder, 'val.target'), 'w')
f.write('\n'.join(d[N:]))
f.close()
f = open(os.path.join(folder, 'test.target'), 'w')
f.write('\n'.join(d[N:]))
f.close()
```
2. Finetune bart on this dataset, i.e.
```sh
./finetune.sh --data_dir /tmp/bartz2/ --train_batch_size=4 --eval_batch_size=4 --output_dir=copybart --num_train_epochs 3 --model_name_or_path facebook/bart-large --n_val=1000 --n_test=1000 --task=translation
```
Caveat: I commented out `--fp16` in `finetune.sh`  

3. Generate text with the finetuned model
```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model = AutoModelForSeq2SeqLM.from_pretrained(f'/home/marcotcr/work/transformers/examples/seq2seq/copybart//best_tfmr')
tokenizer = AutoTokenizer.from_pretrained(f'/home/marcotcr/work/transformers/examples/seq2seq/copybart/best_tfmr')
model.to('cuda');
for text in d[N+1:N+10]:
    print(text)
    inputs = tokenizer([text], max_length=1024, return_tensors='pt', truncation=True).to('cuda')
    a = model.generate(**inputs,  early_stopping=True, num_beams=4, max_length=100, early_stoppy=True)
    dec = tokenizer.batch_decode(a, skip_special_tokens=True, clean_up_tokenization_spaces=False)
    print(dec[0])
    print()
```

Output (first few lines):
```
A black man walks away with a large basket full of items on his head.
A black man walks away with a large basket full of items on his head of a woman-on-shoreline-a-chosie-centre-of items in a basket full-items on their head.A fellow-side of the basket.-

A person is doing a bicycle trick over the rocks while a bystander gives a thumbs up.
A person is doing a bicycle trick over the rocks while a bystander gives a thumbs up.A couple of people are afoot in a bicycle trunk over the Rocks while a passerer gives an thumbs up to the bystander is a thumb up.T-up.

Female with long black hair and wearing eyeglasses, a blue shirt, sitting next to a male in a striped shirt, at a table, cutting into a chocolate cake with a single red candle in it.
Female with long black hair and wearing eyeglasses, a blue shirt, sitting next to a male in a striped shirt, at a table, cutting into a chocolate cake with a single red candle in it.The cake is a blue shirts, a purple-colored tissue-
```

## Expected behavior
I would expect it to emit the EOS token after copying the sentence. It seems to learn the right behavior up to that point (it copies the sentence), and I saw this in my real seq2seq tasks as well (i.e. it did the right thing but failed to stop). This behavior is also present in `test_generations.txt`.

I also tried training on a dataset where the target sentences ended in `
`, but that didn't change it.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 - `transformers` version: 3.0.2 (pulled from master this morning)
- Platform: Linux-5.3.0-61-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Comments:
Hmm, as far as I know the pretrained BART models do produce the EOS token and can actually generate pretty short summaries. For example if you look into the test `test_cnn_summarization_same_as_fairseq` you can see that the summaries are actually quite small (some are much smaller than `max_length`)

Also you could try to replace this line:
```python
a = model.generate(**inputs,  early_stopping=True, num_beams=4, max_length=100, early_stoppy=True)
```

with 

```python
a = model.generate(**inputs,  early_stopping=True, num_beams=4, max_length=100, length_penalty=2.0, no_repeat_ngram_size=3)
```

Also pinging @sshleifer here.

That test helped me figure it out, thanks. `min_length` is set to some default value, I'm guessing around `50`. If I set `min_length=0`, it works as expected. It sounds from the documentation that `0` should be the default value, so I guess there is a bug either in the documentation or in the code:
> min_length – (optional) int The min length of the sequence to be generated. Between 0 and infinity. Default to 0.

I'd close this, but I'm leaving it open just in case you guys want to fix this mismatch.
Thanks,

Interesting! Usually `config.min_length` should be set to 0 by default. Not sure why it is not set to `0` in your case. Thanks for the feedback though!

There is a bigger problem somewhere I suspect. This started happening around generation_utils.py time, and is happening in the blenderbot PR as well.  Also see #5656 
We should be able to generate EOS with min_length=50. 
I'll try to take a look later in the week if the mystery remains unsolved. 

Does early_stopping=True/False make any difference?

No early_stopping=True/False is not making any difference.
And setting config.min_length = 0 is still not working for my case as the fine-tuned model still producing truncated outputs.

@tromedlov22 you also tried BART? This issue is about BART.

@marcotcr the confusion is caused by the fact that by default we use `config.task_specific_params['summarization']`.
The way to override is to save the desired config locally.

```python
In [8]: AutoConfig.from_pretrained('facebook/bart-large').task_specific_params['summarization']
Out[8]:
{'early_stopping': True,
 'length_penalty': 2.0,
 'max_length': 142,
 'min_length': 56,
 'no_repeat_ngram_size': 3,
 'num_beams': 4}
```

We should probably add a `logger.info` statement that we are using task specific params.


Same issue here. I've been using BART-large-cnn (and -xsum) to finetune on my dataset. By setting `max_tokens=620`, and `min_tokens=0, 300, 500, and 600`, the BART still produces truncated (incomplete) sentences, which does not make sense to me. Any workaround/solution to this? @sshleifer Thanks!