`eos_token_id` breaks the `T5ForConditionalGeneration`
Author: girishponkiya
Pull request: no
Labels: name: Ex: Generation

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `T5ForConditionalGeneration`

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

```python
from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration

T5_PATH = 't5-base' # "t5-small", "t5-base", "t5-large", "t5-3b", "t5-11b"

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # My envirnment uses CPU

t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)
t5_config = T5Config.from_pretrained(T5_PATH)
t5_mlm = T5ForConditionalGeneration.from_pretrained(T5_PATH, config=t5_config).to(DEVICE)

text = 'India is a <extra_id_0> of the world. 
'
encoded = t5_tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')
input_ids = encoded['input_ids'].to(DEVICE)

# I want to set <extra_id_1> as end of a sequence.
eos_token_id = t5_tokenizer.encode('<extra_id_1>')[0]

outputs = t5_mlm.generate(input_ids=input_ids, 
                          num_beams=200, num_return_sequences=20,
                          eos_token_id=eos_token_id,  # <-- This is causes the error. Just removing it works fine.
                          max_length=5)

results = [t5_tokenizer.decode(output, skip_special_tokens=False, clean_up_tokenization_spaces=False) for output in outputs]
print(results)
```


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
* [x] Just playing with the `T5` model

## To reproduce

Steps to reproduce the behavior:

1. Start a new Google Colab notebook,
2. Install transformers using `pip install git+https://github.com/huggingface/transformers`,
3. Restart the runtime, and
4. Run the above snippet.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
AssertionError                            Traceback (most recent call last)

<ipython-input-86-aae5bd45e59c> in <module>()
      2                           num_beams=200, num_return_sequences=20, max_length=5,
      3                         #   repetition_penalty=1.2,
----> 4                           eos_token_id=eos_token_id,
      5                           )
      6 # print(outputs)

2 frames

/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17 

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id)
    993                 vocab_size=vocab_size,
    994                 encoder_outputs=encoder_outputs,
--> 995                 attention_mask=attention_mask,
    996             )
    997         else:

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in _generate_beam_search(self, input_ids, cur_len, max_length, min_length, do_sample, early_stopping, temperature, top_k, top_p, repetition_penalty, no_repeat_ngram_size, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, decoder_start_token_id, batch_size, num_return_sequences, length_penalty, num_beams, vocab_size, encoder_outputs, attention_mask)
   1359                     next_scores[batch_idx, :num_beams] == beam_scores.view(batch_size, num_beams)[batch_idx]
   1360                 ), "If batch_idx is not done, final next scores: {} have to equal to accumulated beam_scores: {}".format(
-> 1361                     next_scores[:, :num_beams][batch_idx], beam_scores.view(batch_size, num_beams)[batch_idx],
   1362                 )
   1363 

AssertionError: If batch_idx is not done, final next scores: tensor([-3.5015, -4.4583, -4.5677, -4.5808, -4.7325, -4.8779, -5.1127, -5.2106,
        -5.2147, -5.3292, -5.3575, -5.5227, -5.6846, -5.7290, -5.8227, -5.8522,
        -5.8537, -5.9930, -6.0119, -6.0440, -6.0469, -6.0496, -6.0739, -6.0959,
        -6.1007, -6.1125, -6.1519, -6.2492, -6.2850, -6.3033, -6.3035, -6.3677,
        -6.3798, -6.4438, -6.4508, -6.4511, -6.4993, -6.5149, -6.5269, -6.5330,
        -6.5335, -6.5662, -6.6071, -6.6222, -6.6331, -6.6437, -6.6454, -6.6660,
        -6.7163, -6.7690, -6.7841, -6.8139, -6.8147, -6.8157, -6.8297, -6.8297,
        -6.8326, -6.8448, -6.8493, -6.8763, -6.8866, -6.8871, -6.9026, -6.9046,
        -6.9078, -6.9475, -6.9544, -6.9654, -6.9672, -6.9735, -6.9749, -6.9795,
        -7.0020, -7.0024, -7.0124, -7.0310, -7.0837, -7.1405, -7.1407, -7.1420,
        -7.1652, -7.1675, -7.1788, -7.1800, -7.1875, -7.2146, -7.2148, -7.2294,
        -7.2538, -7.2765, -7.2775, -7.2859, -7.3110, -7.3120, -7.3189, -7.3403,
        -7.3438, -7.3479, -7.3535, -7.3592, -7.3733, -7.3740, -7.3842, -7.3848,
        -7.3882, -7.3889, -7.4192, -7.4192, -7.4211, -7.4657, -7.4662, -7.4774,
        -7.4776, -7.4862, -7.4999, -7.5150, -7.5156, -7.5187, -7.5214, -7.5253,
        -7.5365, -7.5376, -7.5547, -7.5566, -7.5815, -7.5922, -7.5940, -7.5960,
        -7.5996, -7.6115, -7.6128, -7.6406, -7.6485, -7.6619, -7.6744, -7.6752,
        -7.6804, -7.6875, -7.6928, -7.6979, -7.7064, -7.7172, -7.7229, -7.7429,
        -7.7431, -7.7482, -7.7510, -7.7576, -7.7619, -7.7641, -7.7655, -7.7721,
        -7.7787, -7.7792, -7.7842, -7.7859, -7.8005, -7.8083, -7.8087, -7.8128,
        -7.8145, -7.8184, -7.8185, -7.8197, -7.8228, -7.8275, -7.8381, -7.8478,
        -7.8572, -7.8598, -7.8677, -7.8712, -7.8755, -7.8820, -7.8867, -7.9135,
        -7.9308, -7.9485, -7.9613, -7.9629, -7.9649, -7.9706, -7.9714, -7.9739,
        -7.9740, -7.9757, -7.9779, -7.9809, -7.9858, -7.9964, -7.9992, -8.0047,
        -8.0086, -8.0105, -8.0255, -8.0500, -8.0550, -8.0604, -8.0811, -8.0858]) have to equal to accumulated beam_scores: tensor([-4.7325, -5.1127, -5.2106, -5.3292, -5.8227, -5.8522, -6.0739, -6.0959,
        -6.1007, -6.1125, -6.2492, -6.3033, -6.3035, -6.3798, -6.4438, -6.4508,
        -6.4511, -6.4993, -6.5269, -6.5330, -6.5662, -6.6331, -6.6660, -6.7163,
        -6.8139, -6.8157, -6.8297, -6.8297, -6.8493, -6.8763, -6.8871, -6.9026,
        -6.9078, -6.9475, -6.9735, -6.9749, -7.0020, -7.0024, -7.0837, -7.1405,
        -7.1420, -7.1652, -7.1800, -7.1875, -7.2146, -7.2294, -7.2538, -7.2765,
        -7.2775, -7.2859, -7.3189, -7.3403, -7.3438, -7.3535, -7.3592, -7.3733,
        -7.4192, -7.4192, -7.4657, -7.4662, -7.4774, -7.4999, -7.5150, -7.5156,
        -7.5187, -7.5253, -7.5365, -7.5547, -7.5566, -7.5815, -7.5922, -7.5940,
        -7.5996, -7.6115, -7.6128, -7.6406, -7.6619, -7.6744, -7.6752, -7.6804,
        -7.6875, -7.7172, -7.7229, -7.7429, -7.7431, -7.7482, -7.7510, -7.7619,
        -7.7641, -7.7655, -7.7787, -7.7792, -7.7842, -7.8083, -7.8087, -7.8145,
        -7.8184, -7.8185, -7.8197, -7.8228, -7.8275, -7.8478, -7.8572, -7.8598,
        -7.8677, -7.8712, -7.8755, -7.8820, -7.8867, -7.9613, -7.9629, -7.9706,
        -7.9714, -7.9739, -7.9740, -7.9757, -7.9779, -7.9809, -7.9858, -7.9964,
        -7.9992, -8.0047, -8.0086, -8.0255, -8.0500, -8.0550, -8.0604, -8.0811,
        -8.0858, -8.0862, -8.0880, -8.1197, -8.1198, -8.1261, -8.1331, -8.1337,
        -8.1364, -8.1509, -8.1633, -8.1681, -8.1759, -8.1843, -8.2007, -8.2019,
        -8.2094, -8.2223, -8.2236, -8.2504, -8.2580, -8.2653, -8.2748, -8.2778,
        -8.2782, -8.2860, -8.2931, -8.2932, -8.2963, -8.3047, -8.3111, -8.3161,
        -8.3215, -8.3255, -8.3393, -8.3439, -8.3567, -8.3608, -8.3706, -8.3810,
        -8.3858, -8.4023, -8.4076, -8.4111, -8.4153, -8.4198, -8.4335, -8.4365,
        -8.4390, -8.4397, -8.4480, -8.4511, -8.4532, -8.4594, -8.4663, -8.4681,
        -8.4686, -8.4720, -8.4726, -8.4778, -8.4781, -8.4786, -8.4885, -8.5027,
        -8.5173, -8.5181, -8.5226, -8.5298, -8.5308, -8.5341, -8.5463, -8.5466])
```

## Expected behavior

If I remove the `eos_token_id` argument from `t5_mlm.generate(...)` (refer the given code snippet), it returns the following (expected) output:
```
['<extra_id_0> cornerstone<extra_id_1>',
 '<extra_id_0> part<extra_id_1> developing',
 '<extra_id_0> huge part<extra_id_1>',
 '<extra_id_0> big part<extra_id_1>',
 '<extra_id_0> beautiful part<extra_id_1>',
 '<extra_id_0> very important part',
 '<extra_id_0> part<extra_id_1> larger',
 '<extra_id_0> unique part<extra_id_1>',
 '<extra_id_0> part<extra_id_1> developed',
 '<extra_id_0> part<extra_id_1> large',
 '<extra_id_0> beautiful country in',
 '<extra_id_0> part of the',
 '<extra_id_0> small part<extra_id_1>',
 '<extra_id_0> part<extra_id_1> great',
 '<extra_id_0> part<extra_id_1> bigger',
 '<extra_id_0> country in the',
 '<extra_id_0> large part<extra_id_1>',
 '<extra_id_0> part<extra_id_1> middle',
 '<extra_id_0> significant part<extra_id_1>',
 '<extra_id_0> part<extra_id_1> vast']
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Google Colab
- Python version: 3.6
- PyTorch version (GPU?): 1.5.0+cu101
- Tensorflow version (GPU?): N/A
- Using GPU in script?:  No
- Using distributed or parallel set-up in script?: No


Comments:
Hi @girishponkiya,

Thanks for the detailed error description. I was not able to reproduce the error with the newest version of transformers, 2.9.0.

See google colab here: https://colab.research.google.com/drive/14TQzMBPuaNb6aZ_NoVFQrr_ZTJS2mCr3 . Can you check again with a newer version of transformers?

Hi @patrickvonplaten,

Because of a bug in `tokenizers` (v0.7.0), the following code sets a wrong value to `eos_token_id`:
```python
# I want to set <extra_id_1> as end of a sequence.
eos_token_id = t5_tokenizer.encode('<extra_id_1>')[0]
```

The correct value of `eos_token_id` should be 32098 (index of <extra_id_1>).  Just replace the above snippet with the following to reproduce the error:
```python
# I want to set <extra_id_1> as end of a sequence.
eos_token_id = 32098  # t5_tokenizer.encode('<extra_id_1>')[0]
```

PS: for more details on the tokenizer bug, you may refer #4021 

same issue with GPT-2 to generate texts using beam search, eos_token_id set to 50256(
)

I had the same issue with GPT-2 and fixed it by hacking the code in `transformers/modeling_utils.py`
from
<pre>
if eos_token_id is not None and all(
    (token_id % vocab_size).item() <b>is not</b> eos_token_id for token_id in next_tokens[batch_idx]
):
    assert torch.all(
        next_scores[batch_idx, :num_beams] == beam_scores.view(batch_size, num_beams)[batch_idx]
    ), "If batch_idx is not done, final next scores: {} have to equal to accumulated beam_scores: {}".format(
        next_scores[:, :num_beams][batch_idx], beam_scores.view(batch_size, num_beams)[batch_idx],
    )
</pre>
to
<pre>
if eos_token_id is not None and all(
    (token_id % vocab_size).item() <b>!=</b> eos_token_id for token_id in next_tokens[batch_idx]
):
    assert torch.all(
        next_scores[batch_idx, :num_beams] == beam_scores.view(batch_size, num_beams)[batch_idx]
    ), "If batch_idx is not done, final next scores: {} have to equal to accumulated beam_scores: {}".format(
        next_scores[:, :num_beams][batch_idx], beam_scores.view(batch_size, num_beams)[batch_idx],
    )
</pre>
Hope it could help.

@junyann : Thanks, that's not a hack, should be the right fix. Irrespective of whether the the issues are solely caused by this, the "is not" comparison is wrong and should be fixed.

@patrickvonplaten : Thoughts?

Encountered the same assertion failure today and was looking to reproduce, ended up here.