Using the T5 model with huggingface's mask-fill pipeline
Author: p-christ
Pull request: no
Labels: name: wontfix

Does anyone know if it is possible to use the T5 model with hugging face's mask-fill pipeline? The below is how you can do it using the default model but i can't seem to figure out how to do is using the T5 model specifically?

```
from transformers import pipeline
nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in ' + nlp_fill.tokenizer.mask_token)

```

Trying this for example raises the error "TypeError: must be str, not NoneType" because nlp_fill.tokenizer.mask_token is None.

```
nlp_fill = pipeline('fill-mask',model="t5-base", tokenizer="t5-base")
nlp_fill('Hugging Face is a French company based in ' + nlp_fill.tokenizer.mask_token)
```

Stack overflow [question](https://stackoverflow.com/questions/61408753/using-the-t5-model-with-huggingfaces-mask-fill-pipeline) 

Comments:
Correct me if I'm wrong @patrickvonplaten, but I don't think T5 is trained on masked language modeling (and does not have a mask token) so will not work with this pipeline.

Yeah, `T5` is not trained on the conventional "Bert-like" masked language modeling objective. It does a special encoder-decoder masked language modeling (see docs [here](https://huggingface.co/transformers/model_doc/t5.html#training)), but this is not really supported in combination with the `mask-fill` pipeline at the moment.

Hi @patrickvonplaten, is there any plan to support `T5` with the `mask-fill` pipeline in the near future?

`T5` is an encoder-decoder model so I don't really see it as a fitting model for the `mask-fill` task. 

Could we use the following workaround?

* `<extra_id_0>` could be considered as a mask token
* Candidate sequences for the mask-token could be generated using a code, like: 
```python
from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration

T5_PATH = 't5-base' # "t5-small", "t5-base", "t5-large", "t5-3b", "t5-11b"

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # My envirnment uses CPU

t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)
t5_config = T5Config.from_pretrained(T5_PATH)
t5_mlm = T5ForConditionalGeneration.from_pretrained(T5_PATH, config=t5_config).to(DEVICE)

# Input text
text = 'India is a <extra_id_0> of the world. 
'

encoded = t5_tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')
input_ids = encoded['input_ids'].to(DEVICE)

# Generaing 20 sequences with maximum length set to 5
outputs = t5_mlm.generate(input_ids=input_ids, 
                          num_beams=200, num_return_sequences=20,
                          max_length=5)

_0_index = text.index('<extra_id_0>')
_result_prefix = text[:_0_index]
_result_suffix = text[_0_index+12:]  # 12 is the length of <extra_id_0>

def _filter(output, end_token='<extra_id_1>'):
    # The first token is <unk> (inidex at 0) and the second token is <extra_id_0> (indexed at 32099)
    _txt = t5_tokenizer.decode(output[2:], skip_special_tokens=False, clean_up_tokenization_spaces=False)
    if end_token in _txt:
        _end_token_index = _txt.index(end_token)
        return _result_prefix + _txt[:_end_token_index] + _result_suffix
    else:
        return _result_prefix + _txt + _result_suffix

results = list(map(_filter, outputs))
results
```
Output:
```
['India is a cornerstone of the world. 
',
 'India is a part of the world. 
',
 'India is a huge part of the world. 
',
 'India is a big part of the world. 
',
 'India is a beautiful part of the world. 
',
 'India is a very important part of the world. 
',
 'India is a part of the world. 
',
 'India is a unique part of the world. 
',
 'India is a part of the world. 
',
 'India is a part of the world. 
',
 'India is a beautiful country in of the world. 
',
 'India is a part of the of the world. 
',
 'India is a small part of the world. 
',
 'India is a part of the world. 
',
 'India is a part of the world. 
',
 'India is a country in the of the world. 
',
 'India is a large part of the world. 
',
 'India is a part of the world. 
',
 'India is a significant part of the world. 
',
 'India is a part of the world. 
']
```

@girishponkiya Thanks for your example! Unfortunately, I can't reproduce your results. I get

```
['India is a _0> of the world. 
',
 'India is a  ⁇ extra of the world. 
',
 'India is a India is  of the world. 
',
 'India is a  ⁇ extra_ of the world. 
',
 'India is a a  of the world. 
',
 'India is a [extra_ of the world. 
',
 'India is a India is an of the world. 
',
 'India is a of the world of the world. 
',
 'India is a India. of the world. 
',
 'India is a is a of the world. 
',
 'India is a India  ⁇  of the world. 
',
 'India is a Inde is  of the world. 
',
 'India is a ] of the of the world. 
',
 'India is a . of the world. 
',
 'India is a _0 of the world. 
',
 'India is a is  ⁇  of the world. 
',
 'India is a india is  of the world. 
',
 'India is a India is the of the world. 
',
 'India is a -0> of the world. 
',
 'India is a  ⁇ _ of the world. 
']
```

Tried on CPU, GPU, 't5-base' and 't5-3b' — same thing.

Could you please mention the version of torch, transformers and tokenizers? 

I used the followings:
* torch: 1.5.0+cu101
* transformers: 2.8.0
* tokenizers: 0.7.0

`tokenizers` in the latest version of `transformers` has a bug. Looking at your output, I believe you are using a buggy version of tokenizers. 

@girishponkiya I'm using 

```
transformers 2.9.0
tokenizers 0.7.0
torch 1.4.0
```

Tried tokenizers-0.5.2 transformers-2.8.0 — now it works, thank you!

Thanks to @takahiro971. He pointed out this bug in #4021. 

@girishponkiya  thanks a lot for your above code. Your example works but if I run your above code instead with the text :

`text = "<extra_id_0> came to power after defeating Stalin"
`

I get the following error:

```
/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in _generate_beam_search(self, input_ids, cur_len, max_length, min_length, do_sample, early_stopping, temperature, top_k, top_p, repetition_penalty, no_repeat_ngram_size, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, decoder_start_token_id, batch_size, num_return_sequences, length_penalty, num_beams, vocab_size, encoder_outputs, attention_mask)
   1354             # test that beam scores match previously calculated scores if not eos and batch_idx not done
   1355             if eos_token_id is not None and all(
-> 1356                 (token_id % vocab_size).item() is not eos_token_id for token_id in next_tokens[batch_idx]
   1357             ):
   1358                 assert torch.all(

UnboundLocalError: local variable 'next_tokens' referenced before assignment
```

Any ideas of the cause?



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Can I use multiple masking in a same sentence?

> Can I use multiple masking in a same sentence?

In this case, you need to set max_length higher or not set it at all.
This might produce more outputs than you need though.


outputs = t5_mlm.generate(input_ids=input_ids, 
                          num_beams=200, num_return_sequences=20,
                          max_length=5)

is very slow. How can I improve performance?

@jban-x3 I think you should try smaller values for `num_beams`

> Could we use the following workaround?
> 
> * `<extra_id_0>` could be considered as a mask token
> * Candidate sequences for the mask-token could be generated using a code, like:
> 
> ```python
> from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration
> 
> T5_PATH = 't5-base' # "t5-small", "t5-base", "t5-large", "t5-3b", "t5-11b"
> 
> DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # My envirnment uses CPU
> 
> t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)
> t5_config = T5Config.from_pretrained(T5_PATH)
> t5_mlm = T5ForConditionalGeneration.from_pretrained(T5_PATH, config=t5_config).to(DEVICE)
> 
> # Input text
> text = 'India is a <extra_id_0> of the world. 
'
> 
> encoded = t5_tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')
> input_ids = encoded['input_ids'].to(DEVICE)
> 
> # Generaing 20 sequences with maximum length set to 5
> outputs = t5_mlm.generate(input_ids=input_ids, 
>                           num_beams=200, num_return_sequences=20,
>                           max_length=5)
> 
> _0_index = text.index('<extra_id_0>')
> _result_prefix = text[:_0_index]
> _result_suffix = text[_0_index+12:]  # 12 is the length of <extra_id_0>
> 
> def _filter(output, end_token='<extra_id_1>'):
>     # The first token is <unk> (inidex at 0) and the second token is <extra_id_0> (indexed at 32099)
>     _txt = t5_tokenizer.decode(output[2:], skip_special_tokens=False, clean_up_tokenization_spaces=False)
>     if end_token in _txt:
>         _end_token_index = _txt.index(end_token)
>         return _result_prefix + _txt[:_end_token_index] + _result_suffix
>     else:
>         return _result_prefix + _txt + _result_suffix
> 
> results = list(map(_filter, outputs))
> results
> ```
> 
> Output:
> 
> ```
> ['India is a cornerstone of the world. 
',
>  'India is a part of the world. 
',
>  'India is a huge part of the world. 
',
>  'India is a big part of the world. 
',
>  'India is a beautiful part of the world. 
',
>  'India is a very important part of the world. 
',
>  'India is a part of the world. 
',
>  'India is a unique part of the world. 
',
>  'India is a part of the world. 
',
>  'India is a part of the world. 
',
>  'India is a beautiful country in of the world. 
',
>  'India is a part of the of the world. 
',
>  'India is a small part of the world. 
',
>  'India is a part of the world. 
',
>  'India is a part of the world. 
',
>  'India is a country in the of the world. 
',
>  'India is a large part of the world. 
',
>  'India is a part of the world. 
',
>  'India is a significant part of the world. 
',
>  'India is a part of the world. 
']
> ```

Nice tool! 
may I ask a question?
what if I have several masked places to predict?

```python
t5('India is a <extra_id_0> of the <extra_id_1>. 
',max_length=5)
```
I got:
[{'generated_text': 'part world'}]

It seems only the first masked place is predicted.

> > Could we use the following workaround?
> > 
> > * `<extra_id_0>` could be considered as a mask token
> > * Candidate sequences for the mask-token could be generated using a code, like:
> > 
> > ```python
> > from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration
> > 
> > T5_PATH = 't5-base' # "t5-small", "t5-base", "t5-large", "t5-3b", "t5-11b"
> > 
> > DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # My envirnment uses CPU
> > 
> > t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)
> > t5_config = T5Config.from_pretrained(T5_PATH)
> > t5_mlm = T5ForConditionalGeneration.from_pretrained(T5_PATH, config=t5_config).to(DEVICE)
> > 
> > # Input text
> > text = 'India is a <extra_id_0> of the world. 
'
> > 
> > encoded = t5_tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')
> > input_ids = encoded['input_ids'].to(DEVICE)
> > 
> > # Generaing 20 sequences with maximum length set to 5
> > outputs = t5_mlm.generate(input_ids=input_ids, 
> >                           num_beams=200, num_return_sequences=20,
> >                           max_length=5)
> > 
> > _0_index = text.index('<extra_id_0>')
> > _result_prefix = text[:_0_index]
> > _result_suffix = text[_0_index+12:]  # 12 is the length of <extra_id_0>
> > 
> > def _filter(output, end_token='<extra_id_1>'):
> >     # The first token is <unk> (inidex at 0) and the second token is <extra_id_0> (indexed at 32099)
> >     _txt = t5_tokenizer.decode(output[2:], skip_special_tokens=False, clean_up_tokenization_spaces=False)
> >     if end_token in _txt:
> >         _end_token_index = _txt.index(end_token)
> >         return _result_prefix + _txt[:_end_token_index] + _result_suffix
> >     else:
> >         return _result_prefix + _txt + _result_suffix
> > 
> > results = list(map(_filter, outputs))
> > results
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Output:
> > ```
> > ['India is a cornerstone of the world. 
',
> >  'India is a part of the world. 
',
> >  'India is a huge part of the world. 
',
> >  'India is a big part of the world. 
',
> >  'India is a beautiful part of the world. 
',
> >  'India is a very important part of the world. 
',
> >  'India is a part of the world. 
',
> >  'India is a unique part of the world. 
',
> >  'India is a part of the world. 
',
> >  'India is a part of the world. 
',
> >  'India is a beautiful country in of the world. 
',
> >  'India is a part of the of the world. 
',
> >  'India is a small part of the world. 
',
> >  'India is a part of the world. 
',
> >  'India is a part of the world. 
',
> >  'India is a country in the of the world. 
',
> >  'India is a large part of the world. 
',
> >  'India is a part of the world. 
',
> >  'India is a significant part of the world. 
',
> >  'India is a part of the world. 
']
> > ```
> 
> Nice tool! may I ask a question? what if I have several masked places to predict?
> 
> ```python
> t5('India is a <extra_id_0> of the <extra_id_1>. 
',max_length=5)
> ```
> 
> I got: [{'generated_text': 'part world'}]
> 
> It seems only the first masked place is predicted.

Seems like <extra_id_0> is filled with "part", and <extra_id_1> is filled with "world"?

Hi @beyondguo,

> may I ask a question?
> what if I have several masked places to predict?
> ```python
> t5('India` is a <extra_id_0> of the <extra_id_1>. 
',max_length=5)
> ```

You need to modify the `_filter` function. 

The (original) function in [my code](https://github.com/huggingface/transformers/issues/3985#issuecomment-622981083) extracts the word sequence between `<extra_id_0>` and `<extra_id_1>` from the output, and the word sequence is being used to replace the `<extra_id_0>` token in the input. In addition to it, you have to extract the word sequence between `<extra_id_1>` and `<extra_id_2>` to replace the `<extra_id_1>` token from the input. Now, as the system needs to generate more tokens, one needs to set a bigger value for the `max_length` argument of `t5_mlm.generate`.

How is it possible to use the workaround with target words like in the pipeline? 
E.g. targets=["target1", "target2"] and get the probabilities definitely for those targets?