Sized Fill-in-the-blank or Multi Mask filling with T5
Author: ramsrigouthamg
Pull request: no
Labels: name: wontfix

In the Google T5 paper they mentioned :
 For example, with the input, “I love peanut butter and _4_ sandwiches,” the outputs looked like:
I love peanut butter and jelly, which is what makes good sandwiches.

How do I achieve Multi Mask filling with T5 in hugging face transformers?
Code samples please :) 
![image](https://user-images.githubusercontent.com/1754080/80286349-98d96300-8748-11ea-9e68-573f4f3a8e61.png)


Comments:
Did you discover any way of doing this?

@edmar Nothing solid yet! I Will update if I do.

Also looking for a solution
https://github.com/google-research/text-to-text-transfer-transformer/issues/133

@franz101 @edmar 
The closest thing I could come up with : 

```
from transformers import RobertaTokenizer, RobertaForMaskedLM
import torch

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaForMaskedLM.from_pretrained('roberta-base')

sentence = "Tom has fully <mask> <mask> <mask> illness."


token_ids = tokenizer.encode(sentence, return_tensors='pt')
# print(token_ids)
token_ids_tk = tokenizer.tokenize(sentence, return_tensors='pt')
print(token_ids_tk)


masked_position = (token_ids.squeeze() == tokenizer.mask_token_id).nonzero()
masked_pos = [mask.item() for mask in masked_position ]
print (masked_pos)


with torch.no_grad():
    output = model(token_ids)

last_hidden_state = output[0].squeeze()

print ("\n\n")
print ("sentence : ",sentence)
print ("\n")
list_of_list =[]
for mask_index in masked_pos:
    mask_hidden_state = last_hidden_state[mask_index]
    idx = torch.topk(mask_hidden_state, k=5, dim=0)[1]
    words = [tokenizer.decode(i.item()).strip() for i in idx]
    list_of_list.append(words)
    print (words)
    
best_guess = ""
for j in list_of_list:
    best_guess = best_guess+" "+j[0]

print ("\nBest guess for fill in the blank :::",best_guess)
```

The output is : 
['Tom', 'Ġhas', 'Ġfully', '<mask>', '<mask>', '<mask>', 'Ġillness', '.']
[4, 5, 6]

sentence :  Tom has fully <mask> <mask> <mask> illness.

['recovered', 'returned', 'recover', 'healed', 'cleared']
['from', 'his', 'with', 'to', 'the']
['his', 'the', 'her', 'mental', 'this']

Best guess for fill in the blank :::  recovered from his

@ramsrigouthamg @edmar @franz101 Any update on how to do that ?

@Diego999 The above-provided answer is the best I have. 
Google hasn't released the pretrained multi-mask fill model.

@ramsrigouthamg Thanks! This is also similar to BART architecture where they mask a span of text. A similar thread is available here https://github.com/huggingface/transformers/issues/4984 

@Diego999 There are few things that you can possibly explore. There is Spanbert https://huggingface.co/SpanBERT/spanbert-base-cased but I didn't explore on how to use it.
Then there is Google pegasus that is trained with sentences as masks. https://ai.googleblog.com/2020/06/pegasus-state-of-art-model-for.html

This example might be useful:
https://github.com/huggingface/transformers/issues/3985

@ramsrigouthamg Any luck with it yet on T5?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I think the `__4__` is a special token, and they use the same text to text framework (seq2seq, instead of masked-lm) to train this task. 
And that's why the screenshot above says:
> train it to fill in the blank with **approximately** 4 words

is there any updates or simple example code for doing this? thanks!

```
from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration

T5_PATH = 't5-base' # "t5-small", "t5-base", "t5-large", "t5-3b", "t5-11b"

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # My envirnment uses CPU

t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)
t5_config = T5Config.from_pretrained(T5_PATH)
t5_mlm = T5ForConditionalGeneration.from_pretrained(T5_PATH, config=t5_config).to(DEVICE)

# Input text
text = 'India is a <extra_id_0> of the world. 
'

encoded = t5_tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')
input_ids = encoded['input_ids'].to(DEVICE)

# Generaing 20 sequences with maximum length set to 5
outputs = t5_mlm.generate(input_ids=input_ids, 
                          num_beams=200, num_return_sequences=20,
                          max_length=5)

_0_index = text.index('<extra_id_0>')
_result_prefix = text[:_0_index]
_result_suffix = text[_0_index+12:]  # 12 is the length of <extra_id_0>

def _filter(output, end_token='<extra_id_1>'):
    # The first token is <unk> (inidex at 0) and the second token is <extra_id_0> (indexed at 32099)
    _txt = t5_tokenizer.decode(output[2:], skip_special_tokens=False, clean_up_tokenization_spaces=False)
    if end_token in _txt:
        _end_token_index = _txt.index(end_token)
        return _result_prefix + _txt[:_end_token_index] + _result_suffix
    else:
        return _result_prefix + _txt + _result_suffix

results = list(map(_filter, outputs))
results
```

does the above work?