How can I apply BERT to a cloze task?
Author: Deep1994
Pull request: no

Hi, I have a dataset like : 

From Monday to Friday most people are busy working or studying, but in the evenings and weekends they are free and  _  themselves.

And there are four candidates for the missing blank area:

 ["love", "work", "enjoy", "play"], here "enjoy" is the correct answer, it is a cloze-style task, and it looks like the maskLM in the BERT,  the difference is that I don't want to search the candidate from all the tokens  but the four given candidates, how can I do this? It looks like negtive sampling method. Do you have any idea? Thank you! 

Comments:
I think that you best option would be to use the masked language modeling head and restrict the output of the softmax layer to your candidates.

I think the following code does the job:

```
import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = 'From Monday to Friday most people are busy working or studying, '\
       'but in the evenings and weekends they are free and _ themselves.'
tokenized_text = tokenizer.tokenize(text)

masked_index = tokenized_text.index('_')
tokenized_text[masked_index] = '[MASK]'

candidates = ['love', 'work', 'enjoy', 'play']
candidates_ids = tokenizer.convert_tokens_to_ids(candidates)

indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)

segments_ids = [0] * len(tokenized_text)

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

language_model = BertForMaskedLM.from_pretrained('bert-base-uncased')
language_model.eval()

predictions = language_model(tokens_tensor, segments_tensors)
predictions_candidates = predictions[0, masked_index, candidates_ids]
answer_idx = torch.argmax(predictions_candidates).item()

print(f'The most likely word is "{candidates[answer_idx]}".')
```

When run, this code prints: 

```
The most likely word is "enjoy".
```

The solution of @rodgzilla looks good. Don't hesitate to re-open the issue if you have other questions.

Just a note that this solution does not help you if any of your candidates are out of your model's whole-word vocabulary. (A work-around is required to deal with BERT's reliance on word-piece tokens.)

> > I think that you best option would be to use the masked language modeling head and restrict the output of the softmax layer to your candidates.
> > I think the following code does the job:
> > ```
> > import torch
> > from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM
> > 
> > tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
> > text = 'From Monday to Friday most people are busy working or studying, '\
> >        'but in the evenings and weekends they are free and _ themselves.'
> > tokenized_text = tokenizer.tokenize(text)
> > 
> > masked_index = tokenized_text.index('_')
> > tokenized_text[masked_index] = '[MASK]'
> > 
> > candidates = ['love', 'work', 'enjoy', 'play']
> > candidates_ids = tokenizer.convert_tokens_to_ids(candidates)
> > 
> > indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
> > 
> > segments_ids = [0] * len(tokenized_text)
> > 
> > tokens_tensor = torch.tensor([indexed_tokens])
> > segments_tensors = torch.tensor([segments_ids])
> > 
> > language_model = BertForMaskedLM.from_pretrained('bert-base-uncased')
> > language_model.eval()
> > 
> > predictions = language_model(tokens_tensor, segments_tensors)
> > predictions_candidates = predictions[0, masked_index, candidates_ids]
> > answer_idx = torch.argmax(predictions_candidates).item()
> > 
> > print(f'The most likely word is "{candidates[answer_idx]}".')
> > ```
> > 
> > 
> > When run, this code prints:
> > ```
> > The most likely word is "enjoy".
> > ```
> 
> 
Thanks, you solution is good