How to get masked word prediction probabilities
Author: Oxi84
Pull request: no
Labels: name: wontfix

Original sentence: i love apples. there are a lot of fruits in the world that i like, but apples would be my favorite fruit.
Masked sentence: i love apples . there are a lot of fruits in the world that i [MASK] , but apples would be my favorite fruit .

When I run through the pytorch version of bert, I get the following representations of probabilities:

Best predicted word: ['love'] tensor(12.7276, grad_fn=)
Other words along with their probabilities:
['like'] tensor(10.2872, grad_fn=)
['miss'] tensor(8.8226, grad_fn=)
['know'] tensor(8.5971, grad_fn=)
['am'] tensor(7.9407, grad_fn=)
['hate'] tensor(7.9209, grad_fn=)
['mean'] tensor(7.8873, grad_fn=)
['enjoy'] tensor(7.8813, grad_fn=)
['want'] tensor(7.6885, grad_fn=)
['prefer'] tensor(7.5712, grad_fn=)

I am quite sure that this does not mean that probability for word "love" is proportional to 12.7276 and for word "like" is 10.2872.
I also know that the summ of all func(this number) thought the whole vocabulary is 1. But I do not know what the func is?

Thanks

Comments:
I'm interested in an answer, too. A score/probability would help to select the best word for a masked token.

You are looking for the softmax function: https://pytorch.org/docs/stable/nn.html?highlight=softmax#torch.nn.functional.softmax

Thanks Thomas, I'll give it a try.

Thanks,
So you say that for score x1 (where all the scores are x1,x2,.. xn) :
probability_x1 = (exp(^x1)/(exp(^x1) + exp(^x2)  +  .. exp(^xn))



@Oxi84 could you share how you obtained the masked word probabilities? I have been trying to do that on my custom data. That is, I want to pretrain my own model and then do masked word prediction on new data.

@rvoak The quickstart guide [here](https://github.com/huggingface/pytorch-transformers/blob/master/docs/source/quickstart.md#bert-example) shows a nice example of how to do masked word prediction. 
Replace 
```
# confirm we were able to predict 'henson'
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
```
with something like this (e.g. if you want the top k predicted tokens):

```
top_k = 10
probs = torch.nn.functional.softmax(predictions[0, mask_idx], dim=-1)
top_k_weights, top_k_indices = torch.topk(probs, top_k, sorted=True)

for i, pred_idx in enumerate(top_k_indicies):
    predicted_token = tokenizer.convert_ids_to_tokens([pred_idx])[0]
    token_weight = top_k_weights[i]
```



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> @rvoak The quickstart guide [here](https://github.com/huggingface/pytorch-transformers/blob/master/docs/source/quickstart.md#bert-example) shows a nice example of how to do masked word prediction.
> Replace
> 
> ```
> # confirm we were able to predict 'henson'
> predicted_index = torch.argmax(predictions[0, masked_index]).item()
> predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
> ```
> 
> with something like this (e.g. if you want the top k predicted tokens):
> 
> ```
> top_k = 10
> probs = torch.nn.functional.softmax(predictions[0, mask_idx], dim=-1)
> top_k_weights, top_k_indices = torch.topk(probs, top_k, sorted=True)
> 
> for i, pred_idx in enumerate(top_k_indicies):
>     predicted_token = tokenizer.convert_ids_to_tokens([pred_idx])[0]
>     token_weight = top_k_weights[i]
> ```

very good example thanks! is there a version for RoBERTa and other models?

Hi @yuchenlin, you can use the recently added `fill-mask` pipeline to do so:

```py
>>> from transformers import pipeline
>>> nlp = pipeline("fill-mask", model="roberta-base")
>>> nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")
[
    {'sequence': "
 This is the best thing I've done in my life.
", 'score': 0.8024354577064514, 'token': 626}, 
    {'sequence': "
 This is the best thing I've heard in my life.
", 'score': 0.031355079263448715, 'token': 1317}, 
    {'sequence': "
 This is the best thing I've learned in my life.
", 'score': 0.027319395914673805, 'token': 2435}, 
    {'sequence': "
 This is the best thing I've seen in my life.
", 'score': 0.026892054826021194, 'token': 450}, 
    {'sequence': "
 This is the best thing I've experienced in my life.
", 'score': 0.02160099521279335, 'token': 2984}
]
```

We're in the process of adding example usage for common tasks (question answering, sequence classification, mask filling etc), you can follow the progress in https://github.com/huggingface/transformers/pull/2850. There already is an example for mask filling.

Hey @LysandreJik, does the fill-mask also support whole word mask prediction, or does it only work on subword level?

> Hi @yuchenlin, you can use the recently added `fill-mask` pipeline to do so:
> 
> ```python
> >>> from transformers import pipeline
> >>> nlp = pipeline("fill-mask", model="roberta-base")
> >>> nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")
> [
>     {'sequence': "
 This is the best thing I've done in my life.
", 'score': 0.8024354577064514, 'token': 626}, 
>     {'sequence': "
 This is the best thing I've heard in my life.
", 'score': 0.031355079263448715, 'token': 1317}, 
>     {'sequence': "
 This is the best thing I've learned in my life.
", 'score': 0.027319395914673805, 'token': 2435}, 
>     {'sequence': "
 This is the best thing I've seen in my life.
", 'score': 0.026892054826021194, 'token': 450}, 
>     {'sequence': "
 This is the best thing I've experienced in my life.
", 'score': 0.02160099521279335, 'token': 2984}
> ]
> ```
> 
> We're in the process of adding example usage for common tasks (question answering, sequence classification, mask filling etc), you can follow the progress in #2850. There already is an example for mask filling.

Is it possible to give this an input word for the mask and get probabilities back for that specific word?

Also is it possible to request the top N sentences rather than the default returned?

Edit: Never mind on this specific question! I found out by setting:

`nlp.topk = 20`

before doing:

` nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")`

It now returns 20.

Sure, you can do that using the recently added `targets` (in `v3.1.0`):

```py
>>> from transformers import pipeline
>>> nlp = pipeline("fill-mask", model="roberta-base")
>>> nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.", targets=[' experienced'])
[
    {
        'sequence': "
This is the best thing I've experienced in my life.
", 
        'score': 0.022622672840952873, 
        'token': 2984, 
        'token_str': 'Ġexperienced'
    }
]
```

Please note the space before the word, because we're using the [RoBERTa tokenizer](https://huggingface.co/transformers/model_doc/roberta.html#robertatokenizer) which is a Byte-level BPE tokenizer that has a different behaviour according to the spaces before tokens.

@LysandreJik So very helpful! Thank you so much!

@LysandreJik If a word is at the start of a sentence, should it also have a space in front of it?:

```
nlp(f"{nlp.tokenizer.mask_token} talk about the rules of the game first.", targets=[' We\'ll'])
```
Which gives me:
```
The specified target token ` We'll` does not exist in the model vocabulary. Replacing with `ĠWe`.
[{'sequence': '
 We talk about the rules of the game first
', 'score': 8.493712812196463e-06, 'token': 166, 'token_str': 'ĠWe'}]
```

Or

```
nlp(f"{nlp.tokenizer.mask_token} talk about the rules of the game first.", targets=['We\'ll'])
```
Which gives me:
```
The specified target token `We'll` does not exist in the model vocabulary. Replacing with `We`.
[{'sequence': '
We talk about the rules of the game first
', 'score': 0.12082401663064957, 'token': 170, 'token_str': 'We'}]
```

How to predict a word that is seperated into several tokens. For example, DOTA2(name for a popular game)?