How to predict missing word [MASK] using Robert
Author: Oxi84
Pull request: no

I am reading the docs and I still cannot figure out how to I predict missing word in a sentence using Robert. With bert this is described at https://huggingface.co/pytorch-transformers/quickstart.html

    # Mask a token that we will try to predict back with `BertForMaskedLM`
    masked_index = 8
    tokenized_text[masked_index] = '[MASK]'
    assert tokenized_text == ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a',     'puppet', '##eer', '[SEP]']

Robert example is something I do not understand. What is the output of that? 

    import torch
    from pytorch_transformers import RobertaTokenizer, RobertaForMaskedLM

    tokenizer = RobertaTokenizer.from_pretrained('roberta- 
    base',cache_dir="/var/software/Models/robert/")
    model = RobertaForMaskedLM.from_pretrained('roberta- 
    base',cache_dir="/var/software/Models/robert/")
    input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
    outputs = model(input_ids, masked_lm_labels=input_ids)
    loss, prediction_scores = outputs[:2]
    print("prediction_scores",prediction_scores,len(prediction_scores))

     output:
    A sequence with no special tokens has been passed to the RoBERTa model. This model requires 
    special tokens in order to work. Please specify add_special_tokens=True in your encoding.
    prediction_scores tensor([[[33.6519, -3.9080, 24.2591,  ...,  2.8165,  4.9966, 12.8938],
         [ 5.8086, -4.2237, 16.1383,  ..., -1.0431, -0.8348,  3.5343],
         [ 0.3336, -4.1881, 10.7825,  ...,  0.7295,  0.9056,  3.7928],
         [ 0.2897, -4.4614,  8.1219,  ..., -3.9978,  0.1261, -1.4313],
         [ 3.3684, -4.0727, 10.7862,  ...,  1.7704, -2.2975,  3.9174],
         [ 2.0526, -4.9519, 18.1501,  ..., -4.2190, -5.0759,  1.4990]]],
       grad_fn=<AddBackward0>) 1






Comments:
Basically, the problem is that the model is called masked language model, but it does not mask anything.

If i want to get token distribution for word "dog", but the model sees word dog, because its not masked, so it use the word in prediction.  Input should not be "Hello, my dog is cute", but something like that "Hello, my [MASK] is cute".

How do I do this?

Maybe there is another way to specify that the word is masked by index or something?


When I put:

     input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)
     top_predicted_words:  dog puppy pet pup Dog dogs guy husband kid housedog brother girl friend job cat post

    input_ids = torch.tensor(tokenizer.encode("Hello, my wife is cute")).unsqueeze(0)
    top_predicted_words:  wife spouse marriage Wife bride husband head house family wives culture throat




Ok, i got it, I should use <mask> instead of [MASK] and <pad> instead of [PAD].

I find robera-base to be around 7% faster than bert-base, but a bit less precise.

> Ok, i got it, I should use instead of [MASK] and instead of [PAD].
> 
> I find robera-base to be around 7% faster than bert-base, but a bit less precise.

@Oxi84 Hi, do you mean should use [PAD] instead of [MASK]? I am having the same problem here. Could you plz share the whole code snippet? Thanks very much.