How to predict masked whole word which was tokenized as sub-words for bert-base-multilingual-cased
Author: ksopyla
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Hello,
I have started working with pytorch-transformers and want to use it to predict masked words in polish. I use ' bert-base-multilingual-cased' pre-trained model and want to predict masked words which very often are tokenized into sub-word.
My question is how can I predict the whole word?
When I predict each token separately the results are poor. especially when I try to concatenate those predicted tokens

Here is sample code showing the problem

```python
import torch
from pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM
import logging
logging.basicConfig(level=logging.INFO)

USE_GPU = 1
# Device configuration
device = torch.device('cuda' if (torch.cuda.is_available() and USE_GPU) else 'cpu')

# Load pre-trained model tokenizer (vocabulary)
pretrained_model = 'bert-base-multilingual-cased'
tokenizer = BertTokenizer.from_pretrained(pretrained_model)
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
mask1 = 13
mask2 = 14
mask3 = 15
tokenized_text[mask1] = '[MASK]'
tokenized_text[mask2] = '[MASK]'
tokenized_text[mask3] = '[MASK]'
assert tokenized_text == ['[CLS]', 'Who', 'was', 'Jim', 'Hen', '##son', '?', '[SEP]', 'Jim', 'Hen', '##son', 'was', 'a', '[MASK]', '[MASK]', '[MASK]', '[SEP]']

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)
segments_ids = [0, 0, 0, 0, 0, 0, 0,0, 1, 1, 1, 1, 1, 1, 1,1,1]

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])


# Load pre-trained model (weights)
model = BertForMaskedLM.from_pretrained(pretrained_model)
model.eval()

# If you have a GPU, put everything on cuda
tokens_tensor = tokens_tensor.to(device)
segments_tensors = segments_tensors.to(device)
model.to(device)

# Predict all tokens
with torch.no_grad():
    outputs = model(tokens_tensor, token_type_ids=segments_tensors)
    predictions = outputs[0]

# get predicted tokens

#prediction for mask1
predicted_index = torch.argmax(predictions[0, mask1]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token) #returns "baseball"


#prediction for mask2
predicted_index = torch.argmax(predictions[0, mask2]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token) #returns "actor"


#prediction for mask3
predicted_index = torch.argmax(predictions[0, mask3]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token) # returns "."
```




Comments:
Hi I don't have any good solution for your use-case, unfortunately.

There are two "Whole-Word_masking" models for Bert (see the [list here](https://huggingface.co/pytorch-transformers/pretrained_models.html)) that would be better at guessing full words but they are only in English unfortunately.

SpanBert (whose open-sourcing we are still waiting) may also be better but I think they also only trained an English model...

@thomwolf thanks for the reply. 
can you specify what exactly the problem lies in? I know the model is not capable of properly tokenizing polish. 

Assuming I have pre-trained model for Polish, or just working with English text how can I predict a sequence of masked two or three tokens side by side

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


if i want to fine-tune with my dataset,what should i do?

> if i want to fine-tune with my dataset,what should i do?

Hi tom1125, to fine-tune you can run this script:

export TRAIN_FILE=/path/to/dataset/wiki.train.raw
export TEST_FILE=/path/to/dataset/wiki.test.raw

python run_language_modeling.py \
    --output_dir=output \
    --model_type=roberta \
    --model_name_or_path=roberta-base \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm

Explained here: https://huggingface.co/transformers/examples.html#roberta-bert-and-masked-language-modeling