XLM model masked word prediction Double Language 
Author: valdrox
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I am trying to generate in-context word translations. 

For instance, if the target language is french and "well" is the word to translate.
- I walked to the well. -> the translation for "well" should be "puit" 
- I am doing well. -> the translation for "well" should be "bien"

I have a simple solution inspired from https://github.com/huggingface/transformers/issues/1842#issuecomment-555734728 and https://github.com/qiang2100/BERT-LS

I am basically concatenating a sentence clone, masking the second target word and changing its language inside the "langs" tensor.

The code is something like this :

```
from transformers import XLMTokenizer, XLMWithLMHeadModel
import torch

# model
model_string = "xlm-mlm-tlm-xnli15-1024"

# load tokenizer
tokenizer = XLMTokenizer.from_pretrained(model_string)

# encode sentence with a masked token in the middle
encoded_array = tokenizer.encode(
    "That is a well. That is a " + tokenizer.mask_token + ".")
sentence = torch.tensor([encoded_array])

# Identify the masked token position
masked_index = torch.where(sentence == tokenizer.mask_token_id)[1].tolist()[0]

# Load model
model = XLMWithLMHeadModel.from_pretrained(model_string)

# Load languages
language_id_from = tokenizer.lang2id['en']  # 0
language_id_to = tokenizer.lang2id['fr']  # 0
languages_array = [language_id_from] * len(encoded_array)
languages_array[masked_index] = language_id_to

langs = torch.tensor(languages_array)
langs = langs.view(1, -1)

# Get the five top answers
result = model(input_ids=sentence, langs=langs)
prediction_scores = result[0]

result = prediction_scores[:, masked_index].topk(20).indices
result = result.tolist()[0]
print(tokenizer.decode(result))

```

I've been getting some positive results, and not so positive results (with longer sentences). For example:

"I walked to the well" : well -> Puit is the 3rd result. [easily identifiable as correct!]
"I felt well." : well -> Bien is the 6th result. [easily identifiable as correct!]
"If the sentence is too convoluted, it won't translate well." -> The results are all parts of correct answers but wrong individually. Like "correct" and "ly" in french.  [very hard to piece together answer]
"After a hard day's work like yesterday, I really like to go jump in the well to cool down." -> all results are in English for some reason.  [no french at all!]
Sentences with the animal meaning of "bat" don't work at all. I don't think it knows a bat is an animal.

My questions. 
- Should I be looking at something else than xlm; is there a better way of doing this already out there, or existing solution? I would rather avoid a cloud service like Yandex translate, as I will want to do this a lot!
- If this can be achieved with xlm, is there a way to force a particular language, and full words?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.