RoBERTa support
Author: sleepinyourhat
Pull request: no

https://twitter.com/sleepinyourhat/status/1151940994688016384

The code/parameters aren't out yet, but I figure it couldn't hurt to put in an obnoxious feature request now!

Comments:
Working on the code/paper release as we speak :) It largely follows the existing masked_lm implementation in fairseq. Happy to help get this integrated here.

Hi @myleott great news :) I'm really excited about the release 🤗 I've some questions: do you plan to perform any comparisons between RoBERTa and BERT on NER (CoNLL-2003)? 

I've read the [Cloze-driven Pretraining of Self-attention Networks](https://arxiv.org/abs/1903.07785) paper, and if I recall correctly, the implementation is currently done in the `bi_trans_lm` branch in `fairseq`, but do you have any updates on that? It would be awesome if a pre-trained CNN model from that paper could also be integrated into `pytorch-transformers` 😍

Sounds great @myleott. Keep us updated about the release!

Models and README are uploaded: https://github.com/pytorch/fairseq/tree/master/examples/roberta. We submitted the paper to arXiv today so it should be out Sunday evening.

> I've some questions: do you plan to perform any comparisons between RoBERTa and BERT on NER (CoNLL-2003)?

We haven't yet, but it would be interesting to explore. RoBERTa was trained on considerably more data than BERT, so I expect it would do well on NER tasks.

Paper is [out](https://arxiv.org/abs/1907.11692). Thanks @myleott!

Work in progress in #964 feel free to chime in :)

Example of how RoBERTa can be used to predict a masked token.

import torch
from pytorch_transformers import RobertaTokenizer, RobertaForMaskedLM

tokenizer = RobertaTokenizer.from_pretrained('roberta-large')
model = RobertaForMaskedLM.from_pretrained('roberta-large')
model.eval()

if torch.cuda.is_available(): model.to('cuda') #if we have a GPU
text = 'I believe my sister is <mask<m>> because she eats a lot of vegetables .'

tokenized_text = tokenizer.tokenize(text)
masked_index  = tokenized_text.index(<mask<m>>)+1

#add_special_tokens adds a <s
> to the beginning and </s
> to the end of the text
input_ids = torch.tensor(tokenizer.encode(text,add_special_tokens=True)).unsqueeze(0)  
input_ids_tensor = input_ids.to("cuda")

#with torch.no_grad():
outputs = model(input_ids_tensor, masked_lm_labels=input_ids_tensor)
loss, prediction_scores = outputs[:2]

#predicted_index = torch.argmax(prediction_scores[0, masked_index]).item()
#predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]

predicted_k_indexes = torch.topk(prediction_scores[0, masked_index],k=20)
predicted_logits_list = predicted_k_indexes[0] 
predicted_indexes_list = predicted_k_indexes[1]

for i, item in enumerate(predicted_indexes_list):
    the_index = predicted_indexes_list[i].item()
    print("word and logits",tokenizer.decode(the_index),predicted_logits_list[i].item())


Hi @pwolff, at first glance it looks ok to me. You don't need to send the `masked_lm_labels` if you don't use the loss though.

@thomwolf hello, I trained the robert on my customized corpus following the fairseq instruction. I am confused how to generate the robert vocab.json and also merge.txt because I want to use the pytorch-transformer RoBERTaTokenizer.

@stefan-it  hello, I trained the robert on my customized corpus following the fairseq instruction. I am confused how to generate the robert vocab.json and also merge.txt because I want to use the pytorch-transformer RoBERTaTokenizer.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@songtaoshi I think this can be done via `subword-nmt`, see this note:

https://github.com/pytorch/fairseq/issues/1163#issuecomment-534098220



is this still an issue?

Nope, RoBERTa support was shipped in [v1.1.0](https://github.com/huggingface/transformers/releases/tag/1.1.0)

Thanks all!