ImportError: cannot import name 'RobertaLMHeadModel'
Author: lighteternal
Pull request: no

Hi all, I was just trying to run a text generation script for low-resource languages, and therefore experimented with XLM-R and initially with Roberta, using the documentation for RobertaForCausalLM:
here: https://huggingface.co/transformers/model_doc/roberta.html#robertaforcausallm

I am running into the import error shown in the title. See code snippet and error message below. I also experimented with different Tensorflow and transformers versions to no avail. I suspect that the model classes have changed (or the documentation may not be up to date with the current version). I also tried importing RobertaForCausalLM but it returned the same error.

## Environment info     
- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-112-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- Tensorflow version: 2.3.1


### Who can help
@LysandreJik , @TevenLeScao
 
Model I am using (**Roberta**, **XLM-Roberta**):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce
Steps to reproduce the behavior:

1. Run:
```
from transformers import RobertaTokenizer, RobertaLMHeadModel, RobertaConfig
import torch
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
config = RobertaConfig.from_pretrained("roberta-base")
config.is_decoder = True
model = RobertaLMHeadModel.from_pretrained('roberta-base', config=config, return_dict=True)
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
prediction_logits = outputs.logits
```

Error: `ImportError: cannot import name 'RobertaLMHeadModel'`

If this script runs succesfully, I 'd like to re-run it for XMLRoberta (changing the imports and model names of course).
Many thanks!

Comments:
Hello, the `RobertaLMHeadModel` is a PyTorch model, you would need to have PyTorch installed to import it.

If you want to use the TensorFlow variant, you should use the `TFRobertaLMHeadModel`

Yeap, sorry for not including it in the Environment info, I have torch 1.6 installed. 
The following script:
```
import transformers 
print(transformers.__version__)
import torch 
print(torch.__version__)
import tensorflow 
print(tensorflow.__version__)

from transformers import RobertaTokenizer, RobertaLMHeadModel, RobertaConfig
```
Returns:
```
3.1.0
1.6.0
2.3.1
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-4-aecd14032a4d> in <module>
      6 print(tensorflow.__version__)
      7 
----> 8 from transformers import RobertaTokenizer, RobertaLMHeadModel, RobertaConfig

ImportError: cannot import name 'RobertaLMHeadModel'
```

The tflow variant returns the same error.

My bad, I read too fast! The error is probably because you're trying to import `RobertaLMHeadModel`, but as it can be seen in your first post, the model is actually `RobertaForCausalLM`. Can you successfully load that model? We plan on having uniform naming for these models so that the `CausalLM` and `LMHeadModel` have the same naming soon.

Many thanks for the quick replies! :)

Yes, it can be loaded  that way. However,  after running the following script: 

```
import torch
from transformers import RobertaTokenizer, RobertaForCausalLM, RobertaConfig
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
config = RobertaConfig.from_pretrained("roberta-base")
config.is_decoder = True
model = RobertaForCausalLM.from_pretrained('roberta-base', config=config, return_dict=True)
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
prediction_logits = outputs.logits
```

The following error appears: 
```
TypeError                                 Traceback (most recent call last)
<ipython-input-99-f1066c26064d> in <module>
      4 config = RobertaConfig.from_pretrained("roberta-base")
      5 config.is_decoder = True
----> 6 model = RobertaForCausalLM.from_pretrained('roberta-base', config=config, return_dict=True)
      7 inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
      8 outputs = model(**inputs)

~/translation/DimPapSandbox/greek_text_generation/tsflow23/lib/python3.6/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    921 
    922         # Instantiate model.
--> 923         model = cls(config, *model_args, **model_kwargs)
    924 
    925         if state_dict is None and not from_tf:

TypeError: __init__() got an unexpected keyword argument 'return_dict'
```

If I remove the `return_dict` argument, another error comes up:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-100-44c62bef9ec6> in <module>
      7 inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
      8 outputs = model(**inputs)
----> 9 prediction_logits = outputs.logits

AttributeError: 'tuple' object has no attribute 'logits' 
```

If there's a working snippet using the Roberta or XLM-Roberta for text generation, it would be much appreciated. 

Indeed, there's an issue with the docstrings here. I'm fixing it in #7642.

Have you taken a look at the summary of text generation [here](https://huggingface.co/transformers/task_summary.html#text-generation)?

Please note that RoBERTa has not been trained to do text generation, but to do mask in-filling, so using a pre-trained RoBERTa model to do generation would yield bad results.

Τhanks, I was actually interested in XLM-R (looking for low-resource language text generation) but I stumbled upon the RoBERTa example shown above first so I thought I could just swap the model and it would work. I can confirm that the following code works with xlm-roberta-large: 

```
from transformers import AutoModelWithLMHead, AutoTokenizer
model = AutoModelWithLMHead.from_pretrained("xlm-roberta-large")
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
prompt = "Σήμερα ο καιρός"  # means: "Today the weather", in Greek"
inputs = tokenizer.encode(PADDING_TEXT + prompt, add_special_tokens=False, return_tensors="pt")
prompt_length = len(tokenizer.decode(inputs[0], skip_special_tokens=True, clean_up_tokenization_spaces=True))
outputs = model.generate(inputs, max_length=250, do_sample=True, top_p=0.95, top_k=60)
generated = prompt + tokenizer.decode(outputs[0])[prompt_length:]
print(generated)
```
However the generated output is not really useful (repeating the word "weather"): 

`Σήμερα ο καιρός ιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός καιρός`

Ι understood that XLM-R had a CLM checkpoint but maybe I was wrong. In any case, if you are aware of any pretrained models that could I could try for text prediction (I am interested in Greek, where GPT-2 does not really shine) it would be great. Otherwise, we can close this issue. :)

@lighteternal Have you checked whether this model is any good for your use case? https://huggingface.co/nikokons/gpt2-greek?text=%CE%A3%CE%AE%CE%BC%CE%B5%CF%81%CE%B1+%CE%BF+%CE%BA%CE%B1%CE%B9%CF%81%CF%8C%CF%82

model author is @nikkon3

Hi @julien-c, yes I have tried it in the past. It is notably better compared to the vanilla GPT-2 in most cases (the latter "has" Greek tokens in its vocabulary, but the relative corpus that was used must have been extremely small for any useful inference).  However even the `nikokons/gpt-2-greek` is sometimes generating sentences that, while syntactically OK, are not relevant to the input context. Probably a larger and more diverse training corpus would help. 

I have been experimenting a while with this, and my conclusion is that for now the most "robust" generations for Greek are made by masked LM which are repurposed to causal ones, e.g. If I use a BERT-like model, I put the mask at the end of the unfinished sentence:
`"This is a great <mask> ..."` Of course this comes with the problem that I have to reuse the mask's result to feed it as input in case I want more than one token to be predicted. Autoregressive models either return non-sense or drift away from the input really quickly. 

yes @lighteternal the dataset that is used for gpt2-greek is not large . It is trained on about 5 GB of text with the main source to be from Greek Wikipedia .