Repository: huggingface/transformers

Issue:
ImportError: cannot import name 'RobertaLMHeadModel'
Hi all, I was just trying to run a text generation script for low-resource languages, and therefore experimented with XLM-R and initially with Roberta, using the documentation for RobertaForCausalLM:
here: https://huggingface.co/transformers/model_doc/roberta.html#robertaforcausallm

I am running into the import error shown in the title. See code snippet and error message below. I also experimented with different Tensorflow and transformers versions to no avail. I suspect that the model classes have changed (or the documentation may not be up to date with the current version). I also tried importing RobertaForCausalLM but it returned the same error.

## Environment info     
- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-112-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- Tensorflow version: 2.3.1


### Who can help
@LysandreJik , @TevenLeScao
 
Model I am using (**Roberta**, **XLM-Roberta**):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce
Steps to reproduce the behavior:

1. Run:
```
from transformers import RobertaTokenizer, RobertaLMHeadModel, RobertaConfig
import torch
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
config = RobertaConfig.from_pretrained("roberta-base")
config.is_decoder = True
model = RobertaLMHeadModel.from_pretrained('roberta-base', config=config, return_dict=True)
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
prediction_logits = outputs.logits
```

Error: `ImportError: cannot import name 'RobertaLMHeadModel'`

If this script runs succesfully, I 'd like to re-run it for XMLRoberta (changing the imports and model names of course).
Many thanks!


Discussion:
Hello, the `RobertaLMHeadModel` is a PyTorch model, you would need to have PyTorch installed to import it.

If you want to use the TensorFlow variant, you should use the `TFRobertaLMHeadModel`
Yeap, sorry for not including it in the Environment info, I have torch 1.6 installed. 
The following script:
```
import transformers 
print(transformers.__version__)
import torch 
print(torch.__version__)
import tensorflow 
print(tensorflow.__version__)

from transformers import RobertaTokenizer, RobertaLMHeadModel, RobertaConfig
```
Returns:
```
3.1.0
1.6.0
2.3.1
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-4-aecd14032a4d> in <module>
      6 print(tensorflow.__version__)
      7 
----> 8 from transformers import RobertaTokenizer, RobertaLMHeadModel, RobertaConfig

ImportError: cannot import name 'RobertaLMHeadModel'
```

The tflow variant returns the same error.
My bad, I read too fast! The error is probably because you're trying to import `RobertaLMHeadModel`, but as it can be seen in your first post, the model is actually `RobertaForCausalLM`. Can you successfully load that model? We plan on having uniform naming for these models so that the `CausalLM` and `LMHeadModel` have the same naming soon.
Many thanks for the quick replies! :)

Yes, it can be loaded  that way. However,  after running the following script: 

```
import torch
from transformers import RobertaTokenizer, RobertaForCausalLM, RobertaConfig
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
config = RobertaConfig.from_pretrained("roberta-base")
config.is_decoder = True
model = RobertaForCausalLM.from_pretrained('roberta-base', config=config, return_dict=True)
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
prediction_logits = outputs.logits
```

The following error appears: 
```
TypeError                                 Traceback (most recent call last)
<ipython-input-99-f1066c26064d> in <module>
      4 config = RobertaConfig.from_pretrained("roberta-base")
      5 config.is_decoder = True
----> 6 model = RobertaForCausalLM.from_pretrained('roberta-base', config=config, return_dict=True)
      7 inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
      8 outputs = model(**inputs)

~/translation/DimPapSandbox/greek_text_generation/tsflow23/lib/python3.6/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    921 
    922         # Instantiate model.
--> 923         model = cls(config, *model_args, **model_kwargs)
    924 
    925         if state_dict is None and not from_tf:

TypeError: __init__() got an unexpected keyword argument 'return_dict'
```

If I remove the `return_dict` argument, another error comes up:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-100-44c62bef9ec6> in <module>
      7 inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
      8 outputs = model(**inputs)
----> 9 prediction_logits = outputs.logits

AttributeError: 'tuple' object has no attribute 'logits' 
```

If there's a working snippet using the Roberta or XLM-Roberta for text generation, it would be much appreciated. 

Resolving patch:
diff --git a/src/transformers/modeling_roberta.py b/src/transformers/modeling_roberta.py
--- a/src/transformers/modeling_roberta.py
+++ b/src/transformers/modeling_roberta.py
@@ -758,13 +758,13 @@ def forward(
 
         Example::
 
-            >>> from transformers import RobertaTokenizer, RobertaLMHeadModel, RobertaConfig
+            >>> from transformers import RobertaTokenizer, RobertaForCausalLM, RobertaConfig
             >>> import torch
 
             >>> tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
-            >>> config = RobertaConfig.from_pretrained("roberta-base")
+            >>> config = RobertaConfig.from_pretrained("roberta-base", return_dict=True)
             >>> config.is_decoder = True
-            >>> model = RobertaLMHeadModel.from_pretrained('roberta-base', config=config, return_dict=True)
+            >>> model = RobertaForCausalLM.from_pretrained('roberta-base', config=config)
 
             >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
             >>> outputs = model(**inputs)


Test patch: