Repository: huggingface/transformers

Issue:
convert_roberta: AttributeError when converting CamemBERT model.pt to pytorch_model.bin
Hi,

I trained a CamemBERT model with the fairseq library which gave me the following files:
- dict.txt: vocabulary coming from the sentencepiece model
- sentencepiece.bpe.model
- model.pt

Now I am trying to convert the model.pt into pytorch_model.bin and config.json as mentionned here ([fairseq/issues#1514](https://github.com/pytorch/fairseq/issues/1514)) and here ([transformers/issue#1850](https://github.com/huggingface/transformers/issues/1850)), by using the conversion script of the transformers library ([transfomers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py)). The goal is to use those files with fast-bert.

However, using this command line:
```shell
python convert_roberta_original_pytorch_checkpoint_to_pytorch.py --roberta_checkpoint_path ./ --pytorch_dump_folder_path ./ --classification_head
```

I get the following error:

```python
AttributeError                            Traceback (most recent call last)
<ipython-input-27-ea791887ff26> in <module>
----> 1 convert_roberta_original_pytorch_checkpoint_to_pytorch.convert_roberta_checkpoint_to_pytorch(CAMEMBERT_PATH, CAMEMBERT_PATH, True)

~/anaconda3/envs/NLP/lib/python3.7/site-packages/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py in convert_roberta_checkpoint_to_pytorch(roberta_checkpoint_path, pytorch_dump_folder_path, classification_head)
     48     roberta = FairseqRobertaModel.from_pretrained(roberta_checkpoint_path)
     49     roberta.eval()  # disable dropout
---> 50     roberta_sent_encoder = roberta.model.decoder.sentence_encoder
     51     config = RobertaConfig(
     52         vocab_size=roberta_sent_encoder.embed_tokens.num_embeddings,

~/anaconda3/envs/NLP/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    592                 return modules[name]
    593         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 594             type(self).__name__, name))
    595 
    596     def __setattr__(self, name, value):

AttributeError: 'RobertaModel' object has no attribute 'decoder'
```

And indeed when I check the fairseq/pytorch RobertaModel has no decoder attribute. 

Am I doing this wrong ? I see no other conversion script to fit my CamemBERT model so I guess the RoBERTa one is the good one.



Discussion:
maybe @sshleifer has an idea
I can help! 
On fairseq master (as of 5/28/20), that class seems to no longer have a `decoder` attribute.

I think you want to change the `roberta.model.decoder` references to `self.model.encoder`, but hard to know without seeing the `state_dict`/handling the model interactively.

The best way to debug is to either instantiate the fairseq model in jupyter/ipython or set a breakpoint and see what the attributes are.

If you are stuck, feel free to upload your `model.pt` to some cloud storage and I can give it a shot!


Hey @sshleifer!

I did what you suggested and it worked, thanks a lot. You have to replace all the references to `roberta.model.decoder` with `roberta.model.encoder` as the attributes were just renamed.

On the other hand, I can't figure out what happened to `roberta.args.num_classes` that is used for the classification_head flag, which makes it useless for now.

I would gladly commit the fix but I'm not a powergit user, so I'll leave it to the pros.

Thanks again!

***

Edit : the error that comes up with the flag. 

```python
Traceback (most recent call last):
  File "converter.py", line 179, in <module>
    args.roberta_checkpoint_path, args.pytorch_dump_folder_path, args.classification_head
  File "converter.py", line 62, in convert_roberta_checkpoint_to_pytorch
    config.num_labels = roberta.args.num_classes
AttributeError: 'Namespace' object has no attribute 'num_classes'
```

I think `num_classes` will be like something like

`roberta.model.classification_heads[some_key].out_proj.weight.shape[0]`
There is likely only one possible key.
I just checked with a model fine-tuned on MNLI and the key is classification_heads['mnli'], is this what you expected?
sounds right! Don't lose that head!

Resolving patch:
diff --git a/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py b/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py
--- a/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py
+++ b/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py
@@ -47,7 +47,7 @@ def convert_roberta_checkpoint_to_pytorch(
     """
     roberta = FairseqRobertaModel.from_pretrained(roberta_checkpoint_path)
     roberta.eval()  # disable dropout
-    roberta_sent_encoder = roberta.model.decoder.sentence_encoder
+    roberta_sent_encoder = roberta.model.encoder.sentence_encoder
     config = RobertaConfig(
         vocab_size=roberta_sent_encoder.embed_tokens.num_embeddings,
         hidden_size=roberta.args.encoder_embed_dim,
@@ -59,7 +59,7 @@ def convert_roberta_checkpoint_to_pytorch(
         layer_norm_eps=1e-5,  # PyTorch default used in fairseq
     )
     if classification_head:
-        config.num_labels = roberta.args.num_classes
+        config.num_labels = roberta.model.classification_heads["mnli"].out_proj.weight.shape[0]
     print("Our BERT config:", config)
 
     model = RobertaForSequenceClassification(config) if classification_head else RobertaForMaskedLM(config)
@@ -126,12 +126,12 @@ def convert_roberta_checkpoint_to_pytorch(
         model.classifier.out_proj.bias = roberta.model.classification_heads["mnli"].out_proj.bias
     else:
         # LM Head
-        model.lm_head.dense.weight = roberta.model.decoder.lm_head.dense.weight
-        model.lm_head.dense.bias = roberta.model.decoder.lm_head.dense.bias
-        model.lm_head.layer_norm.weight = roberta.model.decoder.lm_head.layer_norm.weight
-        model.lm_head.layer_norm.bias = roberta.model.decoder.lm_head.layer_norm.bias
-        model.lm_head.decoder.weight = roberta.model.decoder.lm_head.weight
-        model.lm_head.decoder.bias = roberta.model.decoder.lm_head.bias
+        model.lm_head.dense.weight = roberta.model.encoder.lm_head.dense.weight
+        model.lm_head.dense.bias = roberta.model.encoder.lm_head.dense.bias
+        model.lm_head.layer_norm.weight = roberta.model.encoder.lm_head.layer_norm.weight
+        model.lm_head.layer_norm.bias = roberta.model.encoder.lm_head.layer_norm.bias
+        model.lm_head.decoder.weight = roberta.model.encoder.lm_head.weight
+        model.lm_head.decoder.bias = roberta.model.encoder.lm_head.bias
 
     # Let's check that we get the same results.
     input_ids: torch.Tensor = roberta.encode(SAMPLE_TEXT).unsqueeze(0)  # batch of size 1


Test patch: