convert_roberta: AttributeError when converting CamemBERT model.pt to pytorch_model.bin
Author: ghost
Pull request: no

Hi,

I trained a CamemBERT model with the fairseq library which gave me the following files:
- dict.txt: vocabulary coming from the sentencepiece model
- sentencepiece.bpe.model
- model.pt

Now I am trying to convert the model.pt into pytorch_model.bin and config.json as mentionned here ([fairseq/issues#1514](https://github.com/pytorch/fairseq/issues/1514)) and here ([transformers/issue#1850](https://github.com/huggingface/transformers/issues/1850)), by using the conversion script of the transformers library ([transfomers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py)). The goal is to use those files with fast-bert.

However, using this command line:
```shell
python convert_roberta_original_pytorch_checkpoint_to_pytorch.py --roberta_checkpoint_path ./ --pytorch_dump_folder_path ./ --classification_head
```

I get the following error:

```python
AttributeError                            Traceback (most recent call last)
<ipython-input-27-ea791887ff26> in <module>
----> 1 convert_roberta_original_pytorch_checkpoint_to_pytorch.convert_roberta_checkpoint_to_pytorch(CAMEMBERT_PATH, CAMEMBERT_PATH, True)

~/anaconda3/envs/NLP/lib/python3.7/site-packages/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py in convert_roberta_checkpoint_to_pytorch(roberta_checkpoint_path, pytorch_dump_folder_path, classification_head)
     48     roberta = FairseqRobertaModel.from_pretrained(roberta_checkpoint_path)
     49     roberta.eval()  # disable dropout
---> 50     roberta_sent_encoder = roberta.model.decoder.sentence_encoder
     51     config = RobertaConfig(
     52         vocab_size=roberta_sent_encoder.embed_tokens.num_embeddings,

~/anaconda3/envs/NLP/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    592                 return modules[name]
    593         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 594             type(self).__name__, name))
    595 
    596     def __setattr__(self, name, value):

AttributeError: 'RobertaModel' object has no attribute 'decoder'
```

And indeed when I check the fairseq/pytorch RobertaModel has no decoder attribute. 

Am I doing this wrong ? I see no other conversion script to fit my CamemBERT model so I guess the RoBERTa one is the good one.


Comments:
maybe @sshleifer has an idea

I can help! 
On fairseq master (as of 5/28/20), that class seems to no longer have a `decoder` attribute.

I think you want to change the `roberta.model.decoder` references to `self.model.encoder`, but hard to know without seeing the `state_dict`/handling the model interactively.

The best way to debug is to either instantiate the fairseq model in jupyter/ipython or set a breakpoint and see what the attributes are.

If you are stuck, feel free to upload your `model.pt` to some cloud storage and I can give it a shot!



Hey @sshleifer!

I did what you suggested and it worked, thanks a lot. You have to replace all the references to `roberta.model.decoder` with `roberta.model.encoder` as the attributes were just renamed.

On the other hand, I can't figure out what happened to `roberta.args.num_classes` that is used for the classification_head flag, which makes it useless for now.

I would gladly commit the fix but I'm not a powergit user, so I'll leave it to the pros.

Thanks again!

***

Edit : the error that comes up with the flag. 

```python
Traceback (most recent call last):
  File "converter.py", line 179, in <module>
    args.roberta_checkpoint_path, args.pytorch_dump_folder_path, args.classification_head
  File "converter.py", line 62, in convert_roberta_checkpoint_to_pytorch
    config.num_labels = roberta.args.num_classes
AttributeError: 'Namespace' object has no attribute 'num_classes'
```


I think `num_classes` will be like something like

`roberta.model.classification_heads[some_key].out_proj.weight.shape[0]`
There is likely only one possible key.

I just checked with a model fine-tuned on MNLI and the key is classification_heads['mnli'], is this what you expected?

sounds right! Don't lose that head!

Hey @sshleifer,
Sorry if that's not the right place to ask but I couldn't find an answer to that question anywhere: is there a script like this one to convert a model.pt trained on gpu to a model.bin ? or should this script works both for cpu and gpu models ?
Thanks!

Did our library save the `model.pt`?

The filenames don't really matter if the contents of the file are a `state_dict`.
So it may be as simple as, from the terminal,
```bash
mv model.pt pytorch_model.bin
```
The library doesn't care if a `state_dict` was saved on gpu or cpu.

If that fails try to run `torch.load('model.pt', map_location='cpu')` in an interactive environment and see if it's a state dict.



Thanks for the answer @sshleifer! HuggingFace was working well, it was my nvidia apex installation that was broken and returned errors in fast-bert that confused me. All works well now!