Different outputs in using convert_roberta_original_pytorch_checkpoint_to_pytorch.py
Author: lhaausing
Pull request: no

# 🐛 Bug
We have an issue in converting roberta model from fairseq format to huggingface format. The conversion function provided in the transformers library gives you different outputs when you pass sample data through the fairseq and huggingface-transformers versions seperately.

⬇️Problem with our pretrained model
There's a difference between two output tensors. The actual number is⬇️

huggingface transformers output:

>  tensor([[[ 2.1787e+01, -4.7770e+00,  6.1631e+00,  ..., -4.6316e+00,
>           -4.7297e+00, -3.9510e-01],
>          [ 2.0051e+00, -2.7158e+00,  5.2598e+00,  ..., -2.3681e+00,
>           -2.0179e+00, -1.5263e-02],
>          [-2.7891e+00, -4.7558e+00,  5.3717e+00,  ..., -4.5290e+00,
>           -3.8888e+00, -5.7892e-02],
>          [ 1.3125e+00, -3.9378e+00,  6.7551e+00,  ..., -3.6842e+00,
>           -3.4968e+00,  5.4736e-01],
>          [-3.4706e+00, -7.7992e+00,  1.6678e+01,  ..., -6.1806e+00,
>           -7.4419e+00, -8.5062e-02]]], grad_fn=<AddBackward0>)

fairseq output:

>  tensor([[[21.2672, -4.8905,  6.2439,  ..., -4.8653, -4.9650, -1.6207],
>          [ 1.4856, -2.8294,  5.3406,  ..., -2.6018, -2.2533, -1.2408],
>          [-3.3087, -4.8693,  5.4525,  ..., -4.7626, -4.1241, -1.2835],
>          [ 0.7930, -4.0513,  6.8359,  ..., -3.9179, -3.7322, -0.6782],
>          [-3.9902, -7.9127, 16.7589,  ..., -6.4142, -7.6773, -1.3106]]],
>        grad_fn=<AddBackward0>)

abs difference:

>  tensor([[[0.5195, 0.1135, 0.0808,  ..., 0.2336, 0.2354, 1.2256],
>          [0.5195, 0.1135, 0.0808,  ..., 0.2336, 0.2354, 1.2256],
>          [0.5195, 0.1135, 0.0808,  ..., 0.2336, 0.2354, 1.2256],
>          [0.5195, 0.1135, 0.0808,  ..., 0.2336, 0.2354, 1.2256],
>          [0.5195, 0.1135, 0.0808,  ..., 0.2336, 0.2354, 1.2256]]],
>        grad_fn=<AbsBackward>)

The same issue happens when we try to convert the default roberta-base model from fairseq format into transformers format.
We have change some source code from fairseq to register our model name and architecture(Just changes in some hyperparameters).
Our initial guess is that there are some parameters isn't or is wrongly loaded.

The error looks like this:

> max_absolute_diff = 1.2255859375
> Do both models output the same tensors? 💩
> Traceback (most recent call last):
>   File "convert_roberta_original_pytorch_checkpoint_to_pytorch.py", line 181, in <module>
>     args.roberta_checkpoint_path, args.pytorch_dump_folder_path, args.classification_head
>   File "convert_roberta_original_pytorch_checkpoint_to_pytorch.py", line 160, in convert_roberta_checkpoint_to_pytorch
>     raise Exception("Something went wRoNg")
> Exception: Something went wRoNg

Thanks a lot for helping.

## Information

Model I am using (Bert, XLNet ...): RoBERTa

Language I am using the model on (English, Chinese ...): English

## Expected behavior
Explanation about the function (or further contact in refining the function?)


## Environment info

- `transformers` version: the default version
- Platform: NYU Prince Cluster
- Python version: python 3.7
- PyTorch version (GPU?): No GPU
- Tensorflow version (GPU?): No GPU
- Using GPU in script?: No GPU
- Using distributed or parallel set-up in script?: No GPU


Comments:
I had a similar problem. There is a bug in conversion script, this pull request fixes the issue: https://github.com/huggingface/transformers/pull/3642

@sdadas Thanks a lot. I've pulled the latest change and it works now.

Great to hear, closing!