Disparitry with Fairseq Roberta implementation for predicting the mask token
Author: hamediramin
Pull request: no
Labels: name: Core: Modeling

## ❓ Questions & Help
When experimenting with the fill mask functionality on the fairseq repo I realized there is a disparity with the results I get from huggingface implementation. Wondering if there is a mismatch between the model released here and their latest release.

Thanks

Comments:
We would need more information

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Also cc @joeddav 

Ok, I've reproduced an issue here.

```python
sentence = ' My favorite type of cheese is <mask>!'
roberta_fairseq = torch.hub.load('pytorch/fairseq', 'roberta.base')
roberta_hf = pipeline('fill-mask', model='roberta-base')

roberta_fairseq.fill_mask(sentence)
"""[(' My favorite type of cheese is goat!', 0.10264863073825836, ' goat'),
    (' My favorite type of cheese is cream!', 0.07233648002147675, ' cream'),
    (' My favorite type of cheese is broccoli!', 0.057516228407621384,' broccoli'),
    (' My favorite type of cheese is bacon!', 0.037444233894348145, ' bacon'),
    (' My favorite type of cheese is ham!', 0.03281955048441887, ' ham')]"""

roberta_hf(sentence)
"""[{'sequence': '
 My favorite type of cheese is goat!
',
     'score': 0.09398240596055984,
     'token': 24791},
    {'sequence': '
 My favorite type of cheese is cream!
',
     'score': 0.07240654528141022,
     'token': 6353},
    {'sequence': '
 My favorite type of cheese is broccoli!
',
     'score': 0.06303773820400238,
     'token': 34803},
    {'sequence': '
 My favorite type of cheese is bacon!
',
     'score': 0.04124978929758072,
     'token': 18599},
    {'sequence': '
 My favorite type of cheese is jack!
',
     'score': 0.03125162795186043,
     'token': 10267}]"""
```

They don't align when I use roberta large either. cc @sshleifer.

Probably want to compare the encoded input IDs. If they are the same, then there is a model difference (which is a big deal). Otherwise it can be attributed to a different tokenisation (e.g. difference in spaces or difference in adding special tokens).

The input IDs do match (again, when an empty space is prepended in fairseq's case): `[    0,  1308,  2674,  1907,     9,  7134,    16, 50264,   328,     2]`. 

I doesn't look like a pipelines issue:

```python
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
roberta_hf = RobertaForMaskedLM.from_pretrained('roberta-base')
roberta_fairseq = torch.hub.load('pytorch/fairseq', 'roberta.base')
roberta_fairseq.model.eval()

sequence = 'My favorite type of cheese is gouda!'
tokens = torch.tensor([tokenizer.encode(sequence)])
hf_out = roberta_hf.forward(tokens)[0]
fairseq_out = roberta_fairseq.model.forward(tokens)[0]

torch.mean(torch.abs(fairseq_out - hf_out)).item()
# 0.053489409387111664
```

Masked LM outputs differ by an average of `0.053`.

I don't have any time to look into this further, but from looking at the named parameters of both implementations, it seems that there is a difference in the LM head, that is, everything that comes after the encode (embedding + 12 layers). I can't pinpoint it exactly, but it seems note-worthy that the weight LM head in fairseq, comes from the token embedding weights

https://github.com/pytorch/fairseq/blob/4923f34790761f41170fd88cd06e4d00ab0c527c/fairseq/models/roberta/model.py#L286-L291

I am not sure in how far that is the same in transformers, but it seemed peculiar. Particularly, it seems that `transformers` does the 'None' case here, whereas it should actually take the token embedding weights.

https://github.com/pytorch/fairseq/blob/4923f34790761f41170fd88cd06e4d00ab0c527c/fairseq/models/roberta/model.py#L211-L213

I might be completely wrong, though. Other input welcome.

**Tl;dr**

Transformers does this

```python
self.decoder = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
```

whereas fairseq does

```python
self.lm_head = RobertaLMHead(
    embed_dim=args.encoder_embed_dim,
    output_dim=len(dictionary),
    activation_fn=args.activation_fn,
    weight=self.sentence_encoder.embed_tokens.weight,
)
...
# `weight` is **not** None
if weight is None:
    weight = nn.Linear(embed_dim, output_dim, bias=False).weight
self.weight = weight
```

I'd be very interested to know _why_ fairseq does this. Why would you want specifically those weights to be the same? 

I implemented my findings in https://github.com/huggingface/transformers/pull/2928.

The following snippet (based on the problem stated above) runs as expected, with identical results for both `transformers` and `fairseq`.

```python
import torch
from transformers import RobertaTokenizer, RobertaForMaskedLM, pipeline

# init HuggingFace
tokenizer_hf = RobertaTokenizer.from_pretrained('roberta-base')
roberta_hf = RobertaForMaskedLM.from_pretrained('roberta-base')
# init fairseq
roberta_fairseq = torch.hub.load('pytorch/fairseq', 'roberta.base')
roberta_fairseq.model.eval()

# LM test
sequence = 'My favorite type of cheese is gouda!'
tokens = torch.tensor([tokenizer_hf.encode(sequence)])
hf_out = roberta_hf.forward(tokens)[0]
fairseq_out = roberta_fairseq.model.forward(tokens)[0]
print(torch.mean(torch.abs(fairseq_out - hf_out)).item())
# should be 0.0

# ---
# pipeline test fill-mask
sentence = ' My favorite type of cheese is <mask>!'
print(roberta_fairseq.fill_mask(sentence))
roberta_hf = pipeline('fill-mask', tokenizer=tokenizer_hf, model='roberta-base')
print(roberta_hf(sentence))
# should have identical predictions
```

Resolved by #2958