Pre-trained BERT-LM missing LM Head - returns random token predictions
Author: iliaschalkidis
Pull request: no
Labels: name: Core: Modeling

name: New model

# 🐛 Bug

## Information

I released Greek BERT, almost a week ago and so far I'm exploring its use by running some benchmarks in Greek datasets. Although Greek BERT works just fine for sequence tagging (`AutoModelForTokenClassification`) and text classification (`AutoModelForSequenceClassification`), there are issues when we try to to use it as Language Model (`AutoModelWithLMHead`) in order to predict masked tokens. The bug was originally reported in (https://github.com/nlpaueb/greek-bert/issues/1) by  @jzbjyb.

## To reproduce

- `transformers` version:
- Platform: Linux / Mac OS
- Python version: 3.7
- PyTorch version (GPU?): 1.0.1
- Tensorflow version (GPU?): 2.1
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

The model has been trained using the official BERT release (https://github.com/google-research/bert) originally converted from Tensorflow checkpoints using the library script:

```
python transformers/convert_bert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path /home/ichalkidis/greek_bert/variables --bert_config_file=/home/ichalkidis/greek_bert/config.json --pytorch_dump_path=/home/ichalkidis/greek_bert/pytorch_model.bin
```

and then exported accompanied by the tokenizer files using:

```python
from transformers import BertModel
from transformers import BertConfig, BertTokenizer
model_path = '/home/ichalkidis/greek_bert/'

bert = BertModel.from_pretrained(model_path + 'pytorch_model.bin',
                                         config=BertConfig().from_pretrained(model_path + 'config.json'))
bert.save_pretrained('/home/ichalkidis/bert-base-greek-uncased-v1/')

tokenizer = BertTokenizer.from_pretrained(model_path+'vocab.txt')
tokenizer.save_pretrained('/home/ichalkidis/bert-base-greek-uncased-v1/')
```


You can replicate the inconsistent behaviour of the LM  with the following script:

```python
import torch
from transformers import *

text = 'Είναι ένας [MASK] άνθρωπος.'
tokenizer_greek = AutoTokenizer.from_pretrained('nlpaueb/bert-base-greek-uncased-v1')
lm_model_greek = AutoModelWithLMHead.from_pretrained('nlpaueb/bert-base-greek-uncased-v1')
input_ids = tokenizer_greek.encode(text)
print(tokenizer_greek.convert_ids_to_tokens(input_ids))
# ['[CLS]', 'ειναι', 'ενας', '[MASK]', 'ανθρωπος', '.', '[SEP]']
outputs = lm_model_greek(torch.tensor([input_ids]))[0]
print(tokenizer_greek.convert_ids_to_tokens(outputs[0, 3].max(0)[1].item()))
# the most plausible prediction for [MASK] is changing in every single run
```

It is obvious that the LM Head (layer) is missing from `pytorch_model.bin`, my main questions and report are:
* How we could preserve this layer moving from the TF checkpoint to the final PyTorch model? 
* Is it possible to serve the model both in Pytorch and TF binaries?

Comments:
My guess would be that you'd need to load (and then save) with `BertForPreTraining` rather than `BertModel`.

https://github.com/huggingface/transformers/blob/20fc18fbda3669c2f4a3510e0705b2acd54bff07/src/transformers/modeling_bert.py#L806

@BramVanroy you're my hero for today! Appreciated man! 

Test Cases:

```python
import torch
from transformers import *

model_path = '/Users/kiddothe2b/Downloads/bert-base-greek-uncased-v2'
tokenizer_greek = AutoTokenizer.from_pretrained(model_path)
lm_model_greek = AutoModelWithLMHead.from_pretrained(model_path)

# ================ EXAMPLE 1 ================
text_1 = 'O ποιητής έγραψε ένα [MASK] .'
# EN: The poet wrote a [MASK] . '
input_ids = tokenizer_greek.encode(text_1)
print(tokenizer_greek.convert_ids_to_tokens(input_ids))
# ['[CLS]', 'o', 'ποιητης', 'εγραψε', 'ενα', '[MASK]', '.', '[SEP]']
outputs = lm_model_greek(torch.tensor([input_ids]))[0]
print(tokenizer_greek.convert_ids_to_tokens(outputs[0, 5].max(0)[1].item()))
# the most plausible prediction for [MASK] is "song"

# ================ EXAMPLE 2 ================
text_2 = 'Είναι ένας [MASK] άνθρωπος.'
# EN: He is a [MASK] person. '
input_ids = tokenizer_greek.encode(text_1)
print(tokenizer_greek.convert_ids_to_tokens(input_ids))
# ['[CLS]', 'ειναι', 'ενας', '[MASK]', 'ανθρωπος', '.', '[SEP]']
outputs = lm_model_greek(torch.tensor([input_ids]))[0]
print(tokenizer_greek.convert_ids_to_tokens(outputs[0, 3].max(0)[1].item()))
# the most plausible prediction for [MASK] is "good"

# ================ EXAMPLE 3 ================
text_3 = 'Είναι ένας [MASK] άνθρωπος και κάνει συχνά [MASK].'
# EN: He is a [MASK] person he does frequently [MASK]. '
input_ids = tokenizer_greek.encode(text_3)
print(tokenizer_greek.convert_ids_to_tokens(input_ids))
# ['[CLS]', 'ειναι', 'ενας', '[MASK]', 'ανθρωπος', 'και', 'κανει', 'συχνα', '[MASK]', '.', '[SEP]']
outputs = lm_model_greek(torch.tensor([input_ids]))[0]
print(tokenizer_greek.convert_ids_to_tokens(outputs[0, 8].max(0)[1].item()))
# the most plausible prediction for the second [MASK] is "trips"
```

I will release the updated version later today. Although I think this is a very important technical detail and needs to be referred in the examples.

Is it possible to load the same saved model with TF2 binding?



Tensorflow checkpoints can be loaded when using `from_pretrained`. Have a look at the documentation, particularly this line:

https://github.com/huggingface/transformers/blob/20fc18fbda3669c2f4a3510e0705b2acd54bff07/src/transformers/modeling_utils.py#L317

Ok, so it is also suggested to upload the initial TF checkpoint files [ `variables.data-00000-of-00001`, `variables.index`, `variables.meta`] through the CLI, otherwise people cannot use the model in TF2 with `AutoModelWith.from_pretrained()`? 


I am not sure to be honest. Perhaps someone else can help you further along.

In the past for another project, this (`from_pt=True`) did the dirty trick for me:

```python
bert = BERT.from_pretrained(model_path+'pytorch_model.bin',
                                               from_pt=True,
                                               config=BertConfig().from_pretrained(model_path+'config.json'))
```
but I definitely not recommend this...

I have already uploaded the TF checkpoint files (`model_ckpt.data-00000-of-00001`, `model_ckpt.index`, `model_ckpt.meta`) in model's folder, so please feel free to troubleshoot.

Looks like you managed to do it in the meantime?

**For reference here's how to do it:** starting with version 2.5.0 thanks to this commit https://github.com/huggingface/transformers/pull/2765/commits/961c69776f8a2c95b92407a086848ebca037de5d

You can now just do 
```python
tf_model = TFAutoModelForPreTraining.from_pretrained(
    "nlpaueb/bert-base-greek-uncased-v1", 
    from_pt=True
)
tf_model.save_pretrained(dirname)
```

**And then you can upload the TF weights using the CLI.**

Of course, if you have the model in a local folder, you don't need to use the remote id.

cc @LysandreJik 

> Looks like you managed to do it in the meantime?
> 
> **For reference here's how to do it:** starting with version 2.5.0 thanks to this commit [961c697](https://github.com/huggingface/transformers/commit/961c69776f8a2c95b92407a086848ebca037de5d)
> 
> You can now just do
> 
> ```python
> tf_model = TFAutoModelForPreTraining.from_pretrained(
>     "nlpaueb/bert-base-greek-uncased-v1", 
>     from_pt=True
> )
> tf_model.save_pretrained(dirname)
> ```
> 
> Of course, if you have the model in a local folder, you don't need to use the remote id.
> 
> cc @LysandreJik

Does that imply that the CLI upload should only upload PyTorch checkpoints? (I suppose for saving space.) I am asking because the documentation emphasises that loading PT checkpoints to TF and the other way around is quite slow. Also, if all checkpoints are indeed PyTorch only, it might be useful to set from_pt=True automatically when a model is fetched from the HF bucket (since those would then all contain PT checkpoints anyway).

Thanx @julien-c and @BramVanroy 

Indeed I followed a very similar process:

```python

from transformers import TFBertForPreTraining
from transformers import BertConfig
model_path = '/home/ichalkidis/greek_bert/'
bert = TFBertForPreTraining.from_pretrained(model_path + 'pytorch_model.bin',
                                         config=BertConfig().from_pretrained(model_path + 'config.json'), from_pt=True)

bert.save_pretrained('/home/ichalkidis/bert-base-greek-uncased-v1/')
```

From now on, we also serve the `tf_model.h5` and everyone will be able to load the model in TF2 without any further issue.

Ah, it seems that I misunderstood, then. In terms of the models themselves, you can upload the tf_model.h5 and the pytorch_model.bin, and when someone requests `/home/ichalkidis/bert-base-greek-uncased-v1/` based on the framework (TF or PT), the appropriate model (.h5 or .bin) is downloaded?

@BramVanroy, yes that's how it works! You can also explicitely specify `from_pt`/`from_tf` to `from_pretrained` for the model to fetch the other framework's checkpoint and convert it.

Yep @BramVanroy , I added the line "And then you can upload the TF weights using the CLI." to my comment above to try and make that clearer

I guess we can close this issue now @iliaschalkidis?

Of course @julien-c . In the end, this was just a misunderstanding, not a bug at all. Thank all of you for your help!