Not getting expected results for freshly-pretrained BERT 
Author: danyaljj
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I have a freshly trained BERT (using the original BERT) which doesn't seem to function as expected after moving it to PyTorch/Transformers. I don't have any syntax errors anywhere. So I am not sure how to go about understanding the bug. Wondering if anyone has suggestions for me. 
Below I am providing details on every single step of my training process.

Here is a comment from my training: 
```
2020-05-14 08:36:41,144 :  Saving checkpoints for 1875000 into gs:// .... /perbert_L-12_H-768_A-12/model.ckpt.
2020-05-14 08:37:07,703 :  loss = 1.7460283, step = 1875000 (1664.300 sec)
2020-05-14 08:37:07,704 :  global_step/sec: 15.0213
2020-05-14 08:37:07,704 :  examples/sec: 1922.73
2020-05-14 08:37:07,705 :  Enqueue next (25000) batch(es) of data to infeed.
2020-05-14 08:37:07,705 :  Dequeue next (25000) batch(es) of data from outfeed.
```
As you can see, the loss function is pretty low after being trained for about 180k steps. 

Now I had it converted with `convert_tf_checkpoint_to_pytorch`: 
```python
from pytorch_pretrained_bert.convert_tf_checkpoint_to_pytorch import convert_tf_checkpoint_to_pytorch

model = "perbert_L-12_H-768_A-12"
checkpoint = "1875000"

convert_tf_checkpoint_to_pytorch(
    f'/models/{model}/model.ckpt-{checkpoint}.index',
    bert_config_file=f'models/{model}/bert_config.json',
    pytorch_dump_path=f"models/{model}/saved_model_{checkpoint}")
```

Then I load the results in masked language model to see whether it does something reasonable or not: 
```python 
import torch
from transformers import BertConfig, BertTokenizer, BertForMaskedLM

dir = "/Users/danielk/ideaProjects/farsi-language-models/src/models/perbert_L-12_H-768_A-12/"
checkpoint = 1875000
tokenizer = BertTokenizer.from_pretrained(dir)
config = BertConfig.from_json_file(dir + '/bert_config.json')
model = BertForMaskedLM.from_pretrained(dir + '/saved_model_' + str(checkpoint), config=config)

string = "بزرگترین اقتصاد دنیا در حال فروپاشی موقت است. این امر مساله ای کاملا طبیعی است"
tokens = tokenizer.tokenize(string)
print("tokens: " + tokens)
print("indices: " +  tokenizer.encode(string))
input_ids = torch.tensor([tokenizer.encode(string)])
outputs = model(input_ids)

predictions = outputs[0]
for masked_index, _ in enumerate(tokens):
    print(" * input string: " + tokens[masked_index])
    predictions_numpy = predictions[0, masked_index].cpu().data.numpy()
    predictions_with_index = list(enumerate(predictions_numpy))
    predictions_with_index = sorted(predictions_with_index, key=lambda x: -x[1])
    top_indices = [x for x in predictions_with_index[:10]]
    predicted_tokens = tokenizer.convert_ids_to_tokens([x[0] for x in top_indices])
    print(predicted_tokens)
    print("...")
```
The output of the tokenization is correct; the tokens are reasonable and they're mapped to the right indices: 
```
tokens: ['بزرگترین', 'اقتصاد', 'دنیا', 'در', 'حال', 'فروپاشی', 'موقت', 'است', '[UNK]', 'این', 'امر', 'مساله', 'ای', 'کاملا', 'طبیعی', 'است']
indices: [2, 989, 499, 655, 6, 166, 16459, 3756, 13, 1, 11, 1312, 3703, 36, 1089, 1070, 13, 3]
```

However, the output of the masking is not as expected: 
```
 * input string: بزرگترین
['Plugin', '##AN', '##AZ', 'ورزشکاران', 'خانوارها', 'items', 'Moha', 'بکنید', '##Style', 'سهیلی']
...
 * input string: اقتصاد
['استان', 'آنها', 'نماد', 'বন', 'ویتنام', 'محكوم', '##جاری', '##sc', 'public', 'آلبرت']
...
 * input string: دنیا
['##Lang', 'public', 'آی', 'یافتهاند', 'check', 'یادتون', 'ضریب', 'رصدخانه', 'تزئینات', 'رف']
...
 * input string: در
['##Lang', 'Plugin', 'বন', '##BX', 'گرن', 'مغز', 'منجمد', 'فضل', 'سرمایشی', 'public']
...
 * input string: حال
['##Lang', 's', 'تمهیدات', 'خخخ', 'Product', 'حلزون', '##ten', '##تیپ', '##sc', 'آی']
...
 * input string: فروپاشی
['##Pack', 'বন', 'باشد', 'عجایب', '##File', '##Lang', '##column', 's', '##BG', '334']
...
 * input string: موقت
['بن', 'عاشورایی', 'تایباد', 'Vivo', 'بلوغ', '##dani', 'اشتیاق', '1351', '[UNUSED_194]', '##أفادت']
...
 * input string: است
['85', 'مشار', '##گرایان', '##جاری', 'while', '##cause', 'ژیل', 'مولد', 'نقلیه', 'رحم']
...
 * input string: [UNK]
['آنها', 'چسبندگی', 'سرمایشی', '##یس', 'کلی', 'آی', 'آزمایشی', '##گ', 'مولد', 'سینماهای']
...
 * input string: این
['اسپانیایی', '##زآپ', 'سهیلی', 'رسیدند', 'استان', 'بن', 'کریر', '##پد', 'تایباد', '##ث']
...
 * input string: امر
['ش', 'বন', '##آيند', 'اعتباری', 'نامرئی', '##ملی', 'public', 'اومد', 'یافتهاند', 'موعظه']
...
 * input string: مساله
['إيران', 'বন', 'public', '##سمح', 'wp', 'نماد', 'شئون', 'website', '##تیپ', '##اعتراض']
...
 * input string: ای
['مردمان', '[UNUSED_203]', 'خورده', 'public', 'لهم', 'آنها', 'کعبه', 'الهی', 'رصدخانه', 'سرمایشی']
...
 * input string: کاملا
['public', '77', 'مروج', 'تایباد', '##aghi', 'منصب', '##Lang', '[UNUSED_203]', '##198', 'مغز']
...
 * input string: طبیعی
['##Lang', 'Plugin', '##den', '##مفید', 'وزنه', 'دلهره', 'check', 'Vivo', 'Biz', 'خواهان']
...
 * input string: است
['ju', '##Lang', 'عقربه', 'استان', 'اثرانگشت', '##aghi', 'انزلی', 'چسبندگی', '##tis', 'فرماید']
...
```
The above output that the masked language model has returned is quite random. Clearly, something is off. 

I briefly inspected one of my tfrecord files, but they also seem to be reasonable; here is one example block: 
```json
{
  "features": {
    "feature": {
      "masked_lm_positions": {
        "int64List": {
          "value": [
            "2",
            "3",
            "5",
            "15",
            "23",
            "27",
            "33",
            "41",
            "55",
            "60",
            "72",
            "74",
            "87",
            "97",
            "103",
            "104",
            "106",
            "112",
            "0",
            "0"
          ]
        }
      },
      "masked_lm_ids": {
        "int64List": {
          "value": [
            "5601",
            "33",
            "4125",
            "12",
            "6",
            "11635",
            "9808",
            "178",
            "8",
            "287",
            "6",
            "3213",
            "6",
            "23",
            "8",
            "814",
            "613",
            "12466",
            "0",
            "0"
          ]
        }
      },
      "masked_lm_weights": {
        "floatList": {
          "value": [
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            1.0,
            0.0,
            0.0
          ]
        }
      },
      "segment_ids": {
        "int64List": {
          "value": [
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0"
          ]
        }
      },
      "input_ids": {
        "int64List": {
          "value": [
            "2",
            "2824",
            "4",
            "4",
            "6",
            "4",
            "599",
            "5388",
            "16",
            "16598",
            "4513",
            "5",
            "2824",
            "2219",
            "217",
            "5891",
            "7",
            "3117",
            "4710",
            "24069",
            "16",
            "69",
            "5",
            "4",
            "127",
            "166",
            "3386",
            "4",
            "178",
            "13",
            "1",
            "278",
            "26085",
            "5791",
            "5",
            "1401",
            "1801",
            "5",
            "2348",
            "127",
            "905",
            "4",
            "13",
            "1",
            "2824",
            "1127",
            "8447",
            "7144",
            "18646",
            "5",
            "168",
            "14763",
            "9",
            "6",
            "7103",
            "4",
            "3878",
            "1978",
            "5",
            "430",
            "287",
            "1",
            "3",
            "12466",
            "23",
            "814",
            "6",
            "14259",
            "6",
            "4426",
            "46",
            "453",
            "4",
            "707",
            "4",
            "46",
            "6",
            "4426",
            "365",
            "338",
            "7935",
            "9",
            "6",
            "748",
            "6",
            "4426",
            "44",
            "4",
            "476",
            "1",
            "1",
            "7",
            "12466",
            "10",
            "12053",
            "1070",
            "6",
            "4",
            "1832",
            "44",
            "8012",
            "7968",
            "46",
            "4",
            "4",
            "49",
            "4",
            "287",
            "14",
            "6",
            "14259",
            "707",
            "4",
            "1",
            "1348",
            "44",
            "6849",
            "62",
            "6931",
            "31",
            "1",
            "3",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0"
          ]
        }
      },
      "input_mask": {
        "int64List": {
          "value": [
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "1",
            "0",
            "0",
            "0",
            "0",
            "0",
            "0"
          ]
        }
      },
      "next_sentence_labels": {
        "int64List": {
          "value": [
            "1"
          ]
        }
      }
    }
  }
}
```

Here are some thoughts: 
 - Could the model conversion somehow go wrong without raising any errors? 
 - Persian is a right-to-left language; could that be a factor? In theory, it should not be an issue, since from the perspective of the either code (TF and PyTorch) these are just strings/byres sequenced after each other. 





Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.