Transformer-XL: Convert lm1b model to PyTorch
Author: stefan-it
Pull request: no
Labels: name: Help wanted

name: wontfix

Hi,

I wanted to convert the TensorFlow checkpoint for the ` lm1b` model to PyTorch with the `convert_transfo_xl_checkpoint_to_pytorch.py` script.

I downloaded the checkpoint with the [download.sh](https://github.com/kimiyoung/transformer-xl/blob/master/tf/sota/download.sh) script. 

Then I called the convert script with:

```bash
$ python3 convert_transfo_xl_checkpoint_to_pytorch.py --pytorch_dump_folder_path converted --tf_checkpoint_path
/mnt/transformer-xl/tf/sota/pretrained_xl/tf_lm1b/model/checkpoint
```

Then the following error message is returned:

```bash
2019-02-19 22:46:54.693060: W tensorflow/core/util/tensor_slice_reader.cc:95] Could not open /mnt/transformer-xl/tf/sota/pretrained_xl/tf_lm1b/model/checkpoint: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
Traceback (most recent call last):
  File "convert_transfo_xl_checkpoint_to_pytorch.py", line 116, in <module>
    args.transfo_xl_dataset_file)
  File "convert_transfo_xl_checkpoint_to_pytorch.py", line 81, in convert_transfo_xl_checkpoint_to_pytorch
    model = load_tf_weights_in_transfo_xl(model, config, tf_path)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_pretrained_bert/modeling_transfo_xl.py", line 141, in load_tf_weights_in_transfo_xl
    init_vars = tf.train.list_variables(tf_path)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/checkpoint_utils.py", line 95, in list_variables
    reader = load_checkpoint(ckpt_dir_or_file)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/checkpoint_utils.py", line 64, in load_checkpoint
    return pywrap_tensorflow.NewCheckpointReader(filename)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 382, in NewCheckpointReader
    return CheckpointReader(compat.as_bytes(filepattern), status)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/errors_impl.py", line 548, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file /mnt/transformer-xl/tf/sota/pretrained_xl/tf_lm1b/model/checkpoint: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
```

I'm using the *0.6.1* version of `pytorch-pretrained-BERT` and the latest `tf-nightly-gpu` package that ships TensorFlow 1.13dev.

Comments:
Hm, I guess I was using the wrong `checkpoint` file? When I used `/mnt/transformer-xl/tf/sota/pretrained_xl/tf_lm1b/model/model.ckpt-1191000` weights are loaded, but another error occurs:

```bash
Loading TF weight transformer/r_r_bias/Adam_1 with shape [24, 16, 80]
Loading TF weight transformer/r_w_bias with shape [24, 16, 80]
Loading TF weight transformer/r_w_bias/Adam with shape [24, 16, 80]
Loading TF weight transformer/r_w_bias/Adam_1 with shape [24, 16, 80]
Traceback (most recent call last):
  File "convert_transfo_xl_checkpoint_to_pytorch.py", line 116, in <module>
    args.transfo_xl_dataset_file)
  File "convert_transfo_xl_checkpoint_to_pytorch.py", line 81, in convert_transfo_xl_checkpoint_to_pytorch
    model = load_tf_weights_in_transfo_xl(model, config, tf_path)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_pretrained_bert/modeling_transfo_xl.py", line 169, in load_tf_weights_in_transfo_xl
    assert pointer.shape == array.shape
AssertionError: (torch.Size([3, 1024]), (3, 1280))
```

Ok, the `TransfoXLConfig` for the `lm1b` model is a bit different. I tried:

```python
config = TransfoXLConfig(vocab_size_or_config_json_file=793472,
                 cutoffs=[0, 60000, 100000, 640000, 793472],
                 d_model=1280,
                 d_embed=1280,
                 n_head=16,
                 d_head=80,
                 d_inner=8192,
                 div_val=4,
                 pre_lnorm=False,
                 n_layer=24,
                 tgt_len=32,
                 ext_len=0,
                 mem_len=128,
                 clamp_len=-1,
                 same_length=True,
                 proj_share_all_but_first=False,
                 attn_type=0,
                 sample_softmax=-1,
                 adaptive=True,
                 tie_weight=True,
                 dropout=0.0,
                 dropatt=0.0,
                 untie_r=True,
                 init="normal",
                 init_range=0.01,
                 proj_init_std=0.01,
                 init_std=0.02)
```

which seems not to be 100% correct. Where do I get the model json configuration from (so I can easily pass it to the `convert_transfo_xl_checkpoint_to_pytorch.py` script 🤔

Hi Stefan,
You have to create the configuration yourself indeed 🙂
I usually do it by looking at the training parameters of the Tensorflow code related to the model you are trying to load.

The voab `cutoffs` were wrong. I changed the configuration to:

```python
config = TransfoXLConfig(vocab_size_or_config_json_file=793472,
                 cutoffs=[60000, 100000, 640000],
                 d_model=1280,
                 d_embed=1280,
                 n_head=16,
                 d_head=80,
                 d_inner=8192,
                 div_val=4,
                 pre_lnorm=False,
                 n_layer=24,
                 tgt_len=32,
                 ext_len=0,
                 mem_len=128,
                 clamp_len=-1,
                 same_length=True,
                 proj_share_all_but_first=False,
                 attn_type=0,
                 sample_softmax=-1,
                 adaptive=True,
                 tie_weight=True,
                 dropout=0.0,
                 dropatt=0.0,
                 untie_r=True,
                 init="normal",
                 init_range=0.01,
                 proj_init_std=0.01,
                 init_std=0.02,
                 )
```

And then the `transformer/adaptive_softmax/cutoff_0/proj` key wasn't found in the `tf_weights` dict:

```python
transformer/adaptive_softmax/cutoff_0/proj
Traceback (most recent call last):
  File "convert_transfo_xl_checkpoint_to_pytorch.py", line 142, in <module>
    args.transfo_xl_dataset_file)
  File "convert_transfo_xl_checkpoint_to_pytorch.py", line 107, in convert_transfo_xl_checkpoint_to_pytorch
    model = load_tf_weights_in_transfo_xl(model, config, tf_path)
  File "/mnt/pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling_transfo_xl.py", line 150, in load_tf_weights_in_transfo_xl
    assert name in tf_weights
AssertionError
```


That's probably a question of weights or projection tying, try to set `tie_weight` or `proj_share_all_but_first` to `False` (the correct value should be indicated in Google/CMU hyper-parameters for lm1b).

(I can convert this model later if you don't manage to but not before next week unfortunately)

Thanks for your help @thomwolf ! I'll try to find the correct configuration settings.

We are currently trying to integrate the Transformer-XL model into [flair](https://github.com/zalandoresearch/flair), and we would really like to use a larger (in terms of training size) model for downstream tasks like NER :)

Here's the last configuration I tried:

```json
{
  "adaptive": true,
  "attn_type": 0,
  "clamp_len": -1,
  "cutoffs": [
    60000,
    100000,
    640000
  ],
  "d_embed": 1280,
  "d_head": 80,
  "d_inner": 8192,
  "d_model": 1280,
  "div_val": 4,
  "dropatt": 0.0,
  "dropout": 0.1,
  "ext_len": 0,
  "init": "normal",
  "init_range": 0.01,
  "init_std": 0.02,
  "mem_len": 32,
  "n_head": 16,
  "n_layer": 24,
  "n_token": 793472,
  "pre_lnorm": false,
  "proj_init_std": 0.01,
  "same_length": true,
  "sample_softmax": -1,
  "tgt_len": 32,
  "tie_weight": true,
  "untie_r": true,
  "proj_share_all_but_first": false,
  "proj_same_dim": false,
  "tie_projs": [
    true,
    false,
    true
  ]
}
````

Unfortunately, an error is thrown. @thomwolf it would be awesome if you can take a look on this :)

Did you manage to convert this model @stefan-it?

Sadly, I couldn't managed to convert it (I tried several options)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


 @stefan-it did you ever manage to convert this model?

Hi @irugina, unfortunately, I wasn't able to convert the model 😞