Model conversion from PyTorch to TF2 doesn't work properly for ALBERT
Author: singletongue
Pull request: no
Labels: name: wontfix

name: Core: Modeling

# 🐛 Bug

## Information

The model conversion script `convert_pytorch_checkpoint_to_tf2.py` seems to be not working properly for ALBERT models.
It fails on the pre-trained models officially released by Google which are converted to PyTorch models with `convert_albert_original_tf_checkpoint_to_pytorch.py`.

## To reproduce

```
$ wget https://storage.googleapis.com/albert_models/albert_base_v2.tar.gz
$ tar xzf albert_base_v2.tar.gz
$ cd albert_base/
$ python -m transformers.convert_albert_original_tf_checkpoint_to_pytorch --tf_checkpoint_path model.ckpt-best --pytorch_dump_path ./pytorch_model.bin --albert_config_file albert_config.json
$ python -m transformers.convert_pytorch_checkpoint_to_tf2 --tf_dump_path ./ --model_type albert --pytorch_checkpoint_path ./pytorch_model.bin --config_file albert_config.json --compare_with_pt_model
...
Max absolute difference between models outputs 17.709423065185547
Traceback (most recent call last):
  File "/home/m-suzuki/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/m-suzuki/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 499, in <module>
    only_convert_finetuned_models=args.only_convert_finetuned_models,
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 428, in convert_all_pt_checkpoints_to_tf
    compare_with_pt_model=compare_with_pt_model,
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 351, in convert_pt_checkpoint_to_tf
    assert diff <= 2e-2, "Error, model absolute difference is >2e-2: {}".format(diff)
AssertionError: Error, model absolute difference is >2e-2: 17.709423065185547
```

Same error for ALBERT v1 models.

## Expected behavior

Max absolute difference between models outputs should be <= 2e-2

## Environment info

Observed on both of the following environments: 

- `transformers` version: 2.5.1
- Platform: Darwin-19.3.0-x86_64-i386-64bit
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
<!-- -->
- `transformers` version: 2.5.1
- Platform: Linux-4.15.0-58-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Proposed a fix that didn't work out-of-the-box with official ALBERT models. Still looking into it, will keep you posted.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Closing since it seems solved in #4076. Thank you!