Convert pytorch models back to tensorflow
Author: chrislarson1
Pull request: yes

Added a file that converts pytorch models that have been trained/finetuned back to tensorflow. This currently supports the base BERT models (uncased/cased); conversion for other BERT models will be added in the future.

Comments:
Changed filename from convert_hf_checkpoint_to_tf.py to convert_pytorch_checkpoint_to_tf.py for consistency.

I use this to convert the fine-tuned pytorch model to TF and convert this converted TF back to pytorch model. The prediction result seems incorrect with the converted-converted pytorch model.  

I compare the dumped stat_dict, it seems the differences lie in :

encoder.layer.{d}.attention.self.query.weight
encoder.layer.{d}.attention.self.key.weight
encoder.layer.{d}.attention.self.value.weight

What model are you trying to convert @Qiuzhuang? Per the docstring, only the BertModel is currently supported. I will add support other models in the near future.

Hi @chrislarson1, I use BertModel as follows:
BertModel(config).from_pretrained(pretrain_model_dir)
where pretrain_model_dir is (domain-pretraining + task specific classifier) training model dir.

I write the sample test code read the pytorch model and tf-pytorch model via BertModel.

![1558888271296](https://user-images.githubusercontent.com/334695/58384647-bec92580-8016-11e9-8a60-3e86b738542b.jpg)


@chrislarson1 we need to transpose attention Q/K/V weight as well,  here is the fixing:

if any(attention_weight in var_name for attention_weight in ["dense.weight", "attention.self.query", "attention.self.key", "attention.self.value"]):
  torch_tensor = torch_tensor.T
  tf_tensor = assign_tf_var(tensor=torch_tensor, name=tf_name)

I am able to train pytorch fine-tuned model and then convert to tensorflow model for serving purpose. E.g. using bert-as-service. The results are consistent.

Thanks @Qiuzhuang, your change has been added.

Thanks, @chrislarson1 and @Qiuzhuang.
This feature in interesting indeed.
I think it could be nice to add a check that output of the PyTorch and TensorFlow models are identical (on at least one example). For the Bert model, I did a simple notebook (see the notebook folder) but it can also be a script or a test.
Do you think you can add something like that @chrislarson1 ?

# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620?src=pr&el=h1) Report
> Merging [#620](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/3763f8944dc3fef8afb0c525a2ced8a04889c14f?src=pr&el=desc) will **decrease** coverage by `0.79%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master     #620     +/-   ##
=========================================
- Coverage   68.23%   67.43%   -0.8%     
=========================================
  Files          18       19      +1     
  Lines        3976     4023     +47     
=========================================
  Hits         2713     2713             
- Misses       1263     1310     +47
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...retrained\_bert/convert\_pytorch\_checkpoint\_to\_tf.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvY29udmVydF9weXRvcmNoX2NoZWNrcG9pbnRfdG9fdGYucHk=) | `0% <0%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620?src=pr&el=footer). Last update [3763f89...716cc1c](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/620?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


No problem @thomwolf, I've added a notebook that checks correctness.

Ok great, let's merge it!

Hi, 

I am trying to write a converter for RobertaForSequenceClassification to tensorflow using this script as a guide. I had a question regarding this.

Why did we take a transpose of the layers here at all? Is it because of tensorflow treating its layers differently than pytorch?

Also, if the dense.weight layers are being transposed, then in s sequence classification model, will the out_proj layer need to be transposed as well?

Hi @justachetan, the answer lies in how linear transformations are represented in pytorch vs. tensorflow; they are not the same. In Pytorch, weights in a network **often** get wrapped in the torch.nn.Linear class, which store transposed versions of the weights that would get saved in tensorflow for an equivalent projection (see the example below).

```
>>> import numpy as np
>>> import tensorflow as tf
>>> import torch
>>> from torch.functional import F
>>> x = np.ones([10, 4])
>>> W = np.ones([5, 4])
>>> tf.matmul(x, W.T).shape
(10, 5)
>>> F.linear(torch.Tensor(x), torch.Tensor(W)).detach().numpy().shape
(10, 5)
```