Huge Drop of Accuracy when Converting TF Model to PyTorch Model

username: arianna
cooked: <p>I am trying to convert a pre-trained TensorFlow model to PyTorch and evaluate it. I import the TensorFlow model and then convert each layer into Numpy arrays. My PyTorch model has the exact same set up except all the layer shapes are flipped. I convert each Numpy array into the appropriate shape (using the torch permute function) and insert it into my PyTorch model, but the accuracy is nowhere near what it was on TensorFlow. I have a feeling it has something to do with the flipped shapes, but I’m not quite sure how to fix it. Is there something I’m missing?</p>

username: gphilip
cooked: <p>What do you mean when you say that all the layer shapes are flipped? Can you give a toy example with a couple of layers?</p>

username: arianna
cooked: <p>Here is an example of the shapes of the same layer in mobileNetV2 for the tf vs pytorch model:<br>
Tensorflow: Conv1/kernel:0: (3, 3, 3, 32)<br>
Pytorch: features.0.0.weight torch.Size([32,3,3,3])</p>

username: arianna
cooked: <p>Most of the layers in Tensorflow are the reverse of the Pytorch layers, except the depthwise layers. Those are (2,3,0,1). For example:<br>
Tensorflow: expanded_conv_depthwise/depthwise_kernel:0: (3, 3, 32, 1)<br>
Pytorch: features.1.conv.0.0.weight torch.Size([32,1,3,3])</p>

username: gphilip
cooked: <p>What are you using to do this import from TensorFlow to PyTorch?</p>

username: FeelTheOatmeal
cooked: <p><a class="mention" href="/u/gphilip">@gphilip</a><br>
We are importing a TensorFlow model, going layer by layer and converting each layer’s Tensor into a numpy array. Then we take each numpy array, convert it to a torch tensor and permute it to the “correct” shape.</p>

username: gphilip
cooked: <p>The way in which TensorFlow implements a certain type of layer may be drastically different from the way in which PyTorch implements the same layer. Permuting the indices till they look like the correct shape, may not be enough.</p>
<p>If you want to do this import on your own, you may have to drill deep into the innards of both libraries to see how each type of layer that you are interested in, translates to tensors in each library.</p>

username: FeelTheOatmeal
cooked: <p>Can you explain why this may be?<br>
The model we are using is mobilenetv1, we made sure that the ordering of layers is exactly the same, names of layers correspond to each other etc. A conv2d layer should be a Conv2d layer, a depthwise conv2d is a depthwise conv2d in both. They should produce the same output on both frameworks regardless of code implementation because the operation definition is the same. Can you maybe give an example as to how this might not be the case?</p>

username: gphilip
cooked: <p>I am not saying that this <em>must</em> be the case, just that this <em>may</em> be the case. From what I understand from this conversation, you already noticed that the shapes of layers are flipped across the two libraries. If the layers are implemented in essentially the same way in both libraries, why would such a non-trivial difference exist?</p>
<p>So—if I understood you correctly—you have discovered one type of mismatch between the two libraries’ implementation of the selfsame thing. Given this, all I am saying is that it would be prudent to <em>not</em> assume that things are nearly the same across the two libraries. If you wish to do a thorough job of the porting, I feel you should cross-check the implementation of each type of layer in either library.</p>

username: FeelTheOatmeal
cooked: <p>Thats a fair point, was hoping it would be an easy port but will try this when we get some more time.</p>

username: gphilip
cooked: <p>You may want to take a look at <a href="https://onnx.ai/" rel="noopener nofollow ugc">ONNX</a>. I found a <a href="https://github.com/onnx/tutorials/blob/master/tutorials/PytorchTensorflowMnist.ipynb" rel="noopener nofollow ugc">tutorial for converting a PyTorch model to TensorFlow</a> using ONNX. The other direction should also be possible, though I couldn’t find an example in five minutes of searching.</p>

username: FeelTheOatmeal
cooked: <p>Will take a look definitely cool! But the goal was really to do it from scratch. Maybe I can look at ONNX’s internals for how it converts them</p>

username: sans_sense
cooked: <p>Conv2d and BatchNorm need some attention while porting. I posted about the constant eps value handling in my <a href="https://discuss.pytorch.org/t/porting-batch-normalization-from-tf-to-pytorch/116790">message</a>  about porting batch normalization. I have ported resnet architectures from Tensorflow to Pytorch and there is no loss, the errors mostly creep in because of constants like eps values, shapes, stride implementation differences as I mentioned in this <a href="https://discuss.pytorch.org/t/convnd-pick-2nd-instead-of-1st-element-for-stride-2/116788">message</a>. In my experience flipping is the NCH and NHC convention (N being batch, C being channel and H being height), if you flip everything while reading from TF model and setting it in Torch, it will work.</p>