Convert Tensorflow Model to PyTorch

username: k2lin
cooked: <p>Hi</p>
<p>I’m porting a Tensorflow model to PyTorch in order to use it as an initialization.<br>
It mainly consists of Conv3D and LayerNorm.<br>
I have copied the weights to my PyTorch model, but I am not sure if I have done it correctly.</p>
<p>I followed this guide:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://cdn-static-1.medium.com/_/fp/icons/favicon-rebrand-medium.3Y6xpZ-0FSdWDnPM3hSBIA.ico" class="site-icon" width="32" height="32">
      <a href="https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28" target="_blank" title="04:14PM - 10 August 2019" rel="nofollow noopener">Medium – 10 Aug 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/333;"><img src="https://miro.medium.com/max/1200/1*KfD4ZOtBktmZcbBNNa6l4g.png" class="thumbnail"></div>

<h3><a href="https://medium.com/huggingface/from-tensorflow-to-pytorch-265f40ef2a28" target="_blank" rel="nofollow noopener">🌓 From TensorFlow to PyTorch</a></h3>

<p>Friends and users of our open-source tools are often surprised how fast 🚀 we reimplement the latest SOTA pretrained TensorFlow models to…</p>

  <p><span class="label1">Reading time: 7 min read</span>
    </p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>And did the following:</p>
<ol>
<li>Load the weights with the following code and find their corresponding layers.</li>
</ol>
<pre><code class="lang-auto">    tf_vars = []
    init_vars = tf.train.list_variables(ckpt_path)
    for name, shape in init_vars:
        array = tf.train.load_variable(ckpt_path, name)
        tf_vars.append((name, array.squeeze()))
</code></pre>
<ol start="2">
<li>
<p>Assign the weights to the corresponding Conv3D layers<br>
And permute them with [4, 3, 0, 1, 2]<br>
My knowledge is that PyTorch stores weights in [out_channels, in_channels, depth, height, width] order.<br>
And Tensorflow stores weights in [depth, height, width, in_channels, out_channels] order.<br>
So I first permute the Tensorflow array to fit the PyTorch format.</p>
</li>
<li>
<p>Use GroupNorm with only 1 group to simulate LayerNorm’s behavior in Tensorflow<br>
Because LayerNorm in PyTorch acts a bit weird for images, I use GroupNorm’s implementation instead.<br>
The weights (gamma) and bias (beta) are assigned accordingly.</p>
</li>
</ol>
<p>Are the above correct? If so, then I think the problem would be in the implementation.<br>
If not, please let me know which is incorrect.<br>
Thanks in advance.</p>

username: k2lin
cooked: <p>I was able to get a reasonable difference (around 1e-5) between the first-layer outputs of both framework by changing the permute to [4, 3, 2, 0, 1].<br>
But the second layer generates different outputs.<br>
I have checked that GroupNorm with group 1 is really close to LayerNorm.<br>
The layer just looks like this:</p>
<pre><code class="lang-auto">        self.conv1_1 = nn.Sequential(
            nn.Conv3d(in_c, nf, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.GroupNorm(1, nf)
        )
        self.conv1_2 = nn.Sequential(
            nn.Conv3d(nf, nf*2, kernel_size=3, stride=2, padding=1),
            nn.ReLU(inplace=True),
            nn.GroupNorm(1, nf*2)
        )
</code></pre>
<p>The big difference happens at the output of conv1_2.<br>
Does anyone know how to fix this?</p>