TensorFlow/Keras to PyTorch translation

username: mavewrik
cooked: <p>Hello, I am trying to translate a TensorFlow/Keras model to PyTorch, but specifically facing an issue in writing the PyTorch equivalent of a Dense layer in the TensorFlow/Keras code.</p>
<p>original TensorFlow/Keras code</p>
<pre><code class="lang-auto">inputs = (256, 256, 1)

model = Sequential()
# encoder
model.add(Convolution2D(32, (3,3), input_shape=inputs, \
                        activation='relu', padding='same'))
model.add(MaxPooling2D((2,2), padding='same'))
model.add(Convolution2D(64, (3,3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2), padding='same'))
model.add(Convolution2D(128, (3,3), activation='relu', padding='same'))
model.add(MaxPooling2D((2,2), padding='same'))

model.add(Dense(128, activation='relu'))

# decoder
model.add(UpSampling2D((2,2)))
model.add(Convolution2D(128, (3,3), activation='sigmoid', padding='same'))
model.add(UpSampling2D((2,2)))
model.add(Convolution2D(64, (3,3), activation='sigmoid', padding='same'))
model.add(UpSampling2D((2,2)))
model.add(Convolution2D(1, (3,3), activation='sigmoid', padding='same'))
</code></pre>
<p>and the TensorFlow/Keras model summary is as follows:</p>
<pre><code class="lang-auto">Model: "sequential"
_________________________________________________________________
 Layer (type)                       Output Shape          Param #   
=================================================================
 conv2d (Conv2D)                  (None, 256, 256, 32)      320       
                                                                 
 max_pooling2d (MaxPooling2D)     (None, 128, 128, 32)      0                                                                 
                                                                 
 conv2d_1 (Conv2D)                (None, 128, 128, 64)      18496     
                                                                 
 max_pooling2d_1 (MaxPooling 2D)  (None, 64, 64, 64)        0                                                 
                                                                 
 conv2d_2 (Conv2D)                (None, 64, 64, 128)       73856     
                                                                 
 max_pooling2d_2 (MaxPooling 2D)  (None, 32, 32, 128)       0                          
                                                                 
 dense (Dense)                    (None, 32, 32, 128)       16512     
                                                                 
 up_sampling2d (UpSampling2D)     (None, 64, 64, 128)       0                                  
                                                                 
 conv2d_3 (Conv2D)                (None, 64, 64, 128)       147584    
                                                                 
 up_sampling2d_1 (UpSampling 2D)  (None, 128, 128, 128)     0                                
                                                                 
 conv2d_4 (Conv2D)                (None, 128, 128, 64)      73792     
                                                                 
 up_sampling2d_2 (UpSampling  2D) (None, 256, 256, 64)      0         
                                                                 
 conv2d_5 (Conv2D)                (None, 256, 256, 1)       577       
                                                                 
=================================================================
Total params: 331,137
Trainable params: 331,137
Non-trainable params: 0
_________________________________________________________________

</code></pre>
<p>My attempt at writing the PyTorch equivalent is here:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        
        # encoder
        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, padding='same')
        self.pool1 = nn.MaxPool2d(kernel_size=2, padding=0)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding='same')
        self.pool2 = nn.MaxPool2d(kernel_size=2, padding=0)
        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding='same')
        self.pool3 = nn.MaxPool2d(kernel_size=2, padding=0)
        
        # fully-connected
        self.fc1 = nn.Linear(128*16*16, 128)
        
        # decoder
        self.up1 = nn.Upsample(scale_factor=2)
        self.conv4 = nn.Conv2d(128, 128, kernel_size=3, padding='same')
        self.up2 = nn.Upsample(scale_factor=2)
        self.conv5 = nn.Conv2d(128, 64, kernel_size=3, padding='same')
        self.up3 = nn.Upsample(scale_factor=2)
        self.conv6 = nn.Conv2d(64, 1, kernel_size=3, padding='same')
        
        # calculate output sizes
        self.conv1_out = get_output_shape(self.conv1, (1, 1, 256, 256))
        self.pool1_out = get_output_shape(self.pool1, self.conv1_out)
        self.conv2_out = get_output_shape(self.conv2, self.pool1_out)
        self.pool2_out = get_output_shape(self.pool2, self.conv2_out)
        self.conv3_out = get_output_shape(self.conv3, self.pool2_out)
        self.pool3_out = get_output_shape(self.pool3, self.conv3_out)
        
        self.fc1_out = get_output_shape(self.fc1, self.pool3_out)
        
        # self.up1_out = get_output_shape(self.up1, self.fc1_out)
        # self.conv4_out = get_output_shape(self.conv4, self.up1_out)
        # self.up2_out = get_output_shape(self.up2, self.conv4_out)
        # self.conv5_out = get_output_shape(self.conv5, self.up2_out)
        # self.up3_out = get_output_shape(self.up3, self.conv5_out)
        # self.conv6_out = get_output_shape(self.conv6, self.up3_out)
        
    
    def forward(self, x):
        
        # encoder activations
        x = F.relu(self.conv1(x))
        x = self.pool1(x)
        x = F.relu(self.conv2(x))
        x = self.pool2(x)
        x = F.relu(self.conv3(x))
        x = self.pool3(x)
        
        # x = x.view(x.size(0), -1)
        # fully-connected activation
        x = F.relu(self.fc1(x))
        
        # decoder activations
        x = self.up1(x)
        x = F.sigmoid(self.conv4(x))
        x = self.up2(x)
        x = F.sigmoid(self.conv5(x))
        x = self.up3(x)
        x = F.sigmoid(self.conv6(x))
</code></pre>
<p>I am getting a dimension mismatch error in the Linear layer of PyTorch code I’ve written. When I do a <code>print(model)</code> for an instance of the PyTorch class I’ve written, I get the following:</p>
<pre><code class="lang-auto">Net(
  (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (conv3): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (pool3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (fc1): Linear(in_features=32768, out_features=128, bias=True)
  (up1): Upsample(scale_factor=2.0, mode=nearest)
  (conv4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (up2): Upsample(scale_factor=2.0, mode=nearest)
  (conv5): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1), padding=same)
  (up3): Upsample(scale_factor=2.0, mode=nearest)
  (conv6): Conv2d(64, 1, kernel_size=(3, 3), stride=(1, 1), padding=same)
)
</code></pre>
<hr>
<p>Some side info: While debugging, I found a StackOverflow post which had a code snippet to print output size of each layer for the PyTorch code, and I’m mentioning it below, if required:</p>
<pre><code class="lang-auto">def get_output_shape(model, image_dim):
    return model(torch.rand(*(image_dim))).data.shape
</code></pre>
<p>Now, using this code snippet above, I get the output sizes of the first three conv-pool pairs correct for the PyTorch code I’ve written (it can be verified from the TensorFlow/Keras model summary above, and also keep in mind that in PyTorch the number of kerne;s comes first and then the spatial dimension, while it is the opposite in TensorFlow/Keras) :</p>
<pre><code class="lang-auto">torch.Size([1, 32, 256, 256])
torch.Size([1, 32, 128, 128])

torch.Size([1, 64, 128, 128])
torch.Size([1, 64, 64, 64])

torch.Size([1, 128, 64, 64])
torch.Size([1, 128, 32, 32])
</code></pre>
<p>However, it gives me an error of dimension mismatch while trying to find output size of the <code>fc1</code> layer in PyTorch, as follows:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_10614/273569339.py in &lt;module&gt;
----&gt; 1 model = Net()
      2 
      3 print(model)

/tmp/ipykernel_10614/1565241974.py in __init__(self)
     32         self.pool3_out = get_output_shape(self.pool3, self.conv3_out)
     33 
---&gt; 34         self.fc1_out = get_output_shape(self.fc1, self.pool3_out)
     35 
     36         # self.up1_out = get_output_shape(self.up1, self.fc1_out)

/tmp/ipykernel_10614/1833495027.py in get_output_shape(model, image_dim)
      1 def get_output_shape(model, image_dim):
----&gt; 2     return model(torch.rand(*(image_dim))).data.shape

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1188         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1190             return forward_call(*input, **kwargs)
   1191         # Do not call functions when jit is used
   1192         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input)
    112 
    113     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 114         return F.linear(input, self.weight, self.bias)
    115 
    116     def extra_repr(self) -&gt; str:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (4096x32 and 32768x128)
</code></pre>
<p>Any help would be appreciated.</p>
<hr>
<p>EDIT: I got to know that Keras does implicit flattening before sending it to Dense, and that is an error in my PyTorch code, but even after adding the flattening code as follows, I get a dimension mismatch error when trying to run torch-summary.</p>
<p>flattening code (goes before <code>self.fc1 = nn.Linear(...)</code>)</p>
<pre><code class="lang-auto">x = x.view(x.size(0), -1)
</code></pre>
<p>dimension mismatch error when trying to get model summary using torch-summary</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/torchsummary/torchsummary.py in summary(model, input_data, batch_dim, branching, col_names, col_width, depth, device, dtypes, verbose, *args, **kwargs)
    139             with torch.no_grad():
--&gt; 140                 _ = model.to(device)(*x, *args, **kwargs)  # type: ignore[misc]
    141         except Exception as e:

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1189                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1190             return forward_call(*input, **kwargs)
   1191         # Do not call functions when jit is used

/tmp/ipykernel_149/842565426.py in forward(self, x)
     57         # fully-connected activation
---&gt; 58         x = F.relu(self.fc1(x))
     59 

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1207 
-&gt; 1208         result = forward_call(*input, **kwargs)
   1209         if _global_forward_hooks or self._forward_hooks:

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input)
    113     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 114         return F.linear(input, self.weight, self.bias)
    115 

RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x131072 and 32768x128)

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_149/566945719.py in &lt;module&gt;
----&gt; 1 summary(model, (1,256,256))

/opt/conda/lib/python3.7/site-packages/torchsummary/torchsummary.py in summary(model, input_data, batch_dim, branching, col_names, col_width, depth, device, dtypes, verbose, *args, **kwargs)
    144                 "Failed to run torchsummary. See above stack traces for more details. "
    145                 "Executed layers up to: {}".format(executed_layers)
--&gt; 146             ) from e
    147         finally:
    148             if hooks is not None:

RuntimeError: Failed to run torchsummary. See above stack traces for more details. Executed layers up to: [Conv2d: 1-1, MaxPool2d: 1-2, Conv2d: 1-3, MaxPool2d: 1-4, Conv2d: 1-5, MaxPool2d: 1-6]
</code></pre>

username: J_Johnson
cooked: <p>I can’t speak as to what the TF model should or shouldn’t do. That said, your Pytorch model has 3x MaxPool2d layers that are cutting the size in half per dim. And your input size is 256x256. Also, you have 128 channels coming out of your CNN encoder.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e10a31b8e954f73b386dc14a88b0e6b021ae9fc9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e10a31b8e954f73b386dc14a88b0e6b021ae9fc9" title="Screenshot_2023-03-05-23-25-15-55_3e99d9242afcaf161dfb7a0d3002274f"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e10a31b8e954f73b386dc14a88b0e6b021ae9fc9_2_225x500.jpeg" alt="Screenshot_2023-03-05-23-25-15-55_3e99d9242afcaf161dfb7a0d3002274f" data-base62-sha1="w6NgR7pd7qUyHCUUSIMWAQrKbbP" width="225" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e10a31b8e954f73b386dc14a88b0e6b021ae9fc9_2_225x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e10a31b8e954f73b386dc14a88b0e6b021ae9fc9_2_337x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e10a31b8e954f73b386dc14a88b0e6b021ae9fc9_2_450x1000.jpeg 2x" data-dominant-color="85817A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2023-03-05-23-25-15-55_3e99d9242afcaf161dfb7a0d3002274f</span><span class="informations">1080×2400 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So it’s giving the correct output size for the Pytorch version. You can either adjust your Linear layer size by 4x or run through one more Conv2d + MaxPool2d layer.</p>

username: J_Johnson
cooked: <aside class="quote no-group" data-username="mavewrik" data-post="1" data-topic="174083">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mavewrik/48/57990_2.png" class="avatar"> mavewrik:</div>
<blockquote>
<pre><code class="lang-auto">Model: "sequential"
_________________________________________________________________
 Layer (type)                       Output Shape          Param #   
=================================================================
 conv2d (Conv2D)                  (None, 256, 256, 32)      320       
                                                                 
 max_pooling2d (MaxPooling2D)     (None, 128, 128, 32)      0                                                                 
                                                                 
 conv2d_1 (Conv2D)                (None, 128, 128, 64)      18496     
                                                                 
 max_pooling2d_1 (MaxPooling 2D)  (None, 64, 64, 64)        0                                                 
                                                                 
 conv2d_2 (Conv2D)                (None, 64, 64, 128)       73856     
                                                                 
 max_pooling2d_2 (MaxPooling 2D)  (None, 32, 32, 128)       0                          
                           
 dense (Dense)                    (None, 32, 32, 128)       16512     
         
</code></pre>
</blockquote>
</aside>
<p>I had another glance at your TF model print out. They are using 128 in_features. So what’s likely happening is something like the following:</p>
<pre><code class="lang-auto">self.fc1 = nn.Linear(128, 128)
...
#forward pass 

x = self.pool3(x)

N, c, h, w = x.shape
x = x.reshape(N, c, h*w).permute(0,2,1)

x = F.relu(self.fc1(x))

x = x.permute(0,2,1).reshape(N, c, h, w)
</code></pre>

username: mavewrik
cooked: <p>Thank you, this works and I get the following PyTorch model summary using torch-summary:</p>
<pre><code class="lang-auto">==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
├─Conv2d: 1-1                            [-1, 32, 256, 256]        320
├─MaxPool2d: 1-2                         [-1, 32, 128, 128]        --
├─Conv2d: 1-3                            [-1, 64, 128, 128]        18,496
├─MaxPool2d: 1-4                         [-1, 64, 64, 64]          --
├─Conv2d: 1-5                            [-1, 128, 64, 64]         73,856
├─MaxPool2d: 1-6                         [-1, 128, 32, 32]         --
├─Linear: 1-7                            [-1, 1024, 128]           16,512
├─Upsample: 1-8                          [-1, 128, 64, 64]         --
├─Conv2d: 1-9                            [-1, 128, 64, 64]         147,584
├─Upsample: 1-10                         [-1, 128, 128, 128]       --
├─Conv2d: 1-11                           [-1, 64, 128, 128]        73,792
├─Upsample: 1-12                         [-1, 64, 256, 256]        --
├─Conv2d: 1-13                           [-1, 1, 256, 256]         577
==========================================================================================
Total params: 331,137
Trainable params: 331,137
Non-trainable params: 0
Total mult-adds (G): 2.47
==========================================================================================
Input size (MB): 0.25
Forward/backward pass size (MB): 41.50
Params size (MB): 1.26
Estimated Total Size (MB): 43.01
==========================================================================================
</code></pre>
<p>But a follow-up question: the output dimension for the TF model for the Dense layer is <code>(None, 32, 32, 128)</code>, however for the PyTorch model’s Linear layer is <code>[-1, 1024, 128]</code>. I don’t understand why.</p>

username: J_Johnson
cooked: <aside class="quote no-group" data-username="mavewrik" data-post="4" data-topic="174083">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mavewrik/48/57990_2.png" class="avatar"> mavewrik:</div>
<blockquote>
<p>But a follow-up question: the output dimension for the TF model for the Dense layer is <code>(None, 32, 32, 128)</code>, however for the PyTorch model’s Linear layer is <code>[-1, 1024, 128]</code>. I don’t understand why.</p>
</blockquote>
</aside>
<p>32 x 32 = 1024</p>
<p>After the Linear layer matmul and bias addition operations are complete, the code in my previous reply permutes the H x W dim back to the end, before reshaping them to separate dims to feed into the decoder CNN.</p>
<p>When you run a tensor through a Pytorch Linear layer, it will <a href="https://mathinsight.org/matrix_vector_multiplication#:~:text=To%20define%20multiplication%20between%20a,number%20of%20rows%20in%20x." rel="noopener nofollow ugc">matmul</a> the last dim with the weights, while all other dims are effectively treated as batches. This conforms with how I’ve seen Google’s research team(who primarily use TF) handle operations in their ViT paper for the initial patchify stem.</p>