Does torch.squeeze() affect the gradients?

arjung: <p>I modified the WaveNet architecture for binary classification:</p>
<pre><code class="lang-auto">class CustomConv(nn.Module):
    def __init__(self):
        super(CustomConv, self).__init__()
        
        self.conv1 = nn.Conv1d(in_channels=1, out_channels=256, kernel_size=3, padding=1)
        self.blocks = self.build_conv_block(num_blocks, 256)
        self.conv2 = nn.Conv1d(in_channels=256, out_channels=128, kernel_size=1)
        self.conv3 = nn.Conv1d(in_channels=128, out_channels=1, kernel_size=1)
        self.act = nn.ReLU()
        self.linear = nn.Linear(2000, 1)
        self.sigmoid = nn.Sigmoid()

    def build_conv_block(self, num_layers, num_channels):
        block = []
        for _ in range(num_layers):
            for i in range(12):
                block.append(ConvBlock(num_channels, 2**i))
        return nn.Sequential(*block)

    def forward(self, x):
        x = self.conv1(x)
        x = self.act(x)
        _,x = self.blocks((x,0))
        x = self.act(x)
        x = self.conv2(x)
        x = self.act(x)
        x = self.conv3(x)
        x = torch.squeeze(x)
        x = self.linear(x)
        x = self.sigmoid(x)

        return x
</code></pre>
<p>My model hasn’t been training very well, and I was wondering if <code>torch.squeeze()</code> could have any (negative) effect on the gradients produced during backprop, or any effect on training in general. Is there better practice for something like this? Any ideas?</p>

ptrblck: <p>Usually you would flatten the activation via</p>
<pre><code class="lang-python">x = x.view(x.size(0), -1)
</code></pre>
<p>to create a 2-dimensional tensor out of the 4-dimensional conv output.<br>
If you are using <code>torch.squeeze(x)</code>, this will remove a variable size of dimensions.<br>
Assuming that you are using a <code>batch_size&gt;1</code>, at least the channel dimension will be removed.<br>
This would yield an activation of <code>[batch_size, h, w]</code>, which is then passed to the linear layer.<br>
<code>nn.Linear</code> takes an input as <code>[batch_size, *, in_features]</code>, where <code>*</code> denotes additional dimensions. The linear layer will be applied on each of these additional dimensions and I’m not sure, if that’s what you really want in this case.</p>
<p>Could you replace the squeeze with the <code>view</code> operation and check the results again?</p>

arjung: <p>Thanks for your response!</p>
<p>I think you’re right – <code>view</code> is better than <code>squeeze</code> in this case as my <code>batch_size&gt;1</code>. So thank you for pointing this out.</p>
<p>When I retrained the model with <code>view</code>, interestingly enough, I got the same accuracy as what I observed with <code>squeeze</code>. I’m not sure why this was the case (any ideas?), but nonetheless thank you for helping me make my code better reflect my intentions.</p>

ptrblck: <p>Are you using <code>nn.BCELoss</code> as the criterion?</p>
<p>I would assume the model to work quite differently if you pass a 2-dimensional tensor vs. a 3-dimensional one into <code>nn.Linear</code>.</p>

arjung: <p>Yes, I am using <code>nn.BCELoss</code> as the criterion.</p>
<p>Yeah, I’m not sure why the model gives me the same accuracy. Perhaps I made a mistake – let me double check and retrain both models and get back to you if the accuracies vary.</p>