Parameters almost same in Depthwise Separable Convolution (when compared to a standard CNN)?

username: Waqas
cooked: <p>Hi,</p>
<p>I am new to using PyTorch and really like the Pythonic approach it offers. Currently I’m trying to do a Neural Net performance comaprison between a simple Depthwise Separable Convolutional Neural Net and a standard Convolutional Neural net. The model architecture of both is simple:</p>
<p>The following are done 3 times<br>
Convolution-&gt; Batch Normalization-&gt; ReLU -&gt; Pool<br>
Followed by flattening it to 3 outputs in two steps.</p>
<p>The code is as following:</p>
<p><strong>Standard Conv. Net architecture:</strong></p>
<pre><code class="lang-auto">in_features = 304  #in_features for Flatten(linear) layer

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 4, kernel_size=2, padding=1)
        self.bn1 = nn.BatchNorm2d(num_features=4)
        self.relu1 = nn.ReLU()
        self.pool1 = nn.MaxPool2d(2)        
        
        self.conv2 = nn.Conv2d(4, 8, kernel_size=2, padding=1)
        self.bn2 = nn.BatchNorm2d(num_features=8)
        self.relu2 = nn.ReLU()
        self.pool2 = nn.MaxPool2d(2)        

        self.conv3 = nn.Conv2d(8, 16, kernel_size=2, padding=1)
        self.bn3 = nn.BatchNorm2d(num_features=16)
        self.relu3 = nn.ReLU()
        self.pool3 = nn.MaxPool2d(2)        

        self.fc1   = nn.Linear(in_features, 36)
        self.relu4 = nn.ReLU()
        self.fc2   = nn.Linear(36, 3)
        
    def forward(self, x):
        out = self.pool1(self.relu1(self.bn1(self.conv1(x))))
        out = self.pool2(self.relu2(self.bn2(self.conv2(out))))
        out = self.pool3(self.relu3(self.bn3(self.conv3(out))))
        
        out = out.view(out.size(0), -1)
        out = self.relu4(self.fc1(out))
        out = self.fc2(out)

        return out

standardCNN = Net()  # defining an instance of our network
</code></pre>
<p><strong>Depthwise Separable Conv. Net architecture:</strong></p>
<pre><code class="lang-auto">class depthwise_separable_conv(nn.Module):
    def __init__(self):
        super(depthwise_separable_conv, self).__init__()
        self.depthwise1 = nn.Conv2d(1, 1, kernel_size=2, padding=1, groups=1)
        self.pointwise1 = nn.Conv2d(1, 4, kernel_size=1)
        self.bn1 = nn.BatchNorm2d(num_features=4)
        self.relu1 = nn.ReLU()
        self.pool1 = nn.MaxPool2d(2)        
        
        self.depthwise2 = nn.Conv2d(4, 4, kernel_size=2, padding=1, groups=4)
        self.pointwise2 = nn.Conv2d(4, 8, kernel_size=1)
        self.bn2 = nn.BatchNorm2d(num_features=8)
        self.relu2 = nn.ReLU()
        self.pool2 = nn.MaxPool2d(2)
        
        self.depthwise3 = nn.Conv2d(8, 8, kernel_size=2, padding=1, groups=8)
        self.pointwise3 = nn.Conv2d(8, 16, kernel_size=1)
        self.bn3 = nn.BatchNorm2d(num_features=16)
        self.relu3 = nn.ReLU()
        self.pool3 = nn.MaxPool2d(2)        
        
        self.dense1 = nn.Linear(304, 36)
        self.relu4 = nn.ReLU()
        self.dense2 = nn.Linear(36,3)
        
    def forward(self, x):
        out = self.pool1(self.relu1(self.bn1(self.pointwise1(self.depthwise1(x)))))
        out = self.pool2(self.relu2(self.bn2(self.pointwise2(self.depthwise2(out)))))
        out = self.pool3(self.relu3(self.bn3(self.pointwise3(self.depthwise3(out)))))
        
        out = out.view(out.size(0), -1)
        out = self.relu4(self.dense1(out))
        out = self.dense2(out)
        
        return out

dsCNN_model = depthwise_separable_conv()
</code></pre>
<p>Now, here’s the important part. As the Standard Convolutions require a large number of parameters as compared to depthwise, I was hoping to get reduced number of parameters for my <code>dsCNN_model</code>. But to my surprise, I got these results.</p>
<p><strong>Parameters:</strong></p>
<pre><code class="lang-auto">total_params = 0
for parameter in standardCNN.parameters():
    if parameter.requires_grad:
        total_params += np.prod(parameter.size())
print(total_params)
&gt;&gt;11831

total_params = 0
for parameter in dsCNN_model.parameters():
    blah blah...        
print(total_params)
&gt;&gt;11404
</code></pre>
<p>Both having almost the same with a mere reduction of <strong>400</strong> parameters (from ~11800 to ~11400).</p>
<p>I wanted to ask, is it that this is normal behavior of Depthwise model and we do not see that much of a differnce in smaller models? Or is there something wrong with my approach, or in the model architecture that is causing this? I’m actually trying to get a model with least parameters so the inference is possible on a hardware with less computational resources. Also, what other approaches are there in PyTorch’s arsenal that can be used for this purpose.</p>
<p>Any help would be really appreciated… Thanks!</p>

username: Eta_C
cooked: <p>kernel_size <strong>K</strong>, in_channels <strong>M</strong>, out_channels <strong>N</strong>, assume that you have no bias.<br>
naive convolution has <strong>KKMN</strong> parameters.<br>
separable convolution has <strong>KKM + MN</strong> parameters.</p>
<p>Let me try to caculate <strong>KKMN - (KKM + MN)</strong>.</p>
<pre><code class="lang-python">nn.Conv2d(1, 4, kernel_size=2, padding=1)
</code></pre>
<p>2x2x4x1 - 2x2x4 - 4x1 = -4</p>
<pre><code class="lang-python">nn.Conv2d(4, 8, kernel_size=2, padding=1)
</code></pre>
<p>2x2x8x4 - 2x2x8 - 8x4 = 64</p>
<pre><code class="lang-python">nn.Conv2d(8, 16, kernel_size=2, padding=1)
</code></pre>
<p>2x2x16x8 - 2x2x16 - 16x8 = 320</p>
<p>Finally… (-4 + 64 + 320) = 380, maybe I’m wrong, but the value is really small…</p>

username: Waqas
cooked: <p>Thanks a bunch for this explanation!! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<p>A couple of things I would ask…</p>
<p>Correct me if I’m wrong here. So, putting it in mathematical terms. If I want this difference to be large,</p>
<p><strong>KKMN - (KKM + MN)</strong>  = KKMN - M(KK + N) = <strong>M(KKN - KK - N)</strong></p>
<p>I should have a large <strong>M</strong>, and should try other values for <strong>K</strong> and <strong>N</strong>. This is the correct approach, right?</p>
<p>And second, although doing this might give me a large difference, but possibly will take me away from my main objective of having least number of params to begin with. If that is correct, can you shed some light on how can I tweak PyTorch model architecture to have a model with fewer params (because I had a model on Keras with same structure and had about 3000 paramters)? Or is having fewer parameters even a good metric for a model to utilize less resources during inference?</p>

username: Eta_C
cooked: <p>Some networks, such as MobileNet and ResNet, have <strong>N = 512</strong> or <strong>N = 256</strong>.<br>
Assume that <strong>K = 2</strong> and <strong>N = M/2</strong>, we have<br>
<strong>KKMN - (KKM + MN) = 4MM/2 - (4M + MM/2) = 1.5MM - 4M</strong>.<br>
What a big number it is!</p>

username: Waqas
cooked: <p>Right, that makes a lot of sense…<br>
Thanks again!</p>

username: solvingPuzzles
cooked: <p>Also: the majority of your parameters are in this one layer:</p>
<p>self.dense1 = nn.Linear(304, 36)</p>
<p>(304 in_channels) * (36 out_channels) = 10944</p>
<p>So, you might consider averagepooling to 1x1xChannels right before dense1. And, make dense1 a convolution with 304 in_channels, 36 out_channels, kernel_size of 1, and groups=32 or 64 or whatever number you like. That ought to bring your parameter count way down.</p>

username: Waqas
cooked: <p>WHOA!! That really did work…<br>
I have my parameters down from <strong>~11k</strong> to <strong>~2k</strong> and I can manually control them as well. That was a big help <a class="mention" href="/u/solvingpuzzles">@solvingPuzzles</a>… <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>But, the Convolution layer had to be given the number of input channels after AvgPool which happened to be <strong>16</strong> not 16*19=<strong>304</strong> (or perhaps did I misunderstood something here?) but it is working now. Just for reference, here are the changes I made:</p>
<pre><code class="lang-auto">        ...

        self.bn3 = nn.BatchNorm2d(num_features=16)
        self.relu3 = nn.ReLU()
        self.pool3 = nn.AvgPool2d(2)
        
        self.conv = nn.Conv2d(16, x, kernel_size=1, groups=16)
        
        self.relu4 = nn.ReLU()
        self.dense2 = nn.Linear(x*19,3)

        ...
</code></pre>
<p>I’ll just have to evaluate and compare their performance soon enough…</p>