Conv2d certain values for groups and out_channels don't work

vmirly1: <p>I am playing with the groups option in <code>torch.nn.Conv2d(..)</code>. It appears that both in_channels and out_channels must be divisible by groups. But in theory, it is not necessary, for example, if I have <code>in_channels=3</code>, and <code>groups=3</code>, then <code>out_channels=8</code> should give me the operation shown in the figure, but this will raise an error, saying “out_cnhannels is not divisable by groups”.</p>
<p>This works for in_channels=3, out_channels=9, and groups=3:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; conv = torch.nn.Conv2d(in_channels=3, out_channels=9, 
                       kernel_size=(3,3), stride=1, 
                       padding=0, dilation=1, 
                       groups=3, bias=True)

&gt;&gt;&gt; print(conv.weight.data.size())
torch.Size([9, 1, 3, 3])
</code></pre>
<p>So, I think <code>out_channels=8</code> should work as well. Isn’t that right? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6ef68adb9d9910f82ffabdc53adc3abd842f6885.png" alt="Sharing-kernels" data-base62-sha1="fPCJEQVDmSELxuDpfVuSNGr04YZ" width="564" height="484"></p>

rasbt: <p>From what I understand, the number of channels in one kernel tensor is equal to the number of channels in the input image. E.g., if you set</p>
<pre><code>c = torch.nn.Conv2d(in_channels=4, out_channels=8, kernel_size=(3, 3), groups=1)
</code></pre>
<p>you will have 8 kernel tensors with 4 channels each, corresponding to the following scenario:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7c0376d265db0433eef43c6f7501346f4bb31a7f.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c0376d265db0433eef43c6f7501346f4bb31a7f" title="28%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7c0376d265db0433eef43c6f7501346f4bb31a7f_2_419x500.png" alt="28%20PM" data-base62-sha1="hH4CWWNaiicImgM7XYcHj3NKJ1d" width="419" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7c0376d265db0433eef43c6f7501346f4bb31a7f_2_419x500.png, https://discuss.pytorch.org/uploads/default/original/2X/7/7c0376d265db0433eef43c6f7501346f4bb31a7f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/7c0376d265db0433eef43c6f7501346f4bb31a7f.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">28%20PM</span><span class="informations">583×695 10.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then, if you change it to</p>
<pre><code>c = torch.nn.Conv2d(in_channels=4, out_channels=8, kernel_size=(3, 3), groups=2)
</code></pre>
<p>the 8 kernel tensors will be divided such that 2 of them are used for the first 2 image channels, and 2 of them are used for the other 2 image channels. The results are stacked, so if each kernel tensor has 4 channels, you have 2*4=8 output channels and 2 of them will be used for the other 2 image channels.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/363ca33ab0e4fc2b0a075b19bb7fa666cad52230.png" data-download-href="https://discuss.pytorch.org/uploads/default/363ca33ab0e4fc2b0a075b19bb7fa666cad52230" title="37%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/363ca33ab0e4fc2b0a075b19bb7fa666cad52230_2_380x500.png" alt="37%20PM" data-base62-sha1="7JNHkxWAoIyzjJ4DPjpy4W9Ny7K" width="380" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/363ca33ab0e4fc2b0a075b19bb7fa666cad52230_2_380x500.png, https://discuss.pytorch.org/uploads/default/original/2X/3/363ca33ab0e4fc2b0a075b19bb7fa666cad52230.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/363ca33ab0e4fc2b0a075b19bb7fa666cad52230.png 2x" data-dominant-color="F9F7F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">37%20PM</span><span class="informations">501×659 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Now, if you change it to sth like this</p>
<pre><code>c = torch.nn.Conv2d(3, 9, (3, 3), groups=3)
</code></pre>
<p>you would end up with 1 kernel channel per image channel:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/41ca66ffb9ad8b6e3c09cac8b1887be3713897f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/41ca66ffb9ad8b6e3c09cac8b1887be3713897f3" title="58%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/41ca66ffb9ad8b6e3c09cac8b1887be3713897f3_2_369x500.png" alt="58%20PM" data-base62-sha1="9o0Gcc4h4Jgy44GtVA7I4tEg26n" width="369" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/41ca66ffb9ad8b6e3c09cac8b1887be3713897f3_2_369x500.png, https://discuss.pytorch.org/uploads/default/original/2X/4/41ca66ffb9ad8b6e3c09cac8b1887be3713897f3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/41ca66ffb9ad8b6e3c09cac8b1887be3713897f3.png 2x" data-dominant-color="F9FAF8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">58%20PM</span><span class="informations">540×731 12.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Finally, if you write sth like you suggested:</p>
<pre><code>c = torch.nn.Conv2d(in_channels=3, out_channels=8, kernel_size=(3, 3), groups=3)
</code></pre>
<p>the code will complain, because it wouldn’t now how to evenly distribute the 8 kernels among the 3 input image channels. Eg., the first channel might get 3 kernels, the 2nd gets 3, and the third gets 2? That’s possible but ambiguous.</p>

rasbt: <p><a class="mention" href="/u/smth">@smth</a><br>
Hm, I was just thinking that it would maybe help to have a parameter called <code>num_kernel_channels_per_input_channels</code> (or probably a shorter name ;)) instead of <code>groups</code> in future? The default could be</p>
<pre><code>num_kernel_channels_per_input_channels = None
</code></pre>
<p>which would set <code>num_kernel_channels_per_input_channels</code> equal to the number of channels in the input image. This could make the scenarios above a bit more intuitive and probably more user-friendly.</p>

vmirly1: <p>I see; I thought that each kernel would be shared with all three input channels so given kernel W_k and the 3 input channels X1 X2 and X3, we could have the following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d8413ba54800be45fb93e83c373109a0ac149c11.png" alt="groups" data-base62-sha1="uR4Tt3mcj1rqprritmJbWVNOWIx" width="613" height="410"></p>

rasbt: <p>Hmm, I don’t think that’s possible in PyTorch without workarounds, but maybe someone else knows more …</p>
<p>Eg as a work around, you could split the input image into 3 matrices, and for each matrix, you could use the same convolutional layer. And then, you could average the results over the three outputs. In pseudo code, sth like this:</p>
<pre><code class="lang-python">x1 = X[:, 0, :, :]
x2 = X[:, 1, :, :]
x3 = X[:, 2, :, :]

o1 = conv2d_1(x1, in_channels=1, out_channels=1...)
o2 = conv2d_1(x2, in_channels=1, out_channels=1...)
o3 = conv2d_1(x3, in_channels=1, out_channels=1...)

o = (o1 + o2 + o3) / 3
</code></pre>

smth: <p>you can do that by folding the channels dimension into batch dimension, and unfolding after finishing.<br>
For example:</p>
<pre><code class="lang-auto">input = Tensor(N, C, H, W)
layer = nn.Conv2d(1, 1, kernel_size)

# first fold C into N
input = input.view(N*C, 1, input.size(2), input.size(3))
out = layer(input)

# unfold
out = out.view(N, C, out.size(2), out.size(3))
</code></pre>

michaelklachko: <p>Current docs don’t mention that out_channels should also be divisible by the number of groups:</p>
<p><a href="https://pytorch.org/docs/0.1.12/_modules/torch/nn/functional.html#conv2d" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/0.1.12/_modules/torch/nn/functional.html#conv2d</a></p>
<p>That should be added, IMO.</p>

Jingshuai: <p>As I understand and verified, kernel_numbers=out_channels*(in_channels/groups).<br>
For instance, nn.Conv2d(6, 16, kernel_size=(5,5) ,groups=2) makes the learnable parameters of the layer be: torch.Size([16, 3, 5, 5]).</p>

cadip92: <p>You have explained the scenario for a convolution with just 1 output channel. Let’s suppose the no.of input channels are 16 and the no.of output channels are 8 (i.e width multipler = 0.5 as cited in MobileNet paper).<br>
In this case, can I write nn.Conv2d(1,8,3) where 1 is the input channel after folding, 8 are the no.of output channels and 3 is the kernel size ?<br>
Is my understanding correct ?</p>

barto: <p>Sorry for being so late to the discussion but I just had to comment on this as your second example confused me</p>
<pre data-code-wrap="py"><code class="lang-py">c = torch.nn.Conv2d(in_channels=4, out_channels=8, kernel_size=(3, 3), groups=2)
</code></pre>
<p>so I had to check if my understanding of group convolution was correct. To me it seems that you are saying that each kernel tensor has 4 input channels within each group, and you are also saying that each group only has 2 filters? But clearly if you print out the weight shape you get <code>torch.Size([8, 2, 3, 3])</code> showing that the input channel is 2, not 4. So my understanding is that in this example there are 2 groups, where each group has 4 filters and each of these 3x3 filter has 2 channels (not 4 channels). So for each group, the filters each take turn on convolving the input (only looking at 2 input channels, one group looks at one half of the input channels and the other group looks at the other half) producing a depth output of 4 because there are 4 filters in a group. Now, since there are 2 groups we will get a total of 8 depth (4+4) on the output, because each group produces 4 depth and therefore stacking them together produces 8 depth.</p>
<p>My understanding comes from this <a href="https://youtube.com/clip/UgkxnvsdgvFp2ogftHNjFceBj7-2td90VoJA?si=aVIio0OQF9l1iSzo" rel="noopener nofollow ugc">animation</a> at 1:24 where they have 2 groups, and thus has halved the input channels, the output depth is still determined by the number of filters in total (because at the end you just stack the intermediate outputs anyways).</p>
<p>This understanding is consistent even with the third example <code>c = torch.nn.Conv2d(3, 9, (3, 3), groups=3)</code>, where the weight shape is shown to be <code>torch.Size([9, 1, 3, 3])</code>. Here we have 3 groups, 3 in_channels and 9 out_channels, so each group will have 9/3=3 filters and each filter will have 3/3=1 in_channel. This means that each group will produce a depth output of 3 because there are 3 filters in a group. But we have 3 groups, so in total when stacking the output we will get a depth of 9 in the end.</p>
<p>And that’s kinda the point of using grouped convolution, by splitting up the computations into groups this effectively reduces the number of input channels on each filter, while still reaping the benefits of the potentially increased output channels in later conv layers to capture richer features. This ultimately reduces the computational cost from O(c_i * c_o) to O(g * c_i/g * c_o/g) = O(c_i * c_o / g), which makes it g times faster, where g is the number of groups, c_i is the input channels and c_o the output channels according to <a href="https://d2l.ai/" rel="noopener nofollow ugc">d2l</a> chapter 8.6.5. This also reduces the number of parameters in the same way.</p>