Conv2d groups=2 on single GPU

username: ttbb
cooked: <p>I searched on the forum, but couldn’t find any topic with exactly same problem described below.</p>
<p>Currently, I do some research on computer vision deep learning topic.<br>
I noticed, that in “Conv2d”, there is an option to specify a “groups” value.<br>
According to PyTorch docs:<br>
At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.</p>
<p>If I’m right, groups can also be used to split the load on many GPUs, if groups=2, split on 2 GPUs (original AlexNet was trained that way on many less powerfull GPUs)</p>
<p>My question is, what happens, if I specify <strong>groups=2</strong>, but have only a <strong>single GPU</strong>? Is the half of the channels from the output of the previous layer dropped in case of single GPU, and information is lost? Or does pyTorch concatenates the two groups and feeds to the next layer as if there weren’t any groups (or groups=1) ?</p>
<p>I train my network currently on Google Colab with GPU, and as far as I know, colab assigns just one GPU to the session.</p>
<p>Excerpt from models state dict (conv2, conv4, conv5 with groups=2):</p>
<p>conv.conv1_s1.weight 	 torch.Size([<strong>96</strong>, 3, 25, 7])<br>
conv.conv1_s1.bias 	 torch.Size([96])<br>
conv.conv2_s1.weight 	 torch.Size([192, <strong>48</strong>, 3, 3])<br>
conv.conv2_s1.bias 	 torch.Size([192])<br>
conv.conv3_s1.weight 	 torch.Size([<strong>256</strong>, 192, 3, 3])<br>
conv.conv3_s1.bias 	 torch.Size([256])<br>
conv.conv4_s1.weight 	 torch.Size([<strong>256</strong>, <strong>128</strong>, 3, 3])<br>
conv.conv4_s1.bias 	 torch.Size([256])<br>
conv.conv5_s1.weight 	 torch.Size([192, <strong>128</strong>, 3, 3])<br>
conv.conv5_s1.bias 	 torch.Size([192])<br>
fc6.fc6_s1.weight 	 torch.Size([512, 6336])<br>
fc6.fc6_s1.bias 	 torch.Size([512])<br>
fc7.fc7.weight 	 torch.Size([256, 2048])<br>
fc7.fc7.bias 	 torch.Size([256])<br>
classifier.fc8.weight 	 torch.Size([24, 256])<br>
classifier.fc8.bias 	 torch.Size([24])<br>
Total parameters:  4868056</p>
<p>Many Thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ttbb" data-post="1" data-topic="74700">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/82dd89/48.png" class="avatar"> ttbb:</div>
<blockquote>
<p>If I’m right, groups can also be used to split the load on many GPUs, if groups=2, split on 2 GPUs (original AlexNet was trained that way on many less powerfull GPUs)</p>
</blockquote>
</aside>
<p>You could achieve the same output by splitting the input to two GPUs, calling each conv individually, and concatenating the output, as with <code>groups=2</code>.<br>
However, <code>groups=2</code> will not split the conv to two GPUs. Instead it will use the explained method as if you are using two conv layers side by side. The operation will still use a single GPU.</p>

username: ttbb
cooked: <p>Actually, my goal is not to split on 2 GPUs. I tested my network with groups=2 and groups=1, and wondering if I loose some information when applying groups=2, because printing the state dict (if I do it correctly) shows in the second layer only half of the channels from the first layer.</p>
<p>That means with groups=2:</p>
<ol>
<li>layer output channels : 96   --&gt; 2.layer input channels : 48   (see first post)</li>
</ol>
<p>But with groups=1:</p>
<ol>
<li>layer output channels : 96   --&gt; 2.layer input channels : 96   (as expected)</li>
</ol>
<p>I print the state dict this way:</p>
<pre><code class="lang-auto">    for param_tensor in net.state_dict():
      print(param_tensor, "\t", net.state_dict()[param_tensor].size())
</code></pre>
<p><strong>So the main question from my perspective is, if I loose some information when using groups=2 ?</strong></p>

username: ptrblck
cooked: <p>If you are using <code>groups=2</code>, then each filter will have <code>0.5*in_channels</code>, as the operation is comparable to two separate convolutions using half of the input.</p>
<p>This also means that the number of parameters in the conv layer will be lowered.</p>
<aside class="quote no-group" data-username="ttbb" data-post="3" data-topic="74700">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/82dd89/48.png" class="avatar"> ttbb:</div>
<blockquote>
<p>So the main question from my perspective is, if I loose some information when using groups=2 ?</p>
</blockquote>
</aside>
<p>The input will still be processes by one or the other group, so the convolution will still use the complete input information.<br>
If you refer to the capacity (number of parameters) as information, then the grouped conv will have less capacity.</p>

username: ttbb
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="74700">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>by one <strong>or</strong> the other</p>
</blockquote>
</aside>
<p>Do you really mean “or” ? How can it be if the convolution uses the complete input information ? Just to be on the safe side, both groups are processed, right ?  (Sorry for repeated question, I’m a newbie)</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Sorry for the misleading message.<br>
The input will completely processed. Each group will see half the input channels, and producing half the output channels, and both subsequently concatenated (from the docs).</p>
<p>This code demonstrates the behavior better:</p>
<pre><code class="lang-python">conv = nn.Conv2d(6, 6, 3, 1, 1, groups=2, bias=False)
x = torch.randn(1, 6, 24, 24)
out = conv(x)

filters1 = conv.weight[:3]
o1 = F.conv2d(x[:, :3], filters1, padding=1)

filters2 = conv.weight[3:]
o2 = F.conv2d(x[:, 3:], filters2, padding=1)

out_manual = torch.cat((o1, o2), dim=1)

print((out - out_manual).abs().max())
&gt; tensor(3.5763e-07, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>