How to access weights of each group in a grouped convolution?

username: b1d4fc9f712b67d006e1
cooked: <p>a convolution layer with g groups by definition conducts g convolution operations, each has weight of size (out//g, in//g, k, k). How to access those weights?</p>

username: ptrblck
cooked: <p>If you’ve created the conv layer as a module attribute (e.g. <code>self.conv = nn.Conv2d(...)</code>), you can access the weight matrix via:</p>
<pre><code class="lang-python">model = MyModel()
kernels = model.conv.weight
</code></pre>

username: b1d4fc9f712b67d006e1
cooked: <p>in this case</p>
<pre><code class="lang-auto">kernels.size()
</code></pre>
<p>will return (out, in//g, k, k).</p>
<p>I would like to obtain g kernels, each one has size (out//g, in//g, k, k), corresponding to the g side by side conv layers</p>

username: ptrblck
cooked: <p>You could probably use <code>torch.chunk</code> to get the paired kernels:</p>
<pre><code class="lang-python">groups = 6
conv = nn.Conv2d(6, 12, 3, 1, 1, groups=groups, bias=False)
kernels = torch.stack(conv.weight.chunk(groups, 0))

# Reference
x = torch.randn(1, 6, 24, 24)
out = conv(x)

# Comparison
o1 = F.conv2d(x[:, :1], kernels[0], padding=1)
print(torch.allclose(o1, out[:, :2]))
</code></pre>

username: b1d4fc9f712b67d006e1
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="47868">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>groups = 6 conv = nn.Conv2d(6, 12, 3, 1, 1, groups=groups, bias=False) kernels = torch.stack(conv.weight.chunk(groups, 0)) # Reference x = torch.randn(1, 6, 24, 24) out = conv(x) # Comparison o1 = F.conv2d(x[:, :1], kernels[0], padding=1) print(torch.allclose(o1, out[:, :2]))</p>
</blockquote>
</aside>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F
groups = 6
conv = nn.Conv2d(6, 12, 3, 1, 1, groups=groups, bias=False)
kernels = torch.stack(conv.weight.chunk(groups, 0))
# Reference
x = torch.randn(1, 6, 24, 24)
out = conv(x)
# Comparison
o1 = F.conv2d(x[:, :1], kernels[0], padding=1)
print(torch.allclose(o1, out[:, :2]))
False
</code></pre>
<p>should i expect this False message?</p>

username: ptrblck
cooked: <p>No, as it returns <code>True</code> on my machine.<br>
I’ve run the code 1000 times in a loop and don’t get a <code>False</code> value.</p>

username: b1d4fc9f712b67d006e1
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="47868">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>groups = 6 conv = nn.Conv2d(6, 12, 3, 1, 1, groups=groups, bias=False) kernels = torch.stack(conv.weight.chunk(groups, 0)) # Reference x = torch.randn(1, 6, 24, 24) out = conv(x) # Comparison o1 = F.conv2d(x[:, :1], kernels[0], padding=1) print(torch.allclose(o1, out[:, :2]))</p>
</blockquote>
</aside>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/29aa389c4a8a54bf1bdb893dffb31abc46437023.png" data-download-href="https://discuss.pytorch.org/uploads/default/29aa389c4a8a54bf1bdb893dffb31abc46437023" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/29aa389c4a8a54bf1bdb893dffb31abc46437023_2_690x489.png" alt="image" data-base62-sha1="5WAgWTXLY6exrKO5M49ZdpHTgdB" width="690" height="489" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/29aa389c4a8a54bf1bdb893dffb31abc46437023_2_690x489.png, https://discuss.pytorch.org/uploads/default/original/2X/2/29aa389c4a8a54bf1bdb893dffb31abc46437023.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/29aa389c4a8a54bf1bdb893dffb31abc46437023.png 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">953×676 66.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>then i have no idea why…</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using?</p>

username: b1d4fc9f712b67d006e1
cooked: <p>Thanks, i am currently using 1.1.0<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/616873b47ed2863b6bcf6b0afa5c89b797ae32dd.png" data-download-href="https://discuss.pytorch.org/uploads/default/616873b47ed2863b6bcf6b0afa5c89b797ae32dd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/616873b47ed2863b6bcf6b0afa5c89b797ae32dd_2_690x181.png" alt="image" data-base62-sha1="dTI6jUlqaHwwoVitJmQmy9aNPYh" width="690" height="181" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/616873b47ed2863b6bcf6b0afa5c89b797ae32dd_2_690x181.png, https://discuss.pytorch.org/uploads/default/original/2X/6/616873b47ed2863b6bcf6b0afa5c89b797ae32dd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/616873b47ed2863b6bcf6b0afa5c89b797ae32dd.png 2x" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">782×206 18.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>That’s strange.<br>
I’ve tested the code again on <code>1.1.0</code> as well as <code>1.1.0.dev20190502</code> and both seem to run without an issue:</p>
<pre><code class="lang-python">
import torch
from torch import nn
from torch.nn import functional as F

for _ in range(1000):
    groups = 6
    conv = nn.Conv2d(6, 12, 3, 1, 1, groups=groups, bias=False)
    kernels = torch.stack(conv.weight.chunk(groups, 0))
    # Reference
    x = torch.randn(1, 6, 24, 24)
    out = conv(x)
    # Comparison
    o1 = F.conv2d(x[:, :1], kernels[0], padding=1)
    if not torch.allclose(o1, out[:, :2]):
        print('False')
</code></pre>

username: ptrblck
cooked: <p>Maybe we are unlucky and triggering <a href="https://github.com/pytorch/pytorch/issues/21597" rel="nofollow noopener">this issue</a>.<br>
Could you change the number of input channels and run the test again?</p>

username: b1d4fc9f712b67d006e1
cooked: <pre><code class="lang-auto">import torch
from torch import nn
from torch.nn import functional as F
for _ in range(1000):
    groups = 12
    conv = nn.Conv2d(12, 12, 3, 1, 1, groups=groups, bias=False)
    kernels = torch.stack(conv.weight.chunk(groups, 0))
    # Reference
    x = torch.randn(1, 12, 24, 24)
    out = conv(x)
    # Comparison
    o1 = F.conv2d(x[:, :1], kernels[0], padding=1)
    if not torch.allclose(o1, out[:, :2]):
        print('False')
</code></pre>
<p>still…</p>

username: ptrblck
cooked: <p><code>o1</code> in your example will have the shape <code>[1, 1, 24, 24]</code>, while <code>out[:, :2]</code> will be <code>[1, 2, 24, 24]</code>.<br>
Could you change the condition to <code>if not torch.allclose(o1, out[:, :1])</code> and run it again?</p>