How to apply different kernels to each example in a batch when using convolution?

username: postBG
cooked: <p>F.conv2d only supports applying the same kernel to all examples in a batch.<br>
However, I want to apply different kernels to each example. How can I do this?</p>
<p>The most naive approach seems the code below:</p>
<pre><code class="lang-auto">def parallel_conv2d(inputs, filters, stride=1, padding=1):
  batch_size = inputs.size(0)

  output_slices = [F.conv2d(inputs[i:i+1], filters[i], bias=None, stride=stride, padding=padding).squeeze(0)
                   for i in range(batch_size)]
  return torch.stack(output_slices, dim=0)
</code></pre>
<p>Is there any better and efficient implementation or APIs that I can utilize?</p>

username: ptrblck
cooked: <p>You could use a grouped convolution, which would use a single filter for a group of input channel, and in the case of a depthwise convolution an own filter for each input channel.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/nn.html?highlight=conv2d#torch.nn.Conv2d">docs</a> for more information.</p>

username: postBG
cooked: <p>Hi, Thanks for your reply.<br>
However, I think there is a misunderstanding. I apologize that my question was not clear enough.</p>
<p>My question means that I want to apply different kernels to each image, not different channels.<br>
More specifically, the arguments of the parallel_conv2d are:<br>
inputs: mini_batch x in_channel x iH x iW tensor - batch of input “images”<br>
filters: mini_batch x out_channel x in_channel x KH x kW tensor - a batch of convolutional kernels</p>
<p>What I want to do is applying filters[i] to inputs[i].</p>

username: ptrblck
cooked: <p>Thanks for the update and I clearly misunderstood the use case.<br>
I think if the kernel shapes are different, you would need to use a loop and concatenate the output afterwards, as the filters cannot be stored directly in a single tensor.</p>
<p>However, if the kernels have all the same shape, the grouped conv approach might still work.<br>
Here is a small example using convolutions with <code>in_channels=3</code>, <code>out_channels=15</code> for a batch size of 10:</p>
<pre><code class="lang-python"># Setup
N, C, H, W = 10, 3, 24, 24
x = torch.randn(N, C, H, W)

# Create filterset for each sample
weights = []
for _ in range(N):
    weight = nn.Parameter(torch.randn(15, 3, 5, 5))
    weights.append(weight)

# Apply manually
outputs = []
for idx in range(N):
    input = x[idx:idx+1]
    weight = weights[idx]
    output = F.conv2d(input, weight, stride=1, padding=2)
    outputs.append(output)

outputs = torch.stack(outputs)
outputs = outputs.squeeze(1) # remove fake batch dimension
print(outputs.shape)
&gt; torch.Size([10, 15, 24, 24])

# Use grouped approach
weights = torch.stack(weights)
weights = weights.view(-1, 3, 5, 5)
print(weights.shape)
&gt; torch.Size([150, 3, 5, 5])
# move batch dim into channels
x = x.view(1, -1, H, W)
print(x.shape)
&gt; torch.Size([1, 30, 24, 24])
# Apply grouped conv
outputs_grouped = F.conv2d(x, weights, stride=1, padding=2, groups=N)
outputs_grouped = outputs_grouped.view(N, 15, 24, 24)

# Compare
print((outputs - outputs_grouped).abs().max())
tensor(1.3351e-05, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>If this approach could work, I would recommend to profile both approaches and see, if my suggestion is faster for your workload.</p>

username: postBG
cooked: <p>Hi, thanks for your reply.<br>
I think it’s the best trick that I can choose.</p>

username: naru-byte
cooked: <p>This code was very helpful because I want to do same thing by using conv3d.</p>
<p>However when I actually ran a below program, I wonder this code may be not work in case of using conv3d.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

#Setup
N, T, C, H, W = 10, 24, 3, 24, 24
x = torch.stack([torch.full(size=(C,T,H,W), fill_value=float(n)) for n in range(N)])
for x_elem in x:
    print(x_elem.abs().max())

# Create filterset for each sample
OC = 15
weights = []
for _ in range(N):
    weight = nn.Parameter(torch.full(size=(OC,C,5,5,5), fill_value=float(_)))
    weights.append(weight)
    print(weight.abs().max())

# Apply manually
outputs = []
for idx in range(N):
    input = x[idx:idx+1]
    weight = weights[idx]
    output = F.conv3d(input, weight, stride=1, padding=2)
    outputs.append(output)


outputs = torch.stack(outputs)
outputs = outputs.squeeze(1) # remove fake batch dimension
# outputs = torch.cat(outputs, dim=0)
print(outputs.shape)

for output in outputs:
    print(output.abs().max())

# Use grouped approach
weights = torch.stack(weights)
weights = weights.view(-1, C, 5, 5, 5)
print(weights.shape)

# move batch dim into channels
x = x.view(1, -1, T, H, W)
print(x.shape)

# Apply grouped conv
outputs_grouped = F.conv3d(x, weights, stride=1, padding=2, groups=N)
outputs_grouped = outputs_grouped.view(10, 15, 24, 24, 24)
print(outputs_grouped.shape)
# Compare
print((outputs - outputs_grouped).abs().max())

for output_grouped in outputs_grouped:
    print(outputs_grouped.abs().max())

</code></pre>
<p>outputs</p>
<pre><code class="lang-auto">tensor(0.)
tensor(1.)
tensor(2.)
tensor(3.)
tensor(4.)
tensor(5.)
tensor(6.)
tensor(7.)
tensor(8.)
tensor(9.)
tensor(0., grad_fn=&lt;MaxBackward1&gt;)
tensor(1., grad_fn=&lt;MaxBackward1&gt;)
tensor(2., grad_fn=&lt;MaxBackward1&gt;)
tensor(3., grad_fn=&lt;MaxBackward1&gt;)
tensor(4., grad_fn=&lt;MaxBackward1&gt;)
tensor(5., grad_fn=&lt;MaxBackward1&gt;)
tensor(6., grad_fn=&lt;MaxBackward1&gt;)
tensor(7., grad_fn=&lt;MaxBackward1&gt;)
tensor(8., grad_fn=&lt;MaxBackward1&gt;)
tensor(9., grad_fn=&lt;MaxBackward1&gt;)
torch.Size([10, 15, 24, 24, 24])

for output in outputs:
    print(output.abs().max())
&gt;tensor(0., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(1500., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(3375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(6000., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(9375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(13500., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(18375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(24000., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)

print(weights.shape)
&gt;torch.Size([150, 3, 5, 5, 5])

print(x.shape)
&gt;torch.Size([1, 30, 24, 24, 24])

print(outputs_grouped.shape)
&gt;torch.Size([10, 15, 24, 24, 24])

print((outputs - outputs_grouped).abs().max())
&gt;tensor(0., grad_fn=&lt;MaxBackward1&gt;)

for output_grouped in outputs_grouped:
    print(outputs_grouped.abs().max())
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
&gt;tensor(30375., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>Displayed max values of <code>outputs</code> in this code gradually increase with increasing <code>torch.full</code> value.<br>
On the other hand,  all <code>outputs_grouped</code> 's max values are same.</p>
<p>I think it might be caused by <code>outputs_grouped = outputs_grouped.view(10, 15, 24, 24, 24)</code> . But I couldn’t this solution.</p>
<p>I would be grateful if you could tell me about your experience.</p>

username: ptrblck
cooked: <p>Your code works fine and shows a zero absolute difference between the manual and grouped approach.<br>
The issue with the constant print values is raised, because you are using <code>print(outputs_grouped.abs().max())</code> instead of <code>print(output_grouped.abs().max())</code> in the last loop.</p>

username: naru-byte
cooked: <p>I made a very stupid mistake.</p>
<p>I confirm this code correctly work. Thank you very much.</p>

username: Qizhou
cooked: <p>The trick of group convolution is wonderful！Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/heart_eyes.png?v=9" title=":heart_eyes:" class="emoji" alt=":heart_eyes:"></p>

username: Sutharsan_Mahendren
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="84848">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto"># Setup
N, C, H, W = 10, 3, 24, 24
x = torch.randn(N, C, H, W)

# Create filterset for each sample
weights = []
for _ in range(N):
    weight = nn.Parameter(torch.randn(15, 3, 5, 5))
    weights.append(weight)

# Apply manually
outputs = []
for idx in range(N):
    input = x[idx:idx+1]
    weight = weights[idx]
    output = F.conv2d(input, weight, stride=1, padding=2)
    outputs.append(output)

outputs = torch.stack(outputs)
outputs = outputs.squeeze(1) # remove fake batch dimension
print(outputs.shape)
&gt; torch.Size([10, 15, 24, 24])

# Use grouped approach
weights = torch.stack(weights)
weights = weights.view(-1, 3, 5, 5)
print(weights.shape)
&gt; torch.Size([150, 3, 5, 5])
# move batch dim into channels
x = x.view(1, -1, H, W)
print(x.shape)
&gt; torch.Size([1, 30, 24, 24])
# Apply grouped conv
outputs_grouped = F.conv2d(x, weights, stride=1, padding=2, groups=N)
outputs_grouped = outputs_grouped.view(N, 15, 24, 24)

# Compare
print((outputs - outputs_grouped).abs().max())
tensor(1.3351e-05, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
</blockquote>
</aside>
<p>what is the reason for this minimal deviation of both values?</p>

username: ptrblck
cooked: <p>The difference is created due to the limited floating point precision and a potentially different order of operations through the usage of different algorithms (manual vs. grouped conv in this case).</p>

username: Sutharsan_Mahendren
cooked: <p>Thank you for the reply. You are right. I checked it on the tensors with random binary values. It became zero</p>