How can I come up with a "do nothing" kernel for a Conv1d?

username: shamoons
cooked: <p>I have:</p>
<pre><code class="lang-auto">        print('\ninp', inp.min(), inp.mean(), inp.max())
        print(inp)
        out = self.conv1(inp)
        print('\nout1', out.min(), out.mean(), out.max())
        print(out)
        quit()
</code></pre>
<p>My min, mean and max for my inp is: <code>inp tensor(9.0060e-05) tensor(0.1357) tensor(2.4454)</code></p>
<p>For my output, I have: <code>out1 tensor(4.8751, grad_fn=&lt;MinBackward1&gt;) tensor(21.8416, grad_fn=&lt;MeanBackward0&gt;) tensor(54.9332, grad_fn=&lt;MaxBackward1&gt;)</code></p>
<p>My <code>self.conv1</code> is:</p>
<pre><code class="lang-auto">        self.conv1 = torch.nn.Conv1d(
            in_channels=161,
            out_channels=161,
            kernel_size=11,
            stride=1,
            padding=5)
        self.conv1.weight.data = torch.zeros(self.conv1.weight.data.size())
        self.conv1.weight.data[:, :, 5] = 1.0
        self.conv1.bias.data = torch.zeros(self.conv1.bias.data.size())
</code></pre>
<p>So my weights look like: <code>tensor([0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.])</code></p>
<p>So if I understand how convolution works, this should produce the same output. But it doesn’t.</p>
<p>What am I doing wrong? I realize that there’s some summing going on, but then how would I have an identity kernel?</p>

username: ptrblck
cooked: <p>Note that each kernel has the same number of channels as your input activation, which will be part of the summation, so you would need a depthwise convolution, where each filter acts on a single input channel.</p>

username: shamoons
cooked: <p>How would I do a depthwise convolution in my case?</p>

username: shamoons
cooked: <p>Also - is there any advantage to doing a “depth wise separable convolution”? It seems then the contextual information from the other channels won’t be factored in.</p>

username: shamoons
cooked: <p>Also, I just tried:</p>
<pre><code class="lang-auto">        self.conv1 = torch.nn.Conv1d(
            in_channels=feature_dim,
            out_channels= feature_dim,
            kernel_size=kernel_sizes[0],
            stride=1,
            groups= feature_dim,
            padding=kernel_sizes[0] // 2)
</code></pre>
<p>This still doesn’t quite get it:</p>
<pre><code class="lang-auto">inp tensor(3.2217e-05) tensor(0.1359) tensor(1.9529) torch.Size([1, 161, 32])

out1 tensor(2.0011e-07, grad_fn=&lt;MinBackward1&gt;) tensor(0.0008, grad_fn=&lt;MeanBackward0&gt;) tensor(0.0121, grad_fn=&lt;MaxBackward1&gt;) torch.Size([1, 161, 32])
</code></pre>

username: shamoons
cooked: <p>I guess I’m not fully understanding this.</p>
<p>If my input is 10 steps of 161-dimension each, then in the equation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da7eeadbe449e086be0ceac218d5ce3cfdb1b8af.png" data-download-href="https://discuss.pytorch.org/uploads/default/da7eeadbe449e086be0ceac218d5ce3cfdb1b8af" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da7eeadbe449e086be0ceac218d5ce3cfdb1b8af_2_690x98.png" alt="image" data-base62-sha1="vaU0ufYDEJM1Ekslx7Xff3PBKNN" width="690" height="98" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da7eeadbe449e086be0ceac218d5ce3cfdb1b8af_2_690x98.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da7eeadbe449e086be0ceac218d5ce3cfdb1b8af_2_1035x147.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/a/da7eeadbe449e086be0ceac218d5ce3cfdb1b8af.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1360×194 8.77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What is input(Ni, k)?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="shamoons" data-post="4" data-topic="73919">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shamoons/48/21191_2.png" class="avatar"> shamoons:</div>
<blockquote>
<p>is there any advantage to doing a “depth wise separable convolution”? It seems then the contextual information from the other channels won’t be factored in.</p>
</blockquote>
</aside>
<p>I can’t give you a general answer, as the advantages/disadvantages will depend on your actual use case.<br>
For your use case to get the identical output, you would need to use a depthwise convolution, otherwise the input channels will be summed.</p>
<aside class="quote no-group" data-username="shamoons" data-post="5" data-topic="73919">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shamoons/48/21191_2.png" class="avatar"> shamoons:</div>
<blockquote>
<p>This still doesn’t quite get it:</p>
</blockquote>
</aside>
<p>You are most likely getting a different result due to the bias.<br>
Here is an example yielding the same result:</p>
<pre data-code-wrap="python"><code class="lang-python">conv = nn.Conv1d(5, 5, kernel_size=11, stride=1, padding=5, groups=5, bias=False)
with torch.no_grad():
    conv.weight.zero_()
    conv.weight[:, :, 5] = 1.

x = torch.randn(2, 5, 24)
out = conv(x)
print((out == x).all())
&gt; tensor(True)
</code></pre>
<aside class="quote no-group" data-username="shamoons" data-post="6" data-topic="73919">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shamoons/48/21191_2.png" class="avatar"> shamoons:</div>
<blockquote>
<p>What is input(Ni, k)?</p>
</blockquote>
</aside>
<p><code>N_i</code> is used to index the batch dimension, while <code>k</code> is used to index the input channel dimension.</p>
<p>Have a look at <a href="http://cs231n.github.io/convolutional-networks/#conv">CS231n - Convolution</a> for a detailed description about the shape and workflow of convolutions. While the post is focused on 2D convolutions, the general method also applies for 1D convs.</p>