Gradient backprop for different ways of slicing a tensor

jayz: <p>Say we have two methods of slicing for computing an output of a convolutional layer:</p>
<pre><code class="lang-auto"># N  - batch size
# CI - number of in channels
# CO - number of out channels 
# T  - sequence length (think of e.g. a time series 0, ..., t, ..., T)

# model
conv = nn.Conv1d(CI,CO,kernel_size=1)

# input 
x = torch.rand(N,CI,T)

# method 1
out1 = conv( x[:,:,-5:] )

# method 2
out2 = conv( x )[:,:,-5:]
</code></pre>
<p>Theoretically, out1 and out2 should result in the same gradient updates for the conv kernel, but out1 requires less computation, as it only convolves the last 5 elements instead of the whole tensor x. In both cases, we cut away all elements but the 5 last ones, so that the gradients are not influenced by any but the last 5 points. Is that right? Or do the earlier points in the T-dimension of x also influence the gradient computation (for method 2)?</p>

KFrank: <p>Hi Jay!</p>
<aside class="quote no-group" data-username="jayz" data-post="1" data-topic="157838" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayz/48/46742_2.png" class="avatar"> jayz:</div>
<blockquote>
<p>Theoretically, out1 and out2 should result in the same gradient updates for the conv kernel,</p>
</blockquote>
</aside>
<p>This is correct.</p>
<aside class="quote no-group">
<blockquote>
<p>Or do the earlier points in the T-dimension of x also influence the gradient computation (for method 2)?</p>
</blockquote>
</aside>
<p>No,* because you have <code>kernel_size = 1</code>, neighboring points don’t get<br>
mixed with one another, so the earlier points in <code>x</code> don’t affect the later<br>
points in <code>out1</code> and <code>out2</code>.</p>
<p>See this example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt;
&gt;&gt;&gt; conv1 = torch.nn.Conv1d (3, 3, kernel_size = 1)
&gt;&gt;&gt; conv2 = torch.nn.Conv1d (3, 3, kernel_size = 1)
&gt;&gt;&gt;
&gt;&gt;&gt; with torch.no_grad():   # initialize the two convs identically
...     _ = conv2.weight.copy_ (conv1.weight)
...     _ = conv2.bias.copy_ (conv1.bias)
...
&gt;&gt;&gt; x = torch.rand (1, 3, 10)
&gt;&gt;&gt;
&gt;&gt;&gt; out1 = conv1 (x[:, :, -5:])
&gt;&gt;&gt; out2 = conv2 (x)[:, :, -5:]
&gt;&gt;&gt; torch.equal (out1, out2)
True
&gt;&gt;&gt;
&gt;&gt;&gt; (out1**2).sum().backward()
&gt;&gt;&gt; (out2**2).sum().backward()
&gt;&gt;&gt; torch.equal (conv1.weight.grad, conv2.weight.grad)
False
&gt;&gt;&gt; torch.equal (conv1.bias.grad, conv2.bias.grad)
False
&gt;&gt;&gt; torch.allclose (conv1.weight.grad, conv2.weight.grad)
True
&gt;&gt;&gt; torch.allclose (conv1.bias.grad, conv2.bias.grad)
True
</code></pre>
<p>*) To be precise, the gradient computations are <em>mathematically</em> equivalent,<br>
but do differ in (mathematically-equivalent) orders of the operations that<br>
lead to differing floating-point round-off error.  That’s why the <code>equal()</code> tests<br>
for the gradients return <code>False</code>, but the <code>allclose()</code> tests return <code>True</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

jayz: <p>Absolutely great example, thanks much for the perfect explanation!</p>
<p><a class="mention" href="/u/kfrank">@KFrank</a>: maybe you have solution for this one, too? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> kind of stuck there…</p><aside class="quote quote-modified" data-post="1" data-topic="157813">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayz/48/46742_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/joint-parameter-multivariate-rnn/157813">Joint Parameter Multivariate RNN</a> 
  </div>
  <blockquote>
    Hello, I am trying to extract joint features from H time series based on a single RNN. I have data tensors X of shape N x C x H x L, where N batch size, C is the number of features used as the input_size for the RNN, and L is the sequence length. 
I’d like to use a single RNN to extract features from the H parallel time series. For this, I loop through the H rows (N x C x L) of the tensor X and feed them one after the other into the same RNN. The class CustomRNN below defines this RNN and takes …
  </blockquote>
</aside>