Adding avergaed tensor to a single channel in another tensor

falmasri: <p>I have a tensor of size b,c,w,h and another tensor of the same size. both of them are encoded features as a result of previous conv layers.</p>
<p>the first tensor is avergaed to have the size b,1,w,h. I want to add this tensor to the second one but in a single channel only. what I mean</p>
<p>x1 = torch.mean(x1, dim=1, keepdim=True)<br>
x2[:, 0:1] += x1</p>
<p>produced the error if in place operation. " one of the variables needed for gradient computation has been modified by an inplace operation"</p>
<p>the two tensors must keep gradient flow.</p>

KFrank: <p>Hi Falmasri!</p>
<aside class="quote no-group" data-username="falmasri" data-post="1" data-topic="148647" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/falmasri/48/11265_2.png" class="avatar"> falmasri:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">x2[:, 0:1] += x1 
</code></pre>
</blockquote>
</aside>
<p>Use:</p>
<pre data-code-wrap="python"><code class="lang-python">x2 = x2[:, 0:1] + x1
# or  x3 = x2[:, 0:1] + x1
</code></pre>
<p>Some comments:</p>
<p><code>x2[:, 0:1] += x1</code> does modify <code>x2</code> inplace, because you are indexing<br>
into <code>x2</code>.</p>
<p>This is <em>not necessarily</em> a problem, but it can break the computation<br>
graph – as appears to be happening in your use case – depending<br>
upon what else is going on in your forward pass.</p>
<p>The solution is to not modify the tensor in question inplace.</p>
<p>Note, <code>x2 = x2[:, 0:1] + x1</code> does not modify the original tensor to<br>
which the python variable <code>x2</code> referred.  Instead, it creates a new tensor<br>
and then sets the python variable <code>x2</code> to refer to this new tensor.  (If<br>
autograd’s computation graph needs the original <code>x2</code> tensor for the<br>
backward pass, autograd will keep a separate reference to it so that<br>
it won’t get garbage-collected when <code>x2</code> no longer refers to it.)</p>
<p>Best.</p>
<p>K. Frank</p>

falmasri: <p>Thank you KFrank, but this doesn’t answer my question. Instead it creates a new tensor. I want to update a single channel value inside the tensor x2.</p>

KFrank: <p>Hi Falmasri!</p>
<aside class="quote no-group" data-username="falmasri" data-post="3" data-topic="148647" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/falmasri/48/11265_2.png" class="avatar"> falmasri:</div>
<blockquote>
<p>it creates a new tensor. I want to update a single channel value inside the tensor x2.</p>
</blockquote>
</aside>
<p>Sometimes autograd needs to reuse tensors from the forward pass in<br>
the backward pass.  If you modify such a tensor inplace, you will break<br>
the computation graph and get the error message you quoted, “one of<br>
the variables needed for gradient computation has been modified by an<br>
inplace operation.”</p>
<p>You have a choice:  You can “update a single channel value inside the<br>
tensor x2” and break the computation graph, or you can “create a new<br>
tensor” and have the backward pass work.</p>
<p>That’s just how pytorch works.</p>
<p>Best.</p>
<p>K. Frank</p>

falmasri: <p>I solved indirectly, but there is better way</p>
<pre><code class="lang-auto">x1 = torch.mean(x1, dim=1, keepdim=True)
z = torch.zeros(m.shape[0], l, m.shape[2], m.shape[3]).to(m.device)
z[:, i:i+1] = x1
x2 = x2 + z
</code></pre>