Setting part of the training value into zero

username: Yupjun
cooked: <p>When I training my network, I want to set a specific part of latent space into the zero value based on the label.</p>
<p>For example, after input values come out from convolution layer, there would be (B, C, H, W) 4-D tensor value would come out.</p>
<p>I want to make half of the Channel part into zero based on the label.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b4fed33427e269806ec13e429cabe57edadff0fc.png" alt="image" data-base62-sha1="pP9U5SQ2PHB7SKOFGtz8fHlGodm" width="435" height="202"></p>
<p>Like this but I got an error like this.</p>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 128, 15, 15]], which is output 0 of ReluBackward1, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</p>
<p>How can I assign a specific part of the tensor value into zero?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>This problem raises in two kind of situations:</p>
<ol>
<li>When you use <code>inplace=True</code> in your code and somehow regarding your model, backpropagation needs the values unchanged.</li>
<li>When you change a value like the image you have attached and backprop needs previously value again to compute gradient.</li>
</ol>
<p>In the first situation if cannot change the way you have implemented your model, you can just set <code>inplace=False</code> and it will be ok.</p>
<p>But in the second one, you have to preserve let assume <code>X</code> value.</p>
<p>Here is an small example you can test:</p>
<pre data-code-wrap="python"><code class="lang-python">A = torch.nn.Parameter(torch.randn(3, 3))
    B = torch.randn(2, 4, 3)
    B[0, :, :] = B[0, :, :].mm(A)
    loss = B.sum()
    loss.backward()
</code></pre>
<p>This will give you same error you have posted. And here is the solution:</p>
<pre data-code-wrap="python"><code class="lang-python">A = torch.nn.Parameter(torch.randn(3, 3))
B = torch.randn(2, 4, 3)
z = B[0, :, :].mm(A)
loss = z.sum()
loss.backward()
</code></pre>
<p>As you can see, in the second example, <code>B</code> matrix, in your case <code>X</code> has not been changed so gradient could be computed.</p>
<p>By the way, operations like <code>x += y</code>, <code>x[:, ...] = y</code> etc will be considered <code>inplace</code> operation.</p>
<aside class="quote" data-post="1" data-topic="51574">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/082a83b8eb9177051864/48/13609_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-this-code-will-throw-out-an-inplace-error/51574">Why this code will throw out an inplace error</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    A = torch.nn.Parameter(torch.randn(3, 3))
    B = torch.randn(2, 4, 3)
    B[0, :, :] = B[0, :, :].mm(A)
    loss = B.sum()
    loss.backward()

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4, 3]] is at version 1; expected version 0 instead. 
How could I solve this problem? <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title="slight_smile" alt="slight_smile" class="emoji">
  </blockquote>
</aside>

<p>Good luck</p>

username: Yupjun
cooked: <p>So, I think I solved the problem.</p>
<ol>
<li>I used inplace=False in previous layer</li>
<li>assigned 0 value in specific tensor</li>
<li>sent to next layer</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/827d22bdb9350219f1fb1ca797fb80dbf321aedc.png" data-download-href="https://discuss.pytorch.org/uploads/default/827d22bdb9350219f1fb1ca797fb80dbf321aedc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/827d22bdb9350219f1fb1ca797fb80dbf321aedc.png" alt="image" data-base62-sha1="iCmaJGtHqZPqr8A32G0aQ76Zts8" width="690" height="220" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">960×307 7.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you very much Nikronic!</p>

username: Satish1901
cooked: <p>Hello Yupjun<br>
The way you are setting parts of your latent vector. Does it take a lot of memory?<br>
Do you get : Cuda out of memory error</p>

username: Yupjun
cooked: <p>Nope It works fine with me. Sorry for late response</p>