Gradient computation issue due to inplace operation, unsure how to debug for custom model

Zahl: <p>I am getting a gradient computation error and unsure how to fix it, ‘one of the variables needed for gradient computation has been modified by an inplace operation’. I am trying to create a model where the inputs can undergo some arbitraty tranformation specified by a function during the network execution. In this simple example the two transformations are adding or multiplying the top 2 elements of the three element input and storing result in third element. The results of this are then mixed by the model by weighting with alphas, the actual parameters of the model to give the output. Now I understand that I am doing some inplace operations with these transformations of the input but I want to learn the alpha parameters of the model and believe all the necessary variables exsist to do this and do not get overwritten anywhere. Am I making a theoretical mistake or not understanding why PyTorch is not happy with what im doing?</p>
<pre><code class="lang-auto">def addb(s):
  s[2] = s[0] + s[1]
  return s

def multb(s):
  s[2] = s[0]*s[1]
  return s

skills = [addb, multb]

class AddS(nn.Module):
    def __init__(self, num_skills, state_len) -&gt; None:
        super().__init__()
        self.num_skills = num_skills
        self.state_len = state_len
        self.alphas = torch.ones(2)
        self.alphas = torch.nn.parameter.Parameter(self.alphas)
        self.softmax = torch.nn.Softmax(dim = 0)
    
    def forward(self, x):
      out = torch.zeros(x.size()[0], x.size()[1])
      alphas_p = self.softmax(self.alphas)

      inter = torch.zeros(x.size()[0], self.num_skills, self.state_len)

      for k in range(x.size()[0]):
        for i in range(self.num_skills):
          inter[k, i] = skills[i](x[k].clone())
        out[k] = alphas_p[0]*inter[k,0] + alphas_p[1]*inter[k,1]

      return out

input, labels = torch.tensor([[1,2,0], [2,3,0]]), torch.tensor([3., 5.])

model = AddS(2, 3)

criterion = nn.MSELoss()

outputs = model(input)

loss = criterion(outputs[:, 2], labels)

loss.backward()
</code></pre>

KFrank: <p>Hi Sahil!</p>
<aside class="quote no-group" data-username="Zahl" data-post="1" data-topic="170133" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zahl/48/44762_2.png" class="avatar"> Zahl:</div>
<blockquote>
<pre><code class="lang-auto">        out[k] = alphas_p[0]*inter[k,0] + alphas_p[1]*inter[k,1]

      return out
</code></pre>
</blockquote>
</aside>
<p>The tensor <code>out</code> is in the computation graph and assigning to an element<br>
of a tensor by indexing into it is an inplace operation, hence your error.</p>
<p>Best.</p>
<p>K. Frank</p>

Zahl: <p>Hi KFrank,</p>
<p>Thanks for your response, this is something that I’ve done before in other models and it doesn’t cause an issue, so I have yet to properly understand when PyTorch is ok with an inplace operations. In fact for this code example I was actually able to find a fix by replacing</p>
<pre><code class="lang-auto">out[k] = alphas_p[0]*inter[k,0] + alphas_p[1]*inter[k,1]
</code></pre>
<p>with</p>
<pre><code class="lang-auto">out[k] = alphas_p @ inter[k].clone()
</code></pre>
<p>The cloning being the key fix.</p>
<p>However, this is not my real problem, this was supposed to be a simplified example. I will try adapt it to be closer to my real issue.</p>

KFrank: <p>Hi Sahil!</p>
<aside class="quote no-group" data-username="Zahl" data-post="3" data-topic="170133" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zahl/48/44762_2.png" class="avatar"> Zahl:</div>
<blockquote>
<p>I have yet to properly understand when PyTorch is ok with an inplace operations.</p>
</blockquote>
</aside>
<p>An inplace modification of a tensor that is in the computation graph does<br>
not necessarily cause an error – it depends on whether the that tensor’s<br>
<em>value</em> is actually used in the backward pass.  For some explanation, please<br>
see this post:</p>
<aside class="quote quote-modified" data-post="3" data-topic="169583">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/when-inplace-operation-are-allowed-and-when-not/169583/3">When inplace operation are allowed and when not</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Jie (and Srishti)! 

The part that can be confusing is that a given tensor used in the forward 
pass may or may not be used in the backward pass.  It depends on the math 
of the gradient computation (and the details of how the relevant backward() 
function is implemented).  It’s not always obvious. 
Here are some explanatory examples: 
The derivative of exp (x) is exp (x).  So it makes sense that pytorch saves 
the output of exp (x) when it is computed during the forward pass so that it 
can …
  </blockquote>
</aside>

<aside class="quote no-group">
<blockquote>
<p>In fact for this code example I was actually able to find a fix by replacing</p>
<pre><code class="lang-auto">out[k] = alphas_p[0]*inter[k,0] + alphas_p[1]*inter[k,1]
</code></pre>
<p>with</p>
<pre><code class="lang-auto">out[k] = alphas_p @ inter[k].clone()
</code></pre>
<p>The cloning being the key fix.</p>
</blockquote>
</aside>
<p>From your first post:</p>
<aside class="quote no-group" data-username="Zahl" data-post="1" data-topic="170133">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zahl/48/44762_2.png" class="avatar"> Zahl:</div>
<blockquote>
<pre><code class="lang-auto">          inter[k, i] = skills[i](x[k].clone())
        out[k] = alphas_p[0]*inter[k,0] + alphas_p[1]*inter[k,1]
</code></pre>
</blockquote>
</aside>
<p><code>inter</code> is also in the computation graph and is also being modified inplace<br>
(assignment to indexed elements).  Because cloning <code>inter</code> fixes your inplace<br>
error, modifying <code>inter</code> via indexing was likely the cause of your error.</p>
<p>Best.</p>
<p>K. Frank</p>

Zahl: <p>Hi K. Frank,</p>
<p>Ah, thanks for that. I think it is slightly clearer now. I guess cloning meant new variables were created, none of which were modified in place and so it works out. Ok so now moving towards my actual model I have introduced a small change, now for each input the add tranformation takes place on one copy of the input, the multiply on another copy, these are stored in inter and then weighted sum to give x_k, I then take x_k and simply just repeat. I take copies of x_k run each of the transformations and weighted sum. I get the same inplace operation error. Not sure what the problem is now as I clone and dont inplace modify those things. Interestingly the error message points to an empty tensor, not sure what to make of that ‘[torch.FloatTensor []], which is output 0 of AsStridedBackward0’.</p>
<p>Here is the modified forward</p>
<pre><code class="lang-auto">    def forward(self, x):
      out = torch.zeros(x.size()[0], x.size()[1])
      alphas_p = self.softmax(self.alphas)

      inter = torch.zeros(x.size()[0], self.num_skills, self.state_len)
      inter2 = torch.zeros(x.size()[0], self.num_skills, self.state_len)

      for k in range(x.size()[0]):
        for i in range(self.num_skills):
          inter[k, i] = skills[i](x[k].clone())
        x_k = alphas_p @ inter[k].clone()
        for i in range(self.num_skills):
          inter2[k, i] = skills[i](x_k.clone())
        out[k] = alphas_p @ inter2[k].clone()

      return out
</code></pre>
<p>Many thanks,<br>
S.</p>

KFrank: <p>Hi Sahil!</p>
<aside class="quote no-group" data-username="Zahl" data-post="5" data-topic="170133" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zahl/48/44762_2.png" class="avatar"> Zahl:</div>
<blockquote>
<p>I get the same inplace operation error.</p>
</blockquote>
</aside>
<p>Try wrapping your training loop in a <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly" rel="noopener nofollow ugc">with autograd.detect_anomaly():</a> block.<br>
When the backward-pass inplace error occurs, you should get a backtrace<br>
to the forward-pass inplace modification that is the root cause of the error.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">      for k in range(x.size()[0]):
        for i in range(self.num_skills):
          inter[k, i] = skills[i](x[k].clone())
        x_k = alphas_p @ inter[k].clone()
        for i in range(self.num_skills):
          inter2[k, i] = skills[i](x_k.clone())
        out[k] = alphas_p @ inter2[k].clone()
</code></pre>
</blockquote>
</aside>
<p>I’m not sure where the issue is.  But some comments:</p>
<p>You are still modifying <code>inter</code>, <code>inter2</code>, and <code>out</code> inplace.  Note that<br>
<code>inter[k].clone()</code> isn’t cloning <code>inter</code>, but, rather, <code>inter[k]</code>, a<br>
different tensor (that is a view into <code>inter</code>).</p>
<p>You never use all of <code>inter</code> and <code>inter2</code> at once – just their <code>k</code> slices.<br>
You might try something like:</p>
<pre data-code-wrap="python"><code class="lang-python">    def forward(self, x):
      out_list = []
      alphas_p = self.softmax(self.alphas)

      inter_k = torch.zeros(self.num_skills, self.state_len)
      inter2_k = torch.zeros(self.num_skills, self.state_len)

      for k in range(x.size()[0]):
        for i in range(self.num_skills):
          inter_k[i] = skills[i](x[k])
        x_k = alphas_p @ inter_k
        for i in range(self.num_skills):
          inter2_[i] = skills[i](x_k)
        out.list.append  = (alphas_p @ inter2_k)
      
      out = torch.stack (out_list)
      
      return out
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Mary84: <p>I really appreciate your help. Thank you for going above and beyond!</p>