RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 6890]], which is output 0 of SelectBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detect

julyaugust12345: <p>I am trying to reproduct AvatarCLIP in pytorch.<br>
And I got the following error:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 6890]], which is output 0 of SelectBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).<br>
It would be very appreciate if you help me.</p>
<pre><code class="lang-auto">    def get_pose(self, text_feature: Tensor) -&gt; Tensor:
        latent_code = nn.Parameter(torch.randn(32))
        cls = getattr(torch.optim, self.optim_name)
        optimizer = cls([latent_code], **self.optim_cfg)
        for i in tqdm(range(self.num_iteration)):
            new_latent_code = latent_code.to(self.device).unsqueeze(0)
            new_pose = self.vp.decode(new_latent_code)['pose_body']
            new_pose = new_pose.contiguous().view(-1)
            clip_feature = self.get_pose_feature(new_pose).squeeze(0)
            loss = 1 - F.cosine_similarity(clip_feature, text_feature, dim=-1)
            loss = loss.mean()
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        return pose_padding(new_pose.detach())

    def get_topk_poses(self, text: str) -&gt; Tensor:
        text_feature = self.get_text_feature(text)
        poses = [self.get_pose(text_feature) for _ in range(self.topk)]
        poses = self.sort_poses_by_score(text, poses)
        poses = torch.stack(poses, dim=0)
        return poses
</code></pre>
<pre><code class="lang-auto">    def get_pose(self, text_feature: Tensor) -&gt; Tensor:
        pose = nn.Parameter(torch.randn(63))
        cls = getattr(torch.optim, self.optim_name)
        optimizer = cls([pose], **self.optim_cfg)
        for i in tqdm(range(self.num_iteration)):
            new_pose = pose.to(self.device)
            clip_feature = self.get_pose_feature(new_pose).squeeze(0)
            loss = 1 - F.cosine_similarity(clip_feature, text_feature, dim=-1)
            loss = loss.mean()
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        return pose_padding(pose.data).to(self.device)

    def get_topk_poses(self, text: str) -&gt; Tensor:
        text_feature = self.get_text_feature(text)
        poses = [self.get_pose(text_feature) for _ in range(self.topk)]
        poses = self.sort_poses_by_score(text, poses)
        poses = torch.stack(poses, dim=0)
        return poses
</code></pre>

KFrank: <p>Hi July!</p>
<aside class="quote no-group" data-username="julyaugust12345" data-post="1" data-topic="195884" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/julyaugust12345/48/67160_2.png" class="avatar"> julyaugust12345:</div>
<blockquote>
<p>inplace operation: [torch.cuda.FloatTensor [1, 6890]]</p>
</blockquote>
</aside>
<p>Can you identify a tensor in your forward pass that has shape <code>[1, 6890]</code>?<br>
See if you can find where it is being modified.</p>
<aside class="quote no-group">
<blockquote>
<p>Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
</aside>
<p>Try using <code>set_detect_anomaly (True)</code>.  Doing so provides additional<br>
information that can be very helpful in tracking down the cause of your<br>
error.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    def get_pose(self, text_feature: Tensor) -&gt; Tensor:
</code></pre>
</blockquote>
</aside>
<p>There’s a lot of code that you haven’t posted that could be causing your<br>
error.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    def get_pose(self, text_feature: Tensor) -&gt; Tensor:
        ...
        return pose_padding(pose.data).to(self.device)
</code></pre>
</blockquote>
</aside>
<p>You’ve posted two nearly identical versions of <code>get_pose()</code> (and we can’t<br>
tell which one is being run when you get your error).</p>
<p>Note, this second version uses <code>pose.data</code>.  <code>.data</code> is deprecated (in the<br>
public api) and can lead to errors, so you shouldn’t be using it.</p>
<p>For suggestions about how to find and fix inplace-modification errors,<br>
please see the following post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<p>Good luck!</p>
<p>K. Frank</p>