RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation (New)

username: Zador
cooked: <p>Hi I am stuck with an error and I do not know what I am looking for. I have seen previous posts with a similar error but their fixes did not apply for my case.</p>
<p>The error summary:</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an in place<br>
operation: [torch.cuda.HalfTensor [64, 32]], which is output 0 of TBackward, is at version 2; expected<br>
version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its<br>
gradient. The variable in question was changed in there or anywhere later. Good luck!```</p>
</blockquote>
<p>Full error:</p>
<blockquote>
<p>[W python_anomaly_mode.cpp:104] Warning: Error detected in AddmmBackward. Traceback of forward call that caused the error:<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/train.py”, line 123, in <br>
train.fit(n_epochs=n_epochs, vb_size=vb_size, train_dataset=train_dataset,<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/framework.py”, line 111, in fit<br>
for i, data in (enumerate(tqdm(self.train_loader)) if self.verbose &gt;= 2 else enumerate(self.train_loader)):<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/tqdm/std.py”, line 1180, in <strong>iter</strong><br>
for obj in iterable:<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 521, in <strong>next</strong><br>
data = self._next_data()<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/utils/data/dataloader.py”, line 561, in _next_data<br>
data = self._dataset_fetcher.fetch(index)  # may raise StopIteration<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in fetch<br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py”, line 44, in <br>
data = [self.dataset[idx] for idx in possibly_batched_index]<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/dataset.py”, line 81, in <strong>getitem</strong><br>
return self.load_file(scene_idx)<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/dataset.py”, line 118, in load_file<br>
mesh.fusion(features[i], view[:,:-1], view[:,-1].long(), depth[i])<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/mesh/mesh.py”, line 150, in fusion<br>
self.fc_fusion(index, latent_features_flat)<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/mesh/mesh.py”, line 309, in fc_fusion<br>
self.feature_volume[inds_0, inds_1, inds_2, :] = self.fusion_learner(torch.cat(inputs_i), self.feature_volume[inds_0, inds_1, inds_2, :])<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/learned_fusion.py”, line 13, in forward<br>
return self.fc(a)<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/nn/modules/module.py”, line 1051, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/nn/modules/linear.py”, line 96, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/nn/functional.py”, line 1847, in linear<br>
return torch._C.<em>nn.linear(input, weight, bias)<br>
(function <em>print_stack)<br>
0%|                                 | 0/600 [00:22&lt;?, ?it/s]<br>
Traceback (most recent call last):<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/train.py”, line 123, in <br>
train.fit(n_epochs=n_epochs, vb_size=vb_size, train_dataset=train_dataset,<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/framework.py”, line 140, in fit<br>
loss = self.train_step(inputs[idx], torch.tensor(targets[idx], device=self.device, dtype=torch.long), optimizer, last=last</em>)<br>
File “/cluster/work/riner/patakiz/asl/asldoc-2021-SA-Zador-How-to-fuse/framework.py”, line 213, in train_step<br>
loss.backward(retain_graph=retain_graph</em>)<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/_tensor.py”, line 255, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/cluster/home/patakiz/miniconda3/envs/myenv/lib/python3.9/site-packages/torch/autograd/<strong>init</strong>.py”, line 147, in backward<br>
Variable._execution_engine.run_backward(</p>
</blockquote>
<p>I am using two neural networks. <code>self.net</code> and <code>self.fusion_learner</code>. These are stored in my <code>Framework</code> class which handles most things. Additionally I have a dataset where I store  copy of <code>self.fusion_learner</code> as I am nesting the fusion learner network in a data loader since I am in part doing learned data preprocessing.</p>
<p><a href="http://self.net" rel="noopener nofollow ugc">self.net</a>:</p>
<pre><code class="lang-auto">class Task(nn.Module):
    def __init__(self, fusion=None, n_mult=None, kmeans_strat=None):
        super(Task, self).__init__()
        mult = 1

        self.conv1 =  nn.Conv3d(32*mult, 16, kernel_size=[3,3,3], stride=2, padding=0, bias=False)
        self.bnorm1 = nn.BatchNorm3d(16)
        self.relu = nn.ReLU(False)
        self.conv2 = nn.Conv3d(16, 16, kernel_size=1, stride=1, padding=0, bias=False)
        self.dropout = nn.Dropout3d(0.2)

        self.conv3 = nn.Conv3d(16, 8, kernel_size=[3,3,3], stride=2, padding=0, bias=False)
        self.bnorm2 = nn.BatchNorm3d(8)
        self.conv4 = nn.Conv3d(8, 8, kernel_size=1, stride=1, padding=0)
        self.conv5 = nn.Conv3d(8, 8, kernel_size=1, stride=1, padding=0)

        self.convtranspose5 = nn.ConvTranspose3d(8, 8, kernel_size=1, stride=1, padding=0)
        self.convtranspose4 = nn.ConvTranspose3d(8, 8, kernel_size=1, stride=1, padding=0)
        self.convtranspose3 = nn.ConvTranspose3d(8, 16, kernel_size=[3,3,3], stride=2, padding=0)
        self.convtranspose2 = nn.ConvTranspose3d(16, 16, kernel_size=1, stride=1, padding=0)
        self.convtranspose1 = nn.ConvTranspose3d(16, 32, kernel_size=[3,3,3], stride=2, padding=0)
        self.convtranspose_out = nn.Conv3d(32, 14, 1, 1, 0)

    def forward(self, feature_volume):
        x = self.relu(self.bnorm1(self.conv1(feature_volume)))
        x = self.relu(self.bnorm1(self.conv2(x)))
        x = self.relu(self.bnorm1(self.conv2(x)))
        x = self.relu(self.bnorm2(self.conv3(x)))
        x = self.relu(self.bnorm2(self.conv4(x)))
        x = self.relu(self.bnorm2(self.conv5(x)))
        x = self.dropout(x)

        x = self.relu(self.convtranspose5(x))
        x = self.relu(self.convtranspose4(x))
        x = self.relu(self.convtranspose3(x))
        x = self.relu(self.convtranspose2(x))
        x = self.relu(self.convtranspose2(x))
        x = self.relu(self.convtranspose1(x))
        x = self.convtranspose_out(x)
        return x
</code></pre>
<p>self.fusion_learner:</p>
<pre><code class="lang-auto">class FC_fusion(nn.Module):
    def __init__(self):
        super(FC_fusion, self).__init__()

        self.fc = nn.Linear(64, 32)

    def forward(self, X_new, X_fused):
        if X_fused.dim()==1:
            X_fused = X_fused[None,:]
        a = torch.cat([X_new, X_fused], dim=1)
        return self.fc(a)
</code></pre>
<p>Forward pass of <a href="http://self.net" rel="noopener nofollow ugc">self.net</a> is not interesting: <code>self.net(input)</code>.<br>
I pass through self.fusion_learner many times:</p>
<pre><code class="lang-auto">...
for i in range(np.max(batches_len)):
    inputs_i = []
    sorted_indices_i = []
    for j in range(len(batches_len)):
        if i &lt; batches_len[j]:
            inputs_i.append(packed_inputs_list[j][bs_sum[j]:batch_sizes_list[j][i]])
            bs_sum[j]+=batch_sizes_list[j][i]
            sorted_indices_i.append(sorted_indices_list[j][:inputs_i[-1].shape[0]])
    sorted_indices_i = torch.cat(sorted_indices_i)
    inds_0, inds_1, inds_2 = indices[0][sorted_indices_i], indices[1][sorted_indices_i], indices[2][sorted_indices_i]

    self.feature_volume[inds_0, inds_1, inds_2, :] = self.fusion_learner(torch.cat(inputs_i), self.feature_volume[inds_0, inds_1, inds_2, :])
</code></pre>
<p>In fact, I run this multiple times before any backpropagation. Here the output of the net updates this feature volume, which is used as an input to the net in addition to a new vector. As a reminder, all of this is done inside of the DataLoader (i.e. this is done in the . In the end the “<strong>getitem</strong>” function (cant do __ __ without bold).</p>
<p>Here is how I define my optimizer:</p>
<pre><code class="lang-auto">optimizer = Adam(list(self.net.parameters()) + list(self.fusion_learner.parameters()), eps=1e-04)
</code></pre>
<p>and here is how I do the optimization step:</p>
<pre><code class="lang-auto">optimizer.zero_grad()
pred_flat, target_flat = self.net_step(inputs, classes)
loss = self.crit(pred_flat, target_flat)
if last == False:
    retain_graph_=True
else:
    retain_graph_=False
loss.backward(retain_graph=retain_graph_)
optimizer.step()
</code></pre>
<p>Of course throughout this process there are many more lines of code (self.net_step contains <a href="http://self.net" rel="noopener nofollow ugc">self.net</a>() for example) but I cant include everything.</p>
<p>Does anybody have any ideas?</p>

username: ptrblck
cooked: <p>Based on the stacktrace I would guess that this inplace manipulation is disallowed:</p>
<pre><code class="lang-python">self.feature_volume[inds_0, inds_1, inds_2, :] = self.fusion_learner(torch.cat(inputs_i), self.feature_volume[inds_0, inds_1, inds_2, :])
</code></pre>
<p>so try to replace it with assigning the result to a new tensor instead.</p>