Torch.max(), inplace operation

username: JUNHAOYAN
cooked: <p>I have been trying to figure this issue out by myself for a day but I have no idea what is going on here. The detailed error is shown below:</p>
<pre><code class="lang-auto">K:\Anaconda\envs\cv\lib\site-packages\torch\autograd\__init__.py:197: UserWarning: Error detected in MaxBackward0. Traceback of forward call that caused the error:
  File "K:\BaiduSyncdisk\CodeProject\pythonProject\DeepLearningForComputerVision\ObjectDetection\train.py", line 166, in &lt;module&gt;
    main(args)
  File "K:\BaiduSyncdisk\CodeProject\pythonProject\DeepLearningForComputerVision\ObjectDetection\train.py", line 94, in main
    mean_loss, lr = train_one_epoch(model, optimizer, train_data_loader,
  File "K:\BaiduSyncdisk\CodeProject\pythonProject\DeepLearningForComputerVision\ObjectDetection\train_utils\train_eval_utils.py", line 33, in train_one_epoch
    loss_dict = model(images, cats, bboxes)
  File "K:\Anaconda\envs\cv\lib\site-packages\torch\nn\modules\module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "K:\BaiduSyncdisk\CodeProject\pythonProject\DeepLearningForComputerVision\ObjectDetection\models\YOLO1.py", line 227, in forward
    pred_max_conf, max_conf_idx = torch.max(pred_conf, dim=-1)
  File "K:\Anaconda\envs\cv\lib\site-packages\torch\fx\traceback.py", line 57, in format_stack
    return traceback.format_stack()
 (Triggered internally at C:\cb\pytorch_1000000000000\work\torch\csrc\autograd\python_anomaly_mode.cpp:119.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "K:\BaiduSyncdisk\CodeProject\pythonProject\DeepLearningForComputerVision\ObjectDetection\train.py", line 166, in &lt;module&gt;
    main(args)
  File "K:\BaiduSyncdisk\CodeProject\pythonProject\DeepLearningForComputerVision\ObjectDetection\train.py", line 94, in main
    mean_loss, lr = train_one_epoch(model, optimizer, train_data_loader,
  File "K:\BaiduSyncdisk\CodeProject\pythonProject\DeepLearningForComputerVision\ObjectDetection\train_utils\train_eval_utils.py", line 55, in train_one_epoch
    losses.backward()
  File "K:\Anaconda\envs\cv\lib\site-packages\torch\_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "K:\Anaconda\envs\cv\lib\site-packages\torch\autograd\__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [98]] is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
<p>And here is the code produce this error (a part of it). It’s in a for loop, and the <strong>pred_conf</strong> is a <strong>[98, 2]</strong> tensor, I use <strong>torch.max()</strong> to produce the max value and the corresponding index along the last axis, which is the <strong>pred_max_conf[98, ]</strong> and <strong>max_conf_idx[98, ]</strong> respectively. The <strong>mask</strong> is a <strong>[98, ]</strong> boolean tensor, which is used to point out the position with / without objects. The <strong>self.conf_loss</strong> is used to calculate the loss between pred and gt, and then accumulate along batch.<br>
I dont think I have any inplace operations in <strong>pred_max_conf</strong>, so I’m so confusing.<br>
I’m very thankful if anybody can help!!!</p>
<pre><code class="lang-auto">        for pred_bbox, pred_conf, gt_bbox, gt_conf, mask in zip(pred_bboxes, pred_confs,
                                                                gt_bboxes, gt_confs,
                                                                masks):
            # todo: one of the variables needed for gradient computation has been modified by an inplace operation
            pred_max_conf, max_conf_idx = torch.max(pred_conf, dim=-1)
            max_conf_idx *= 5
            # confidence loss
            # balance loss between grids w and w/o objects
            a1, a2 = torch.sum(mask) / mask.size(0), torch.sum(~mask) / mask.size(0)

            loss_conf_w_obj = loss_conf_w_obj + self.conf_loss(pred_max_conf[mask], gt_conf[mask]) * a2
            loss_conf_wo_obj = loss_conf_wo_obj + self.conf_loss(pred_max_conf[~mask], gt_conf[~mask]) * a1

            # only count images with front ground and then filter grids without any objects
            gt_bbox = gt_bbox[mask]
            pred_bbox = pred_bbox[mask]
            if pred_bbox.numel() &gt; 0:
                pred_bbox = self.boxer.filter_bboxes(pred_bbox, max_conf_idx[mask])
                # coord loss
                loss_coord = loss_coord + self.coord_loss(pred_bbox, gt_bbox)
        loss_coord = loss_coord / B
        loss_conf_w_obj = loss_conf_w_obj / B
        loss_conf_wo_obj = loss_conf_wo_obj / B
        loss_class = self.ce_loss(pred_classes.permute(0, 2, 1), gt_cats.long())

        total_loss = 5 * loss_coord + 0.5 * loss_conf_wo_obj + loss_conf_w_obj + loss_class
</code></pre>
<p>the function self.conf_loss()</p>
<pre><code class="lang-auto">    @staticmethod
    def conf_loss(pred, gt):
        pred = torch.sigmoid(pred)
        loss = (pred - gt) ** 2

        return torch.sum(loss.reshape(-1)) / loss.size(0)
</code></pre>

username: JUNHAOYAN
cooked: <aside class="quote no-group" data-username="JUNHAOYAN" data-post="1" data-topic="165123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/junhaoyan/48/54571_2.png" class="avatar"> JUNHAOYAN:</div>
<blockquote>
<p><code>loss_dict </code></p>
</blockquote>
</aside>
<p>the pytorch version is 1.13.0</p>

username: KFrank
cooked: <p>Hi Junhao!</p>
<aside class="quote no-group quote-modified" data-username="JUNHAOYAN" data-post="1" data-topic="165123" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/junhaoyan/48/54571_2.png" class="avatar"> JUNHAOYAN:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [98]] is at version 1; expected version 0 instead.
</code></pre>
<p>…</p>
<pre><code class="lang-auto">            max_conf_idx *= 5
</code></pre>
</blockquote>
</aside>
<p><code>*=</code> is an in-place operation.  From what you’ve said, <code>max_conf_idx</code> will<br>
be a <code>LongTensor</code> of shape <code>[98]</code>, so this is almost certainly your culprit.</p>
<p>As a work-around you could try <code>max_conf_idx = 5 * max_conf_idx</code>.</p>
<p>(This version creates a new tensor and sets the python reference<br>
<code>max_conf_idx</code> to refer to it, leaving the old version of <code>max_conf_idx</code><br>
unchanged for use in that part of the backward pass.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: JUNHAOYAN
cooked: <p>Thank you so much! It works! But I’m quit confusing.<br>
<strong>max_conf_idx.requires_grad == False</strong>, I think the <strong>max_conf_idx</strong> is not used for gradient computation, that’s why I haven’t thought this problem is raised by the <strong>max_conf_idx</strong>.</p>

username: KFrank
cooked: <p>Hi Junhao!</p>
<aside class="quote no-group" data-username="JUNHAOYAN" data-post="4" data-topic="165123" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/junhaoyan/48/54571_2.png" class="avatar"> JUNHAOYAN:</div>
<blockquote>
<p>I think the <strong>max_conf_idx</strong> is not used for gradient computation</p>
</blockquote>
</aside>
<p>Autograd does not <em>backpropagate</em> through <code>max_conf_idx</code> (that is, it does<br>
not track what upstream variables <code>max_conf_idx</code> might depend on), but it<br>
does <em>use</em> <code>max_conf_idx</code>  in the gradient computation.</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.ones (3, requires_grad = True)
&gt;&gt;&gt; idx = torch.tensor ([2, 1, 0])   # long tensor, no grad possible
&gt;&gt;&gt; f = torch.tensor ([3.0, 5.0, 7.0])
&gt;&gt;&gt;
&gt;&gt;&gt; loss = (f * t[idx]).sum()
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; t.grad   # backward uses idx to map grads to the correct elements of t
tensor([7., 5., 3.])
&gt;&gt;&gt;
&gt;&gt;&gt; t.grad = None
&gt;&gt;&gt;
&gt;&gt;&gt; loss = (f * t[idx]).sum()
&gt;&gt;&gt; idx *= 0   # in-place operation on idx -- correct mapping is lost
&gt;&gt;&gt; loss.backward()   # raises inplace error
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.LongTensor [3]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
</code></pre>
<p>(Note, <code>idx</code> is a LongTensor, so it can’t have <code>requires_grad = True</code> and<br>
you can’t backpropagate through it, but it is part of the forward pass and is<br>
still needed to perform the gradient computation correctly.)</p>
<p>Best.</p>
<p>K. Frank</p>