RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8, 1, 120, 224]], which is output 0 of SumBackward1, is at version 1; expected version 0 instead. Hint: the backtrace fu

Veshnu_Ramakrishnan: <p>I am trying to run end-to-end stereo code.</p>
<p>I am getting the following error.</p>
<pre><code class="lang-auto">python_anomaly_mode.cpp:104] Warning: Error detected in MaxPool2DWithIndicesBackward. Traceback of forward call that caused the error:
  File "train.py", line 85, in &lt;module&gt;
    trainer.fit(stereo_model, data_module)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit
    self._run(model)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run
    self._dispatch()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _dispatch
    self.accelerator.start_training(self)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in run_stage
    return self._run_train()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1045, in _run_train
    self.fit_loop.run()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 130, in advance
    batch_output = self.batch_loop.run(batch, self.iteration_count, self._dataloader_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 101, in run
    super().run(batch, batch_idx, dataloader_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 148, in advance
    result = self._run_optimization(batch_idx, split_batch, opt_idx, optimizer)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 202, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 396, in _optimizer_step
    model_ref.optimizer_step(
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1618, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 209, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 129, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 292, in optimizer_step
    make_optimizer_step = self.precision_plugin.pre_optimizer_step(
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 59, in pre_optimizer_step
    result = lambda_closure()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 236, in _training_step_and_backward_closure
    result = self.training_step_and_backward(split_batch, batch_idx, opt_idx, optimizer, hiddens)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 537, in training_step_and_backward
    result = self._training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 307, in _training_step
    training_step_output = self.trainer.accelerator.training_step(step_kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 193, in training_step
    return self.training_type_plugin.training_step(*step_kwargs.values())
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 172, in training_step
    return self.model.training_step(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/model.py", line 63, in training_step
    output, all_outputs = self(batch)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/model.py", line 52, in forward
    output, all_outputs = self.model(left_image, right_image)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "models/hdrn_alpha_stereo.py", line 394, in forward
    disparity = self.disparity_refinement(left_image, disparity_small, matchability, cost_volume)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "models/hdrn_alpha_stereo.py", line 278, in forward
    base_downsample = self.base_downsample(disparity)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/modules/pooling.py", line 162, in forward
    return F.max_pool2d(input, self.kernel_size, self.stride,
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/_jit_internal.py", line 405, in fn
    return if_false(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 718, in _max_pool2d
    return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
 (function _print_stack)
Traceback (most recent call last):
  File "train.py", line 85, in &lt;module&gt;
    trainer.fit(stereo_model, data_module)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 553, in fit
    self._run(model)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run
    self._dispatch()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 986, in _dispatch
    self.accelerator.start_training(self)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 996, in run_stage
    return self._run_train()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1045, in _run_train
    self.fit_loop.run()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 200, in advance
    epoch_output = self.epoch_loop.run(train_dataloader)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 130, in advance
    batch_output = self.batch_loop.run(batch, self.iteration_count, self._dataloader_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 101, in run
    super().run(batch, batch_idx, dataloader_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 148, in advance
    result = self._run_optimization(batch_idx, split_batch, opt_idx, optimizer)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 202, in _run_optimization
    self._optimizer_step(optimizer, opt_idx, batch_idx, closure)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 396, in _optimizer_step
    model_ref.optimizer_step(
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1618, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 209, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 129, in __optimizer_step
    trainer.accelerator.optimizer_step(optimizer, self._optimizer_idx, lambda_closure=closure, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 292, in optimizer_step
    make_optimizer_step = self.precision_plugin.pre_optimizer_step(
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 59, in pre_optimizer_step
    result = lambda_closure()
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 236, in _training_step_and_backward_closure
    result = self.training_step_and_backward(split_batch, batch_idx, opt_idx, optimizer, hiddens)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 549, in training_step_and_backward
    self.backward(result, optimizer, opt_idx)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 590, in backward
    result.closure_loss = self.trainer.accelerator.backward(result.closure_loss, optimizer, *args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 276, in backward
    self.precision_plugin.backward(self.lightning_module, closure_loss, *args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 78, in backward
    model.backward(closure_loss, optimizer, *args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1481, in backward
    loss.backward(*args, **kwargs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/veshnu/gitcodes/mmstereo/venv/lib/python3.8/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8, 1, 120, 224]], which is output 0 of SumBackward1, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

</code></pre>

ptrblck: <p>It’s a bit hard to narrow down the issue from the stacktrace, but you should check if</p>
<ul>
<li>you are explicitly using inplace operations and replace them with their out-of-place equivalents</li>
<li>you are trying to use stale forward activations as described in <a href="https://discuss.pytorch.org/t/struggling-with-a-runtime-error-related-to-in-place-operations/102222/6">this post</a>
</li>
</ul>

Veshnu_Ramakrishnan: <p>There were some Relu with inplace, which I removed.</p>
<p>I am using pytorch lightning, so not manually calling backward twice.</p>
<p>The error occurs on the backward pass on MaxPool2d layer.</p>