One of the variables needed for gradient computation has been modified by an inplace operation in CrossEntropyLoss()

username: Sivagami_S_N
cooked: <p>In pytorch 1.6,<br>
with torch.autograd.set_detect_anomaly(True):<br>
pointed at torch.nn.CrossEntropyLoss()</p>
<blockquote>
<p>[W python_anomaly_mode.cpp:60] Warning: Error detected in AddmmBackward. Traceback of forward call that caused the error:<br>
(pid=23381)   File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/ray/workers/default_worker.py”, line 124, in <br>
(pid=23381)     ray.worker.global_worker.main_loop()<br>
(pid=23381)   File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/ray/worker.py”, line 421, in main_loop<br>
(pid=23381)     self.core_worker.run_task_loop()<br>
(pid=23381)   File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/ray/function_manager.py”, line 559, in actor_method_executor<br>
(pid=23381)     method_returns = method(actor, *args, **kwargs)<br>
(pid=23381)   File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/ray/tune/trainable.py”, line 261, in train<br>
(pid=23381)     result = self._train()<br>
(pid=23381)   File “/mnt/new_hdd5/sivagami/train_conv/train_conv/trainable_cls.py”, line 638, in _train<br>
(pid=23381)     vector_dict = <a href="http://self.net" rel="noopener nofollow ugc">self.net</a>(data, out_layers=self.out_layers)  # Forward Propagation<br>
(pid=23381)   File “/mnt/new_hdd5/sivagami/train_conv/train_conv/architecture/BaseNet.py”, line 101, in <strong>call</strong><br>
(pid=23381)     store = module(store)<br>
(pid=23381)   File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 723, in _call_impl<br>
(pid=23381)     result = self.forward(*input, **kwargs)<br>
(pid=23381)   File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/torch/nn/modules/linear.py”, line 93, in forward<br>
(pid=23381)     return F.linear(input, self.weight, self.bias)<br>
(pid=23381)   File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/torch/nn/functional.py”, line 1674, in linear<br>
(pid=23381)     ret = torch.addmm(bias, input, weight.t())<br>
(pid=23381)  (function print_stack)<br>
File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py”, line 431, in fetch_result<br>
result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT)<br>
File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/ray/worker.py”, line 1515, in get<br>
raise value.as_instanceof_cause()<br>
ray.exceptions.RayTaskError(RuntimeError): ray::TrainConvTrainableMLNT.train() (pid=23381, ip=192.168.0.24)<br>
File “python/ray/_raylet.pyx”, line 463, in ray._raylet.execute_task<br>
File “python/ray/_raylet.pyx”, line 417, in ray._raylet.execute_task.function_executor<br>
File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/ray/tune/trainable.py”, line 261, in train<br>
result = self._train()<br>
File “/mnt/new_hdd5/sivagami/train_conv/train_conv/trainable_cls.py”, line 691, in _train<br>
grads = torch.autograd.grad(fast_loss, self.net.parameters(), create_graph=True, retain_graph=True, only_inputs=True)<br>
File “/home/cv/.virtualenvs/trainconv_env/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 192, in grad<br>
inputs, allow_unused)<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [512, 5]], which is output 0 of TBackward, is at version 19; e<br>
xpected version 17 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good lu<br>
ck!</p>
</blockquote>
<p>Any solution for this?</p>

username: ptrblck
cooked: <p>Without seeing the code I would recommend to remove obvious inplace operations (e.g. via <code>+=</code> etc. or via the inplace version of some operations such as <code>tensor.add_()</code>).<br>
If that’ doesn’t help, you could add <code>.clone()</code> statements to return tensors to further isolate which operation is causing the inplace manipulation and thus the error.</p>