How to interactively debug pytorch backprop errors?

username: sad_robot
cooked: <p>Hi, I’m a bit frustrated because I thought the whole point in pytorch doing “eager execution” is that it would enable interactive (e.g. pdb) debugging where any error-causing bug would immediately raise an exeception then pdb could interrupt and let you inspect?</p>
<p>I’m already using <code>with torch.autograd.detect_anomaly():</code> but it is giving me a traceback which is probably wrong since it contradicts the reported error (see below). Is there another setting which would give me this ideal interactive pdb debugging scenario I’m talking about?</p>
<p>The error I’m facing is:</p>
<pre><code class="lang-auto">RuntimeError: Expected all tensors to be on the same device, but 
found at least two devices, cpu and cuda:0! (when checking argument 
for argument self in method wrapper_CUDA_softplus_backward)
</code></pre>
<p>But the traceback points to a line which only involves one tensor argument so I have no idea how it is complaining about different devices.</p>
<pre><code class="lang-auto">File "/u/ddeighan/MOR_MoE2/model_agnostic_BNN.py", line 109, in forward
    sigma_params = nn.functional.softplus(self._rho_params)
</code></pre>
<details>
<summary>
Full TraceBack</summary>
<p>/u/ddeighan/MOR_MoE2/JHTDB_sim_op.py:199: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.<br>
with torch.autograd.detect_anomaly():<br>
/u/ddeighan/MOR_MoE2/model_agnostic_BNN.py:94: UserWarning: Anomaly Detection has been enabled. This mode will increase the runtime and should only be enabled for debugging.<br>
with torch.autograd.detect_anomaly():<br>
/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/autograd/graph.py:768: UserWarning: Error detected in SoftplusBackward0. Traceback of forward call that caused the error:<br>
File “”, line 198, in _run_module_as_main<br>
File “”, line 88, in _run_code<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/pdb.py”, line 1795, in <br>
pdb.main()<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/pdb.py”, line 1768, in main<br>
pdb._run(target)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/pdb.py”, line 1646, in _run<br>
self.run(target.code)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/bdb.py”, line 597, in run<br>
exec(cmd, globals, locals)<br>
File “”, line 1, in <br>
File “/u/ddeighan/MOR_MoE2/channel.py”, line 102, in <br>
model = SimModelClass(n_inputs=ndims, n_outputs=ndims, ndims=ndims, n_experts=n_experts, n_layers=n_layers, lr=lr, make_optim=make_optim, T_max=T_max,<br>
File “/u/ddeighan/MOR_MoE2/JHTDB_sim_op.py”, line 197, in <strong>init</strong><br>
super().<strong>init</strong>(*args, simulator=simulator, **kwd_args)<br>
File “/u/ddeighan/MOR_MoE2/JHTDB_sim_op.py”, line 168, in <strong>init</strong><br>
super().<strong>init</strong>(*args, **kwd_args)<br>
File “/u/ddeighan/MOR_MoE2/POU_net.py”, line 240, in <strong>init</strong><br>
model_agnostic_BNN.model_agnostic_dnn_to_bnn(self, train_dataset_size, prior_cfg=prior_cfg)<br>
File “/u/ddeighan/MOR_MoE2/model_agnostic_BNN.py”, line 146, in model_agnostic_dnn_to_bnn<br>
dnn.apply(visit_parametrize)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/modules/module.py”, line 895, in apply<br>
module.apply(fn)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/modules/module.py”, line 895, in apply<br>
module.apply(fn)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/modules/module.py”, line 895, in apply<br>
module.apply(fn)<br>
[Previous line repeated 1 more time]<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/modules/module.py”, line 896, in apply<br>
fn(self)<br>
File “/u/ddeighan/MOR_MoE2/model_agnostic_BNN.py”, line 144, in visit_parametrize<br>
parametrize.register_parametrization(module, name, _BayesianParameterization(param, **prior_cfg))<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/utils/parametrize.py”, line 573, in register_parametrization<br>
parametrizations = ParametrizationList([parametrization], original, unsafe=unsafe)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/utils/parametrize.py”, line 192, in <strong>init</strong><br>
Z = self()<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/utils/parametrize.py”, line 276, in forward<br>
x = self<a>0</a><br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/nn/modules/module.py”, line 1562, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/u/ddeighan/MOR_MoE2/model_agnostic_BNN.py”, line 109, in forward<br>
sigma_params = nn.functional.softplus(self._rho_params)<br>
(Triggered internally at /home/conda/feedstock_root/build_artifacts/libtorch_1727865399751/work/torch/csrc/autograd/python_anomaly_mode.cpp:111.)<br>
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
Traceback (most recent call last):<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/pdb.py”, line 1768, in main<br>
pdb._run(target)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/pdb.py”, line 1646, in _run<br>
self.run(target.code)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/bdb.py”, line 597, in run<br>
exec(cmd, globals, locals)<br>
File “”, line 1, in <br>
File “/u/ddeighan/MOR_MoE2/channel.py”, line 128, in <br>
trainer.fit(model=model, train_dataloaders=train_loader, val_dataloaders=val_dataloaders) #, ckpt_path=ckpt_path)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py”, line 539, in fit<br>
call._call_and_handle_interrupt(<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py”, line 47, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py”, line 575, in _fit_impl<br>
self._run(model, ckpt_path=ckpt_path)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py”, line 982, in _run<br>
results = self._run_stage()<br>
^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py”, line 1026, in _run_stage<br>
self.fit_loop.run()<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py”, line 216, in run<br>
self.advance()<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/fit_loop.py”, line 455, in advance<br>
self.epoch_loop.run(self._data_fetcher)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py”, line 150, in run<br>
self.advance(data_fetcher)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/training_epoch_loop.py”, line 320, in advance<br>
batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py”, line 192, in run<br>
self._optimizer_step(batch_idx, closure)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py”, line 270, in _optimizer_step<br>
call._call_lightning_module_hook(<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py”, line 171, in _call_lightning_module_hook<br>
output = fn(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/core/module.py”, line 1302, in optimizer_step<br>
optimizer.step(closure=optimizer_closure)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/core/optimizer.py”, line 154, in step<br>
step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py”, line 239, in optimizer_step<br>
return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/precision.py”, line 123, in optimizer_step<br>
return optimizer.step(closure=closure, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/optim/lr_scheduler.py”, line 130, in wrapper<br>
return func.<strong>get</strong>(opt, opt.<strong>class</strong>)(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/optim/optimizer.py”, line 484, in wrapper<br>
out = func(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/optim/optimizer.py”, line 89, in _use_grad<br>
ret = func(self, *args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/optim/adam.py”, line 205, in step<br>
loss = closure()<br>
^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/precision.py”, line 109, in _wrap_closure<br>
closure_result = closure()<br>
^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py”, line 146, in <strong>call</strong><br>
self._result = self.closure(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/utils/_contextlib.py”, line 116, in decorate_context<br>
return func(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py”, line 140, in closure<br>
self._backward_fn(step_output.closure_loss)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/loops/optimization/automatic.py”, line 241, in backward_fn<br>
call._call_strategy_hook(self.trainer, “backward”, loss, optimizer)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py”, line 323, in _call_strategy_hook<br>
output = fn(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py”, line 213, in backward<br>
self.precision_plugin.backward(closure_loss, self.lightning_module, optimizer, *args, **kwargs)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/precision.py”, line 73, in backward<br>
model.backward(tensor, *args, **kwargs)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/pytorch_lightning/core/module.py”, line 1097, in backward<br>
loss.backward(*args, **kwargs)<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/_tensor.py”, line 521, in backward<br>
torch.autograd.backward(<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/autograd/<strong>init</strong>.py”, line 289, in backward<br>
_engine_run_backward(<br>
File “/u/ddeighan/miniforge3/envs/uqops2/lib/python3.11/site-packages/torch/autograd/graph.py”, line 768, in _engine_run_backward<br>
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument self in method wrapper_CUDA_softplus_backward)</p>
</details>

username: ptrblck
cooked: <p><code>softplus_backward</code> uses <code>grad, self, beta, threshold</code> as its input arguments as seen <a href="https://github.com/pytorch/pytorch/blob/64957db6c92d2a5817a97c6da3f68488362f8983/tools/autograd/derivatives.yaml#L2213">here</a> so I guess the device mismatch might be raised between <code>grad</code> and <code>self</code>. Do you have a minimal code snippet to reproduce the issue?</p>

username: sad_robot
cooked: <p>Thanks for the insight! So I take it the interactive thing I was hoping for isn’t possible huh?<br>
I will try to get a minimal reproducible example but it is quite difficult &amp; I think it depends on the version of pytorch because this code worked on a different system.</p>