AttributeError: 'NoneType' object has no attribute 'reducer' during DDP backward

username: Apurva_Mhatre
cooked: <p>When training with DDP (spawn/subprocess launcher), the backward pass intermittently crashes with the reported error. The baseline job (without padding or compilation) runs successfully. The issue only arises in a modified setup where the token length is always padded to 200 and the model is trained with <code>torch.compile</code>. Notably, if training is resumed from the latest checkpoint, the job completes without error.</p>
<p>Also when I use LayerNorm this issue does not occur, this issue only occurs with BatchNorm(I am using 1D).</p>
<p>Error log:</p>
<pre><code class="lang-auto">2025-09-09T08:17:46.604-07:00
Traceback (most recent call last): File "/opt/ml/code/sm_entrypoint.py", line 7, in &lt;module&gt;
2025-09-09T08:17:46.604-07:00
entry_point() File "/opt/conda/lib/python3.10/site-packages/otto_training/cli/run_engine.py", line 295, in entry_point
2025-09-09T08:17:46.604-07:00
main(cfg) File "/opt/conda/lib/python3.10/site-packages/otto_training/cli/run_engine.py", line 272, in main
2025-09-09T08:17:46.604-07:00
run(cfg)
2025-09-09T08:17:46.604-07:00
File "/opt/conda/lib/python3.10/site-packages/otto_training/cli/run_engine.py", line 224, in run
2025-09-09T08:17:46.604-07:00
trainer.fit( File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 543, in fit
2025-09-09T08:17:46.604-07:00
call._call_and_handle_interrupt( File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 43, in _call_and_handle_interrupt
2025-09-09T08:17:46.604-07:00
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
2025-09-09T08:17:46.604-07:00
File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 105, in launch
2025-09-09T08:17:46.604-07:00
return function(*args, **kwargs)
2025-09-09T08:17:46.604-07:00
File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 579, in _fit_impl
2025-09-09T08:17:46.604-07:00
self._run(model, ckpt_path=ckpt_path) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 986, in _run
2025-09-09T08:17:46.604-07:00
results = self._run_stage() File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1032, in _run_stage
2025-09-09T08:17:46.604-07:00
self.fit_loop.run() File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 205, in run
2025-09-09T08:17:46.604-07:00
self.advance()
2025-09-09T08:17:46.604-07:00
File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/fit_loop.py", line 363, in advance
2025-09-09T08:17:46.604-07:00
self.epoch_loop.run(self._data_fetcher) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 138, in run
2025-09-09T08:17:46.604-07:00
self.advance(data_fetcher)
2025-09-09T08:17:46.605-07:00
File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/training_epoch_loop.py", line 242, in advance
2025-09-09T08:17:46.605-07:00
batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 191, in run
2025-09-09T08:17:46.605-07:00
self._optimizer_step(batch_idx, closure)
2025-09-09T08:17:46.605-07:00
File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 269, in _optimizer_step
2025-09-09T08:17:46.605-07:00
call._call_lightning_module_hook( File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 157, in _call_lightning_module_hook
2025-09-09T08:17:46.605-07:00
output = fn(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/core/module.py", line 1303, in optimizer_step
2025-09-09T08:17:46.605-07:00
optimizer.step(closure=optimizer_closure) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/core/optimizer.py", line 152, in step
2025-09-09T08:17:46.605-07:00
step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
2025-09-09T08:17:46.605-07:00
File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/strategies/ddp.py", line 270, in optimizer_step
2025-09-09T08:17:46.605-07:00
optimizer_output = super().optimizer_step(optimizer, closure, model, **kwargs) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 239, in optimizer_step
2025-09-09T08:17:46.605-07:00
return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/amp.py", line 80, in optimizer_step
2025-09-09T08:17:46.605-07:00
closure_result = closure() File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 144, in __call__
2025-09-09T08:17:46.605-07:00
self._result = self.closure(*args, **kwargs)
2025-09-09T08:17:46.605-07:00
File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2025-09-09T08:17:46.605-07:00
return func(*args, **kwargs)
2025-09-09T08:17:46.605-07:00
File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 138, in closure
2025-09-09T08:17:46.605-07:00
self._backward_fn(step_output.closure_loss) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/loops/optimization/automatic.py", line 240, in backward_fn
2025-09-09T08:17:46.605-07:00
call._call_strategy_hook(self.trainer, "backward", loss, optimizer) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 309, in _call_strategy_hook
2025-09-09T08:17:46.605-07:00
output = fn(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/strategies/strategy.py", line 213, in backward
2025-09-09T08:17:46.605-07:00
self.precision_plugin.backward(closure_loss, self.lightning_module, optimizer, *args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/plugins/precision/precision.py", line 72, in backward
2025-09-09T08:17:46.605-07:00
model.backward(tensor, *args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/lightning/pytorch/core/module.py", line 1090, in backward
2025-09-09T08:17:46.605-07:00
loss.backward(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward
2025-09-09T08:17:46.605-07:00
torch.autograd.backward( File "/opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward
2025-09-09T08:17:46.605-07:00
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "/opt/conda/lib/python3.10/site-packages/torch/autograd/function.py", line 289, in apply
2025-09-09T08:17:46.605-07:00
return user_fn(self, *args) File "/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 255, in backward
2025-09-09T08:17:46.605-07:00
reducer = ddp_weakref.reducer
2025-09-09T08:17:46.605-07:00
AttributeError: 'NoneType' object has no attribute 'reducer'
</code></pre>
<p>Also attaching loss curves:<br>
The loss spikes and when we resume from the checkpoint, the model converges.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/251bba70c747872b45e0d614c116695ab6ae3101.png" data-download-href="https://discuss.pytorch.org/uploads/default/251bba70c747872b45e0d614c116695ab6ae3101" title="Screenshot 2025-09-09 at 12.02.43 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/251bba70c747872b45e0d614c116695ab6ae3101_2_690x149.png" alt="Screenshot 2025-09-09 at 12.02.43 PM" data-base62-sha1="5ih4AnrpefwCDidzoOx3S6TXDiN" width="690" height="149" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/251bba70c747872b45e0d614c116695ab6ae3101_2_690x149.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/251bba70c747872b45e0d614c116695ab6ae3101_2_1035x223.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/251bba70c747872b45e0d614c116695ab6ae3101_2_1380x298.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2025-09-09 at 12.02.43 PM</span><span class="informations">2148×466 30.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>cc: <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>I haven’t seen this error before. Which PyTorch version are you using?</p>

username: Apurva_Mhatre
cooked: <p>I am using PyTorch 2.2.0. Do you know what this error means in general?</p>