Distributed training with Apex fails at opt_level O0

lvshumang: <p>Hi, I am training a UNet on a single machine with 2 GPU. The codes work fine when only using PyTorch’s distributed package (with DistributedDataParallel). However, when training with Apex, I am getting weird errors if I set opt_level=‘O0’, but trainings with opt_level=‘O1’ are always fine. The error always happen on process 1 at scaled_loss.backward(). Any suggestion is appreciated. Thanks.</p>
<p>Error message:</p>
<p>Traceback (most recent call last):<br>
File “main_V4.py”, line 339, in <br>
mp.spawn(YYY_Proj, args=(2,opt), nprocs=2, join=True)<br>
File “/home/ZZZ/anaconda3/envs/Apex/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 200, in spawn<br>
return start_processes(fn, args, nprocs, join, daemon, start_method=‘spawn’)<br>
File “/home/ZZZ/anaconda3/envs/Apex/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 158, in start_processes<br>
while not context.join():<br>
File “/home/ZZZ/anaconda3/envs/Apex/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 119, in join<br>
raise Exception(msg)<br>
Exception:</p>
<p>– Process 1 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/home/ZZZ/anaconda3/envs/Apex/lib/python3.8/site-packages/torch/multiprocessing/spawn.py”, line 20, in _wrap<br>
fn(i, *args)<br>
File “/home/ZZZ/Documents/YYY_XXX/Apex_Distributed/main_V4.py”, line 191, in YYY_Proj<br>
scaled_loss.backward()<br>
File “/home/ZZZ/anaconda3/envs/Apex/lib/python3.8/site-packages/torch/tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/ZZZ/anaconda3/envs/Apex/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 98, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError</p>
<ul>
<li>set +x<br>
/home/ZZZ/anaconda3/envs/Apex/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 14 leaked semaphore objects to clean up at shutdown<br>
warnings.warn('resource_tracker: There appear to be %d ’</li>
</ul>

ptrblck: <p>Could you post the complete <code>RuntimeError</code>?<br>
We also recommend to use the native mixed-precision training support via <code>torch.cuda.amp</code>, which is available in the nightly binaries and current master, if it’s an option to update.</p>

lvshumang: <p>Thanks for your reply. The strange thing is, what I posted is all the error message I got.</p>

ptrblck: <p>It’s a bit weird that you are seeing this issue with <code>opt_level='O0'</code>, since <code>O0</code> is pure FP32 training and shouldn’t do anything.<br>
Are you calling <code>half()</code> on any tensors or models? Also, I understand that your code is running fine without using apex at all?</p>

lvshumang: <p>I didn’t call half() on any tensors or models. The code is running fine without using apex at all, or using apex with opt_level=‘O1’</p>
<p>Please note that by using apex, I changed from torch’s Adam optimizer to the FusedAdam in apex.optimizers. I will test if by using torch’s adam the error will go away.</p>
<p>I am attaching a screenshot of the error message (sorry I have to blackout the user information). The exact time this error happen seems to be random: some times it happens after 100 epoch.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/2162c9cd3f81e01b59a8f7ed040a261d582285f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/2162c9cd3f81e01b59a8f7ed040a261d582285f1" title="Screenshot from 2020-05-16 18-05-18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/2162c9cd3f81e01b59a8f7ed040a261d582285f1_2_690x474.png" alt="Screenshot from 2020-05-16 18-05-18" data-base62-sha1="4LlpnWO8GK68rluE89ymTs3mFmV" width="690" height="474" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/2162c9cd3f81e01b59a8f7ed040a261d582285f1_2_690x474.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/2162c9cd3f81e01b59a8f7ed040a261d582285f1_2_1035x711.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/1/2162c9cd3f81e01b59a8f7ed040a261d582285f1.png 2x" data-dominant-color="39172E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-16 18-05-18</span><span class="informations">1108×762 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>