Optimizer.step() hangs on linux; multiprocesssing

username: Lewis_Liu
cooked: <p>Background:<br>
I’m doing a distributed PPO (basically gathering data from several worker and training on one learner)</p>
<p>Issue:<br>
Data collection works fine but when I train the network with lines below</p>
<pre><code>            self.critic.optimizer.zero_grad()
            batch_states_values = self.critic.forward(batch_states)
            print('crtitic batch_states_values done')
            critic_loss = F.mse_loss(batch_states_values, batch_REFs)
            print('crtitic critic_loss done')
            critic_loss.backward()
            print('crtitic loss backward done')
            self.critic.optimizer.step()
            print('crtitic step done')
</code></pre>
<p>And the output shows:</p>
<pre><code>crtitic batch_states_values done
crtitic critic_loss done
crtitic loss backward done
</code></pre>
<p>So it appears to be that the program hangs after the loss backward. What could be the cause?<br>
It works fine on my windows workstations but hangs when I run it on a linux machine</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/lewis_liu">@Lewis_Liu</a>, which part of the program is distributed? Since <code>torch.distributed</code> does not support Windows yet, I assume the working version of the program on Windows does not use distributed training?</p>

username: mrshenli
cooked: <p>BTW, which optimizer are you using?</p>

username: Lewis_Liu
cooked: <p>The training isn’t distributed and <code>torch.distributed</code> isn’t used.</p>
<p>By distributed I mean the workers used to collect data are distributed and the network params are send from trainer to these workers through mp.queue.</p>
<p>Once the data are collected and trainer starts to train, the workers stop working so I suppose there’s no interaction between the workers and the trainer. So what appears really strange to me is that the backward is done but step is not.</p>
<p>I’m using the standard optim.Adam</p>

username: mrshenli
cooked: <p>Are you using any CUDA ops? If so, could you please add a <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.synchronize" rel="nofollow noopener"><code>torch.cuda.synchronize()</code></a> before every print to make sure that preceding ops are indeed done instead of still pending in the CUDA stream?</p>

username: Lewis_Liu
cooked: <p>Hi Li,</p>
<p>I just added the line and the prints are the same.</p>
<p>FYI, after it hangs there, I killed the program and it showed this. I’m not sure if this is helpful</p>
<p>Traceback (most recent call last):<br>
File “test.py”, line 23, in <br>
Process Process-2:<br>
p.join()<br>
File “/apps/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/multiprocessing/process.py”, line 140, in join<br>
Process Process-1:<br>
res = self._popen.wait(timeout)<br>
File “/apps/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/multiprocessing/popen_fork.py”, line 48, in wait<br>
return self.poll(os.WNOHANG if timeout == 0.0 else 0)<br>
File “/apps/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/multiprocessing/popen_fork.py”, line 28, in poll<br>
pid, sts = os.waitpid(self.pid, flag)<br>
KeyboardInterrupt</p>

username: mrshenli
cooked: <p>That’s weird, is there a way that we can reproduce this locally, so that we can help debug?</p>

username: Lewis_Liu
cooked: <p>Hi Li,</p>
<p>Thanks for the help. I’m afraid it wouldn’t be easy to do it. I’ll try to find a way to convert it so it can be shared.</p>
<p>Meanwhile, what would you say that might be the cause? Any chance this could be the use of mp.queue or mp.Value in the linux environment? If likely, I can try to avoid or alter the way using them</p>

username: mrshenli
cooked: <p>Are you using <a href="https://pytorch.org/docs/stable/multiprocessing.html" rel="noopener nofollow ugc"><code>torch.multiprocessing.SimpleQueue</code></a>? If yes, does the program guarantee that the owner of the shared data object is still alive when the user uses it when sharing CPU tensors? And are you using <code>spawn</code> to create processes?</p>
<blockquote>
<p>Unlike CPU tensors, the sending process is required to keep the original tensor as long as the receiving process retains a copy of the tensor. The refcounting is implemented under the hood but requires users to follow the next best practices.</p>
</blockquote>

username: Lewis_Liu
cooked: <p>I used spawn by adding the line<br>
mp.set_start_method(“spawn”, force=True)</p>
<p>I used torch.multiprocessing.Queue instead of the SimpleQueue. The owners are always alive</p>

username: Lewis_Liu
cooked: <p>Spawn on windows but not on cluster</p>

username: Lewis_Liu
cooked: <aside class="quote no-group quote-modified" data-username="mrshenli" data-post="9" data-topic="91468">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<blockquote>
<p>Unlike CPU tensors, the sending process is required to keep the original tensor as long as the receiving process retains a copy of the tensor. The refcounting is implemented under the hood but requires users to follow the next best practices.</p>
</blockquote>
</blockquote>
</aside>
<p>Interesting fact, on linux cluster, it works if I change the device to ‘CPU’ instead of using a GPU. But on windows, both devices work</p>

username: bqhuyy
cooked: <p>Have you solved this problem? I have met the same problem when running with multiple machine</p>

username: Lewis_Liu
cooked: <p>Not completely solved but was able to found what was the issue and found a way around. The issue is that the network was somehow shared with other processes. So my practical suggestion would be check everything that might lead to your network being shared/visited. e.g. mistake in using copy.copy or deepcopy to send the statedict</p>