Issue with multiprocessing semaphore tracking

gamma: <p>Hi</p>
<p>I am facing an issue with semaphore tracking while using pytorch multiprocessing over multiple GPUs. I get the following warning message multiple number of times every time and it is slowing down the code execution substantially (8 times slower).</p>
<pre><code class="lang-auto">/anaconda3/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
</code></pre>
<p>I didn’t face this issue a couple of days back and started facing it when I updated my PyTorch to 0.4.1 hence I downgraded it to 0.3.1 (hoping that it was my previous Pytorch version). But, I am still facing the warning and slowdown. I really need to understand the cause of this leakage and how can I fix it; my code is wrapped inside if <strong>name</strong>==‘<strong>main</strong>’ which is one of the solution commonly suggested for this warning.</p>
<p>I would really appreciate any suggestions on this. Please let me know what details should I provide.</p>
<p>Thanks</p>

Michael_Petrochuk: <p>Similarly, I have the same issue and i do not know where it is originating from! I’m using DataLoader with DistributedDataParallel.</p>
<p>Ditto with my name wrapped inside <code>name==main</code></p>

kittoku: <p>I encountered a similar problem with <code>multiprocessing</code> and <code>for</code> loops in CPU. It’s not a fundamental solution, but <code>-W</code> option works for me, e.g. <code>python -W ignore main.py</code>. This makes training stable in my environment.</p>
<p>I tried using <code>warnings</code> module to ignore the warning, but it’s not worked. I think this is because semaphore trackers’ processes are generated internally.</p>
<p>my environment.: linux, python=3.7, pytorch-cpu=0.4.1, start_method=spawn</p>

sbelharbi: <p>I run to the same issue with the warning (when running some preprocessing on GPU), except for now, I didn’t see a slow down in the running time of the code, yet.</p>
<pre><code class="lang-bash">python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
</code></pre>
<p>The message seems harmful, but pretty annoying since it is displayed whenever the multiprocessing takes place.</p>
<p>One way to hide this warning is to ignore it using the Python module <code>warnings</code>. Also, redirecting the <code>stderr</code> may be temporary solution.</p>
<p>However, <a href="https://docs.python.org/3/library/warnings.html#warnings.filterwarnings" rel="nofollow noopener"><code>warnings.filterwarnings()</code></a> DOES not work for this particular warning (and probably for all the warnings of the Python module <code>multiprocessing.semaphor_tracker()</code>. I don’t really understand why. I tried different ways using regular expressions, but nothing seem to work.</p>
<p><strong>Things that  work</strong>:</p>
<ol>
<li>
<code>python -W ignore your_script.py</code> --&gt; ignore ALL the warnings within your script. (not recommended).</li>
<li>
<code>python -W ignore:semaphore_tracker:UserWarning your_script.py</code> or <code>python -W 'ignore:semaphore_tracker:UserWarning' your_script.py</code> seem to ignore exactly the above warning. It can also done by using the environment variable   <a href="https://docs.python.org/3/using/cmdline.html#envvar-PYTHONWARNINGS" rel="nofollow noopener">PYTHONWARNINGS</a>:</li>
</ol>
<pre><code class="lang-bash">export PYTHONWARNINGS='ignore:semaphore_tracker:UserWarning'

python your_script.py
</code></pre>
<p>Python 3.7.1, Pytorch 1.0.0.</p>

aliutkus: <p>Is there some understanding concerning why this warning happens ?</p>
<blockquote>
<p>UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown</p>
</blockquote>
<p>It seems pretty difficult to find the cause for this to me…</p>

aliutkus: <p>still no news on that ? Maybe it’s related to some conflict with tqdm ?</p>
<p>It’s looking ugly to have this displayed all the time…</p>

sbelharbi: <p>If the issue was raised because you tried to use multiprocessing within cuda … It is better to ignore it as mentioned <a href="https://discuss.pytorch.org/t/issue-with-multiprocessing-semaphore-tracking/22943/4?u=sbelharbi">above</a>, and move on (see <a href="https://github.com/pytorch/pytorch/issues/13883#issuecomment-440004307" rel="nofollow noopener">here</a>, and <a href="https://github.com/pytorch/pytorch/issues/2517#issuecomment-452153555" rel="nofollow noopener">here</a>).</p>

zaccharieramzi: <p>I am actually also interested in understanding this error, since I have a similar one using tensorflow, and it uses a lot of memory in my case (too much for my workers which die).</p>

Enamex: <p>I’m getting the same error. Only my latest run had <strong>5779</strong> leaked semaphores <img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=10" title=":upside_down_face:" class="emoji" alt=":upside_down_face:">.<br>
I’m <em>not</em> using PyTorch MP manually; fairseq is using it.<br>
Fairseq master with PyTorch 1.9.</p>

lvrcek: <p>Same issue, I am getting 6 leaked semaphores, and am not using multiprocessing explicitly. The mind-boggling thing about this is that sometimes it happens, sometimes not, and I cannot find any pattern what leads to this. The memory doesn’t seem</p>
<p>I use PyTorch 1.9.0 and DGL 0.7.1 and I’m not sure which one is causing the warning to arise. Would greatly appreciate if anyone had any insight.</p>

jingpengw: <p>I got the same issue. It seems that the dataloader has to be constructed under the “if <strong>name</strong>==‘<strong>main</strong>’:” or in a function. Whenever I move it to a function in a class, it will have this issue…</p>

log2win: <p>I found simple example which leads to “UserWarning: resource_tracker: There appear to be 4 leaked semaphore objects to clean up at shutdown”.<br>
I think it may be helpful to someone, as using barriers with multiprocessing should be pretty common</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp

def foo(barrier = mp.Barrier(1)):
    print("default barrier leads to leak")

# def foo(barrier = None):
#     print("default None not leads to leak")

if __name__ == '__main__':
    processes = []
    bar = mp.Barrier(1)
    mp.set_start_method('spawn', force=True)
    for rank in range(1):
        p = mp.Process(target=foo, args=(bar, ))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()
</code></pre>