Could not unlink the shared memory file, potential race with systemd

jambo6: <p>I want to be able to add some additional flag in the filenames dumped to /dev/shm to ensure no clashes between workers.</p>
<p>When using a DataLoader with multiple workers it dumps a load of files to /dev/shm, <a href="https://github.com/pytorch/pytorch/wiki/Multiprocessing-Technical-Notes" rel="noopener nofollow ugc">as explained in this link</a></p>
<p>Now suppose I have a 4GPU box and run a process on each, each spinning up a multiprocessing dataloader. Each dumps information independently to <code>/dev/shm</code>, I’ve found that training can be fine for a few epochs, then crash due to the following error</p>
<p><code>could not unlink the shared memory file /torch_2999188_550744154</code></p>
<p>Current hypothesis is that two workers have somehow ended up using the same filename, and one has unlinked it meaning another breaks. Not sure if this is true, but is there a way to modify the file location to <code>torch_{rank}_2999188_550744154</code> where rank is the rank number of the process?</p>

ptrblck: <aside class="quote no-group" data-username="jambo6" data-post="1" data-topic="167293">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jambo6/48/52651_2.png" class="avatar"> jambo6:</div>
<blockquote>
<p>Current hypothesis is that two workers have somehow ended up using the same filename, and one has unlinked it meaning another breaks.</p>
</blockquote>
</aside>
<p>That’s an interesting thought, but note that the file name is created via the <code>pid</code> and a random number as seen <a href="https://github.com/pytorch/pytorch/blob/015b05af18b78ca9c77c997bc277eec66b5b1542/aten/src/ATen/MapAllocator.cpp#L36-L51">here</a>:</p>
<pre><code class="lang-auto">TORCH_API std::string NewProcessWideShmHandle()
{
  static std::atomic&lt;uint64_t&gt; counter{0};
  static std::random_device rd;
  std::string handle = "/torch_";
#ifdef _MSC_VER
  handle += c10::guts::to_string(GetCurrentProcessId());
#else
  handle += c10::guts::to_string(getpid());
#endif
  handle += "_";
  handle += c10::guts::to_string(rd());
  handle += "_";
  handle += c10::guts::to_string(counter.fetch_add(1, std::memory_order_relaxed));
  return handle;
}
</code></pre>
<p>Based on this the difference processes should not be able to create the same file as their <code>pid</code> would be different (and additionally they would need to resample the same random number).</p>
<p>In any case, it seems you have a code snippet which can reproduce the error (after a few epochs), so you could add the <code>rank</code> to the file name, rebuild PyTorch, and see if this would help.</p>

jambo6: <aside class="quote no-group" data-username="jambo6" data-post="1" data-topic="167293">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jambo6/48/52651_2.png" class="avatar"> jambo6:</div>
<blockquote>
<p>torch_2999188_550744154</p>
</blockquote>
</aside>
<p>Thanks for the link.</p>
<p>I see, so unlikely this is true.</p>
<p>Can you explain why the above code looks like it should produce three underscores, but the files are always labelled as ‘torch_2999188_550744154’ with two numbers. Is pid definitely being added?</p>

ptrblck: <p>Maybe your PyTorch version is too old and doesn’t use the latest changes?<br>
You can <code>git blame</code> the file and would see when these PRs were merged (e.g. <a href="https://github.com/pytorch/pytorch/pull/60978">this one</a> and <a href="https://github.com/pytorch/pytorch/pull/72232">this one</a>).</p>

jambo6: <p>Ah yes, perfect! I should have thought to check that, thanks very much.</p>

jambo6: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I’ve upgraded to torch1.13 and it definitely works better, however I’m still getting errors<br>
(albeit less frequently)</p>
<p><code>RuntimeError: could not unlink the shared memory file /torch_1333031_3417509106_15235 : No such file or directory (2)</code></p>
<p>It now has the PID attached, but the issue persists.</p>
<p>Is there any straightforward way to just ignore this and turn it into a warning? Seems super rare so I would be happy to just ignore any instances where this happens, I dont know how one would skip through runtime errors in multiprocessing though.</p>

jambo6: <p>Posting a solution in case anyone comes across this error down the line</p>
<p>This solution was found by a colleague, not by me:</p>
<blockquote>
<p>when data is loaded using a DataLoader with num_workers &gt; 0, temporary files in /dev/shm are created and used as temporary storage. Filename is made of the pid of the calling thread, a random number and an incrementing number: (<a href="https://github.com/pytorch/pytorch/blob/v1.13.0/aten/src/ATen/MapAllocator.cpp#L44" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/v1.13.0/aten/src/ATen/MapAllocator.cpp#L44</a>), also as can be seen in <a href="https://github.com/pytorch/pytorch/blob/v1.13.0/aten/src/ATen/MapAllocator.cpp#L345" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/v1.13.0/aten/src/ATen/MapAllocator.cpp#L345</a> the file is immediately unlinked, but because the file descriptor is kept open, it can still be used to store and read data. Systemd by default cleans up various shared memory related resources for non-system users and I think because the user is not logged into the system, it sees the file created in /dev/shm and unlinks it before the code in MapAllocator.cpp#L345 gets to it. So it is a race condition that systemd is very unlikely to win, but sometimes it does. This situation is described much better by <a href="https://www.postgresql.org/docs/current/kernel-resources.html#SYSTEMD-REMOVEIPC" class="inline-onebox" rel="noopener nofollow ugc">PostgreSQL: Documentation: 16: 19.4.&nbsp;Managing Kernel Resources</a></p>
</blockquote>
<p>Solution was to rebuild the image with <code>RemoveIPC=no</code></p>

jambo6: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> is there any way to change the title to something more useful</p>
<p>e.g. could not unlink the shared memory file using torch version where name clash bug has been fixed</p>

ptrblck: <p>Thanks for the follow up and description of the issue! Yes, I can change the title for you.</p>

Hyeonuk_Woo: <p>This <code>RemoveIPC</code> is environmental variable? How can I set this value in torch python script?</p>