Freezing problem while using cuda tensor in multiprocessing environment

iffiX: <p>The main process in the below code snippet will freeze after serveral iterations.</p>
<p>I think it’s related to how the data structure stack in pytorch works, tensors are built upon storage classes, and storage classes are build upon raw cudaMalloc regions. I can understand what <code>recuce_tensor()</code> and <code>rebuild_cuda_tensor()</code> are doing, but I am not sure why creating a new tensor (since g_tensor has been reassigned) after  a blocking starmap would somehow causing the main process to freeze.</p>
<p><strong>Code example</strong></p>
<pre><code class="lang-auto">import itertools as it
import torch as t
import torch.multiprocessing as mp


def infer(id, tensor):
    print(id)
    print(tensor)
    # del tensor immediately doesn't solve the problem
    del tensor

# some global tensor
g_tensor = t.full([1000, 1000], 2, device="cuda:0")
g_tensor.share_memory_()

if __name__ == "__main__":
    ctx = mp.get_context("spawn")
    pool = ctx.Pool(2)
    for i in range(10000000):
        print("start")
        pool.starmap(infer, zip(range(5), it.repeat(g_tensor)))

        # cpu tensors work just fine
        # for cuda tensors:
        # if I delete the global tensor, reassign it with a new cuda tensor
        # or if I use a tensor created dynamically in each iteration
        # the program freezes after 2 iterations.
        # Comment out the following lines and everything will work fine.
        del g_tensor
        g_tensor = t.full([1000, 1000], 2, device="cuda:0")
        g_tensor.share_memory_()
</code></pre>
<h2><strong>Environment</strong></h2>
<ul>
<li>PyTorch Version (e.g., 1.0): 1.1.0</li>
<li>OS (e.g., Linux): Linux</li>
<li>How you installed PyTorch (conda, pip, source): pip</li>
<li>Python version: 3.5</li>
<li>CUDA/cuDNN version: 9.1/7.2.1</li>
</ul>

iffiX: <p><a class="mention" href="/u/mrshenli">@mrshenli</a><br>
I am sorting out my framework today and as of pytorch version 1.5.0, this problem still persists. Is it normal?</p>

mrshenli: <p>Tried this locally, it hangs with a non-deterministic number of iterations (4, 11, etc.), and it hangs at <code>del g_tensor</code>. I suspect it is due to <a href="https://github.com/pytorch/pytorch/blob/master/c10/cuda/CUDACachingAllocator.cpp" rel="nofollow noopener">CUDACachingAllocator</a>, where it might need to keep the memory block alive until all other processes finish using it? But it is a per-process data structure, so it does not have the global view.  I am not sure if this is the reason for this hang.</p>
<p>Call for help cc <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/colesbury">@colesbury</a></p>

colesbury: <p>Thanks for following up on this. I filed an issue to track the problem on GitHub:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/39541" target="_blank">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/39541" target="_blank">Deadlock with shared CUDA tensors and multiprocessing (spawn)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-06-04" data-time="21:48:25" data-timezone="UTC">09:48PM - 04 Jun 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/colesbury" target="_blank">
          <img alt="colesbury" src="https://avatars1.githubusercontent.com/u/655866?v=4" class="onebox-avatar-inline" width="20" height="20">
          colesbury
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
Reported by iffiX in https://discuss.pytorch.org/t/freezing-problem-while-using-cuda-tensor-in-multiprocessing-environment/80000.
(I have moved the code into a main() function to rule out some other multiprocessing bugs)
import...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It looks like the deadlock is in <code>CudaIPCSentData</code> destructor, which looks separate from the caching allocator code.</p>