Stuck at Stream Synchronization

username: AmirFB
cooked: <p>I have designed a real-time scheduler based on LibTorch, using multiple CUDA Contexts and Streams. The problem is sometimes one of the contexts gets stuck suddenly and whatever workload I assign to it gets stuck.</p>
<p>This is a simple code showing how I use it:</p>
<pre><code class="lang-auto">cuCtxSetCurrent(_context);
_context.select();
_stream = at::cuda::getStreamFromPool(false, _context-&gt;index);
at::cuda::setCurrentCUDAStream(_stream);
auto output = sequential-&gt;forward(input);
_stream.synchronize();
</code></pre>
<p>It’s a complex scheduler and I have modified the source for PyTorch to treat each CUDA Context as a separate CUDA device. It’s interesting when using a smaller input size for the network (like 224x224), it works fine but when I use a larger input size (like 512x512) sometimes it gets stuck.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4962ef8d75ed3d7be75a82611474230ebeb8753.png" data-download-href="https://discuss.pytorch.org/uploads/default/f4962ef8d75ed3d7be75a82611474230ebeb8753" title="Screenshot from 2023-06-05 03-48-07"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4962ef8d75ed3d7be75a82611474230ebeb8753_2_602x500.png" alt="Screenshot from 2023-06-05 03-48-07" data-base62-sha1="yTIgHEBNoPQiCSW7oSaXwtBQoCL" width="602" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4962ef8d75ed3d7be75a82611474230ebeb8753_2_602x500.png, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4962ef8d75ed3d7be75a82611474230ebeb8753.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4962ef8d75ed3d7be75a82611474230ebeb8753.png 2x" data-dominant-color="CFDDD4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-06-05 03-48-07</span><span class="informations">606×503 53.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is a screenshot from NSight Systems showing that after a specific time Context 2 has stuck and no more kernels have been scheduled in it. (And all the modules being executed in it are frozen). Using <code>CUDA_LAUNCH_BLOCKING=0</code> also solves the problem but I don’t want to use it because it results in poor performance.</p>
<p>Does anyone have any idea what would be the reason for this?</p>

username: eqy
cooked: <p>Some guesses:</p>
<ol>
<li>
<p>The smaller model(s) may use kernels that have smaller launch bounds, allowing concurrent execution of multiple streams. I wonder if the larger models have more kernels that can only execute one-by-one on the device, exposing some potential deadlocks if the streams have complex dependencies. However, this seems unlikely as we would expect to see similar behavior with <code>CUDA_LAUNCH_BLOCKING=1</code>. (I assume this is what you meant as <code>CUDA_LAUNCH_BLOCKING=0</code> should be the default.</p>
</li>
<li>
<p>I wonder if synchronizing calls e.g., to <code>cudaMalloc</code> occur with the larger models and cause problems as the caching allocator may be running out of free memory—requiring discarded Tensors to be recycled and triggering new calls to <code>cudaMalloc</code>. Does setting <code>PYTORCH_NO_CUDA_MEMORY_CACHING=1</code> in the small-model case also cause problems? If so then I would guess that it could be related to memory usage and caching allocator behavior.</p>
</li>
</ol>

username: AmirFB
cooked: <p>As you mentioned, obviously number 1 is not the case. But I tried <code>PYTORCH_NO_CUDA_MEMORY_CACHING=1</code> even with a smaller size the code does not get stuck, but it really drops the performance, let’s say it gets almost 10 times slower. I think GPU is not happy with parallel <code>cudaMalloc</code> requests. I profiled it and most of the execution time is occupied by <code>cudaMalloc</code> and <code>cudaFree</code>.</p>

username: AmirFB
cooked: <p>I also noticed that when a thread gets stuck (and eventually every other thread which tries to use the same CUDA Context freezes as well), there is an io control call every second in this thread like this. I don’t know what it exactly means (maybe the CPU is trying to reach the GPU but just GPU does not respond properly), but I though it would provide some information.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f9d0b1287d5d83a8cf6ddc83c3f7cfcbf7ca1be6.png" data-download-href="https://discuss.pytorch.org/uploads/default/f9d0b1287d5d83a8cf6ddc83c3f7cfcbf7ca1be6" title="Screenshot from 2023-06-06 22-31-17"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9d0b1287d5d83a8cf6ddc83c3f7cfcbf7ca1be6_2_690x154.png" alt="Screenshot from 2023-06-06 22-31-17" data-base62-sha1="zDY0FhWYvnGhHMcCUOqoCSzZ6US" width="690" height="154" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9d0b1287d5d83a8cf6ddc83c3f7cfcbf7ca1be6_2_690x154.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9d0b1287d5d83a8cf6ddc83c3f7cfcbf7ca1be6_2_1035x231.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/9/f9d0b1287d5d83a8cf6ddc83c3f7cfcbf7ca1be6_2_1380x308.png 2x" data-dominant-color="E3E5E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-06-06 22-31-17</span><span class="informations">1749×391 32.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>