Combining PyTorch and CUDA leads to memory consumption increase when running within the same process

Luis_Leon: <p>Hi everyone!</p>
<p>I have been facing a weird behaviour in memory consumption when building a GStreamer element that combines PyTorch and CUDA.</p>
<p>First, I constructed an element that uses NVMM buffers and I do some computation with CUDA on the incoming buffers in place. It was working like a charm. Even, I was able to place up to six parallel pipelines that use the element without any issue more than the actual GPU usage. However, I wanted to add some object detection functionality to the pipeline and tried to integrate PyTorch into the element. When I just link Torch library to the element, the memory consumption hugely increases, and the pipelines consume ~1.2GB each (it was less than  100MB before).</p>
<p>The memory consumption pattern is like follows (NP=Number of Processes):</p>
<p>Without Torch (left) and with Torch linked-in (right):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/43278a1613ecdce60d65733de1166e0b9a0b44ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/43278a1613ecdce60d65733de1166e0b9a0b44ec" title="Concat"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43278a1613ecdce60d65733de1166e0b9a0b44ec_2_689x251.png" alt="Concat" data-base62-sha1="9A4HBsjj9LDlUuJhwBx5kCwtdk8" width="689" height="251" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43278a1613ecdce60d65733de1166e0b9a0b44ec_2_689x251.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43278a1613ecdce60d65733de1166e0b9a0b44ec_2_1033x376.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/3/43278a1613ecdce60d65733de1166e0b9a0b44ec.png 2x" data-dominant-color="E4E3E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Concat</span><span class="informations">1299×473 60.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the last part, where there is a spike in memory consumption, it happens when releasing the resources.</p>
<p>So far, the experiment is just adding Torch as a dependency in the building system. I am not adding any code (even headers).</p>
<p>For reproducing the issue, I have prepared a toy code where we are able to reproduce the behaviour. In this toy code, I have prepared a couple of pass-thru elements (<code>torchpassthru</code>, <code>cudapassthru</code>): one with Torch linked-in (just that), and another one that uses a similar initialisation/termination as the CUDA code we are using.</p>
<p>The issue can be reproduced by doing:</p>
<pre data-code-wrap="bash"><code class="lang-bash">gst-launch-1.0 videotestsrc ! "video/x-raw,width=640,height=480" ! cudapassthru ! torchpassthru ! fakesink silent=false -vvv
</code></pre>
<p>where the memory consumption only has a spike when closing the pipeline:</p>
<pre data-code-wrap="bash"><code class="lang-bash">gst-launch-1.0 videotestsrc ! "video/x-raw,width=640,height=480" ! torchpassthru ! cudapassthru ! fakesink silent=false -vvv
</code></pre>
<p>where the memory consumption increases from tens of MB to one GB. There is also a spike when closing the pipeline.</p>
<p>The memory scales linearly just by placing more <code>cudapassthru</code> elements. In other words, by placing one, the consumption increases by 1 GB, placing two, it increases to 2GB, and so on. This behaviour does not happen when we only have <code>cudapassthru</code> elements, meaning that the memory keeps in the other of tens of MB. Also, the memory consumption issue does not happen when having the Torch element alone.</p>
<p>I also wanted to mention that this behaviour does not happen when having the pipelines in multiple processes. For example:</p>
<pre data-code-wrap="bash"><code class="lang-bash"># Terminal 1
gst-launch-1.0 videotestsrc ! "video/x-raw,width=640,height=480" ! torchpassthru !  fakesink silent=false -vvv

# Terminal 2
gst-launch-1.0 videotestsrc ! "video/x-raw,width=640,height=480" ! cudapassthru ! cudapassthru ! cudapassthru !  fakesink silent=false -vvv
</code></pre>
<p>I suspect that it can be something related to CUDA contexts. If so:</p>
<ol>
<li>is there a way to make PyTorch and CUDA share the context in some way?</li>
</ol>
<p>In the beginning, I thought it was a PyTorch issue. However, looking at how the memory scales when I add more elements, I started to suspect that there is something else.</p>
<ol start="2">
<li>any clue about why this behaviour can happen in CUDA?</li>
</ol>
<p>I am using PyTorch 1.8 (from NVIDIA) on a Jetson TX2 with JP 4.5.1. The GStreamer version is the default one (1.14.x).</p>
<p>Thanks in advance for any clue you can provide about this issue.</p>
<p>This is the link to the toy code: <a href="https://drive.google.com/file/d/1eRK2r0QxCfiRbMUlkQ3Vr9U7i8bcZTAf/view?usp=sharing" rel="noopener nofollow ugc">download</a></p>