Running multiple model instances on a single GPU performance analysis

Tungdil: <p>Hey,<br>
I’m working with <a href="https://github.com/amoudgl/pygoturn" rel="noopener nofollow ugc">this pytorch based tracker</a>. I’m running the test script with “python3 src/test.py” and observing the fps output, which is around <strong>20 FPS</strong> for my NVIDIA Jetson AGX Orin board. I also modified the script by adding “torch.cuda.max_memory_allocated(device)” to the end of the script to measure the maximum GPU memory allocated by this program, which seems to be around <strong>0.43 GB</strong> for my case.</p>
<p>So I limited the GPU memory usage such that the new limited GPU memory would be just above my measurement of <strong>0.43 GB</strong> with “torch.cuda.set_per_process_memory_fraction(fraction, device)”. Running this modified instance with new limit set to around <strong>0.61 GB</strong> doesn’t affect the FPS performance as expected.</p>
<p>But creating multiple instances of this modified script by using a shell script affects the FPS performance such that each instances FPS measurements drop drastically.<br>
What is the reason behind this occurrence and what is the correct way of solving it such that each limited instance can run around <strong>20 FPS</strong> as well.</p>
<p><strong>The shell script I run to launch multiple instances:</strong><br>
#!/bin/bash</p>
<p>if [ -z “$1” ]; then<br>
echo “Usage: $0 &lt;number_of_instances&gt;”<br>
exit 1<br>
fi</p>
<p>NUM_INSTANCES=$1</p>
<p>echo “Starting $NUM_INSTANCES instances of test.py…”</p>
<p>for i in $(seq 1 $NUM_INSTANCES); do<br>
python3 test.py &amp;<br>
echo “Started instance $i”<br>
done</p>
<p>echo “All instances started. Check the system monitor to observe the effects.”</p>

ptrblck: <p>Besides the used memory each CUDA kernel will also use compute resources, which are shared between all applications. <a href="https://discuss.pytorch.org/t/running-independent-nn-module-instances-in-nn-modulelist-truly-in-parallel-in-pytorch/188166/3">This post</a> explains it in more details with a reference to a great GTC talk.</p>

Tungdil: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for the quick response. I’ll check out the discussion you mentioned and will get back to you. From what you wrote, I sense that it is not a possible or a trivial task as I thought, right? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Yes, it might not be possible if your current workload already saturates the compute resources of your GPU. In this case you won’t be able to run any other kernel in parallel as long as the GPU is busy.</p>

Tungdil: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I see, thanks a lot for your responses. At the mean time I implemented a multi-stream, multiprocess architecture that allows me to run different CUDA streams for each process. Then I held some experiments where I launched different number of instances of the tracker I mentioned above. Since this tracker seems to be using a GPU memory of 0.43 GB, I limited it’s usage to 0.61 GB (0.01% of the current 64 GB memory that Jetson AGX Orin has). And the results are like this:</p>
<p><strong>Number of Tracker Instances Launched &amp; Average FPS</strong><br>
1 Instance &amp; 18-19 FPS<br>
5 Instance &amp; 14-15 FPS<br>
10 Instance &amp; 9-10 FPS<br>
15 Instance &amp; 6-7 FPS<br>
20 Instance &amp; 4-5 FPS</p>
<p>Since the FPS performance decrease as the number of instances increase, I believe we can conclude that even a single instance of this tracker saturates the current GPU workload. Now I’m wondering if there is a way to observe this GPU workload numerically and then maybe limit it somehow. What do you think?</p>

ptrblck: <aside class="quote no-group" data-username="Tungdil" data-post="5" data-topic="200048">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/eb8c5e/48.png" class="avatar"> Tungdil:</div>
<blockquote>
<p>Since the FPS performance decrease as the number of instances increase, I believe we can conclude that even a single instance of this tracker saturates the current GPU workload.</p>
</blockquote>
</aside>
<p>Not necessarily, since you are still seeing some speedup, so a single run might still leave the GPU idle for some time.</p>
<aside class="quote no-group" data-username="Tungdil" data-post="5" data-topic="200048">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/eb8c5e/48.png" class="avatar"> Tungdil:</div>
<blockquote>
<p>Now I’m wondering if there is a way to observe this GPU workload numerically and then maybe limit it somehow. What do you think?</p>
</blockquote>
</aside>
<p>You could profile the workload via e.g. <code>Nsight Systems</code> to see how tightly packed the CUDA kernels are.</p>

Tungdil: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
In order to profile the workload like you described, I used the nsys CLI that comes built-in with Jetpack.</p>
<pre><code class="lang-auto">nsys profile --trace=cuda --cudabacktrace memory --cuda-memory-usage true python3 stream-util.py
</code></pre>
<p>I set the the options accordingly since I aim to observe GPU workload. And then observing the stats with</p>
<pre><code class="lang-auto">nsys stats report3.nsys-rep
</code></pre>
<p>yields this result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f02c0fb511e8263788a44c692677fd4c61b99f54.png" data-download-href="https://discuss.pytorch.org/uploads/default/f02c0fb511e8263788a44c692677fd4c61b99f54" title="4"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f02c0fb511e8263788a44c692677fd4c61b99f54.png" alt="4" data-base62-sha1="ygEZCF7sab5BYruj7ah0dLUpBM8" width="690" height="121" data-dominant-color="393E3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">730×129 24.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aba45305fa52eeb4b441e661af50b72f34cdc3a.png" data-download-href="https://discuss.pytorch.org/uploads/default/6aba45305fa52eeb4b441e661af50b72f34cdc3a" title="3"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aba45305fa52eeb4b441e661af50b72f34cdc3a.png" alt="3" data-base62-sha1="fe9H22vdTgsSGeqY0jerPewL7xo" width="690" height="106" data-dominant-color="373D3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">873×135 24.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c36a88d88f62b65b4d5f9cb3460f73f05eea189d.png" data-download-href="https://discuss.pytorch.org/uploads/default/c36a88d88f62b65b4d5f9cb3460f73f05eea189d" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c36a88d88f62b65b4d5f9cb3460f73f05eea189d_2_690x132.png" alt="2" data-base62-sha1="rSJm4l5EVKqsh2hlkFMGXnjDoGV" width="690" height="132" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c36a88d88f62b65b4d5f9cb3460f73f05eea189d_2_690x132.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c36a88d88f62b65b4d5f9cb3460f73f05eea189d_2_1035x198.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c36a88d88f62b65b4d5f9cb3460f73f05eea189d_2_1380x264.png 2x" data-dominant-color="3D4243"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1583×305 129 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d774651115c3ea6c66d4e27ad4cc8ea376cc56ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/d774651115c3ea6c66d4e27ad4cc8ea376cc56ec" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d774651115c3ea6c66d4e27ad4cc8ea376cc56ec_2_690x285.png" alt="1" data-base62-sha1="uK01UxBqXjUb5YquA5jWntdca96" width="690" height="285" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d774651115c3ea6c66d4e27ad4cc8ea376cc56ec_2_690x285.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d774651115c3ea6c66d4e27ad4cc8ea376cc56ec_2_1035x427.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/7/d774651115c3ea6c66d4e27ad4cc8ea376cc56ec.png 2x" data-dominant-color="393E3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1152×476 126 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Since I’m new to this area I’m not sure if the method I’m applying is correct or how I should interpret this results. Can you help me with that? By the way the script I profiled is running a single instance of the tracker at the moment.</p>
<p>One more thing, initially I also tried setting the <code>--gpu-metrics-device</code> option as well, but I found that Jetson AGX Orin does not support this operation. I used this command to check capabilities.</p>
<pre><code class="lang-auto">nsys profile --trace=cuda --cudabacktrace memory --cuda-memory-usage true --gpu-metrics-device help python3 stream-util.py
</code></pre>
<p>And it returns:</p>
<pre><code class="lang-auto">Possible --gpu-metrics-device values are:
	0: Orin (not supported)
	all: Select all supported GPUs
	none: Disable GPU Metrics [Default]
Some GPUs are not supported. See the user guide: https://docs.nvidia.com/nsight-systems/UserGuide/index.html#gms-introduction

</code></pre>
<p>Thanks a lot in advance.</p>

Tungdil: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hey again, do you have any new opinions about this case?</p>

ptrblck: <p>The summary view might not be really helpful as it does not show the IDLE times between kernels. You could open this profile in the Nsight Systems UI and check the timeline to see how tightly packed the kernels are.</p>