GPU util has 0-100% fluctuation

theorem_on: <p>I’m using 2 GPUs. but all GPU util has fluctuation during NLP training.<br>
I’v been monitoring this using Nvidia-smi.<br>
If there are some solution for high GPU util, I expect training time could be reduced.<br>
I just suspect that large dataset from loaded joblib pkl file might be a factor, so I’ve tried some solutions.<br>
I noticed that when I slice input data, speed of each batch training increases. Is it just a given?</p>
<p>I’v already tried followings</p>
<ol>
<li>num_worker &gt; 0~14 setting</li>
<li>pin_memory = True setting</li>
<li>All data on the memory (dataset/<strong>init</strong>)</li>
<li>When checked CPU usage, it wasn’t that high</li>
</ol>
<p>Are there any other disconnects that could make things better?<br>
Please help this beginner.<br>
Thanks.</p>

ptrblck: <p>Generally, I would recommend profiling your use case before starting to apply optimizations without knowing where the actual bottleneck is. The <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html">performance guide</a> and <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a> might also be helpful.</p>

theorem_on: <p>Thank you for quick answer.<br>
I’ll refer to the links, it will be helpful for me.<br>
Actually, I’ve already profiling this model, but I couldn’t fully understand the result. <code>with profiler.profile</code><br>
Profiler didn’t show about dataloader despite dataloader under the <code>with profiler.profile</code>, did I do it right?<br>
In the result, it show 20% of the cudaFree time why appeared on CPU time? is it related about GPU util?<br>
The addmm layer is where most of the time is spent in CUDA. I think but this one wouldn’t solve GPU util flucuation because this process would be on the GPU.<br>
This may be a stupid question since I’m not familiar with Pytorch yet, but I’d like to address this issue.</p>
<pre><code class="lang-auto">-------------------------------------------------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------
Name		Self CPU %		Self CPU		CPU total %		CPU total		CPU time avg		Self CUDA		Self CUDA %		CUDA total		CUDA time avg		CPU Mem		Self CPU Mem		CUDA Mem		Self CUDA M		em    # of Calls
-------------------------------------------------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------
DistributedDataParallel.forward		1.58%		82.714ms		79.08%		4.153s		2.076s		14.957ms		0.29%		4.239s		2.120s		184 b		-26.79 Mb		2.86 Gb		-2.52 Gb		2
aten::linear		0.60%		31.512ms		21.15%		1.111s		7.506ms		2.990ms		0.06%		1.270s		8.580ms		0 b		0 b		879.35 Mb		0 b		148
aten::addmm		0.20%		10.647ms		20.40%		1.071s		7.238ms		1.262s		24.35%		1.263s		8.535ms		0 b		0 b		879.35 Mb		731.35 Mb		148
cudaFree		20.00%		1.050s		20.00%		1.050s		350.069ms		0.000us		0.00%		0.000us		0.000us		0 b		0 b		0 b		0 b		3
aten::div		8.93%		468.917ms		9.13%		479.685ms		1.057ms		483.799ms		9.33%		483.984ms		1.066ms		1.95 Mb		1.95 Mb		1.94 Gb		1.94 Gb		454
aten::add		7.38%		387.274ms		7.40%		388.333ms		1.961ms		405.858ms		7.83%		405.858ms		2.050ms		3.89 Mb		3.89 Mb		1.65 Gb		1.65 Gb		198
aten::abs		3.54%		185.689ms		7.07%		371.299ms		92.825ms		185.689ms		3.58%		371.333ms		92.833ms		3.89 Mb		1.95 Mb		0 b		0 b		4
aten::mul		5.72%		300.190ms		6.98%		366.603ms		844.707us		313.915ms		6.06%		314.000ms		723.502us		2.92 Mb		2.92 Mb		575.64 Mb		575.64 Mb		434
autograd::engine::evaluate_function: torch::autograd...		1.09%		57.130ms		6.61%		347.215ms		872.399us		11.044ms		0.21%		140.752ms		353.648us		0 b		0 b		-421.77 Mb		-421.77 Mb		398
aten::lt		5.33%		279.649ms		5.33%		279.649ms		69.912ms		279.704ms		5.40%		279.704ms		69.926ms		498.04 Kb		498.04 Kb		0 b		0 b		4
-------------------------------------------------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------		------------
Self CPU time total: 5.251s																												
Self CUDA time total: 5.183s																												
												

</code></pre>

ptrblck: <p>I would recommend profiling the workload with a visual profiler creating a proper timeline view with the kernel launches and execution times. Nsight Systems is the one I’m always using, but I also think the native PyTorch profiler is able to do so creating an output viewable in the browser.<br>
This will allow you to actually see how long each phase takes etc. An example can be found <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">here</a>.</p>