Tegra X1 half precision data with PyTorch

username: jakemdaly
cooked: <p>I am using PyTorch with my Tegra X1 GPU, and running the NVIDIA command line tool nvprof to do some kernel profiling of DCNN layers.<br>
I am seeing, as reported by nvprof, that the half precision (16-bit) workloads have just as many (actually slightly more) FLOPS than when the layers are run with full precision (32-bit). However, about half as much data is being fetched from off-chip. The kernel durations generally seem to be higher for 16-bit also.<br>
My conclusion is that there is some sort of bit packing occuring with the torch.half() data.<br>
Could someone help explain what is going on here?</p>

username: ptrblck
cooked: <p>Once advantage of FP16 operations is that the memory bandwidth will be reduced as only half of the data needs to be transferred to the register for the operation (and back to the global device memory).<br>
Also, TensorCores can be used for suitable operations such as matrix multiplications and convolutions.</p>
<p>I’m not sure why the number of operations should decrease. Could you post a reference for this idea?</p>

username: jakemdaly
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
It makes sense to me why there would be the same number of operations, but not more operations for 16-bit. It sounds like I have a misconception that 16-bit tensors on the host are packed onto 32-bit words to be sent onto GPU? I had hypothesized that because of this, there might be additional unpacking operations on chip that would account for the discrepancy between 16-bit and 32-bit</p>

username: ptrblck
cooked: <p>The are not packed to 32bit, as it would completely get rid of the advantage of the reduced memory bandwidth.<br>
Depending on the operations you are running, you might get some additional operations e.g. for permutations. Some FP16 convolutions use TensorCores in the channels-last memory format, so that a permutation kernel will be called inside the cudnn kernel. However, the number of additional operations should be low compared to the actual workflow, and you should still see a speedup in FP16, if TensorCores are used.</p>

username: jakemdaly
cooked: <p>Sorry <a class="mention" href="/u/ptrblck">@ptrblck</a> I had to gather some more data.<br>
The data I am seeing for workloads that are run at FP32 are slower than FP16.<br>
See runtime column below. Additional information: my group is interested in the per-layer performance of these workloads, and based off the information I’ve collected below I’m struggling to see what the advantage of half precision is: even if there is reduced bandwidth, it seems like the FP16 layers generally take longer to run, so the throughput isn’t as great<br>
Could you shed some light on that? Thanks a lot!<br>
(BTW the rows below are taken as an average over ~30 trials)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a8c70b1f98d2a89c4d29b51994078a2d63d4ac9.png" data-download-href="https://discuss.pytorch.org/uploads/default/7a8c70b1f98d2a89c4d29b51994078a2d63d4ac9" title="Screenshot from 2020-07-06 14-48-40"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a8c70b1f98d2a89c4d29b51994078a2d63d4ac9_2_690x111.png" alt="Screenshot from 2020-07-06 14-48-40" data-base62-sha1="hu78SldViZuzkNm4M760HiSl2at" width="690" height="111" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a8c70b1f98d2a89c4d29b51994078a2d63d4ac9_2_690x111.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a8c70b1f98d2a89c4d29b51994078a2d63d4ac9_2_1035x166.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a8c70b1f98d2a89c4d29b51994078a2d63d4ac9_2_1380x222.png 2x" data-dominant-color="EAEDEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-07-06 14-48-40</span><span class="informations">1602×259 61.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: jakemdaly
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="87057" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The are not packed to 32bit, as it would completely get rid of the advantage of the reduced memory bandwidth.<br>
Depending on the operations you are running, you might get some additional operations e.g. for permutations. Some FP16 convolutions use TensorCores in the channels-last memory format, so that a permutation kernel will be called inside the cudnn kernel. However, the number of additional operations should be low compared to the actual workflow, and you should still see a speedup in FP16, if TensorCores are used.</p>
</blockquote>
</aside>
<p>Another thing I just did a little more research on is that the device I’m using is the Tegra X1, and it looks like it doesn’t support TensorCore technology (although other versions do <a href="https://en.wikipedia.org/wiki/Tegra" class="inline-onebox" rel="noopener nofollow ugc">Tegra - Wikipedia</a>)<br>
If there are no TensorCores, what benefits does FP16 have?</p>

username: ptrblck
cooked: <p>I’m not deeply familiar with the Tegra family. If you don’t have TensorCores, then the benefit would be the lower memory footprint, which might be used to increase the batch size and thus the throughput.<br>
However, depending on your use case, this might not be interesting (e.g. for an inference use case with a single sample).</p>
<p>If you are using cudnn, you might want to activate <code>torch.backends.cudnn.benchmark = True</code> at the beginning of the script to use the cudnn heuristics, which should select the fastest kernels.<br>
Note that the first iteration for each new input shape would be slower due to the benchmarking.</p>
<p>Let me know, if this reduces the FP16 latency.</p>