Bfloat16 has worse performance than float16 for conv2d

StevenTsai: <p>Hi,<br>
I just compared the performance of my model with different parameter data types, and I found that using bfloat16 would get worse performance than float16. Is it expected or not?</p>
<p>For the further investigation, I used simple models to compare the performance of bfloat16 and float16.</p>
<p>exp1.</p>
<blockquote>
<blockquote>
<blockquote>
<p>input = torch.randn(20, 16, 500, 1000)<br>
m = nn.Conv2d(16, 33, 3, stride=2)<br>
for _ in range(1000):<br>
output = m(input)</p>
</blockquote>
</blockquote>
</blockquote>
<p>exp2.</p>
<blockquote>
<blockquote>
<blockquote>
<p>input = torch.randn(10000, 10000)<br>
m = nn.Linear(10000, 20000)<br>
for _ in range(1000):<br>
output = m(input)</p>
</blockquote>
</blockquote>
</blockquote>
<p>For exp1, the execution time of float16/bfloat16/float32 was 2.1/<strong>3.8</strong>/3.2 s. while for exp2, the execution time of float16/bfloat16/float32 was 20.1/<strong>19.5</strong>/33.8 s.<br>
For conv2, the performance of bfloat16 was even was than float32. Is this expected?</p>
<p>By the way, all my experiments were done by A100 GPUs (80 GB) with torch 1.11.0, cuda 11.1.3, NCCL 2.8.4</p>

ptrblck: <aside class="quote no-group" data-username="StevenTsai" data-post="1" data-topic="154373">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/dbc845/48.png" class="avatar"> StevenTsai:</div>
<blockquote>
<p>By the way, all my experiments were done by A100 GPUs (80 GB) with torch 1.11.0, cuda 11.1.3, NCCL 2.8.4</p>
</blockquote>
</aside>
<p>In your code snippet you are not using the GPU, so I’m unsure if the code snippet is wrong or if you are measuring the CPU performance and the GPU information is irrelevant.<br>
If you are using the device, then note that synchronizations are needed.<br>
To enable bfloat16 support for conv layers, you would have to enable the cuDNN v8 API via <code>TORCH_CUDNN_V8_API_ENABLED=1</code> and would need a build with this API enabled via <code>USE_EXPERIMENTAL_CUDNN_V8_API=1</code> (the current nightly binaries should already build with it).</p>

StevenTsai: <p>Actually, my original code was,</p>
<p>exp2.</p>
<blockquote>
<blockquote>
<blockquote>
<p>input = torch.randn(10000, 10000).to(device=_gpu_device, dtype=_type)<br>
m = nn.Linear(10000, 20000).to(device=_gpu_device, dtype=_type)<br>
torch.cuda.synchronize()<br>
import time<br>
t = time.time()<br>
for _ in range(100):<br>
output = m(input)<br>
torch.cuda.synchronize()<br>
time_elapse = time.time() - t<br>
print(f"time_elapse = {time_elapse}")</p>
</blockquote>
</blockquote>
</blockquote>
<p>Exp1 is written in the same way. Since I  wanted to make my exp easy to be read, I just simplified it.<br>
Thanks for you comments, I will tried environment variables you mentioned.</p>

StevenTsai: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , I have tried to set the environment variables but found no gain for bfloat16 performance.</p>
<p>On the other hand, I profiled my experiment using Nvidia Nsight Systems. The diagram showed that different kernels were applied for float16 and bfloat16. Did I run model with bfloat16 correctly?<br>
bfloat16 profiling result<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/35d2741924eafe2b2d66021a8008c5302bb74118.png" alt="bfloat16" data-base62-sha1="7G8cqHXspdxEy26vixuN9VYXFgc" width="560" height="374"></p>

StevenTsai: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
float16 profiling result<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/856b42ed1d699a4139f863548a61ad02ec60c0b9.png" data-download-href="https://discuss.pytorch.org/uploads/default/856b42ed1d699a4139f863548a61ad02ec60c0b9" title="float16"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/856b42ed1d699a4139f863548a61ad02ec60c0b9.png" alt="float16" data-base62-sha1="j2hje7rHp3ZsG2gmC73e1JWo053" width="690" height="452" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">float16</span><span class="informations">992×651 17.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>