Why is there such a huge performance gap between bfloat16, float16, and float32?

yinjun1131: <p>I am trying to reduce the hard disk and memory usage of my model through quantization. The original type of the model is bfloat16. I am trying to perform a forced conversion test on the model using this code to test its performance after conversion<br>
‘’‘python<br>
def convert_bf16_fp16_to_fp32(model):<br>
for param in model.parameters():<br>
if param.dtype == torch.bfloat16 or param.dtype == torch.float16:<br>
param.data = param.data.to(dtype=torch.float16)<br>
for buffer in model.buffers():<br>
if buffer.dtype == torch.bfloat16 or buffer.dtype == torch.float16:<br>
buffer.data = buffer.data.to(dtype=torch.float16)<br>
return model<br>
‘’’<br>
I found that after conversion, if the target type is float32, the speed will be reduced by 1 times, and if the target type is float16, the speed will be reduced by 10 times. Where should I start to investigate how it will be so slow after converting to float16?</p>
<p>If it is because the CPU hardware does not support float16 and needs to be converted to float32 before calculation, what should I do to determine in advance whether the current hardware supports float16, bfloat16, and float32?</p>

yinjun1131: <p>when was float16:token/s was :0.279<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6fd397a147f3bb070b9587203940c41d24c9f30a.png" data-download-href="https://discuss.pytorch.org/uploads/default/6fd397a147f3bb070b9587203940c41d24c9f30a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fd397a147f3bb070b9587203940c41d24c9f30a_2_690x79.png" alt="image" data-base62-sha1="fXgkGwd3ZbOvqIExuzut9p7khdM" width="690" height="79" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fd397a147f3bb070b9587203940c41d24c9f30a_2_690x79.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fd397a147f3bb070b9587203940c41d24c9f30a_2_1035x118.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6fd397a147f3bb070b9587203940c41d24c9f30a_2_1380x158.png 2x" data-dominant-color="333434"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1752×202 54.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>when was bfloat16: token/s was :5.283<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b4f53acf8effeca7c4ca738dd14917b915b4bdd.png" data-download-href="https://discuss.pytorch.org/uploads/default/2b4f53acf8effeca7c4ca738dd14917b915b4bdd" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b4f53acf8effeca7c4ca738dd14917b915b4bdd.png" alt="image" data-base62-sha1="6b8ugN0PQ7ahGPIclhCl8gjnTt3" width="690" height="98" data-dominant-color="353432"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">745×106 11.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>when was float32: token/s was :2.64<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f085f8151eda26004f80c5f1fdf02a35fee5965.png" data-download-href="https://discuss.pytorch.org/uploads/default/3f085f8151eda26004f80c5f1fdf02a35fee5965" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f085f8151eda26004f80c5f1fdf02a35fee5965_2_690x54.png" alt="image" data-base62-sha1="8ZC1ejcuT6dArhSlH6kiwBv6K2h" width="690" height="54" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f085f8151eda26004f80c5f1fdf02a35fee5965_2_690x54.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f085f8151eda26004f80c5f1fdf02a35fee5965_2_1035x81.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f085f8151eda26004f80c5f1fdf02a35fee5965_2_1380x108.png 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1797×141 31.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

yinjun1131: <p>Finally, I forgot to mention that such a huge performance gap was measured in CPUs, and there is no such huge performance gap in GPUs</p>

ptrblck: <p>Not all floating point formats are accelerated on the CPU. <code>autocast</code> should select the appropriate <code>dtype</code> for the specified device.</p>

yinjun1131: <p>Yes, but what I need to know now is how to determine in advance whether this dtype is supported properly on my hardware. If it is not supported, I am curious about the conversion principle.</p>