Autocast with BF16 on CPU slows down model more than 2X
Labels: module: performance, triaged, module: bfloat16, module: amp (automated mixed precision)

### 🐛 Describe the bug

Trying to run inference with autocast and bf16 slows down a Hugging Face BERT model immensely. I would expect inference acceleration.

This makes it hard to just include autocast in a script for inference and have it run bfloat16 or fp16 depending on the availability of CUDA, since the model might be slowed on.


The benchmark is very simple.
Code to replicate: 
```
import torch
from transformers import AutoModelForSequenceClassification

bert_model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased').eval()
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

inp_single = "Bloomberg has decided to publish a new report on global economic situation."

tensor_single = tokenizer(
        inp_single,
        max_length=128,
        pad_to_max_length=True,
        add_special_tokens=True,
        return_tensors="pt",
    )


with torch.no_grad():
    start = time.time()
    for i in range(30):
        bert_model(tensor_single['input_ids'])
duration =time.time() - start
print(f'Normal time: {duration}')

with torch.autocast(device_type='cpu',enabled=True,dtype=torch.bfloat16):
    with torch.no_grad():
        # Warmup
        for i in range(5):
            bert_model(tensor_single['input_ids'])
        start = time.time()    
        for i in range(30):
            bert_model(tensor_single['input_ids'])
            
duration =time.time() - start
print(f'Autocast time: {duration}')
```

Output: 
```
Normal time: 2.8249216079711914
Autocast time: 7.403575658798218
```


System Info: 
Torch Version: 1.10.0
Transformers Version: 4.9.1

CPU Info:
```
/bin/bash /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
stepping        : 7
microcode       : 0x500320a
cpu MHz         : 3202.860
cache size      : 36608 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
:         : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer a/proc/cpuinfo
```

### Versions

Collecting environment information...
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Amazon Linux 2 (x86_64)
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15)
Clang version: Could not collect
CMake version: version 3.22.3
Libc version: glibc-2.26

Python version: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.10.102-99.473.amzn2.x86_64-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.1.105
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 510.47.03
cuDNN version: Probably one of the following:
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.5
/usr/local/cuda-11.1/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.5
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.1
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] numpydoc==1.1.0
[pip3] torch==1.10.0
[pip3] torch-model-archiver==0.5.0b20211117
[pip3] torch-workflow-archiver==0.2.0b20211118
[pip3] torchaudio==0.10.0
[pip3] torchserve==0.5.0b20211117
[pip3] torchtext==0.11.0
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl  
[conda] captum                    0.4.1                         0    pytorch
[conda] cudatoolkit               11.1.1               h6406543_9    conda-forge
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h95df7f1_0    conda-forge
[conda] mkl_fft                   1.3.1            py38h8666266_1    conda-forge
[conda] mkl_random                1.2.2            py38h1abd341_0    conda-forge
[conda] numpy                     1.21.2           py38h20f2e39_0  
[conda] numpy-base                1.21.2           py38h79a1101_0  
[conda] numpydoc                  1.1.0                      py_1    conda-forge
[conda] pytorch                   1.10.0          py3.8_cuda11.1_cudnn8.0.5_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch-model-archiver      0.5.0                    py38_0    pytorch
[conda] torch-workflow-archiver   0.2.0                    py38_0    pytorch
[conda] torchaudio                0.10.0               py38_cu111    pytorch
[conda] torchserve                0.5.0                    py38_0    pytorch
[conda] torchtext                 0.11.0                     py38    pytorch
[conda] torchvision               0.11.1               py38_cu111    pytorch

cc @VitalyFedyunin @ngimel @mcarilli @ptrblck