Why is DeformableDetrForObjectDetection slower with bfloat16 than float32?
Author: DonggeunYu
Pull request: no
Labels: name: Vision

### System Info

- `transformers` version: 4.39.0
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

~~~
import time
from transformers import DeformableDetrForObjectDetection
import torch
from tqdm import tqdm

def main():

    torch_dtype = torch.bfloat16 or torch.float32
    device = torch.device('cuda')

    model = DeformableDetrForObjectDetection.from_pretrained(
        'SenseTime/deformable-detr',
        torch_dtype=torch_dtype,
        device_map=0,
    )

    model = model.train()

    pixel_values = torch.ones([2, 3, 256, 256]).to(torch_dtype).to(device)

    elapsed_time = []
    for i in tqdm(range(400)):
        now = time.time()
        outputs = model(pixel_values=pixel_values)
        elapsed_time.append(time.time() - now)

    print(sum(elapsed_time) / len(elapsed_time))

if __name__ == "__main__":
    main()
~~~

### Expected behavior

When tested on A100 40G, bfloat16 is slower than float32.
Could this be the case in other environments? Or is there a problem?

Comments:
Hi @DonggeunYu , thanks for the issue!

Could you, please, try to profile with your environment what operation slows it down? Maybe this will give some insights. Results will be saved to a file:

```python
import torch
import torch.autograd.profiler as profiler

from tqdm import tqdm
from transformers import DeformableDetrForObjectDetection


def main():

    torch_dtype = torch.float32
    device = torch.device('cuda')

    model = DeformableDetrForObjectDetection.from_pretrained(
        'SenseTime/deformable-detr',
        torch_dtype=torch_dtype,
        device_map=0,
    )

    model = model.train()

    pixel_values = torch.ones([2, 3, 256, 256]).to(torch_dtype).to(device)

    # warmup
    for i in tqdm(range(10)):
        outputs = model(pixel_values=pixel_values)

    with profiler.profile(profile_memory=False, record_shapes=False, use_cuda=True, with_flops=True) as prof:
        for i in tqdm(range(20)):
            outputs = model(pixel_values=pixel_values)

    print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=50), file=open(f"profile_{torch_dtype}.txt", "w"))

if __name__ == "__main__":
    main()
```


@qubvel 
[profile_torch.bfloat16.txt](https://github.com/huggingface/transformers/files/15131446/profile_torch.bfloat16.txt)
[profile_torch.float32.txt](https://github.com/huggingface/transformers/files/15131447/profile_torch.float32.txt)


Based on profiling results it seems like conv2d is slow with `bfloat16`

I found the following information regarding this issue:

Comment 1
https://github.com/pytorch/pytorch/issues/57707#issuecomment-1166656767

> Conv support in bfloat16 should be enabled in the cuDNN v8 API via export TORCH_CUDNN_V8_API_ENABLED=1.
> I don't know what the status of bfloat16 for interpolate/upsample, but couldn't find a commit enabling it.

Comment 2, result with `TORCH_CUDNN_V8_API_ENABLED=1`
https://github.com/Lightning-AI/pytorch-lightning/issues/11933#issuecomment-1181590004

[Relevant issue 1](https://github.com/Lightning-AI/pytorch-lightning/issues/11933)
[Relevant issue 2](https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373)

Please, let me know if it helps 🙂 

> Based on profiling results it seems like conv2d is slow with `bfloat16`
> 
> I found the following information regarding this issue:
> 
> Comment 1 [pytorch/pytorch#57707 (comment)](https://github.com/pytorch/pytorch/issues/57707#issuecomment-1166656767)
> 
> > Conv support in bfloat16 should be enabled in the cuDNN v8 API via export TORCH_CUDNN_V8_API_ENABLED=1.
> > I don't know what the status of bfloat16 for interpolate/upsample, but couldn't find a commit enabling it.
> 
> Comment 2, result with `TORCH_CUDNN_V8_API_ENABLED=1` [Lightning-AI/pytorch-lightning#11933 (comment)](https://github.com/Lightning-AI/pytorch-lightning/issues/11933#issuecomment-1181590004)
> 
> [Relevant issue 1](https://github.com/Lightning-AI/pytorch-lightning/issues/11933) [Relevant issue 2](https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373)
> 
> Please, let me know if it helps 🙂

Thanks, I'll try it.

No improvement...
The results below are using `TORCH_CUDNN_V8_API_ENABLED=1`.
[profile_torch.bfloat16.txt](https://github.com/huggingface/transformers/files/15144695/profile_torch.bfloat16.txt)


Hi @DonggeunYu, thanks for checking this out.

I notices that both files `profile_torch.bfloat16.txt` are completely identical. Could you please check if you provided correct files? I would not expect that 2 runs are completely identical.

Another question is how did you set the env variable, I would recommend to do it before importing torch/transformers.
For example:
```bash
TORCH_CUDNN_V8_API_ENABLED=1 python run_profiling.py
```

I'm so sorry.
Here
[profile_torch.bfloat16.txt](https://github.com/huggingface/transformers/files/15150111/profile_torch.bfloat16.txt)

The method below was used.
`TORCH_CUDNN_V8_API_ENABLED=1 python run_profiling.py`


Probably your cuDNN was not build with experimental features

https://discuss.pytorch.org/t/bfloat16-has-worse-performance-than-float16-for-conv2d/154373
> To enable bfloat16 support for conv layers, you would have to enable the cuDNN v8 API via TORCH_CUDNN_V8_API_ENABLED=1 and would need a build with this API enabled via USE_EXPERIMENTAL_CUDNN_V8_API=1 (the current nightly binaries should already build with it).

Did you try to update cuda/cudnn/torch?

@qubvel 
I don't think the problem is caused by a lower version.

used nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04
CUDA: 12.1
cudnn: 8.9
torch=2.1

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> @qubvel I don't think the problem is caused by a lower version.
> 
> used nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04 CUDA: 12.1 cudnn: 8.9 torch=2.1

hi, have you solved this problem?

> > @qubvel I don't think the problem is caused by a lower version.
> > used nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04 CUDA: 12.1 cudnn: 8.9 torch=2.1
> 
> hi, have you solved this problem?

Couldn't solve it.
I only understand this as a common issue in Conv2D.