DeformableDETR support bfloat16
Author: DonggeunYu
Pull request: yes

# What does this PR do?

This PR for DeformableDetrModel supports bfloat16.

Fixes #29231 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

- vision models: @amyeroberts


Comments:
> Thanks for adding!
> 
> For my own understanding, what is this PR addressing which wasn't addressed in #29013? It seems we're undoing some of the changes there
> 
> Can you run the model's slow tests on GPU and confirm they're all passing?

Since AT_DISPATCH_FLOATING_TYPES_AND_HALF only supports float32 and float16, it was replaced with AT_DISPATCH_FLOATING_TYPES_AND2. AT_DISPATCH_FLOATING_TYPES_AND2 can support float32, float16, bfloat16.
I have completed the bfloat16 learning test. Are you talking about pytest?

@DonggeunYu Great, thanks for explaining. For the tests, I mean can you run:

`RUN_SLOW pytest tests/models/deformable_detr/` on a GPU?

> @DonggeunYu Great, thanks for explaining. For the tests, I mean can you run:
> 
> `RUN_SLOW pytest tests/models/deformable_detr/` on a GPU?

~~~
# RUN_SLOW=1 pytest tests/models/deformable_detr/
===================================== test session starts =====================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.4.0
rootdir: /nas/k8s/dev/mlops/donggeun/tmp/transformers
configfile: pyproject.toml
plugins: timeout-2.2.0, xdist-3.5.0, docs-0.1.0, anyio-4.2.0
collected 158 items

tests/models/deformable_detr/test_image_processing_deformable_detr.py ................  [ 10%]
tests/models/deformable_detr/test_modeling_deformable_detr.py ........................s [ 25%]
sssss..ssssssssss......s..............s......s...ssss.s....ssssssssssss.sssssssssssssss [ 81%]
..s..s.......s..ss............                                                          [100%]

====================================== warnings summary =======================================
../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:121
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:121: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("best_of")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:140
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:140: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("repetition_penalty")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:146
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:146: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("seed")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:152
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:152: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("temperature")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:158
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:158: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("top_k")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:164
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:164: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("top_p")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:170
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:170: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("truncate")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:176
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:176: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("typical_p")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:204
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:204: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("inputs")

../../../../../../../usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:210
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/inference/_text_generation.py:210: PydanticDeprecatedSince20: Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.6/migration/
    @validator("stream")

../../../../../../../usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py:1373
  /usr/local/lib/python3.10/dist-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

src/transformers/deepspeed.py:23
  /nas/k8s/dev/mlops/donggeun/tmp/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(

tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_disk_offload_bin
  /usr/local/lib/python3.10/dist-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.__get__(instance, owner)()

tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_fast_init_context_manager
  /nas/k8s/dev/mlops/donggeun/tmp/transformers/tests/test_modeling_common.py:464: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)

tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_fast_init_context_manager
  /nas/k8s/dev/mlops/donggeun/tmp/transformers/tests/test_modeling_common.py:467: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(

tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_model_outputs_equivalence
tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_model_outputs_equivalence
  /nas/k8s/dev/mlops/donggeun/tmp/transformers/tests/models/deformable_detr/test_modeling_deformable_detr.py:374: UserWarning: Use of index_put_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[indices] = tensor (Triggered internally at ../aten/src/ATen/native/TensorAdvancedIndexing.cpp:708.)
    t[t != t] = 0

tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_model_outputs_equivalence
tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_model_outputs_equivalence
  /nas/k8s/dev/mlops/donggeun/tmp/transformers/tests/models/deformable_detr/test_modeling_deformable_detr.py:374: UserWarning: Use of masked_fill_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[mask] = scalar (Triggered internally at ../aten/src/ATen/native/cuda/Indexing.cu:1564.)
    t[t != t] = 0

tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_pipeline_object_detection
  /usr/local/lib/python3.10/dist-packages/huggingface_hub/repocard.py:105: UserWarning: Repo card metadata block was not found. Setting CardData to empty.
    warnings.warn("Repo card metadata block was not found. Setting CardData to empty.")

tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_training_gradient_checkpointing
  /usr/local/lib/python3.10/dist-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================== 102 passed, 56 skipped, 21 warnings in 1100.81s (0:18:20) ==================
~~~

> Thanks for adding! Just some small nits on the test

thank you! Reflected.

@DonggeunYu Great - last thing is to make sure all the tests on CI are passing. For the `@require_torch_bf16` decorator, you'll need to import it. This should fix most of the issues.

> @DonggeunYu Great - last thing is to make sure all the tests on CI are passing. For the `@require_torch_bf16` decorator, you'll need to import it. This should fix most of the issues.

All pass~!