enable graident checkpointing in DetaObjectDetection and add tests in Swin/Donut_Swin
Author: SangbumChoi
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
I also found missing part that transfomers team did not put.

https://github.com/jozhang97/DETA/blob/f47e50efeb194357fb93a119f196a2e485fffdb5/models/deformable_transformer.py#L225

Also modified donut_swin and swin model for passing CI.

```
root@1c4a6f3ab7da:/mnt/nas2/users/sbchoi/transformers# pytest tests/models/deta/
==================================================================== test session starts ====================================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
rootdir: /mnt/nas2/users/sbchoi/transformers
configfile: pyproject.toml
plugins: hypothesis-6.92.0, anyio-4.2.0, hydra-core-1.3.2
collected 155 items

tests/models/deta/test_image_processing_deta.py ....ss........                                                                                        [  9%]
tests/models/deta/test_modeling_deta.py .........s...............ssssss.ssssssssss......s...............s......s.............sssssssssss.ssssssssssss [ 79%]
sss..s..s.......s.s.ss.sss....ss                                                                                                                      [100%]

===================================================================== warnings summary ======================================================================
../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373
  /opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

src/transformers/deepspeed.py:23
  /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(

../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
  /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]

../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
  /opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_disk_offload_bin
  /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.__get__(instance, owner)()

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:462: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:465: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_gradient_checkpointing
tests/models/deta/test_modeling_deta.py::DetaModelTest::test_training_gradient_checkpointing
  /opt/conda/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
    warnings.warn(

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_model_outputs_equivalence
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:1968: UserWarning: Use of index_put_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[indices] = tensor (Triggered internally at /opt/conda/conda-bld/pytorch_1702400410390/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:708.)
    t[t != t] = 0

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_model_outputs_equivalence
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:1968: UserWarning: Use of masked_fill_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[mask] = scalar (Triggered internally at /opt/conda/conda-bld/pytorch_1702400410390/work/aten/src/ATen/native/cuda/Indexing.cu:1564.)
    t[t != t] = 0

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= 96 passed, 59 skipped, 12 warnings in 56.64s ========================================================

```

```
root@1c4a6f3ab7da:/mnt/nas2/users/sbchoi/transformers# pytest tests/models/swin/
==================================================================== test session starts ====================================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
rootdir: /mnt/nas2/users/sbchoi/transformers
configfile: pyproject.toml
plugins: hypothesis-6.92.0, anyio-4.2.0, hydra-core-1.3.2
collected 189 items

tests/models/swin/test_modeling_swin.py .........ssssss.ssssssssss...............s.......s.......ssssssss.sssssssssssssssssss.s........s.....sss....s [ 57%]
........                                                                                                                                              [ 61%]
tests/models/swin/test_modeling_tf_swin.py ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss                                   [100%]

===================================================================== warnings summary ======================================================================
../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373
  /opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

src/transformers/deepspeed.py:23
  /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(

../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
  /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]

../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
  /opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:462: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:465: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_fast_init_context_manager
  /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.__get__(instance, owner)()

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_for_masked_image_modeling
tests/models/swin/test_modeling_swin.py::SwinModelTest::test_gradient_checkpointing
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/swin/modeling_swin.py:173: FutureWarning: logits attribute is deprecated and will be removed in version 5 of Transformers. Please use the reconstruction attribute to retrieve the final output instead.
    warnings.warn(

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_gradient_checkpointing
tests/models/swin/test_modeling_swin.py::SwinModelTest::test_training_gradient_checkpointing
  /opt/conda/lib/python3.10/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
    warnings.warn(

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_pipeline_image_classification
  /opt/conda/lib/python3.10/site-packages/huggingface_hub/repocard.py:105: UserWarning: Repo card metadata block was not found. Setting CardData to empty.
    warnings.warn("Repo card metadata block was not found. Setting CardData to empty.")

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_torch_fx
  /opt/conda/lib/python3.10/site-packages/torch/overrides.py:110: UserWarning: 'has_cuda' is deprecated, please use 'torch.backends.cuda.is_built()'
    torch.has_cuda,

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_torch_fx
  /opt/conda/lib/python3.10/site-packages/torch/overrides.py:111: UserWarning: 'has_cudnn' is deprecated, please use 'torch.backends.cudnn.is_available()'
    torch.has_cudnn,

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_torch_fx
  /opt/conda/lib/python3.10/site-packages/torch/overrides.py:117: UserWarning: 'has_mps' is deprecated, please use 'torch.backends.mps.is_built()'
    torch.has_mps,

tests/models/swin/test_modeling_swin.py::SwinModelTest::test_torch_fx
  /opt/conda/lib/python3.10/site-packages/torch/overrides.py:118: UserWarning: 'has_mkldnn' is deprecated, please use 'torch.backends.mkldnn.is_available()'
    torch.has_mkldnn,

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= 66 passed, 123 skipped, 17 warnings in 29.65s =======================================================
```

```
root@1c4a6f3ab7da:/mnt/nas2/users/sbchoi/transformers# pytest tests/models/donut/
==================================================================== test session starts ====================================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
rootdir: /mnt/nas2/users/sbchoi/transformers
configfile: pyproject.toml
plugins: hypothesis-6.92.0, anyio-4.2.0, hydra-core-1.3.2
collected 117 items

tests/models/donut/test_image_processing_donut.py ...........                                                                                         [  9%]
tests/models/donut/test_modeling_donut_swin.py ........ssssss..sssssssss.....................s........sssssssssssssssssssssssssss.s........s.....sss. [ 96%]
...
```

@amyeroberts Added test function for each corresponding models!

@amyeroberts I have fixed as you suggested. Overall it has been same as https://github.com/huggingface/transformers/pull/28686 with additional modification. Only modifying as the https://github.com/huggingface/transformers/pull/28686 will pass the CI but when we try in real code it will failure because there are gradient_checkpointing function in backbone.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28615). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts I think it seems good after merging most recent PR?

@amyeroberts all fixed! but could you rerun the CI? It seems just CI got failed to load extension in torchvision

@SangbumChoi Looks great! Yes, we had quite a few issues yesterday because of a new release of pytorch which broke everything 😢  A fix has been merged into main, which should resolve the version issues. Could you rebase? This should update the CI images and trigger another run. 

@amyeroberts all done!

@SangbumChoi Thanks for iterating and another great contribution!