[DETA] Improvement and Sync from DETA especially for training
Author: SangbumChoi
Pull request: yes

# What does this PR do?

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

There are several changes in DETA not only for inference but also for training.
1. Add assign_second_stage argument that used in original DETA
2. add output_proposals for "anchors" that used in assign_first_stage
3. num_boxes normalization based on multi-gpu environment
4. add "enc_outputs" and "auxiliary_outputs" loss while training
5. minor changes in variable name that was not appropriate

I tested with [custom dataset](https://universe.roboflow.com/roboflow-100/cable-damage) with finetuning. As a result original performance after 12 epoch was 1.0 AP, now is above 35 AP with same hyperparameter setting (such as learning rate).

I am not the author or related group to DETA but I am a co-contributor of [DETA](https://github.com/jozhang97/DETA) so I know pretty much all details of DETA. this changes will give a great improvement to user who wants to train/fine-tune DETA with [sagemaker script](https://huggingface.co/jozhang97/deta-swin-large/blob/main/config.json?sagemaker_train=true).

BTW, I think @NielsRogge missed important variable "assign_second_stage" to set True in [config.json](https://huggingface.co/jozhang97/deta-swin-large/blob/main/config.json)

@ArthurZucker Could you review this PR? (I couldn't share my test code for this, sorry about that)

I manually added to enable the auxiliary_loss and second_stage pipeline or see [this link](https://huggingface.co/sbchoi/deta-swin-large/tree/main)
```
  transformer_model.config.auxiliary_loss = cfg.auxiliary_loss
  transformer_model.config.assign_second_stage = cfg.assign_second_stage
```

Comments:
Also due to change of major arguments I think we need to rewrite the test function for swin_backbone.

Hi @SangbumChoi thanks so much for your PR. Could you try to make the CI green by running `make fixup` locally and fixing the errors it gives?

Btw if you would have the time, would be great look into why DETA evaluation numbers on the [open object detection leaderboard](https://huggingface.co/spaces/hf-vision/object_detection_leaderboard) are not as high as reported in the paper. 

@NielsRogge I think my PR will also affect to this leaderboard. I'm not sure if it will have positive or negative but I hope it is positive :)

In a viewpoint of inference pipeline, [the original implementation
](https://huggingface.co/jozhang97/deta-swin-large/discussions/2) of config.json was not using DETA-specific assigner(which was DETAassigner1 and DETAassigner2 in the code that you have written) even though "assign_first_stage=true". You may recheck this to confirm this! 
(See the line of 1941 in modeling_deta.py)

In addition postprocessing pipeline with class_threshold is default by 0.7 however original code doesn't use classification thresholding (=using threshold 0)

And also I will do some extra modification to make all the CI works!

@NielsRogge I updated and passed the CI. There are few changes that you should know! please don't hestitate to ask details

Awesome, I'll take a look in more detail in the next coming days.

Also, I was working on a notebook to fine-tune DETA on a custom dataset (by mainly copying [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) - you can replace `DetrForObjectDetection` by `DetaForObjectDetection`). However, the results were not as expected. If you have time to look into how to successfully fine-tune DETA, that would be awesome.

@NielsRogge I also saw this notebook and my custom code is very very similar to this notebook which is using pytorch_lightning and distributed training.
One important fact is that matching lr_backbone and lr with the same hyperparameter with 2e-4. I also used Swin backbone for default training and gave me 41AP in final (not his balloon but the dataset that i mentioned above)
```
        results = self.postprocess.post_process_object_detection(
            outputs, target_sizes=orig_target_sizes, threshold=0.0
        )
```
Also remind that thresholding and **setting auxiliary_loss to true** is also important

Can you also share the final AP?


@NielsRogge @amyeroberts Ping for reminder

Hi @SangbumChoi I tried out fine-tuning DETA from your branch on the balloon dataset with [this notebook](https://colab.research.google.com/drive/1fyZLR97V9RcWSjPN6djLdPylOjIV95r1?usp=sharing). Initially results didn't look good, but now they do! Had to set `auxiliary_loss=True` when loading the model.

![image](https://github.com/huggingface/transformers/assets/48327001/c8f71e23-e6e4-4241-908b-5772d1ee16af)

The only thing that is still weird is that bounding boxes are shown with confidence > 0.4, even though I've passed in threshold=0.2. Will need to look into this.

Already awesome that training seems to have improved a lot!

Thanks!


@NielsRogge Sounds great, it will have huge improvement. For the threshold part, because it has NMSpostprocess already implemented on the code, maybe overlapped bounding box with 0.2~0.39 might disappear at the visualization. (Check this part)

> Thanks for working on this!
> 
> There's a few changes we might not be able to add because of backwards compatibility and compatibility with the rest of the codebase. Otherwise looks good.

@amyeroberts Hi, I also did some modification and resolved several conversations that is obvious. However, I still opened the conversation that has some misalignment or comments! (I think only one comment is not still fixed which is multi-gpu distribute part!)

@NielsRogge Is there anything that I can help to make this PR merged? 

@amyeroberts would need to approve the PR.

For reference, I don't think anyone was successfully fine-tuning DETA before this PR, so would be great to get it merged. Also haven't shared about this model yet due to differences in reported COCO eval

@amyeroberts Thanks for your suggestion! Everything is fixed as your recommendation 👍🏼 

@amyeroberts There was one mistake importing sorry about that (see last commit)

for the slow test

```
root@0a2b4fe54761:/mnt/nas2/users/sbchoi/transformers# RUN_SLOW=1 pytest tests/models/deta/
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
rootdir: /mnt/nas2/users/sbchoi/transformers
configfile: pyproject.toml
plugins: hypothesis-6.92.0, hydra-core-1.3.2
collected 150 items

tests/models/deta/test_image_processing_deta.py ..............                                                                       [  9%]
tests/models/deta/test_modeling_deta.py ........s...............ssssss.ssssssssss.....s..............s......s.............ssssssssss [ 70%]
s.sssssssssssssss.s..s.......s.s............                                                                                         [100%]

============================================================= warnings summary =============================================================
../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373
  /opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

src/transformers/deepspeed.py:23
  /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(

../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
  /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]


    declare_namespace(pkg)
  /opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

src/transformers/deepspeed.py:23
  /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(

../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
  /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]

../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
  /opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_disk_offload_bin
  /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.__get__(instance, owner)()

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:460: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:463: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_model_outputs_equivalence
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:1894: UserWarning: Use of index_put_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[indices] = tensor (Triggered internally at /opt/conda/conda-bld/pytorch_1702400410390/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:708.)
    t[t != t] = 0

tests/models/deta/test_modeling_deta.py::DetaModelTest::test_model_outputs_equivalence
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:1894: UserWarning: Use of masked_fill_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[mask] = scalar (Triggered internally at /opt/conda/conda-bld/pytorch_1702400410390/work/aten/src/ATen/native/cuda/Indexing.cu:1564.)
    t[t != t] = 0

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================= 100 passed, 50 skipped, 10 warnings in 206.53s (0:03:26) =========================================
```

I think it has no problem. Does this answer your question of 'reduce' function? 
From my training code `accelerate launch --num_processes 4 src/train.py` and `accelerate launch --num_processes 1 src/train.py` both works fine.

@SangbumChoi Yep! That answers everything. Thanks for running the tests. Once the doc test runner finishes I'll merge in. Thanks again for all of your work on this! 

Finally! Thanks for the fast review @amyeroberts