Fix auxiliary loss related code in transformers
Author: SangbumChoi
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

@amyeroberts  https://github.com/huggingface/transformers/pull/28354

There are several things that I summarized
1. deta/table_transformer has no issue
2. conditional_detr had slight issue but it was simple
3. yolos turns out that they don't have auxiliary output for their result. So I removed related changes and did not add aux_loss test. Check [out](https://github.com/hustvl/YOLOS/blob/5717fc29d727dab84ad585c56457b4de1225eddc/models/detector.py#L53) they don't have 'aux_output'.
4. maskformer issue related to configuration file

let me know if other models need this auxiliary loss test!

Comments:
```
- conditional_detr
    
    root@0a2b4fe54761:/mnt/nas2/users/sbchoi/transformers# RUN_SLOW=1 pytest tests/models/conditional_detr/
    ================================================================= test session starts ==================================================================
    platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
    rootdir: /mnt/nas2/users/sbchoi/transformers
    configfile: pyproject.toml
    plugins: hypothesis-6.92.0, hydra-core-1.3.2
    collected 149 items
    
    tests/models/conditional_detr/test_image_processing_conditional_detr.py ..............                                                           [  9%]
    tests/models/conditional_detr/test_modeling_conditional_detr.py .......................ssssss..sssssssss......s..............s......s........s.. [ 63%]
    ..sssssssssss.sssssssssssssss.s..s.......s.............                                                                                          [100%]
    
    =================================================================== warnings summary ===================================================================
    ../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373
    /opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
    
    ```
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\\n")
    
    ```
    
    src/transformers/deepspeed.py:23
    /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(
    
    ../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
    /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]
    
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    /opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
    Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)
    
    tests/models/conditional_detr/test_modeling_conditional_detr.py::ConditionalDetrModelTest::test_disk_offload_bin
    /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.**get**(instance, owner)()
    
    tests/models/conditional_detr/test_modeling_conditional_detr.py::ConditionalDetrModelTest::test_fast_init_context_manager
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:460: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)
    
    tests/models/conditional_detr/test_modeling_conditional_detr.py::ConditionalDetrModelTest::test_fast_init_context_manager
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:463: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(
    
    tests/models/conditional_detr/test_modeling_conditional_detr.py::ConditionalDetrModelTest::test_pipeline_object_detection
    /opt/conda/lib/python3.10/site-packages/huggingface_hub/repocard.py:105: UserWarning: Repo card metadata block was not found. Setting CardData to empty.
    warnings.warn("Repo card metadata block was not found. Setting CardData to empty.")
    
    - - Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
    ===================================================== 101 passed, 48 skipped, 9 warnings in 54.71s =====================================================
- deformable_detr
    
    root@0a2b4fe54761:/mnt/nas2/users/sbchoi/transformers# RUN_SLOW=1 pytest tests/models/deformable_detr/
    ================================================================= test session starts ==================================================================
    platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
    rootdir: /mnt/nas2/users/sbchoi/transformers
    configfile: pyproject.toml
    plugins: hypothesis-6.92.0, hydra-core-1.3.2
    collected 151 items
    
    tests/models/deformable_detr/test_image_processing_deformable_detr.py ..............                                                             [  9%]
    tests/models/deformable_detr/test_modeling_deformable_detr.py .......................ssssss.ssssssssss......s..............s......s............. [ 63%]
    sssssssssss.sssssssssssssss.s..s.......s...............                                                                                          [100%]
    
    =================================================================== warnings summary ===================================================================
    ../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373
    /opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
    
    ```
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\\n")
    
    ```
    
    src/transformers/deepspeed.py:23
    /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(
    
    ../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
    /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]
    
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    /opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
    Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)
    
    tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_disk_offload_bin
    /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.**get**(instance, owner)()
    
    tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_fast_init_context_manager
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:460: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)
    
    tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_fast_init_context_manager
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:463: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(
    
    tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_model_outputs_equivalence
    /mnt/nas2/users/sbchoi/transformers/tests/models/deformable_detr/test_modeling_deformable_detr.py:372: UserWarning: Use of index_put_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[indices] = tensor (Triggered internally at /opt/conda/conda-bld/pytorch_1702400410390/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:708.)
    t[t != t] = 0
    
    tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_model_outputs_equivalence
    /mnt/nas2/users/sbchoi/transformers/tests/models/deformable_detr/test_modeling_deformable_detr.py:372: UserWarning: Use of masked_fill_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[mask] = scalar (Triggered internally at /opt/conda/conda-bld/pytorch_1702400410390/work/aten/src/ATen/native/cuda/Indexing.cu:1564.)
    t[t != t] = 0
    
    tests/models/deformable_detr/test_modeling_deformable_detr.py::DeformableDetrModelTest::test_pipeline_object_detection
    /opt/conda/lib/python3.10/site-packages/huggingface_hub/repocard.py:105: UserWarning: Repo card metadata block was not found. Setting CardData to empty.
    warnings.warn("Repo card metadata block was not found. Setting CardData to empty.")
    
    - - Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
    =============================================== 103 passed, 48 skipped, 11 warnings in 163.84s (0:02:43) ===============================================
- deta
    
    root@0a2b4fe54761:/mnt/nas2/users/sbchoi/transformers# RUN_SLOW=1 pytest tests/models/deta
    ================================================================= test session starts ==================================================================
    platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
    rootdir: /mnt/nas2/users/sbchoi/transformers
    configfile: pyproject.toml
    plugins: hypothesis-6.92.0, hydra-core-1.3.2
    collected 151 items
    
    tests/models/deta/test_image_processing_deta.py ..............                                                                                   [  9%]
    tests/models/deta/test_modeling_deta.py ........s...............ssssss.ssssssssss......s..............s......s.............sssssssssss.sssssssss [ 78%]
    ssssss.s..s.......s.s............                                                                                                                [100%]
    
    =================================================================== warnings summary ===================================================================
    ../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373
    /opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
    
    ```
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\\n")
    
    ```
    
    src/transformers/deepspeed.py:23
    /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(
    
    ../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
    /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]
    
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    /opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
    Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)
    
    tests/models/deta/test_modeling_deta.py::DetaModelTest::test_disk_offload_bin
    /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.**get**(instance, owner)()
    
    tests/models/deta/test_modeling_deta.py::DetaModelTest::test_fast_init_context_manager
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:460: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)
    
    tests/models/deta/test_modeling_deta.py::DetaModelTest::test_fast_init_context_manager
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:463: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(
    
    tests/models/deta/test_modeling_deta.py::DetaModelTest::test_model_outputs_equivalence
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:1894: UserWarning: Use of index_put_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[indices] = tensor (Triggered internally at /opt/conda/conda-bld/pytorch_1702400410390/work/aten/src/ATen/native/TensorAdvancedIndexing.cpp:708.)
    t[t != t] = 0
    
    tests/models/deta/test_modeling_deta.py::DetaModelTest::test_model_outputs_equivalence
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:1894: UserWarning: Use of masked_fill_ on expanded tensors is deprecated. Please clone() the tensor before performing this operation. This also applies to advanced indexing e.g. tensor[mask] = scalar (Triggered internally at /opt/conda/conda-bld/pytorch_1702400410390/work/aten/src/ATen/native/cuda/Indexing.cu:1564.)
    t[t != t] = 0
    
    - - Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
    ==================================================== 101 passed, 50 skipped, 10 warnings in 53.04s =====================================================
- table transformer
    
    root@0a2b4fe54761:/mnt/nas2/users/sbchoi/transformers# RUN_SLOW=1 pytest tests/models/table_transformer/
    ================================================================= test session starts ==================================================================
    platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
    rootdir: /mnt/nas2/users/sbchoi/transformers
    configfile: pyproject.toml
    plugins: hypothesis-6.92.0, hydra-core-1.3.2
    collected 136 items
    
    tests/models/table_transformer/test_modeling_table_transformer.py .....................ssssss..sssssssss......s...............s......s.......... [ 57%]
    ...sssssssssss.sssssssssssssss.s..s..........s............                                                                                       [100%]
    
    tests/models/deta/test_image_processing_deta.py ..............                                                                       [  9%]
    tests/models/deta/test_modeling_deta.py ........s...............ssssss.ssssssssss.....s..............s......s.............ssssssssss [ 70%]
    s.sssssssssssssss.s..s.......s.s............
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")
    
    src/transformers/deepspeed.py:23
    /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    [100%]
    
    ============================================================= warnings summary =============================================================
    ../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373
    /opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
    
    ```
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\\n")
    
    ```
    
    src/transformers/deepspeed.py:23
    /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(
    
    ../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
    /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]
    
    ```
    declare_namespace(pkg)
    
    ```
    
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    /opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
    
    =================================================================== warnings summary ===================================================================
    ../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373
    /opt/conda/lib/python3.10/site-packages/_pytest/config/**init**.py:1373: PytestConfigWarning: Unknown config option: doctest_glob
    
    ```
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\\n")
    
    ```
    
    src/transformers/deepspeed.py:23
    /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(
    
    ../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
    /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]
    
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    ../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871
    /opt/conda/lib/python3.10/site-packages/pkg_resources/**init**.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
    Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)
    
    tests/models/table_transformer/test_modeling_table_transformer.py::TableTransformerModelTest::test_disk_offload_bin
    /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.**get**(instance, owner)()
    
    tests/models/table_transformer/test_modeling_table_transformer.py::TableTransformerModelTest::test_fast_init_context_manager
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:460: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)
    
    tests/models/table_transformer/test_modeling_table_transformer.py::TableTransformerModelTest::test_fast_init_context_manager
    /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:463: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(
    
    tests/models/table_transformer/test_modeling_table_transformer.py::TableTransformerModelTest::test_pipeline_object_detection
    /opt/conda/lib/python3.10/site-packages/huggingface_hub/repocard.py:105: UserWarning: Repo card metadata block was not found. Setting CardData to empty.
    warnings.warn("Repo card metadata block was not found. Setting CardData to empty.")
    
    - - Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
    ================================================ 89 passed, 47 skipped, 9 warnings in 103.04s (0:01:43) ================================================
```

Currently I'm out of GPU so unavailable to test YOLOS :(

@SangbumChoi Thanks for iterating! For MaskFormer, we still need to keep `use_auxiliary_loss`. Rather confusingly, this flag is set in two places: within the `deocder_config` and within the model's config. 

In the maskformer modeling file, it's the [maskformer's config](https://github.com/huggingface/transformers/blob/07bdbebb48a9fe1e748348e4e14ae0b4659e54c4/src/transformers/models/maskformer/modeling_maskformer.py#L1755) - `self.config.use_auxiliary_loss` - which is used, rather than the deocder's - `self.config.decoder_config.auxiliary_loss`. 

I can test YOLOs on my machine

@SangbumChoi It seems that the aux loss tests haven't been added for YOLOs yet (all tests pass - but that's expected :)) 

@amyeroberts For the case of YOLOs auxiliary head doesn't exist at the first place so the test cannot be established!

> @SangbumChoi Thanks for iterating! For MaskFormer, we still need to keep `use_auxiliary_loss`. Rather confusingly, this flag is set in two places: within the `deocder_config` and within the model's config.
> 
> In the maskformer modeling file, it's the [maskformer's config](https://github.com/huggingface/transformers/blob/07bdbebb48a9fe1e748348e4e14ae0b4659e54c4/src/transformers/models/maskformer/modeling_maskformer.py#L1755) - `self.config.use_auxiliary_loss` - which is used, rather than the deocder's - `self.config.decoder_config.auxiliary_loss`.

Ah I thought config.decoder.auxiliary_loss and config.use_auxiliary_loss was the same variable but it should be remained seperately. Let me revert this and fix again!

Voila! Let me know if there are other things to change!

Below is the modified version of maskformer pytest

```
root@bbc12aa272c6:/mnt/nas2/users/sbchoi/transformers# RUN_SLOW=1 pytest tests/models/maskformer/
================================================================================================== test session starts ==================================================================================================
platform linux -- Python 3.10.13, pytest-7.4.4, pluggy-1.0.0
rootdir: /mnt/nas2/users/sbchoi/transformers
configfile: pyproject.toml
plugins: hypothesis-6.92.0, hydra-core-1.3.2
collected 247 items

tests/models/maskformer/test_image_processing_maskformer.py ......................                                                                                                                                [  8%]
tests/models/maskformer/test_modeling_maskformer.py ........ssssss..sssssssss...s.........s.......s.........sssssssss.ssssssssssssssssss.s..s.....s.................                                              [ 54%]
tests/models/maskformer/test_modeling_maskformer_swin.py s........ssssss.ssssssssss..s........ss.......s.......ssssss.sssssssssssssssssssss.s......s.s....................                                        [100%]

=================================================================================================== warnings summary ====================================================================================================
../../../../../opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373
  /opt/conda/lib/python3.10/site-packages/_pytest/config/__init__.py:1373: PytestConfigWarning: Unknown config option: doctest_glob

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

src/transformers/deepspeed.py:23
  /mnt/nas2/users/sbchoi/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
    warnings.warn(

../../../../../opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28
  /opt/conda/lib/python3.10/site-packages/torch/utils/cpp_extension.py:28: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    from pkg_resources import packaging  # type: ignore[attr-defined]

../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
../../../../../opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871
  /opt/conda/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_call_with_segmentation_maps
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/image_processing_maskformer.py:699: FutureWarning: The `pad_and_return_pixel_mask` argument is deprecated and will be removed in v4.27
    warnings.warn(


tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_pipeline_feature_extraction
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_pipeline_image_segmentation
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_fp16
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_instance_segmentation_head
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_instance_segmentation_head_resnet_backbone
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_no_head
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_with_segmentation_maps_and_loss
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/image_processing_maskformer.py:410: FutureWarning: The `size_divisibility` argument is deprecated and will be removed in v4.27. Please use `size_divisor` instead.
    warnings.warn(

tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_image_processor_from_dict_with_kwargs
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_pipeline_feature_extraction
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_pipeline_image_segmentation
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_fp16
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_instance_segmentation_head
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_instance_segmentation_head_resnet_backbone
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_inference_no_head
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelIntegrationTest::test_with_segmentation_maps_and_loss
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/image_processing_maskformer.py:417: FutureWarning: The `max_size` argument is deprecated and will be removed in v4.27. Please use size['longest_edge'] instead.
    warnings.warn(

tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_integration_instance_segmentation
tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_integration_semantic_segmentation
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_pipeline_feature_extraction
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_pipeline_image_segmentation
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/image_processing_maskformer.py:428: FutureWarning: The `reduce_labels` argument is deprecated and will be removed in v4.27. Please use `do_reduce_labels` instead.
    warnings.warn(

tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_integration_instance_segmentation
  /mnt/nas2/users/sbchoi/transformers/tests/models/maskformer/test_image_processing_maskformer.py:300: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(inputs["mask_labels"][0].sum().item(), 41527.0)


  /mnt/nas2/users/sbchoi/transformers/tests/models/maskformer/test_image_processing_maskformer.py:395: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
    self.assertTrue(torch.allclose(inputs["class_labels"][0], torch.tensor(expected_class_labels)))

tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_integration_panoptic_segmentation
  /mnt/nas2/users/sbchoi/transformers/tests/models/maskformer/test_image_processing_maskformer.py:403: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(inputs["mask_labels"][0].sum().item(), 315193.0)

tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_integration_panoptic_segmentation
  /mnt/nas2/users/sbchoi/transformers/tests/models/maskformer/test_image_processing_maskformer.py:404: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(inputs["mask_labels"][1].sum().item(), 350747.0)

tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_integration_semantic_segmentation
  /mnt/nas2/users/sbchoi/transformers/tests/models/maskformer/test_image_processing_maskformer.py:342: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(inputs["mask_labels"][0].sum().item(), 170200.0)

tests/models/maskformer/test_image_processing_maskformer.py::MaskFormerImageProcessingTest::test_integration_semantic_segmentation
  /mnt/nas2/users/sbchoi/transformers/tests/models/maskformer/test_image_processing_maskformer.py:343: DeprecationWarning: Please use assertEqual instead.
    self.assertEquals(inputs["mask_labels"][1].sum().item(), 257036.0)

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_fast_init_context_manager
tests/models/maskformer/test_modeling_maskformer_swin.py::MaskFormerSwinModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:460: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(init_instance.linear.bias, expected_bias, rtol=1e-3, atol=1e-4)

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_fast_init_context_manager
tests/models/maskformer/test_modeling_maskformer_swin.py::MaskFormerSwinModelTest::test_fast_init_context_manager
  /mnt/nas2/users/sbchoi/transformers/tests/test_modeling_common.py:463: FutureWarning: `torch.testing.assert_allclose()` is deprecated since 1.12 and will be removed in a future release. Please use `torch.testing.assert_close()` instead. You can find detailed upgrade instructions in https://github.com/pytorch/pytorch/issues/61844.
    torch.testing.assert_allclose(

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_fast_init_context_manager
  /opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
    return self.fget.__get__(instance, owner)
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer_swin.py:212: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if num_channels != self.num_channels:

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer_swin.py:202: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if width % self.patch_size[1] != 0:

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer_swin.py:205: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if height % self.patch_size[0] != 0:

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer_swin.py:574: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    was_padded = pad_values[3] > 0 or pad_values[5] > 0

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer_swin.py:575: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if was_padded:

tests/models/maskformer/test_modeling_maskformer.py::Mas
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer_swin.py:248: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    should_pad = (height % 2 == 1) or (width % 2 == 1)

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer_swin.py:249: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if should_pad:

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:2443: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if size_prods == 1:

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer.py:537: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if attn_weights.size() != (batch_size * self.num_heads, target_len, source_len):

tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_attentions
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_output_hidden_state
tests/models/maskformer/test_modeling_maskformer.py::MaskFormerModelTest::test_torchscript_simple
  /mnt/nas2/users/sbchoi/transformers/src/transformers/models/maskformer/modeling_maskformer.py:568: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
    if attn_output.size() != (batch_size * self.num_heads, target_len, self.head_dim):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 148 passed, 99 skipped, 69 warnings in 55.89s =====================================================================================

```

@amyeroberts Changed as you suggested! However, do you know the method rerun the CI? it has time out error.

@SangbumChoi I can re-run :) 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28406). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.