Repository: huggingface/transformers
Pull request #23104 (2023-05-02 10:00:13)

Add focalnet backbone
# What does this PR do?
Adds `FocalNetBackbone` class to be used by X-Decoder and possibly other frameworks as FocalNet was published fairly recently.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X ] Did you write any new necessary tests?



HuggingFaceDocBuilderDev: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23104). All of your documentation changes will be reflected on that endpoint.

Patch:
diff --git a/docs/source/en/model_doc/distilbert.mdx b/docs/source/en/model_doc/distilbert.mdx
index cc1e037151..837f0319ec 100644
--- a/docs/source/en/model_doc/distilbert.mdx
+++ b/docs/source/en/model_doc/distilbert.mdx
@@ -19,13 +19,16 @@ specific language governing permissions and limitations under the License.
 <a href="https://huggingface.co/spaces/docs-demos/distilbert-base-uncased">
 <img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
 </a>
+<a href="https://huggingface.co/papers/1910.01108">
+<img alt="Paper page" src="https://img.shields.io/badge/Paper%20page-1910.01108-green">
+</a>
 </div>
 
 ## Overview
 
 The DistilBERT model was proposed in the blog post [Smaller, faster, cheaper, lighter: Introducing DistilBERT, a
 distilled version of BERT](https://medium.com/huggingface/distilbert-8cf3380435b5), and the paper [DistilBERT, a
-distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/abs/1910.01108). DistilBERT is a
+distilled version of BERT: smaller, faster, cheaper and lighter](https://arxiv.org/papers/1910.01108). DistilBERT is a
 small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than
 *bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language
 understanding benchmark.
diff --git a/docs/source/en/model_doc/gpt2.mdx b/docs/source/en/model_doc/gpt2.mdx
index ee80eb2f8b..6288e46eb5 100644
--- a/docs/source/en/model_doc/gpt2.mdx
+++ b/docs/source/en/model_doc/gpt2.mdx
@@ -24,7 +24,7 @@ specific language governing permissions and limitations under the License.
 ## Overview
 
 OpenAI GPT-2 model was proposed in [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by Alec
-Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. It's a causal (unidirectional)
+Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever from [OpenAI](https://huggingface.co/openai). It's a causal (unidirectional)
 transformer pretrained using language modeling on a very large corpus of ~40 GB of text data.
 
 The abstract from the paper is the following:
diff --git a/docs/source/en/model_doc/roberta.mdx b/docs/source/en/model_doc/roberta.mdx
index 7c0818a014..49007409ad 100644
--- a/docs/source/en/model_doc/roberta.mdx
+++ b/docs/source/en/model_doc/roberta.mdx
@@ -19,11 +19,14 @@ specific language governing permissions and limitations under the License.
 <a href="https://huggingface.co/spaces/docs-demos/roberta-base">
 <img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
 </a>
+<a href="https://huggingface.co/papers/1907.11692">
+<img alt="Paper page" src="https://img.shields.io/badge/Paper%20page-1907.11692-green">
+</a>
 </div>
 
 ## Overview
 
-The RoBERTa model was proposed in [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
+The RoBERTa model was proposed in [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://arxiv.org/abs/1907.11692) by Yinhan Liu, [Myle Ott](https://huggingface.co/myleott), Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
 Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google's BERT model released in 2018.
 
 It builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with
diff --git a/docs/source/en/model_doc/t5.mdx b/docs/source/en/model_doc/t5.mdx
index f7665c11ae..58074f1403 100644
--- a/docs/source/en/model_doc/t5.mdx
+++ b/docs/source/en/model_doc/t5.mdx
@@ -19,12 +19,15 @@ specific language governing permissions and limitations under the License.
 <a href="https://huggingface.co/spaces/docs-demos/t5-base">
 <img alt="Spaces" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue">
 </a>
+<a href="https://huggingface.co/papers/1910.10683">
+<img alt="Paper page" src="https://img.shields.io/badge/Paper%20page-1910.10683-green">
+</a>
 </div>
 
 ## Overview
 
-The T5 model was presented in [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf) by Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang,
-Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu.
+The T5 model was presented in [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://arxiv.org/pdf/1910.10683.pdf) by [Colin Raffel](https://huggingface.co/craffel), Noam Shazeer, [Adam Roberts](https://huggingface.co/adarob), Katherine Lee, Sharan Narang,
+Michael Matena, Yanqi Zhou, Wei Li, [Peter J. Liu](https://huggingface.co/peterjliu).
 
 The abstract from the paper is the following:
 
diff --git a/docs/source/ko/_toctree.yml b/docs/source/ko/_toctree.yml
index a9e1ff921d..bd24ee4e5c 100644
--- a/docs/source/ko/_toctree.yml
+++ b/docs/source/ko/_toctree.yml
@@ -8,45 +8,22 @@
   title: 시작하기
 - sections:
   - local: pipeline_tutorial
-    title: 추론을 위한 Pipeline
+    title: Pipeline으로 추론하기
   - local: autoclass_tutorial
-    title: 자동 클래스로 사전 학습된 인스턴스 로드하기
+    title: AutoClass로 사전 학습된 인스턴스 로드하기
   - local: preprocessing
-    title: 전처리
+    title: 데이터 전처리하기
   - local: training
     title: 사전 학습된 모델 미세 조정하기
+  - local: run_scripts
+    title: 스크립트로 학습하기
   - local: accelerate
-    title: 🤗 Accelerate를 활용한 분산 학습
+    title: 🤗 Accelerate로 분산 학습 구성하기
   - local: model_sharing
-    title: 모델 공유하기
-  title: (번역중) 튜토리얼
+    title: 만든 모델 공유하기
+  title: 튜토리얼
 - sections:
   - sections:
-    - local: create_a_model
-      title: 맞춤형 아키텍처 만들기
-    - local: custom_models
-      title: 사용자 정의 모델 공유하기
-    - local: run_scripts
-      title: 스크립트로 학습하기
-    - local: sagemaker
-      title: Amazon SageMaker에서 학습 실행하기
-    - local: in_translation
-      title: (번역중) Converting from TensorFlow checkpoints
-    - local: serialization
-      title: ONNX로 내보내기
-    - local: torchscript
-      title: TorchScript로 내보내기
-    - local: in_translation
-      title: (번역중) Troubleshoot
-    title: (번역중) 일반적인 사용방법
-  - sections:
-    - local: in_translation
-      title: (번역중) Use tokenizers from 🤗 Tokenizers
-    - local: multilingual
-      title: 다국어 모델 추론하기
-    - local: in_translation
-      title: (번역중) Text generation strategies
-    - sections:
       - local: tasks/sequence_classification
         title: 텍스트 분류
       - local: tasks/token_classification
@@ -62,39 +39,68 @@
       - local: tasks/summarization
         title: 요약
       - local: in_translation
-        title: (번역중) Multiple choice
-      title: (번역중) 태스크별 가이드
-      isExpanded: false
-    title: (번역중) 자연어처리
+        title: (번역중) Multiple Choice
+    title: 자연어처리
+    isExpanded: false
   - sections:
-    - local: in_translation
-      title: (번역중) Audio classification
-    - local: in_translation
-      title: (번역중) Automatic speech recognition
+      - local: in_translation
+        title: (번역중) Audio classification
+      - local: in_translation
+        title: (번역중) Automatic speech recognition
     title: (번역중) 오디오
+    isExpanded: false
   - sections:
-    - local: tasks/image_classification
-      title: 이미지 분류
+      - local: tasks/image_classification
+        title: 이미지 분류
+      - local: in_translation
+        title: (번역중) Semantic segmentation
+      - local: in_translation
+        title: (번역중) Video classification
+      - local: in_translation
+        title: (번역중) Object detection
+      - local: in_translation
+        title: (번역중) Zero-shot object detection
+      - local: tasks/zero_shot_image_classification
+        title: 제로샷(zero-shot) 이미지 분류
+      - local: in_translation
+        title: (번역중) Depth estimation
+    title: (번역중) 컴퓨터 비전
+    isExpanded: false
+  - sections:
+      - local: tasks/image_captioning
+        title: 이미지 캡셔닝
+      - local: in_translation
+        title: (번역중) Document Question Answering
+    title: (번역중) 멀티모달
+    isExpanded: false
+  title: 태스크 가이드
+- sections:
     - local: in_translation
-      title: (번역중) Semantic segmentation
+      title: (번역중) Use fast tokenizers from 🤗 Tokenizers
+    - local: multilingual
+      title: 다국어 모델 추론하기
     - local: in_translation
-      title: (번역중) Video classification
+      title: (번역중) Customize text generation strategy
+    - local: create_a_model
+      title: 모델별 API 사용하기
+    - local: custom_models
+      title: 사용자 정의 모델 공유하기
+    - local: sagemaker
+      title: Amazon SageMaker에서 학습 실행하기
+    - local: serialization
+      title: ONNX로 내보내기
+    - local: torchscript
+      title: TorchScript로 내보내기
     - local: in_translation
-      title: (번역중) Object detection
+      title: (번역중) Benchmarks
     - local: in_translation
-      title: (번역중) Zero-shot object detection
-    - local: tasks/zero_shot_image_classification
-      title: 제로샷(zero-shot) 이미지 분류
+      title: (번역중) Notebooks with examples
     - local: in_translation
-      title: (번역중) Depth estimation
-    title: (번역중) 컴퓨터 비전
-  - sections:
-    - local: tasks/image_captioning
-      title: 이미지 캡셔닝
+      title: (번역중) Community resources
     - local: in_translation
-      title: (번역중) Document Question Answering
-    title: (번역중) 멀티모달
-  - sections:
+      title: (번역중) Troubleshoot
+  title: (번역중) 개발자 가이드
+- sections:
     - local: in_translation
       title: (번역중) Overview
     - local: in_translation
@@ -129,8 +135,8 @@
       title: (번역중) Hyperparameter Search using Trainer API
     - local: in_translation
       title: (번역중) XLA Integration for TensorFlow Models
-    title: (번역중) 성능 및 확장성
-  - sections:
+  title: (번역중) 성능 및 확장성
+- sections:
     - local: in_translation
       title: (번역중) How to contribute to transformers?
     - local: in_translation
@@ -143,16 +149,8 @@
       title: (번역중) Testing
     - local: in_translation
       title: (번역중) Checks on a Pull Request
-    title: (번역중) 기여하기
-  - local: notebooks
-    title: (번역중) 🤗 Transformers Notebooks
-  - local: in_translation
-    title: (번역중) Community resources
-  - local: in_translation
-    title: (번역중) Benchmarks
-  - local: in_translation
-    title: (번역중) Migrating from previous packages
-  title: (번역중) How-to 가이드
+  title: (번역중) 기여하기
+
 - sections:
   - local: in_translation
     title: (번역중) Philosophy
@@ -263,6 +261,8 @@
         title: (번역중) ConvBERT
       - local: in_translation
         title: (번역중) CPM
+      - local: in_translation
+        title: (번역중) CPMANT
       - local: in_translation
         title: (번역중) CTRL
       - local: in_translation
@@ -309,6 +309,8 @@
         title: (번역중) GPT-J
       - local: in_translation
         title: (번역중) GPT2
+      - local: in_translation
+        title: (번역중) GPTBigCode
       - local: in_translation
         title: (번역중) GPTSAN Japanese
       - local: in_translation
@@ -361,6 +363,8 @@
         title: (번역중) NLLB-MoE
       - local: in_translation
         title: (번역중) Nyströmformer
+      - local: in_translation
+        title: (번역중) Open-Llama
       - local: in_translation
         title: (번역중) OPT
       - local: in_translation
@@ -460,6 +464,8 @@
         title: (번역중) EfficientFormer
       - local: in_translation
         title: (번역중) EfficientNet
+      - local: in_translation
+        title: (번역중) FocalNet
       - local: in_translation
         title: (번역중) GLPN
       - local: in_translation
@@ -572,6 +578,8 @@
         title: (번역중) CLIPSeg
       - local: in_translation
         title: (번역중) Data2Vec
+      - local: in_translation
+        title: (번역중) DePlot
       - local: in_translation
         title: (번역중) Donut
       - local: in_translation
@@ -592,6 +600,8 @@
         title: (번역중) LiLT
       - local: in_translation
         title: (번역중) LXMERT
+      - local: in_translation
+        title: (번역중) MatCha
       - local: in_translation
         title: (번역중) MGP-STR
       - local: in_translation
@@ -602,6 +612,8 @@
         title: (번역중) Perceiver
       - local: in_translation
         title: (번역중) Pix2Struct
+      - local: in_translation
+        title: (번역중) Segment Anything
       - local: in_translation
         title: (번역중) Speech Encoder Decoder Models
       - local: in_translation
diff --git a/docs/source/ko/notebooks.mdx b/docs/source/ko/notebooks.mdx
deleted file mode 100644
index ead9061833..0000000000
--- a/docs/source/ko/notebooks.mdx
+++ /dev/null
@@ -1,1 +0,0 @@
-# 열심히 번역 중입니다. 조금 이따 만나요!
\ No newline at end of file
diff --git a/src/transformers/__init__.py b/src/transformers/__init__.py
index 445fbb53e2..46be8c9d2c 100644
--- a/src/transformers/__init__.py
+++ b/src/transformers/__init__.py
@@ -1623,6 +1623,7 @@ else:
     _import_structure["models.focalnet"].extend(
         [
             "FOCALNET_PRETRAINED_MODEL_ARCHIVE_LIST",
+            "FocalNetBackbone",
             "FocalNetForImageClassification",
             "FocalNetForMaskedImageModeling",
             "FocalNetModel",
@@ -5178,6 +5179,7 @@ if TYPE_CHECKING:
         )
         from .models.focalnet import (
             FOCALNET_PRETRAINED_MODEL_ARCHIVE_LIST,
+            FocalNetBackbone,
             FocalNetForImageClassification,
             FocalNetForMaskedImageModeling,
             FocalNetModel,
diff --git a/src/transformers/generation/flax_utils.py b/src/transformers/generation/flax_utils.py
index c28e0afbb9..58a2bf13ba 100644
--- a/src/transformers/generation/flax_utils.py
+++ b/src/transformers/generation/flax_utils.py
@@ -463,6 +463,7 @@ class FlaxGenerationMixin:
                 logits_processor=logits_processor,
                 trace=trace,
                 params=params,
+                num_return_sequences=generation_config.num_return_sequences,
                 model_kwargs=model_kwargs,
             )
         else:
@@ -749,6 +750,7 @@ class FlaxGenerationMixin:
         logits_processor: Optional[FlaxLogitsProcessorList] = None,
         trace: bool = True,
         params: Optional[Dict[str, jnp.ndarray]] = None,
+        num_return_sequences: Optional[int] = None,
         model_kwargs: Optional[Dict[str, jnp.ndarray]] = None,
     ):
         """
@@ -793,6 +795,9 @@ class FlaxGenerationMixin:
         eos_token_id = eos_token_id if eos_token_id is not None else self.generation_config.eos_token_id
         length_penalty = length_penalty if length_penalty is not None else self.generation_config.length_penalty
         early_stopping = early_stopping if early_stopping is not None else self.generation_config.early_stopping
+        num_return_sequences = (
+            num_return_sequences if num_return_sequences is not None else self.generation_config.num_return_sequences
+        )
 
         batch_size, num_beams, cur_len = input_ids.shape
 
@@ -996,8 +1001,8 @@ class FlaxGenerationMixin:
         sequences = jnp.where(none_finished[:, None, None], state.sequences, state.running_sequences)
         scores = jnp.where(none_finished[:, None], state.scores, state.running_scores)
 
-        # take best beam for each batch
-        sequences = sequences[:, 0]
-        scores = scores[:, 0]
+        # Take best beams for each batch (the score is sorted in descending order)
+        sequences = flatten_beam_dim(sequences[:, :num_return_sequences, :])
+        scores = flatten_beam_dim(scores[:, :num_return_sequences])
 
         return FlaxBeamSearchOutput(sequences=sequences, scores=scores)
diff --git a/src/transformers/hf_argparser.py b/src/transformers/hf_argparser.py
index b1fa67f458..f808acebe9 100644
--- a/src/transformers/hf_argparser.py
+++ b/src/transformers/hf_argparser.py
@@ -15,6 +15,7 @@
 import dataclasses
 import json
 import sys
+import types
 from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser, ArgumentTypeError
 from copy import copy
 from enum import Enum
@@ -159,7 +160,7 @@ class HfArgumentParser(ArgumentParser):
             aliases = [aliases]
 
         origin_type = getattr(field.type, "__origin__", field.type)
-        if origin_type is Union:
+        if origin_type is Union or (hasattr(types, "UnionType") and isinstance(origin_type, types.UnionType)):
             if str not in field.type.__args__ and (
                 len(field.type.__args__) != 2 or type(None) not in field.type.__args__
             ):
@@ -245,10 +246,23 @@ class HfArgumentParser(ArgumentParser):
             type_hints: Dict[str, type] = get_type_hints(dtype)
         except NameError:
             raise RuntimeError(
-                f"Type resolution failed for f{dtype}. Try declaring the class in global scope or "
+                f"Type resolution failed for {dtype}. Try declaring the class in global scope or "
                 "removing line of `from __future__ import annotations` which opts in Postponed "
                 "Evaluation of Annotations (PEP 563)"
             )
+        except TypeError as ex:
+            # Remove this block when we drop Python 3.9 support
+            if sys.version_info[:2] < (3, 10) and "unsupported operand type(s) for |" in str(ex):
+                python_version = ".".join(map(str, sys.version_info[:3]))
+                raise RuntimeError(
+                    f"Type resolution failed for {dtype} on Python {python_version}. Try removing "
+                    "line of `from __future__ import annotations` which opts in union types as "
+                    "`X | Y` (PEP 604) via Postponed Evaluation of Annotations (PEP 563). To "
+                    "support Python versions that lower than 3.10, you need to use "
+                    "`typing.Union[X, Y]` instead of `X | Y` and `typing.Optional[X]` instead of "
+                    "`X | None`."
+                ) from ex
+            raise
 
         for field in dataclasses.fields(dtype):
             if not field.init:
diff --git a/src/transformers/models/auto/modeling_auto.py b/src/transformers/models/auto/modeling_auto.py
index ccdda1af33..14847c7ad2 100755
--- a/src/transformers/models/auto/modeling_auto.py
+++ b/src/transformers/models/auto/modeling_auto.py
@@ -980,6 +980,7 @@ MODEL_FOR_BACKBONE_MAPPING_NAMES = OrderedDict(
         ("convnext", "ConvNextBackbone"),
         ("convnextv2", "ConvNextV2Backbone"),
         ("dinat", "DinatBackbone"),
+        ("focalnet", "FocalNetBackbone"),
         ("maskformer-swin", "MaskFormerSwinBackbone"),
         ("nat", "NatBackbone"),
         ("resnet", "ResNetBackbone"),
diff --git a/src/transformers/models/focalnet/__init__.py b/src/transformers/models/focalnet/__init__.py
index e082ae26d2..b043a006f9 100644
--- a/src/transformers/models/focalnet/__init__.py
+++ b/src/transformers/models/focalnet/__init__.py
@@ -30,6 +30,7 @@ else:
         "FOCALNET_PRETRAINED_MODEL_ARCHIVE_LIST",
         "FocalNetForImageClassification",
         "FocalNetForMaskedImageModeling",
+        "FocalNetBackbone",
         "FocalNetModel",
         "FocalNetPreTrainedModel",
     ]
@@ -45,6 +46,7 @@ if TYPE_CHECKING:
     else:
         from .modeling_focalnet import (
             FOCALNET_PRETRAINED_MODEL_ARCHIVE_LIST,
+            FocalNetBackbone,
             FocalNetForImageClassification,
             FocalNetForMaskedImageModeling,
             FocalNetModel,
diff --git a/src/transformers/models/focalnet/configuration_focalnet.py b/src/transformers/models/focalnet/configuration_focalnet.py
index 5bfecb5737..c6814e1dda 100644
--- a/src/transformers/models/focalnet/configuration_focalnet.py
+++ b/src/transformers/models/focalnet/configuration_focalnet.py
@@ -47,6 +47,8 @@ class FocalNetConfig(PretrainedConfig):
         use_conv_embed (`bool`, *optional*, defaults to `False`):
             Whether to use convolutional embedding. The authors noted that using convolutional embedding usually
             improve the performance, but it's not used by default.
+        hidden_sizes (`List[int]`, *optional*, defaults to `[192, 384, 768, 768]`):
+            Dimensionality (hidden size) at each stage.
         depths (`list(int)`, *optional*, defaults to `[2, 2, 6, 2]`):
             Depth (number of layers) of each stage in the encoder.
         focal_levels (`list(int)`, *optional*, defaults to `[2, 2, 2, 2]`):
@@ -78,6 +80,14 @@ class FocalNetConfig(PretrainedConfig):
             The epsilon used by the layer normalization layers.
         encoder_stride (`int`, `optional`, defaults to 32):
             Factor to increase the spatial resolution by in the decoder head for masked image modeling.
+        out_features (`List[str]`, *optional*):
+            If used as backbone, list of features to output. Can be any of `"stem"`, `"stage1"`, `"stage2"`, etc.
+            (depending on how many stages the model has). If unset and `out_indices` is set, will default to the
+            corresponding stages. If unset and `out_indices` is unset, will default to the last stage.
+        out_indices (`List[int]`, *optional*):
+            If used as backbone, list of indices of features to output. Can be any of 0, 1, 2, etc. (depending on how
+            many stages the model has). If unset and `out_features` is set, will default to the corresponding stages.
+            If unset and `out_features` is unset, will default to the last stage.
 
     Example:
 
@@ -102,6 +112,7 @@ class FocalNetConfig(PretrainedConfig):
         num_channels=3,
         embed_dim=96,
         use_conv_embed=False,
+        hidden_sizes=[192, 384, 768, 768],
         depths=[2, 2, 6, 2],
         focal_levels=[2, 2, 2, 2],
         focal_windows=[3, 3, 3, 3],
@@ -117,6 +128,8 @@ class FocalNetConfig(PretrainedConfig):
         initializer_range=0.02,
         layer_norm_eps=1e-5,
         encoder_stride=32,
+        out_features=None,
+        out_indices=None,
         **kwargs,
     ):
         super().__init__(**kwargs)
@@ -126,6 +139,7 @@ class FocalNetConfig(PretrainedConfig):
         self.num_channels = num_channels
         self.embed_dim = embed_dim
         self.use_conv_embed = use_conv_embed
+        self.hidden_sizes = hidden_sizes
         self.depths = depths
         self.focal_levels = focal_levels
         self.focal_windows = focal_windows
@@ -141,3 +155,36 @@ class FocalNetConfig(PretrainedConfig):
         self.initializer_range = initializer_range
         self.layer_norm_eps = layer_norm_eps
         self.encoder_stride = encoder_stride
+        self.stage_names = ["stem"] + [f"stage{idx}" for idx in range(1, len(self.depths) + 1)]
+
+        if out_features is not None and out_indices is not None:
+            if len(out_features) != len(out_indices):
+                raise ValueError("out_features and out_indices should have the same length if both are set")
+            elif out_features != [self.stage_names[idx] for idx in out_indices]:
+                raise ValueError("out_features and out_indices should correspond to the same stages if both are set")
+
+        if out_features is None and out_indices is not None:
+            out_features = [self.stage_names[idx] for idx in out_indices]
+        elif out_features is not None and out_indices is None:
+            out_indices = [self.stage_names.index(feature) for feature in out_features]
+        elif out_features is None and out_indices is None:
+            out_features = [self.stage_names[-1]]
+            out_indices = [len(self.stage_names) - 1]
+
+        if out_features is not None:
+            if not isinstance(out_features, list):
+                raise ValueError("out_features should be a list")
+            for feature in out_features:
+                if feature not in self.stage_names:
+                    raise ValueError(
+                        f"Feature {feature} is not a valid feature name. Valid names are {self.stage_names}"
+                    )
+        if out_indices is not None:
+            if not isinstance(out_indices, (list, tuple)):
+                raise ValueError("out_indices should be a list or tuple")
+            for idx in out_indices:
+                if idx >= len(self.stage_names):
+                    raise ValueError(f"Index {idx} is not a valid index for a list of length {len(self.stage_names)}")
+
+        self.out_features = out_features
+        self.out_indices = out_indices
diff --git a/src/transformers/models/focalnet/convert_focalnet_to_hf_format.py b/src/transformers/models/focalnet/convert_focalnet_to_hf_format.py
index a23383e3ab..4aed159280 100644
--- a/src/transformers/models/focalnet/convert_focalnet_to_hf_format.py
+++ b/src/transformers/models/focalnet/convert_focalnet_to_hf_format.py
@@ -56,7 +56,6 @@ def get_focalnet_config(model_name):
         embed_dim = 128
     elif "large" in model_name:
         embed_dim = 192
-        focal_windows = [5, 5, 5, 5]
     elif "xlarge" in model_name:
         embed_dim = 256
     elif "huge" in model_name:
@@ -130,7 +129,10 @@ def convert_focalnet_checkpoint(model_name, pytorch_dump_folder_path, push_to_hu
         "focalnet-small-lrf": "https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_small_lrf.pth",
         "focalnet-base": "https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_base_srf.pth",
         "focalnet-base-lrf": "https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_base_lrf.pth",
-        "focalnet-large": "https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_large_lrf_384.pth",
+        "focalnet-large-lrf-fl3": "https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_large_lrf_384.pth",
+        "focalnet-large-lrf-fl4": "https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_large_lrf_384_fl4.pth",
+        "focalnet-xlarge-lrf-fl3": "https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_xlarge_lrf_384.pth",
+        "focalnet-xlarge-lrf-fl4": "https://projects4jw.blob.core.windows.net/focalnet/release/classification/focalnet_xlarge_lrf_384_fl4.pth",
     }
     # fmt: on
 
diff --git a/src/transformers/models/focalnet/modeling_focalnet.py b/src/transformers/models/focalnet/modeling_focalnet.py
index ff1b75e14b..cfd6468976 100644
--- a/src/transformers/models/focalnet/modeling_focalnet.py
+++ b/src/transformers/models/focalnet/modeling_focalnet.py
@@ -26,7 +26,8 @@ from torch import nn
 from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
 
 from ...activations import ACT2FN
-from ...modeling_utils import PreTrainedModel
+from ...modeling_outputs import BackboneOutput
+from ...modeling_utils import BackboneMixin, PreTrainedModel
 from ...utils import (
     ModelOutput,
     add_code_sample_docstrings,
@@ -209,7 +210,6 @@ class FocalNetEmbeddings(nn.Module):
             embeddings = embeddings * (1.0 - mask) + mask_tokens * mask
 
         embeddings = self.dropout(embeddings)
-
         return embeddings, output_dimensions
 
 
@@ -971,3 +971,81 @@ class FocalNetForImageClassification(FocalNetPreTrainedModel):
             hidden_states=outputs.hidden_states,
             reshaped_hidden_states=outputs.reshaped_hidden_states,
         )
+
+
+@add_start_docstrings(
+    """
+    FocalNet backbone, to be used with frameworks like X-Decoder.
+    """,
+    FOCALNET_START_DOCSTRING,
+)
+class FocalNetBackbone(FocalNetPreTrainedModel, BackboneMixin):
+    def __init__(self, config):
+        super().__init__(config)
+
+        self.stage_names = config.stage_names
+        self.focalnet = FocalNetModel(config)
+
+        self.num_features = [config.embed_dim] + config.hidden_sizes
+        self.out_features = config.out_features if config.out_features is not None else [self.stage_names[-1]]
+        if config.out_indices is not None:
+            self.out_indices = config.out_indices
+        else:
+            self.out_indices = tuple(i for i, layer in enumerate(self.stage_names) if layer in self.out_features)
+
+        # initialize weights and apply final processing
+        self.post_init()
+
+    @add_start_docstrings_to_model_forward(FOCALNET_INPUTS_DOCSTRING)
+    @replace_return_docstrings(output_type=BackboneOutput, config_class=_CONFIG_FOR_DOC)
+    def forward(
+        self,
+        pixel_values: torch.Tensor,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> BackboneOutput:
+        """
+        Returns:
+
+        Examples:
+
+        ```python
+        >>> from transformers import AutoImageProcessor, AutoBackbone
+        >>> import torch
+        >>> from PIL import Image
+        >>> import requests
+
+        >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+        >>> image = Image.open(requests.get(url, stream=True).raw)
+
+        >>> processor = AutoImageProcessor.from_pretrained("microsoft/focalnet-tiny-lrf")
+        >>> model = AutoBackbone.from_pretrained("microsoft/focalnet-tiny-lrf")
+
+        >>> inputs = processor(image, return_tensors="pt")
+        >>> outputs = model(**inputs)
+        ```"""
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+
+        outputs = self.focalnet(pixel_values, output_hidden_states=True, return_dict=True)
+
+        hidden_states = outputs.reshaped_hidden_states
+
+        feature_maps = ()
+        for idx, stage in enumerate(self.stage_names):
+            if stage in self.out_features:
+                feature_maps += (hidden_states[idx],)
+
+        if not return_dict:
+            output = (feature_maps,)
+            if output_hidden_states:
+                output += (outputs.hidden_states,)
+            return output
+
+        return BackboneOutput(
+            feature_maps=feature_maps,
+            hidden_states=outputs.hidden_states if output_hidden_states else None,
+            attentions=None,
+        )
diff --git a/src/transformers/utils/dummy_pt_objects.py b/src/transformers/utils/dummy_pt_objects.py
index 7fe538eccc..1e9845ba9b 100644
--- a/src/transformers/utils/dummy_pt_objects.py
+++ b/src/transformers/utils/dummy_pt_objects.py
@@ -3002,6 +3002,13 @@ class FNetPreTrainedModel(metaclass=DummyObject):
 FOCALNET_PRETRAINED_MODEL_ARCHIVE_LIST = None
 
 
+class FocalNetBackbone(metaclass=DummyObject):
+    _backends = ["torch"]
+
+    def __init__(self, *args, **kwargs):
+        requires_backends(self, ["torch"])
+
+
 class FocalNetForImageClassification(metaclass=DummyObject):
     _backends = ["torch"]
 
diff --git a/utils/check_repo.py b/utils/check_repo.py
index 5bdec16b9e..7280381faf 100644
--- a/utils/check_repo.py
+++ b/utils/check_repo.py
@@ -836,6 +836,7 @@ SHOULD_HAVE_THEIR_OWN_PAGE = [
     "ConvNextBackbone",
     "ConvNextV2Backbone",
     "DinatBackbone",
+    "FocalNetBackbone",
     "MaskFormerSwinBackbone",
     "MaskFormerSwinConfig",
     "MaskFormerSwinModel",


Test patch:
diff --git a/tests/generation/test_flax_utils.py b/tests/generation/test_flax_utils.py
index c6182a2386..647482b88c 100644
--- a/tests/generation/test_flax_utils.py
+++ b/tests/generation/test_flax_utils.py
@@ -158,6 +158,19 @@ class FlaxGenerationTesterMixin:
 
             self.assertListEqual(generation_outputs.tolist(), jit_generation_outputs.tolist())
 
+    def test_beam_search_generate_num_return_sequences(self):
+        config, input_ids, _, max_length = self._get_input_ids_and_config()
+        config.do_sample = False
+        config.max_length = max_length
+        config.num_beams = 2
+        config.num_return_sequences = 2
+
+        for model_class in self.all_generative_model_classes:
+            model = model_class(config)
+
+            generation_outputs = model.generate(input_ids).sequences
+            self.assertEqual(generation_outputs.shape[0], input_ids.shape[0] * config.num_return_sequences)
+
     def test_sample_generate_logits_warper(self):
         config, input_ids, _, max_length = self._get_input_ids_and_config()
         config.do_sample = True
diff --git a/tests/models/focalnet/test_modeling_focalnet.py b/tests/models/focalnet/test_modeling_focalnet.py
index 4ddf8e63b9..75127e5fd3 100644
--- a/tests/models/focalnet/test_modeling_focalnet.py
+++ b/tests/models/focalnet/test_modeling_focalnet.py
@@ -22,6 +22,7 @@ from transformers import FocalNetConfig
 from transformers.testing_utils import require_torch, require_vision, slow, torch_device
 from transformers.utils import cached_property, is_torch_available, is_vision_available
 
+from ...test_backbone_common import BackboneTesterMixin
 from ...test_configuration_common import ConfigTester
 from ...test_modeling_common import ModelTesterMixin, _config_zero_init, floats_tensor, ids_tensor
 
@@ -30,7 +31,12 @@ if is_torch_available():
     import torch
     from torch import nn
 
-    from transformers import FocalNetForImageClassification, FocalNetForMaskedImageModeling, FocalNetModel
+    from transformers import (
+        FocalNetBackbone,
+        FocalNetForImageClassification,
+        FocalNetForMaskedImageModeling,
+        FocalNetModel,
+    )
     from transformers.models.focalnet.modeling_focalnet import FOCALNET_PRETRAINED_MODEL_ARCHIVE_LIST
 
 if is_vision_available():
@@ -48,6 +54,7 @@ class FocalNetModelTester:
         patch_size=2,
         num_channels=3,
         embed_dim=16,
+        hidden_sizes=[32, 64, 128],
         depths=[1, 2, 1],
         num_heads=[2, 2, 4],
         window_size=2,
@@ -67,6 +74,7 @@ class FocalNetModelTester:
         type_sequence_label_size=10,
         encoder_stride=8,
         out_features=["stage1", "stage2"],
+        out_indices=[1, 2],
     ):
         self.parent = parent
         self.batch_size = batch_size
@@ -74,6 +82,7 @@ class FocalNetModelTester:
         self.patch_size = patch_size
         self.num_channels = num_channels
         self.embed_dim = embed_dim
+        self.hidden_sizes = hidden_sizes
         self.depths = depths
         self.num_heads = num_heads
         self.window_size = window_size
@@ -93,6 +102,7 @@ class FocalNetModelTester:
         self.type_sequence_label_size = type_sequence_label_size
         self.encoder_stride = encoder_stride
         self.out_features = out_features
+        self.out_indices = out_indices
 
     def prepare_config_and_inputs(self):
         pixel_values = floats_tensor([self.batch_size, self.num_channels, self.image_size, self.image_size])
@@ -111,6 +121,7 @@ class FocalNetModelTester:
             patch_size=self.patch_size,
             num_channels=self.num_channels,
             embed_dim=self.embed_dim,
+            hidden_sizes=self.hidden_sizes,
             depths=self.depths,
             num_heads=self.num_heads,
             window_size=self.window_size,
@@ -126,6 +137,7 @@ class FocalNetModelTester:
             initializer_range=self.initializer_range,
             encoder_stride=self.encoder_stride,
             out_features=self.out_features,
+            out_indices=self.out_indices,
         )
 
     def create_and_check_model(self, config, pixel_values, labels):
@@ -139,6 +151,35 @@ class FocalNetModelTester:
 
         self.parent.assertEqual(result.last_hidden_state.shape, (self.batch_size, expected_seq_len, expected_dim))
 
+    def create_and_check_backbone(self, config, pixel_values, labels):
+        model = FocalNetBackbone(config=config)
+        model.to(torch_device)
+        model.eval()
+        result = model(pixel_values)
+
+        # verify feature maps
+        self.parent.assertEqual(len(result.feature_maps), len(config.out_features))
+        self.parent.assertListEqual(list(result.feature_maps[0].shape), [self.batch_size, self.image_size, 8, 8])
+
+        # verify channels
+        self.parent.assertEqual(len(model.channels), len(config.out_features))
+        self.parent.assertListEqual(model.channels, config.hidden_sizes[:-1])
+
+        # verify backbone works with out_features=None
+        config.out_features = None
+        model = FocalNetBackbone(config=config)
+        model.to(torch_device)
+        model.eval()
+        result = model(pixel_values)
+
+        # verify feature maps
+        self.parent.assertEqual(len(result.feature_maps), 1)
+        self.parent.assertListEqual(list(result.feature_maps[0].shape), [self.batch_size, self.image_size * 2, 4, 4])
+
+        # verify channels
+        self.parent.assertEqual(len(model.channels), 1)
+        self.parent.assertListEqual(model.channels, [config.hidden_sizes[-1]])
+
     def create_and_check_for_masked_image_modeling(self, config, pixel_values, labels):
         model = FocalNetForMaskedImageModeling(config=config)
         model.to(torch_device)
@@ -191,6 +232,7 @@ class FocalNetModelTest(ModelTesterMixin, unittest.TestCase):
             FocalNetModel,
             FocalNetForImageClassification,
             FocalNetForMaskedImageModeling,
+            FocalNetBackbone,
         )
         if is_torch_available()
         else ()
@@ -204,7 +246,7 @@ class FocalNetModelTest(ModelTesterMixin, unittest.TestCase):
 
     def setUp(self):
         self.model_tester = FocalNetModelTester(self)
-        self.config_tester = ConfigTester(self, config_class=FocalNetConfig, embed_dim=37)
+        self.config_tester = ConfigTester(self, config_class=FocalNetConfig, embed_dim=37, has_text_modality=False)
 
     def test_config(self):
         self.create_and_test_config_common_properties()
@@ -222,6 +264,10 @@ class FocalNetModelTest(ModelTesterMixin, unittest.TestCase):
         config_and_inputs = self.model_tester.prepare_config_and_inputs()
         self.model_tester.create_and_check_model(*config_and_inputs)
 
+    def test_backbone(self):
+        config_and_inputs = self.model_tester.prepare_config_and_inputs()
+        self.model_tester.create_and_check_backbone(*config_and_inputs)
+
     def test_for_masked_image_modeling(self):
         config_and_inputs = self.model_tester.prepare_config_and_inputs()
         self.model_tester.create_and_check_for_masked_image_modeling(*config_and_inputs)
@@ -234,14 +280,14 @@ class FocalNetModelTest(ModelTesterMixin, unittest.TestCase):
     def test_inputs_embeds(self):
         pass
 
-    @unittest.skip(reason="FocalNet Transformer does not use feedforward chunking")
+    @unittest.skip(reason="FocalNet does not use feedforward chunking")
     def test_feed_forward_chunking(self):
         pass
 
     def test_model_common_attributes(self):
         config, _ = self.model_tester.prepare_config_and_inputs_for_common()
 
-        for model_class in self.all_model_classes:
+        for model_class in self.all_model_classes[:-1]:
             model = model_class(config)
             self.assertIsInstance(model.get_input_embeddings(), (nn.Module))
             x = model.get_output_embeddings()
@@ -250,7 +296,7 @@ class FocalNetModelTest(ModelTesterMixin, unittest.TestCase):
     def test_forward_signature(self):
         config, _ = self.model_tester.prepare_config_and_inputs_for_common()
 
-        for model_class in self.all_model_classes:
+        for model_class in self.all_model_classes[:-1]:
             model = model_class(config)
             signature = inspect.signature(model.forward)
             # signature.parameters is an OrderedDict => so arg_names order is deterministic
@@ -309,7 +355,7 @@ class FocalNetModelTest(ModelTesterMixin, unittest.TestCase):
             else (self.model_tester.image_size, self.model_tester.image_size)
         )
 
-        for model_class in self.all_model_classes:
+        for model_class in self.all_model_classes[:-1]:
             inputs_dict["output_hidden_states"] = True
             self.check_hidden_states_output(inputs_dict, config, model_class, image_size)
 
@@ -337,7 +383,7 @@ class FocalNetModelTest(ModelTesterMixin, unittest.TestCase):
         padded_height = image_size[0] + patch_size[0] - (image_size[0] % patch_size[0])
         padded_width = image_size[1] + patch_size[1] - (image_size[1] % patch_size[1])
 
-        for model_class in self.all_model_classes:
+        for model_class in self.all_model_classes[:-1]:
             inputs_dict["output_hidden_states"] = True
             self.check_hidden_states_output(inputs_dict, config, model_class, (padded_height, padded_width))
 
@@ -393,3 +439,14 @@ class FocalNetModelIntegrationTest(unittest.TestCase):
         expected_slice = torch.tensor([0.2166, -0.4368, 0.2191]).to(torch_device)
         self.assertTrue(torch.allclose(outputs.logits[0, :3], expected_slice, atol=1e-4))
         self.assertTrue(outputs.logits.argmax(dim=-1).item(), 281)
+
+
+@require_torch
+class FocalNetBackboneTest(BackboneTesterMixin, unittest.TestCase):
+    all_model_classes = (FocalNetBackbone,) if is_torch_available() else ()
+    config_class = FocalNetConfig
+
+    has_attentions = False
+
+    def setUp(self):
+        self.model_tester = FocalNetModelTester(self)
diff --git a/tests/test_backbone_common.py b/tests/test_backbone_common.py
index 80e68a2f44..6bcf47004b 100644
--- a/tests/test_backbone_common.py
+++ b/tests/test_backbone_common.py
@@ -135,6 +135,8 @@ class BackboneTesterMixin:
             # Verify num_features has been initialized in the backbone init
             self.assertIsNotNone(backbone.num_features)
             self.assertTrue(len(backbone.channels) == len(backbone.out_indices))
+            print(backbone.stage_names)
+            print(backbone.num_features)
             self.assertTrue(len(backbone.stage_names) == len(backbone.num_features))
             self.assertTrue(len(backbone.channels) <= len(backbone.num_features))
             self.assertTrue(len(backbone.out_feature_channels) == len(backbone.stage_names))
diff --git a/tests/utils/test_hf_argparser.py b/tests/utils/test_hf_argparser.py
index 0ad3c9c2ac..a9db072f04 100644
--- a/tests/utils/test_hf_argparser.py
+++ b/tests/utils/test_hf_argparser.py
@@ -15,6 +15,7 @@
 import argparse
 import json
 import os
+import sys
 import tempfile
 import unittest
 from argparse import Namespace
@@ -36,6 +37,10 @@ except ImportError:
     # For Python 3.7
     from typing_extensions import Literal
 
+# Since Python 3.10, we can use the builtin `|` operator for Union types
+# See PEP 604: https://peps.python.org/pep-0604
+is_python_no_less_than_3_10 = sys.version_info >= (3, 10)
+
 
 def list_field(default=None, metadata=None):
     return field(default_factory=lambda: default, metadata=metadata)
@@ -125,6 +130,23 @@ class StringLiteralAnnotationExample:
     foo_str: "List[str]" = list_field(default=["Hallo", "Bonjour", "Hello"])
 
 
+if is_python_no_less_than_3_10:
+
+    @dataclass
+    class WithDefaultBoolExamplePep604:
+        foo: bool = False
+        baz: bool = True
+        opt: bool | None = None
+
+    @dataclass
+    class OptionalExamplePep604:
+        foo: int | None = None
+        bar: float | None = field(default=None, metadata={"help": "help message"})
+        baz: str | None = None
+        ces: list[str] | None = list_field(default=[])
+        des: list[int] | None = list_field(default=[])
+
+
 class HfArgumentParserTest(unittest.TestCase):
     def argparsersEqual(self, a: argparse.ArgumentParser, b: argparse.ArgumentParser):
         """
@@ -167,8 +189,6 @@ class HfArgumentParserTest(unittest.TestCase):
         self.argparsersEqual(parser, expected)
 
     def test_with_default_bool(self):
-        parser = HfArgumentParser(WithDefaultBoolExample)
-
         expected = argparse.ArgumentParser()
         expected.add_argument("--foo", type=string_to_bool, default=False, const=True, nargs="?")
         expected.add_argument("--baz", type=string_to_bool, default=True, const=True, nargs="?")
@@ -176,22 +196,29 @@ class HfArgumentParserTest(unittest.TestCase):
         # and its default must be set to False
         expected.add_argument("--no_baz", action="store_false", default=False, dest="baz")
         expected.add_argument("--opt", type=string_to_bool, default=None)
-        self.argparsersEqual(parser, expected)
 
-        args = parser.parse_args([])
-        self.assertEqual(args, Namespace(foo=False, baz=True, opt=None))
+        dataclass_types = [WithDefaultBoolExample]
+        if is_python_no_less_than_3_10:
+            dataclass_types.append(WithDefaultBoolExamplePep604)
 
-        args = parser.parse_args(["--foo", "--no_baz"])
-        self.assertEqual(args, Namespace(foo=True, baz=False, opt=None))
+        for dataclass_type in dataclass_types:
+            parser = HfArgumentParser(dataclass_type)
+            self.argparsersEqual(parser, expected)
 
-        args = parser.parse_args(["--foo", "--baz"])
-        self.assertEqual(args, Namespace(foo=True, baz=True, opt=None))
+            args = parser.parse_args([])
+            self.assertEqual(args, Namespace(foo=False, baz=True, opt=None))
 
-        args = parser.parse_args(["--foo", "True", "--baz", "True", "--opt", "True"])
-        self.assertEqual(args, Namespace(foo=True, baz=True, opt=True))
+            args = parser.parse_args(["--foo", "--no_baz"])
+            self.assertEqual(args, Namespace(foo=True, baz=False, opt=None))
 
-        args = parser.parse_args(["--foo", "False", "--baz", "False", "--opt", "False"])
-        self.assertEqual(args, Namespace(foo=False, baz=False, opt=False))
+            args = parser.parse_args(["--foo", "--baz"])
+            self.assertEqual(args, Namespace(foo=True, baz=True, opt=None))
+
+            args = parser.parse_args(["--foo", "True", "--baz", "True", "--opt", "True"])
+            self.assertEqual(args, Namespace(foo=True, baz=True, opt=True))
+
+            args = parser.parse_args(["--foo", "False", "--baz", "False", "--opt", "False"])
+            self.assertEqual(args, Namespace(foo=False, baz=False, opt=False))
 
     def test_with_enum(self):
         parser = HfArgumentParser(MixedTypeEnumExample)
@@ -266,21 +293,27 @@ class HfArgumentParserTest(unittest.TestCase):
         self.assertEqual(args, Namespace(foo_int=[1], bar_int=[2, 3], foo_str=["a", "b", "c"], foo_float=[0.1, 0.7]))
 
     def test_with_optional(self):
-        parser = HfArgumentParser(OptionalExample)
-
         expected = argparse.ArgumentParser()
         expected.add_argument("--foo", default=None, type=int)
         expected.add_argument("--bar", default=None, type=float, help="help message")
         expected.add_argument("--baz", default=None, type=str)
         expected.add_argument("--ces", nargs="+", default=[], type=str)
         expected.add_argument("--des", nargs="+", default=[], type=int)
-        self.argparsersEqual(parser, expected)
 
-        args = parser.parse_args([])
-        self.assertEqual(args, Namespace(foo=None, bar=None, baz=None, ces=[], des=[]))
+        dataclass_types = [OptionalExample]
+        if is_python_no_less_than_3_10:
+            dataclass_types.append(OptionalExamplePep604)
+
+        for dataclass_type in dataclass_types:
+            parser = HfArgumentParser(dataclass_type)
+
+            self.argparsersEqual(parser, expected)
+
+            args = parser.parse_args([])
+            self.assertEqual(args, Namespace(foo=None, bar=None, baz=None, ces=[], des=[]))
 
-        args = parser.parse_args("--foo 12 --bar 3.14 --baz 42 --ces a b c --des 1 2 3".split())
-        self.assertEqual(args, Namespace(foo=12, bar=3.14, baz="42", ces=["a", "b", "c"], des=[1, 2, 3]))
+            args = parser.parse_args("--foo 12 --bar 3.14 --baz 42 --ces a b c --des 1 2 3".split())
+            self.assertEqual(args, Namespace(foo=12, bar=3.14, baz="42", ces=["a", "b", "c"], des=[1, 2, 3]))
 
     def test_with_required(self):
         parser = HfArgumentParser(RequiredExample)