RF-DETR: Neural Architecture Search for Real-Time Detection Transformers
Authors: Isaac Robinson
Peter Robicheaux
Matvei Popov
Deva Ramanan
Neehar Peri
Venue: ICLR 2026 Poster (2026)
Keywords: Real-Time Object Detection
Neural Architecture Search
Transfer Learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present RF-DETR, a real-time object detector that achieves pareto-optimal accuracy and latency using Neural Architecture Search.

Abstract: Open-vocabulary detectors achieve impressive performance on COCO, but often fail to generalize to real-world datasets with out-of-distribution classes not typically found in their pre-training. Rather than simply fine-tuning a heavy-weight vision-language model (VLM) for new domains, we introduce RF-DETR, a light-weight specialist detection transformer that discovers accuracy-latency Pareto curves for any target dataset with weight-sharing neural architecture search (NAS). Our approach fine-tunes a pre-trained base network on a target dataset and evaluates thousands of network configurations with different accuracy-latency tradeoffs without re-training. Further, we revisit the "tunable knobs" for NAS to improve the transferability of DETRs to diverse target domains. Notably, RF-DETR significantly improves over prior state-of-the-art real-time methods on COCO and Roboflow100-VL. RF-DETR (nano) achieves 48.0 AP on COCO, beating D-FINE (nano) by 5.3 AP at similar latency, and RF-DETR (2x-large) outperforms GroundingDINO (tiny) by 1.2 AP on Roboflow100-VL while running 20 times as fast. To the best of our knowledge, RF-DETR (2x-large) is the first real-time detector to surpass 60 AP on COCO. Our code is available on GitHub.

Published as a conference paper at ICLR 2026
RF-DETR: NEURAL ARCHITECTURE SEARCH FOR REAL- TIME DETECTION TRANSFORMERS
Neeharr Isaac Robinson', Peter Robicheaux Matvei Popov's Deva Ramanan", Neehar Peri
"Roboflow, *Carnegie Mellon University
ABSTRACT
Open-vocabulary detectors achieve impressive performance COCO, but of- ten fail to generalize to real-world datasets with out-of-distribution classes not typically found in their pre-training. Rather than simply fine-tuning a heavy- weight lision-language model (VLM) for new domains, we introduce RF-DETR, light-weight specialist detection transformer that discovers accuracy- Pareto curves for any target dataset with weight-sharing neural architecture search target dataset
(NAS). Our approach fine-tunes pre-trained base network and evaluates thousands of network configurations with different accuracy-latency tradeoffs withou! re-training. Further, we revisit the "tunable knobs" for NAS to improve the transferability of DETRs to diverse target domains. Notably. RF-DETR significantly improves over prior state-of-the-as real-time methods on COCO and Roboflow 1 (00-VI. RF-DETR (nano) achieves 48.0 AP on COCO, beating D-FINE (nano) by 5.3 AP at similar latency, and RF-DETR (2x-large) outperforms GroundingDINO (tiny) by 1.2 AP Roboflow 100-VL while run- ning 20x as fast. To the best our knowledge, RF-DETR (2x-large) is the first real-time detector to surpass 60 AP on COCO. Our code is available on GitHub.
INTRODUCTION
Object detection is fundamental problem in computer vision that has matured in recent years (Felzenszwalb al., 2009; Lin et al., 2014; Ren al., 2015). Open-vocabulary detectors like GroundingDIINO (Liu et al., 2023) and YOLO-World (Cheng et al., 2024) achieve remarkable zero-shot performance on common categories like car, truck, and pedestrian in. However, state-of-the-art vision-language models (VLMs) still struggle to generalize to out-of-distribution classes, tasks and imaging modalities not typically found in their pre-training (Robicheaux et al., 2025). Fine-tuning VLMs on target dataset significantly improves in-domain performance at the cost of runtime efficiency (due to heavy-weight text encoders) and open- -pca-uocyyylary generalization. In contrast, specialist (i.e.. dlosed-vocabulaee RT-DETR (Zhao et al., 2024) achieve real-time inference, but underperform fined-tuned VLMs like GroundingDINOO. In this paper, we modernize specialist detectors by combining internet-scale pre- training with timetimme& architectures to achieve state-of-the-an performance and fast inference.
object detectors like D-FINE (Peng al., 2024) and
Are Specialist Detectors Over-Optimized for COCO? Sustained progress object detection can be largely attributed to standardized benchmarks like PASCAL VOC (Everingham et al., 2015) and COCO (Lin et al., 2014). However, we find that recent specialist detectors implicitly overfit to COCO at the cost of real-world performance using bespoke model architectures, learning rate schedulers, and augmentation schedulers. Notably, of-e-of-theearr object detectors like YOLOv8 (Jocher et al., 2023) generalize poorly to real-world datasets with significantly different data distri- butions from COCO (e-g.. number of objects per image, number of classes, and dataset size). To address these limitations, present RF-DETR, scheduler-free approach that leverages internet- scale pre- vre-training to generalize to real-world data distributions. To better specialize our model for diverse hardware platforms and dataset characteristics. revisit neural architecture search (NAS) in the context of end-to-end object detection and segmentation.
Rethinking Neural Architecture Search (NAS) for DETRs. NAS discovers iccuracy-latenge tradeoffs by exploring architectural variants within a pre-defined search space. NAS has been previ- ously studied in the context of image classification (Tan & Le, 2019; Cai et al., 2019) and for model

---

Published as a conference paper at ICLR 2026
cub-components like detector backbones Tan al. (2020) and FPNs Ghiasi et al. (2019). Unlike prior work, we explore end-to-end weight-sharing NAS for object detection and segmentation. Our key insight, inspired by OFA (Cai et al., 2019), is that lution, and architectural components like patch size during training. Further, weight-sharing NAS allows us modify inference configurations like the number of decoder layers and query tokens to specialize our strong base model without fine-tuning. We evaluate all model configurations with grid search on a validation set. Importantly, our approach does not evaluate the search space until the base model has been fully-trained on the target dataset. As a result, all possible sub-nets (i.e., model configurations within the search space) achieve strong performance without further tine-tuning, sig- nificantly reducing the computational cost of optimizing for new hardware. Interestingly, we find that sub-nets not explicitly seen during training still achieve high performance (Appendix ??), sug- gesting that RF-DETR can generalize to unseen architectures. Extending RF-DETR for segmenta- tion is also relatively straightforward and only requires adding a lightweight instance segmentation head. We denote this model as RF-DETR-Secs Notably, this allows us to also leverage end-to-end weight-sharing NAS to discover Pareto optimal architectures for real-time instance segmentation.
vary model inputs like image reso-
Standardizing Latency Evaluation. We evaluate our approach COCO (Lin et al., 2014) and Roboflow w0w.:/000vvvv0 VL (RF100-VL) VL) (Robicheaux et al., 2025) and achieve state-of-ther ff-thhean art performance among real-time detectors. RE-DETR (nano) outperforms D-FINE (nano) by 5 5% AP on COCO at comparable run-times, and RF-DETR (2x-large) beats GroundingDINO (tiny) on RF100-VI. at a fraction of the runtime. RF-DETR- SSg (nano) outperforms YOLOv 1 I-Seg (x-large) on COCO while running 4 x as fast. However, comparing RF-DETR's 's latency with prior work remains challenging because reported latency evaluation varies significantly between papers. Notably, each new model re-benchmarks the latency of prior work for fair comparison on their hardware. For example, D- FINE's reported latency evaluation of LW-DETR (Chen et al., 2024a) is 25% faster than originally reported. We identify that this lack of reproducibility can be primarily attributed to GPU power throttling during inference. We find that buffering between forward passes limits power over-draw and standardizes latency evaluation (Table 1).
Contributions. We present three major contributions. First, we introduce RF-DETR, a family of scheduler-free NAS-based detection and segmentation models that outperform prior state-of-the-an on BF100-VL (Robicheaux et al., 2025) and real-time methods with latencies < 40 ms on COCO (Lin et al., 2014) (Figure 1). To the best of our knowledge, RF-DETR is the first real-time detector to exceed 60 mAP on COCO. Next, we explore the "tunable-knobs for weiggt-sharing NAS to improve accuracy-latency tradeoff's for end-to-end object detection (Figure 3). Notably, our use of weight-sharing NAS allows us to leverage large-scale pre-training and effectively transfer to small datasets (Table 4). Lastly, we revisit current benchmarking protocols for measuring latency and propose a simple standardized procedure to improve reproducibility.
2 RELATED WORKS
Neural Architecture Search (NAS) automatically identifies families of model architectures with different accuracy-latency tradeoffs (Zoph & Le, 2016; Zoph et al., 2018; Real et al., 2019; Cai et al., 2018a). Early NAS approaches (Zoph & Le, 2016; Real et al., 2019) focused primarily on maxi- mizing accuracy, with little consideration for efficiency. As result, discovered architectures (c.g.. NASNet and AmoebaNet) were often computationally expensive. More recent hardware-aware NAS methods (Cai al., 2018b; Tan al., 2019; Wu al., 2019) address this limitation by incorporating hardware feedback directly into the search process. However, these methods must repeat the search and training process for each new hardware platform. In contrast, OFA (Cai et al., 2019) proposes a weight-sharing NAS that decouples training and search by simultaneously optimizing thousands of sub-nets with different eccuracy-latene tradeoffs. Contemporary methods typically evaluate NAS for object detection by simply replacing standard backbones with NAS backbones in existing detec- tion frameworks. Unlike prior work, we directly optimize end-to-end object detection accuracy to find Pareto optimal aceuracy-latency tradeoffs for any target dataset.
Real-Time Object Detectors are of significant interest for tions. Historically, two-stage detectors like Mask-RCNN (He et al., 2017) and Hybrid Task Cascade (Chen et al., 2019) achieved state-of-the-an performance at the cost of latency, while single-stage detectors like YOLO (Redmon et al., 2016) and SSD (Liu al., 2016) traded accuracy for state-
and interactive applica-
2

---

Published as a conference paper at ICLR 2026
Objject Detection
Instance Segmentation
A103
1000/8
5-008
30
014001 (2
12.5
15.4
20.0
Latency (75)
Labency Ond)
COCO Object Detection
RF100-VL Object Detection
2
30
cam
VERCUM
01
Labency
Figure 1: Accuracy-Lateney Pareto Curve. We plot the Pareto accuracy-latency frontier for real- time detectors the COCO detection val-set (top left, bottom left), COCO segmentation val-set (top right), and RF100-VL test-set (bottom right). Since RF100-VL contains 100 distinct datasets, select target latencies for the N, S, M, L, XL, 2XL configurations, search for RF-DETR models with latencies within 10% the target and report their average performance after fine-tuning to convergence. Importantly, all points along RF-DETR's continuous Pareto curves for COCO derived from single training run.
of-the-art runtime. However, modern detectors (Zhao et al., 2024) reexamine this accuracy- latency tradeoff, simultaneously improving on both axes. Recent YOLO variants innovate on architecture, data augmentation, and training techniques (Redmon et al., 2016; Wang et al., 2023; 2024; Jocher al., 2023; 2024) improve performance while maintaining fast inference. Despite their effi ciency, most YOLO models rely on non-maximum suppression (NMS), which introduces additional latency. In contrast, DETR (Carion et al., 2020) removes hand-crafted components like NMS and anchor boxes. However, early DETR variants (Zhu et al., 2020; Zhang et al., 2022a; Meng al., 2021; Liu et al., 2022) achieved strong accuracy at the cost of rumtime, limiting their use in real- time applications. Recent works such as RT-DETR (Zhao et al., 2024) and LW-DETR (Chen al., 2024a) have successfully adapted high performance DETRs for real-time applications. Building on LW-DETR, RF-DETR the first real-time detector achieve more than 60 AP COCO.
Vision-Language Models are trained on large-scale, weakly supervised image-text pairs from the web. Such internet-scale pre-training is a key enabler for open-vocabulary object detection (Liu et al., 2023; Cheng et al., 2024). GLIP (Li et al., 2022) frames detection as phrase grounding with single text query, while Detic (Zhou al., 2022) boosts long-tail detection using ImageNet-level supervision (Russakovsky et al., 2015). MQ-Det (Xu et al., 2024) extends GLIP with a learnable module that enables multi-modal prompting. Recent VLMs demonstrate strong zero-shot perfor- mance and are often applied as black-box models in diverse downstream tasks (Ma et al., 2023; Peri et al., 2023; Khurana al., 2024; Osep et al., 2024; Takmaz et al., 2025). However, Robicheaux et al. (2025) find that such models perform poorly when evaluated categories not typically found in their pre-training, requiring further fine-tun tuning. In addition, many vision-language models are prohibitively slow, making them difficult to for real-time tasks. In contrast, RF-DETR combines the fast inference of real-time detectors with the snternet-scale priors of VLMs to achieve state-of- the-art performance RF100-V and all latencies 40 COCO.
3

---

interpret these pixel embeddings segmentation Published as a conference paper at ICLR 2026
Cantical
Ixage
Group
Decoder Grnup
VYT Backbone
Sugrumration
Decoder Group
IFS
Bhokk
JB3
Blok
Preticoal
Nov Windreed Enoder Layer
Windowed Crander 2apers
Projener
Ducs Strer
Speerie
Figure 2: RF-DETR Architecture. RF-DETR uses a pre-trained VIT backbone to extract multi- scale features of the input image. We interleave windowed and non-windowed attention blocks to balance accuracy and latency. Notably, the deformable cross-attention layer and segmentation head both bilinearly interpolate the output of the projector, allowing for consistent spatial organization of features. Lastly, we apply detection and segmentation losses at all decoder layers to facilitate decoder drop out at inference.
RF-DETR: WEIGHT-SHARING NAS WITH FOUNDATION MODELS
In this section, we describe the architecture our base model (Figure 2) and present the "tunable knobs" of our weight-sharing NAS (Figure 3). Further, we highlight the limitations of hand-designed leamiin-rrte and augmentation schedulers, and advocate for a scheduler- free approach.
Incorporating Internet-Scale Priors. RF-DETR modernizes LW-DETR (Chen et al., 2024a) by simplifying its architecture and training procedure to improve generalization to diverse target do- mains. First, replace LW-DETR's CAEv2 (Zhang al., 2022b) backbone with DINOv2 (Oquab et al,, 2023). We find that initializing our backbone with DINOv2's pre-trained weights significantly improves detection accuracy on small datasets. Notably, CAEv2's encoder has 10 layers with patch size of 16, while DINOv2's encoder has 12 layers. Our DINOv2 backbone has more layers and is slower than CAEv2, but we make up for this latency using NAS (discussed next), Lastly, facilitate training on consumer- -rade GPUs via gradient accumulation by using layer norm instead of batch norm the multi-scale projector.
Real-Time Instance Segmentation. Inspired by Li et al. (2023), we add a lightweight instance segmentation head to jointly predict high quality segmentation masks. Our segmentation head bi- linearly interpolates the output of the encoder and learns a lightweight projector to generate pixel embedding map. Specifically, we upsample the same low-resolution feature map for the detection and segmentation heads to ensure that it contains relevant spatial information. Unlike MaskDINO (Li et al., 2023), we do not incorporate multi-scale backbone features in our segmentation head to minimize lateney. Lastly, the output of each decoder layer transformed by a FFN) with the pixel embedding map to gener- ate segmentation masks. Interestingly,
compute the dot product of all projected query token embeddings (at
4

---

Published as a conference paper at ICLR 2026
8x8
32
Smaller Putchres
Larger Patchbes Tewer Thlcens
Fewer Layers
More Layers
More Toleens
Mare Accurate
a) Patch Embedding Interpolation
b) Decoder Layers
Pewer Queries
Fanter
Hiigher
More Windows
More Accurate
Paster
More Accurate
More Accurate
Faster
c) Query Dropping
d) Resolution Interpolation
c) Number of Windows
Figure 3: NAS Search Space. We vary (a) patch size, (b) number of decoder layers, (c) number of queries, (d) image resolution, and (e) number of windows per attention block when evaluating dif- ferent operating points along RF-DETR'S ' Pareto curve. addition to training thousands of network configurations in parallel, find that this "architecture augmentation" serves regularizer and improves generalization.
prototypes (Bolya et al., 2019). Motivated by LW-DETR's observation that pre- pre-training improves DETRs, we pre-train RF-DETR-Sep on Objects-36$ (Shao et al., 2019) psuedo-labeled labeled with SAM2 (Ravi al., 2024) instance masks.
End-to-End Neural Architecture Search. Our weight-sharing NAS evaluates thousands of model configurations with different input image resolutions, patch sizes, window attention blocks, decoder layers, and query tokens. At every training iteration, we uniformly sample random model con- figuration and perform gradient update (Appendix ??). This allows our model efficiently train thousands of sub-nets in parallel, similar to ensemble learning with dropout (Srivastava al., 2014). We find that this weight-sharing NAS approach also serves as a regularizer during training, effec- tively performing "architecture augmentation" To the best of our knowledge, RF-DETR is the first end-to-end weight-sharing NAS applied to object detection and segmentation. We describe each component below.
Patch Size. Smaller patches lead to higher accuracy at greater computational cost. We adopt FlexiVIT-style (Beyer et al., 2023) transformation to interpolate between patch sizes during training.
Number of Decoder Layers. Similar to recent DETRS (Peng et al., 2024; Zhao et al., 2024), we apply regression loss the output of all decoder layers during training. Therefore, we can drop any (or all) decoder blocks during inference. Interestingly, removing the entire decoder during inference effectively turns RF-DETR into single-stage detector. Notably, truncating the decoder also shrinks the size of the segmentation branch, allowing for greater control over segmentation latency.
Number of Query Tokens. Query tokens learn spatial priors for bounding box regression and segmentation. We drop query tokens (ordered by the maximum sigmoid of the corre- sponding class logit per token at the output of the encoder, see Appendix ??) at test time to vary the maximum number of detections and reduce inference latency. The Pareto optimal number of query tokens implicitly encodes dataset statistics about the average number of objects per image in target dataset.
5

---

Published as a conference paper at ICLR 2026
Image Resolution. Higher resolution improves small object detection performance. while lower resolution improves runtime. We pre-allocate N positional embeddings correspond- ing to the largest image resolution divided by the smallest patch size and interpolate these embeddings for smaller resolutions or larger patch sizes.
Number of Windows per Windowed Attention Block. Window attention restricts self- attention to only process fixed number of neighboring tokens. We can add or remove windows per block to balance accuracy, global information mixing, and computational ef- ficiency.
At inference time, we pick specific model configuration to select an operating point on the accuracy-latency Pareto curve. Importantly, different model configurations may have similar pa- rameter counts but significantly different latencies. Similar to Cai ct al. (2019), we see little benefit from fine-tuning the NAS-mined models on COCO (Appendix ??), but note modest improvements from fine-tuning NAS-mined models on RF100-VI. This additional fine-tuning is optional, and is often unnecessary for practical deployment. We posit that RF-DETR benefits from additional fine- tuning on RF100-VL, VL because the "architecture augmentation" regularization requires more than 100 epochs to converge on small datasets. Notably, prior weight-sharing NAS methods (Cai et al., 2019) train in stages and use different learning- grat scheduler per-stage. However, such schedulers make strict assumptions about model convergence, which may not hold across diverse datasets.
Training Schedulers and Augmentations Bias Model Performance. State-of-the-art detectors often require careful hyper-parameter tuning to maximize performance on standard benchmarks. However, such bespoke training procedures implicitly bias the model towards certain dataset char- acteristics (e.g. number of images). Concurrent with DINOv3 (Siméoni et al., 2025), we ob- serve that cosine schedules assume known (fixed) optimization horizon, which impractical for diverse target datasets like those in RF100-VL. Data augmentations introduce similar biases by presuming prior knowledge of dataset properties. For example, prior work leverages aggres- RandomF 1 1ip, RandomResion ze, RandomCrop,
sive data augmentation (c.g., VerticalF YOLOXHSVRa ndomAug, and CachedMixup to increase effective dataset size. However, cer- tain augmentations like VerticalFlip may negatively bias model predictions in saitty-critical domains. For example, a person detector in a self-driving vehicle should not be trained with VerticalFlip 1iipp avoid false positive detections from reflections puddles. Therefore, we limit augmentations to horizontal flips and random crops. Lastly, LW-DETR applies a per-image random resize augmentation, where each image is padded match the largest image in the batch. As a result, most images have significant padding, which introduces window artifacts, and wastes com- putation on padded regions. contrast, we resize images at the batch level to minimize the number of padded pixels per-batch and to ensure that all positional encoding resolutions are equally likely to be seen at train time.
4 EXPERIMENTS
We evaluate RF-DETR COCO and RF100-VL and demonstrate that our approach achieves state- of-the-an art accuracy among all real-time methods. In addition, we identify inconsistencies in standard benchmarking protocols and present a simple standardized procedure improve reproducibility. Following LW-DETR (Chen et al., 2024a), we group models of similar latency into the same size bucket rather than grouping based on parameter count.
Datasets and Metrics. We evaluate RF-DETR on COCO for fair comparison with prior work and on RF100-VI VL evaluate generalization to real-world datasets with significantly different data distributions. Due to the diversity of RF100-VL's 100 datasets, we posit that overall performance on this benchmark is a proxy for transferability to any target domain. We use pycocotools to report standard metrics like mean average precision (mAP) and provide breakdown analysis for AP AP7s. AP Amsmall AP Medium. and Largee Further, we evaluate efficiency by measuring GFLOPs, number of parameters, and inference latency an NVIDIA T4 GPU with Tensor-RT 10.4 and CUDA 12.4.
Standardizing Latency Benchmarking. Despite its maturity, benchmarking object detectors mains inconsistent across prior work. For example, YOLO-based models often omit non- non-maximal suppression (NMS) when computing latency, leading unfair comparisons with end-to-end detec-

---

Published as a conference paper at ICLR 2026
Table 1: Standardizing Lateney Evaluation. Variance in latency measurements can be largely attributed to power throttling and GPU overheating, We mitigate this issue by buffering for 200ms between forward passes. Notably, this benchmarking approach is not designed to measure sustained throughput, but rather ensures reproducible latency measurements. We are unable to reproduce YOLÓv8 and YOLOv1 I's mAP results TensorRT, likely because these models evaluate with multi-class NMS but only use single-class NMS in inference. We use the standard NMS-tuned con- fidence threshold of 0.01. YOLOv8 and YOLOv performance degrades further when quantizied from FP32 to FP16, reaffirming that all models should report latency and accuracy using the same model artifact. Notably, naively quantizing D-FINE to FP16 reduces performance to 0.5 AP. We fix this issue by changing the authors' export code to use ONNX opset 17 (Appendix ??).
Method
Reported
Buffering (FP32)
Buffering (FP-16)
APr005
Latency (ms)
APpa.
Lalency (ms)
APaa.a
Latency (ms)
YOLOVS (M)
50.2
5.80
49.3
148
47.3
54
YOLOx (M)
51.5
4.7
49.7
18.7
48.3
5.2
RE (RTB)
49.0
4,61
49.0
12.2
45.0
4,4
LW-DETR (M)
52.5
5.6
52.6
26.8
52.6
D-FINE (M)
55,1
5.62
55.1
13.9
55.0 (0.5")
5,4
RF-DETR (M)
34.8
20.5
54.7
4,4
Table 2: COCO Detection Evaluation. We compare RE-DETR with popular real-time and open- vocabulary object detectors below. We find that RF-DETR (nano) outperforms D-FINE (nano) and LW-DETR (tiny) by more than AP. RF-DETR significantly outperforms YOLOv8 and YOLOv11, while RF-DETR's nano size achieves performance parity with YOLOv8 and YOLOv1 I''s medium
size model. We denote models that do not support TensorRT execution with a star, and instead report PyTorch latency results. See Appendix ?? for I,, XI, and Max variants of RF-DETR on COCO.
Model
Size
Params.
GFLOPS
Lalency (ms)
AP
APso
AP7s
APs
APM
AP,
Real-Time Object Detection w/ NMS
YOLOv8 (Jocher
2023)
N
32M
8.7
2.1
35.2
49.2
38.3
133
38.8
51.3
YOLOvIl (Jocher ct ol. 2024)
N
26M
6.5
2.2
37.1
51.6
40.4
17.3
40.7
55.6
VOL.0w8 (Incher et al., 2023)
8
11.2M
28.6
2.9
42.4
57.6
46.0
22.2
47.1
59.6
YOLOvI (Jochor (Jochrr ct 2024)
9.4M
21.5
44.1
59.3
47.9
26.1
48.5
62.6
YOLOv& (Jocher ct al., 2023)
25.9M
78.9
47.3
62.5
31.5
27.5
52.9
65.1
YOLOv11 (Jocher et 2024)
M
20.1M
68.0
5.1
48.3
63.6
52.5
29.1
53.8
66.3
Open- Vocabularyb Object Detection Saplly-Speerssed Fine- Tuning)
Grousding (Liu ot 2023)
T
173.0M
1008.3
58.2
End-to-Ead Real- Object Detection
DW (Chen et 20244)
121M
21.4
1.9
41.9
60.7
45.9
47.3
60.0
(Peng at,l, 2024)
38M
7.3
2.1
42.7
60.2
45.4
22.9
46.6
62.1
RE-DETR (Ours)
30.5M
31.9
48.0
67.0
31.4
25.2
53.5
70.0
LWDEET (Chen et al., 2024.a
1466
3188
2.6
48.0
66.8
31.6
26.7
52.5
65.6
(Peng ctal. 2024)
10.2M
25.2
30.6
67.6
55.0
32.6
54.6
66.6
RE-DETR (Ours)
S
32
39.8
3.5
52.9
71.9
37.0
32.0
58.3
73.0
RT-DETR (7hao et ol., 2024)
R18
36.0M
100.0
4.4
49.0
66.6
53.3
32.8
52.1
65.0
LW-DETR (Chen et 2024g)
28
8399
52.6
72.0
36.6
32.5
37.6
70.5
D-FINE (Peng 2024)
M
19.2M
56.6
5.4
55.0
72.6
59.7
37.6
59.4
71.7
RE-DET (Ours)
33.7M
78.8
34.7
73.5
59.2
36.1
59.7
73.8
RF-DETR (Ours)
2XL
126.9M
438.4
172
60.1
78.5
05.3
43.2
64.9
76.2
tors. Additionally, YOLO-based segmentation models measure the latency of generating prototype predictions instead of directly usable per-object masks (Jocher et al., 2024), leading biased run- time measurements. Further, D-FINE's reported latency evaluation of LW-DETR is 25% faster than reported by Chen al. (2024b). We observe that such differences can be attributed to detectable power throttling events, particularly when the GPU overheats (Table 1). In contrast, simply pausing for 200ms between consecutive forward passes largely mitigates power throttling, yielding more sta- ble latency measurements (Appendix ??). Lastly, we find that prior work often reports latency using FP16 quantized models, but evaluates accuracy with FP32 models. However, naive quantization can significantly degrade performance (in some cases dropping performance to near 0 AP). To ensure fair comparison, we advocate for reporting accuracy and latency with the same model artifact. We release stand-alone benchmarking tool GitHub.
Evaluating RF-DETR and RF-DETR-Seg on COCO. COCO (Lin et al., 2014) is a flagship benchmark for object detection and instance segmentation. In Table 2, we compare RF-DETR with leading real-time and vocabuacyy detectors. RF-DETR (nano) beats both D-FINE (nano) and LW-DETR (nano) by more than 5 AP. We see similar trends for small and medium sizes as well. No-
7

---

Published as a conference paper at ICLR 2026
Table 3: COCO Instance Segmentation Evaluation. We compare RF-DETR with popular real- time instance segmentation methods on COCO. Notably, RF-DETR (nano) outperforms all re- ported YOLOv8 and YOLŌv11 model sizes. Further RF-DETR (nano) outperforms FastInst by 5.4%, while running nearly ten times faster. RF-DETR (medium) approaches the performance on MaskDINO at fraction of the runtime. We denote models that do not support TensorRT execution with a star, and instead report PyTorch latency results. Our latencies for YOLOs also include the conversion protos into masks, which are not typically included in prior benchmarks but nonethe-
less contribute meaningfully to practical latency. See Appendix ?? for L, XL, and Max variants of RF-DETR-See on COCO.
Model
Size
# Params.
GFLOPS
Latency (ms)
AP
A050
AP%
AP2
APM
AP2
Real-Time Instance Segmentation w/ NMS
YOLOvE et al. 2023)
3.4M
12.6
3.5
28.3
43.6
29.8
93
31.3
44.3
YOLOvI et al., 2024)
29M
30.0
31.
10.0
47.7
YOLOva (Jocker et al., 2023)
1L.M
42.6
4.2
34.0
53.8
36.0
13.6
38.5
52.2
YOLOv1l (Jocher
2024)
10.1M
35.5
4.6
35.0
55.4
37.1
15.3
39.7
53.9
YOLOx3 (Jocher
2023)
M
27 3M
110.2
7.0
37.3
58.
39.9
16.7
43.0
56.1
YOLOv1l (Jocher et al., 2024)
M
22.4M
123.3
6.9
38.5
60.0
40.9
18.0
44.3
57.6
End-to-End Insfance Segmentation
RF-DETR SSg. (Our))
N
33.6M
50.0
3.4
40.3
63.0
42.6
16.3
45.3
63.6
RF-DETR R---TR-See Seg. (Ours)
53.7M
70.6
4,4
43.1
66.2
45.9
21.9
48.5
64.1
Fasilnst (Hl el., 2023)
R50
29. 79
99.7
39.6'
34.9
56.0
36.2
13.3
38.0
56.8
MaskDINO (La et 2023)
R50
IM
586
242
46.3
69.0
50.7
26.1
49.3
66.
RF-DETR-SS2 (Ours)
M
35 7M
102.0
45.3
68.4
48.8
25.3
50.4
63.3
RE-DETR (Ours)
2XL
3866
435.3
21.8
49.9
65.7
tably, RF-DETR also significantly outperforms YOLOv8 and YOLOv11. RF-DETR (nano) matches the performance of YOLOv8 and YOLOv11 (medium). We use mmdetection's ' implementation of GNO and include their reported AP since they do not release model artifact for Ground- ingDINO fine-tuned on COCO. We benchmark mmGGroundinDDINOS parameter count, GFLOPS, and latency using the released open-vocabulary model. In Table 3, we compare RF-DETR-See with real-time instance segmentation models. RF-DETR-Seg (nano) outperforms YOLOv8 and YOLOv11 at all sizes. Furthermore, RF-DETR-SEE (nano) beats FastInst by 5.4% while running almost ten times faster. Similarly, RF-DETR (x-large) surpasses GroundingDINO (tiny), and RF- DETR-Seg (large) outperforms MaskDINO (R50), fraction of their runtime.
Evaluating RF-DETR on RF100-VI. RF100-VI. VI. is challenging detection benchmark composed of 100 diverse datasets. We report latencies, FI.OPs, and accuracy averaged over all 100 datasets in Table 4. Our results show that RF-DETR (2x-large) outperforms GroundingDIIO and L.MDDet while requiring only a fraction of their runtime. Interestingly, RT-DETR outperforms D-FINE (which is built RT-DETR) at AP indicating that D-FINE's hyperparameters potentially overfit to COCO. We also note that RF-DETR benefits from scaling to larger backbone sizes (Ap- pendix ??). In contrast, YOLOv8 and YOLOv11 consistently underperform DETR-based detectors, and scaling these model families to larger sizes does not improve their performance on RF100-VL. (Figure 1).
Impact of Neural Architecture Search. We ablate the impact of weight-sharing NAS in Table 3. We find that adopting gentler set of hyperparameters compared to LW-DETR (e.g. larger batch size, lower learning rate, and replacing batch normalization with layer normalization) reduces per- formance over LW-DETR by 1.0%. Notably, replacing batch normalization with layer normalization hurts performance, but is necessary to train on consumer hardware. However, replacing LW-DETR's 's CAEv2 backbone with DINOv2 improves performance by 2%. The lower learning rate, in partic- ular, helps preserve DINOv2's pre-trained knowledge, while additional epochs of Objects-365 pre- training further compensate for the slower optimization. Our final model with weight-sharing NAS improves over LW-DETR by 2% without increasing latency.
Impact of Backbone Architecture and Pre- Training. We study the impact of different backbone architectures in RF-DETR. We find that DINOv2 achieves the best performance, outperforming CAEv2 by 2%. Interestingly, despite having fewer parameters than SigLIPy2, SAM2's Hiera-S backbone is considerably slower. This is in contrast with Hiera's 's claim that it is meaningfully faster than equivalently performant ViTs. However, Hiera does not explore latency in the context of Flash Attention kernels, which are highly optimized in compilers such TensorRT. Additionally, existing
8

---

Published as a conference paper at ICLR 2026
Table RF100-VL Evaluation. We compare RF-DETR with real-time and open- vocabuocyyl object detectors on RF100-VL. Interestingly, RF-DETR (2x-large) outperforms GroundingDINO (tiny), and LLMDet (tiny) at a fraction of their runtime. We report the average latency and FLOOPs over all 100 datasets. We note that YOLOv8 and YOLOv11 s latency measurements may be suboptimal because the default tuned NMS threshold of 0.01 may not work well for all datasets in RF100-V VL.
We denote models that do not support TensorRT execution with star, and instead report PyTorch latency results. See Appendix ?? for L, XL, and Max variants of RE-DETR on RFI00-VL.
Model
Size
Params.
GELOPS
Lalency (ms)
AP AP
APTs
AP;
APM
AP,
Real-Time Object Detectors w/ NMS
YOLOv8 (Jocher ot al,, 2023)
N
3.2M
8.7
26
35.0
81.1
59.3
488
44.1
48.0
YOLOv11 (Jocher
2024)
6.5
60.3
49.2
VOLOv& (Incher ai,, 2023)
28.6
56.3
00.9
48.6
YOLOvII (Jocher 2024)
S
9.4M
21.5
3,3
56.4
82.5
61.3
6.5
45.5
48.5
YOLOv8 (locher
2023)
M
25.9M
78.9
5,4
56.5
82.3
60.9
45.7
48.6
YOLOvl (locher et al., 2024)
20.1M
68.0
5.1
37.0
82.5
61.9
7.3
46.1
48.6
Open- Voca-Vocabulrry Object-Detectors
(Fule-Superrised Fine- Tuminn)
GroundingDINO (Liu et al., 2023)
T
173.0M
1008.3
309.9"
62.3
88.8
67.8
39.2
57.7
69.5
LL cl. 20215)
1008.3
62.3
70.3
End-to-Fand Real- Time Object Detectors
LW-DETR (Chon ot 20244)
12.1M
21.4
1.9
57.1
84.7
61.5
31.2
51.8
65.8
D-FINE (Pong st. ail.l.2000 2024)
3.8M
7.3
2.0
58.2
84.4
62.5
32.4
52.9
65.8
RE-DER (Ours)
3M
34.5
37.8
85.1
62.5
30.1
52.2
67.2
RE-DETR Fine- Tuning (Ours)
N
31
34.5
2.5
58.6
85.7
63.0
31.0
53.2
67.6
LW-DETR (Chen et 2024a)
14.6M
31.8
57.4
62.0
65.8
D'FINE et al, 2024)
1022
25.2
3.5
60.3
85.3
65.4
36.6
56.0
68.4
RE-DETK (Ours) RE-DETR w/ Fine- Tuning (Ours)
60.9
66.
34.2
55.7
69.6
33.5M
62.4
3.7
61.2
87.7
06.1
34.9
55.6
09.5
REDETR (Zhao
2024)
M
36.0
100.0
4.3
59.6
85.7
64.6
36.4
54.6
67.3
LW-DETR (Chen et 2024a)
M
2K
81.
39.8
86.8
64.9
34.0
54.4
68.9
D-FINE (Pong ctal, 2ll,,2000 2024)
19.2M
36.6
60.6
85.5
65.8
36.0
56.6
67.5
RE-DETR (Ours)
M
33.5M
86.7
4.6
61.7
88.0
66.9
35.8
56.5
70.0
RETDER Pefiie/Tuunngg (Ours)
86.7
62.0
88.1
56.4
70.2
RE-DETK (Oui)
2XT.
1235M
410.2
15.6
63.3
88.9
69.0
38.7
58.2
71.6
RE-DETR (Ours) w/ Fine- Tuning
2XL
123.5M
410.2
15.6
63.5
89.0
69.2
38.9
58.3
71.7
Table 5: Ablation on Neural Architecture Search. We ablate the impact of each "tunable knob" on accuracy and latency below. Using gentler set of hyperparameters compared to LW-DETR (e.g. smaller batch size. lower learning rate, replacing batch norm with layer norm) reduces performance by 1%. However, we regain this lost performance by replacing LW-DETR's s CAEV2 backbone with DINOv2. Importantly, the lower learning rate and layer-norm allow to better preserve DINOv2's foundational knowledge and allows us to train with larger batch sizes, making weight-sharing NAS more effective. Counterintuitively, introducing weight sharing NAS to the training scheme improves
the performance of the base configuration even though patch size 14 isn't in the NAS search space.
Model
Params.
GFLOPS
Latency (ms)
AP
AP
AP
AP:
APM
AP;
LWWDETT (M)
28
8377
44
52.6
72.0
56.6
32.5
57.6
70.5
.Gentler Hyperparanectrs
28
83.7
4.4
31.6
71.1
55.3
31.7
56.4
69.4
-DINOvZ Backbone
53.6
72.7
34.3
58.3
Additiona 0365 Pie Training
32.3
7822
4.7
54.3
73.4
58.8
35.8
59.2
72.3
Weipht Sharing
78.2
4.7
54.6
73.4
36.3
59.3
72.1
+ Pauch Siee 14 16, Res 560 610
32.3M
78.5
4.7
54,4
73.2
59.1
35.9
59.2
72.1
Imagee Resolution 640 576
32.2M
64.2
4.0
53.6
72.4
58.2
34.8
58.6
72.0
* Windows per Block
32.2M
63.7
4.3
54.3
73.3
58.8
35.6
39.4
73.2
* Decoder Layers 4
33.7M
64.8
4.4
54.6
73.5
59.1
36.0
59.8
73.7
Query Tokens 300
34.6
73.5
36.0
59.8
foundation model families typically do not release lightweight ViT variants such as VIT-S or ViT-T, making it difficult to repurpose such models for real-time applications.
Following prior work, we report all COCO results on the validation set. However, relying solely on the validation for both model se- lection and evaluation can lead to overfitting. For example, D-FINE (which builds RT-DETR) conducts an extensive hyperparameter sweep on COCO's validation set and reports its best model. However, evaluating this configuration on the test set. In contrast, our method achieves state-of -o-the--rttttt performance among all real-time de- tectors both RF100-VL and COCO, demonstrating the robustness of our weight-sharing NAS. In
Rethinking Standard Accuracy Benchmarking Practices.
RF100-VI LLL shows that D-FINE underperforms RT-DETT

---

Published as a conference paper at ICLR 2026
Table Ablation on Backbone. We ablate the impact of using different backbone architectures for RF-DETR below. We find that DINOv2 achieves the highest performance, outperforming CAEv2 by 2.4%. All models are pretrained with 60 epochs of Objects-365 and the "Gentler Hyperparameters" setting. Note that SAM2 and SigLIPv2 perform poorly when evaluated in FP16. Therefore, we
report FP16 TensorRT latency with FP32 ONNX accuracy for these two models as an upper bound
on their performance if optimized for FP16.
LW-DETR (M) Gentler Hyperparameters # Params.
GFLOPS
Latency (ms)
AP
APsa
AP
APs
APır
AP,
w/ CAEy2 V117--16-Iruncatee Backbone
28.3M
83.7
4.4
52.3
71.4
56.3
32.3
36.4
70.0
w/ DINOV2 VTT7S. 14 Backhone
32 3M
78.2
4.7
54.3
73.4
58.8
35.8
59.2
12.3
w/ SigLIPN2 VI17B-32 Backbone"
105.IM
8166
4.8
50.4
70.4
53.7
28.0
55.3
73.0
addition to evaluating on COCO, we advocate that future detectors should also evaluate on datasets with public validation and test splits like RF100-VL.
Limitations. Despite controlling for power throttling and GPU overheating during inference, our latency measurements still have variance of up 0.1ms due to the non-deterministic behavior of TensorRT during compilation. Specifically, TensorRT can introduce power throttling, which in turn affects the resulting engine and leads random fluctuations in latency. Although the measurement of a given TensorRT engine is generally consistent, recompiling the same ONNX artifact can pro- duce slightly different latency results. Therefore, we only report latencies with one digit of precision after the decimal place.
5 CONCLUSION
In this paper, we introduce RF-DETR, state-of-the-art NAS-based method for fine-tuning special- ist end-to-end object detectors for diverse target datasets and hardware platforms. Our approach outperforms prior state-of-the-an real-time methods on COCO and 2F100-VI, improving upon D- FINE (nano) by 5% AP on COCO. Moreover, we highlight that current architectures, learning rate schedulers and augmentation schedulers tailored to maximize performance COCO, suggest- ing that the community should benchmark models on diverse, large-scale datasets to prevent implicit overfitting. Lastly, highlight the high variance in latency benchmarking due to power throttling and propose a standardized protocol to improve reproducibility.
ACKNOWLEDGEMENTS
This work was supported in part by compute provided by NVIDIA DGX. We'd like to thank Brad Dwyer for reviewing early versions of our manuscript.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Korblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flex- ivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 14496-14506, 2023.
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9157-9166, 2019.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018a.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. arXiv preprint arXiv:1812.00332. 2018b.
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.0979/ 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Kai Chen, Jiangmiao Pang. Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Zi- wei Liu, Jianping Shi, Wanli Ouyang, al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4974-
4983, 2019.
Qiang Chen, Xiangbo Su, Xinyu Zhang, Jian Wang, Jiahui Chen, Yunpeng Shen, Chuchu Han, Ziliang Chen, Weixiang Xu, Fanrong Li, et al. Lw-detr: transformer replacement to yolo for real-time detection. arXiv preprint arXiv:2406.0344 034559 2024a.
Qiang Chen, Xiangbo Su. Xinyu Zhang, Jian Wang, Jiahui Chen, Yunpeng Shen, Chuchu Han, Ziliang Chen, Weixiang Xu, Fanrong Li, et al. Lw-detr: a transformer replacement to yolo for real-time detection. arXiv preprint arXiv:2406 2.Xi:22406.03459 2024b.
Tianheng Cheng, Lin Song, Yixiao Gc, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo world: Real-time open-vocabulary object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024.
M. Everingham, M. A. Eslami, Van Gool, C. K. Williams, Winn, and Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111, 2015.
Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained bast- models. IEEE transactions on pattern analysis and ma- chine intelligence, 32(9): 1227-162771655, 2009.
Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, and Wei-Shi Zheng. LImdet: Learning strong open-vocabulary object detectors under the supervision of large language models. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14987-14997, 2025.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid archi- tecture for object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7036-7045, 2019.
Junjie He, Pengyu Li, Yifeng Geng, and Xuansong Xie. Fastinst: A simple query- based model for real-time instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 33663-23677 2023.

---

Published as a conference paper at ICLR 2026
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Glenn Jocher, Jing Qiu, and Ayush Chaurasia, Ultralytics YOLO, January 2023, URL https:
//docs. //docs.uuttaalye.. ultralytics .ttriiccccoo//oooo cowmmooeel////occcccocccccccll soroodessyyllovss
Glenn Jocher, Jing Qiu, and Ayush Chaurasia. Ultralyties YOLOO, January 2024. URL doas.
ultraly Aitralytics ytccs ccom/mode /yyooll.
Mehar Khurana, Neehar Peri, Deva Ramanan, and James Hays, Shelf-supervieed multi-modal pre- training for 3d object detection. arXiv preprint 10X11:2406.10115, 2024.
Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang. Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards unified transformer-based framework for object detection and segmenta- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3041-3054 2023.
L Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10965-10975, 2022.
Tsung-Yr Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang- Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12325) 2022.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 2023.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, 2016.
Yechi Ma, Neehar Peri, Shuoquan Wei, Wei Hua, Deva Ramanan, Yanan Li, and Shu Kong. Long- tailed 3d detection via 2d late fusion. arXiv preprint arXiv:2312. 10986, 2023.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3651-3660, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077/99 2023.
Aljosa Osep, Tim Meinhardt, Francesco Ferroni, Nechar Peri, Deva Ramanan, and Laura Leal-Taixe. Better call sal: Towards learning segment anything in lidar. 2024.
Yansong Peng, Hebei Li, Peixi Wu, Yueyi Zhang, Xiaoyan Sun, and Feng Wu. D-fine: Redefine regression task in detrs as fine-grained distribution refinement. arXiv preprint 2xXx:24400.1842. 2024.
Neehar Peri, Achal Dave, Deva Ramanan, and Shu Kong. Towards long-tailed 3d detection. 2023.
Nikhila Ravi Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao-Yusa Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00711 2024. URL https: / ///axiivvvo .owg///./................. abs / /2408 00714.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780 2019.
12

---

Published as a conference paper at ICLR 2026
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, 2015.
Peter Robicheaux, Matvei Popov, Anish Madan, Isaac Robinson, Joseph Nelson, Deva Ramanan, and Neehar Peri. Roboflow 100-vI: A multi-domain object detection benchmark for vision- language models. arXiv preprint arXiv:2505.2060 2025.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11::211-252, 2015.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. 2019 IEFF/CVF International Conference on Computer Vision (ICCV), pp. 8429-8438, 2019, doi: 10.1109/ 2019.00852.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, al. Dinov3. arXiv preprint arXiv:2508. 1010::5500..0004,, 2025.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1958,1958, 2014.
Ayca Takmaz, Cristiano Saltori, Neehar Peri, Tim Meinhardt, Riccardo de Lutio, Laura Leal-Taixe, and Aljosa Osep. Towards Learning to Complete Anything Lidar. In International Conference on Machine Learning (ICML), 2025.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114. PMLR, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2820-2828, 2019.
Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference computer vision and pullern recognition, pp- 10781-10790, 2020.
Chien Yao Wang, Alexey Bochkovskiy, and Hong- Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7464-7475, 2023.
Chien- Yao Wang, I-Hau Yeh, and Hong- Yuan Mark Liao. Yolov9: Learning what you want to learn using programmable gradient information. In European conference on computer vision, pp. 1-21. Springer, 2024.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10734-10742 2019.
Yifan Xu. Mengdan Zhang, Chaoyou Fu, Peixian Chen, Xiaoshan Yang. Ke Li, and Changsheng Xu. Multi-modal queried object detection in the wild. Advances Neural Information Processing Systems, 36, 2024.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.030m 2022a.
13

---

Published as a conference paper at ICLR 2026
Xinyu Zhang, Jiahui Chen, Junkun Yuan, Qiang Chen, Jian Wang. Xiaodi Wang, Shumin Han, Xiaokang Chen, Jimin Pi, Kun Yao, et al. Cae v2: Context autoencoder with clip target. arXiv preprint arXhv:2211.09799 2022b.
Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos on real-time object detection. In Proceedings of the TEEE/CVF conference on computer vision and pattern recognition, pp. (6965-16974 2024.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In European Conference Computer Vision, pp. 350-368. Springer, 2022.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611.0157,, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
14


Reviews:
[{"review_id": "nZsjNASZTz", "reviewer": "Reviewer_fnAm", "summary": "The paper proposes a weight-sharing, once-for-all style search over DETR variants (“RF-DETR”) to obtain real-time operating points along an accuracy–latency Pareto curve. A single base model is trained while exposing a compact set of architectural knobs (input resolution, patch size, decoder depth, number of queries, windowed vs. global attention). After training, sub-networks are selected by validation without per-subnet fine-tuning. Experiments on COCO and a broader multi-dataset suite show consistent gains over strong real-time baselines, and an instance-segmentation extension indicates broader utility.", "questions": "1. How are sub-networks sampled during training (uniform over knobs, FLOPs-aware, or constrained)? Any coupling constraints to avoid pathological combinations?\n2. Are layers/queries randomly dropped during training to mimic inference-time truncation? Is there loss re-weighting across depths? How exactly are queries ranked at test time?\n3. How many candidates are evaluated during selection, what is the wall-clock/energy budget, and is the same operating point reused across datasets/hardware or re-selected each time?\n4. Could you add controlled re-runs (or a table) where backbones and pre-training are aligned across methods to isolate the effect of your approach?\n5. Could you report results where a single subnet chosen on COCO is evaluated unchanged on other datasets to assess transfer.\n6. Could you provide per-knob sensitivity plots and report variance across random seeds and multiple TensorRT engine builds?\n7. Will you release export scripts, calibration settings, the buffering/throttling harness, and the exact list of selected sub-networks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. A practical blend of weight sharing, once for all selection, and DETR knob tuning that feels deployable, with latency protocols that practitioners can actually follow.\n2. Clear empirical gains against strong real time baselines on COCO and on a broader evaluation suite, and the instance segmentation extension indicates the idea transfers beyond detection.\n3. A search space whose knobs are easy to grasp, with figures and ablations that make the design choices legible.\n4. A simple path to high quality low latency detectors without fine tuning of each subnet, plus useful guidance on latency measurement such as throttling mitigation and artifact consistency.", "weaknesses": "1. The way subnets are sampled during training and the policy or grid used for post training selection are not specified clearly, which hurts reproducibility and interpretation.\n2. The paper does not make clear when or how decoder layers and queries are dropped during training, whether losses are reweighted across depths, or how queries are ranked at test time.\n3. The contribution reads as incremental relative to once for all and weight sharing NAS in vision, and a tighter comparison to prior NAS for detection and backbones is needed to clarify what is new beyond engineering.\n4. Some comparisons mix backbones and pretraining regimes, so stronger parity baselines or controlled reruns would better isolate the benefit of the proposed recipe.\n5. It is unclear how a fixed mined subnet transfers to unseen datasets or domains without reselection, and a cross dataset test would strengthen the robustness claim.\n6. Per knob sensitivity and stability evidence are limited, for example queries versus AP at fixed FLOPs and the interaction of resolution and patch size, and reporting variance with error bars or minimum median maximum would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BMmwUevFdI", "reviewer": "Reviewer_c7Lr", "summary": "The paper proposes a weight-sharing neural architecture search space for finding realtime DETR-style object detectors that are pareto-optimal w.r.t. accuracy and runtime. The resulting models achieve state-of-the-art real-time results on the COCO and Roboflow100-VL object detection benchmarks.", "questions": "* Instance segmentation:  \n  * How is the proposed approach different from Mask DINO?  \n  * “Our segmentation head bilinearly interpolates the output of the FPN and learns a lightweight projector to generate a pixel embedding map” \\- Where is the FPN coming from? I thought this was using a DINO backbone?  \n* There are two inconsistent definitions of latencies at which RF-DETR outperforms prior work: “for all latencies” and “all latencies ≤ 40 ms”. Which of these is correct?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and approachable. The search space is elegantly designed and evidently yields state-of-the-art results. The paper makes a set of great points about suboptimal benchmarking practices in prior work and makes an effort to perform fair comparisons.", "weaknesses": "While the paper does provide significant contributions, it could support more ablations and experiments. For example, I would have loved to see a thorough discussion of how much each of the “tunable knobs” contributes to favorable accuracy-runtime tradeoffs, and how much pareto-optimal “knob-settings” vary between datasets. Furthermore, it would be interesting to see whether specific dataset characteristics, like the prevalence of small objects, have an impact on knobs like patch size. I furthermore find it highly interesting that DINOv2 performs much better on “small datasets”, but there are no systematic comparisons between other backbones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hZVcddLfti", "reviewer": "Reviewer_FU3r", "summary": "The paper introduces RF-DETR, a fast, closed-vocabulary DETR that uses weight-sharing NAS to pick the best accuracy–latency setup (tuning resolution, patch size, decoder depth, queries, and windowing) after a single fine-tune—no extra retraining needed. It swaps in a DINOv2 ViT backbone, adds a lightweight mask head (RF-DETR-Seg), and standardizes latency benchmarking (buffering, same FP model, and counting NMS/mask conversion). Results on COCO and Roboflow100-VL show a new real-time Pareto frontier—e.g., the nano model beats D-FINE (nano) by about 5 AP and the medium model is near GroundingDINO (tiny) while roughly 60× faster—suggesting many recent detectors are implicitly over-optimized for COCO.", "questions": "1) Size taxonomy and fairness: Can you clarify the naming (e.g., “nano”) and provide comparisons under fixed latency/parameter/FLOP budgets or a plot at fixed latency caps? \n2) Latency and throughput: Can you report sustained throughput (QPS) under continuous load without the 200 ms buffer, include full pipeline timing (pre/post-processing, NMS/mask conversion), and show sensitivity to buffer length and FlashAttention usage? \n3) YOLO baselines on RF100-VL: Did you use multi-class NMS and dataset-specific threshold tuning consistent with original inference? Can you provide threshold/post-processing sensitivity analysis to verify robustness? \n4) Mixed-precision/export parity: Are all methods evaluated with identical artifacts (same precision, same export path)? Can you disclose the modified export scripts (e.g., ONNX opset 17) and quantify FP16 accuracy changes? \n5) NAS cost and coverage: What are the GPU-hours for training/search, how much of the search space is sampled, and how do “unseen” subnets perform statistically? \n6) Method details: How is “encoder confidence” defined for query dropping, and what criteria or default policy govern decoder truncation at inference?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Originality here is pragmatic: the paper smartly fuses weight-sharing NAS with DETR and a foundation-model backbone, plus a no-nonsense latency protocol—less theory, more removal of real deployment roadblocks. Quality is strong for an engineering paper: careful ablations, honest discussion of TensorRT variance and FP16 pitfalls, and fair apples-to-apples latency (counting NMS/mask steps) bolster credibility. Clarity is good—the “knobs” are concrete, the scheduler-free recipe is easy to replicate, and the deployment story (post-hoc grid search, decoder/query truncation) is clean. Significance is high for practice: it moves the real-time Pareto frontier, cuts the cost of retargeting to new hardware/domains, and exposes COCO-centric benchmarking biases; the results on RF100-VL make the transfer claims believable.", "weaknesses": "1) Gains appear driven mainly by stronger pretraining (DINOv2, O365+SAM2) and a stable recipe, making NAS’s unique contribution unclear.\n2) Size taxonomy is misleading: the “nano” model (26.9M params) is far larger than baseline “nano” (3–4M), confounding size vs latency comparisons.\n3) The latency protocol (200ms buffering; TensorRT/FlashAttention) stabilizes single-shot timing but can bias rankings and does not reflect sustained throughput.\n4) YOLO baselines on RF100-VL are disadvantaged by COCO-tuned thresholds and multi-/single-class NMS mismatches, likely underestimating their performance.\n5) Mixed-precision/export inconsistencies (FP32 accuracy vs FP16 latency; modified ONNX export) undermine strict parity across methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GQ0JGeMaaP", "reviewer": "Reviewer_HtQ3", "summary": "This paper introduces RF-DETR, a lightweight specialist object detector that uses NAS to discover accuracy-latency Pareto curves for target datasets. The key contributions are: (1) a family of NAS-based detection and segmentation models that outperform prior real-time methods on COCO and Roboflow100-VL; (2) exploration of \"tunable knobs\" in weight-sharing NAS for end-to-end detection, improving transferability; and (3) a standardized latency evaluation protocol to address reproducibility issues.", "questions": "- The weight-sharing NAS involves sampling configurations during training. What is the total training time compared to a non-NAS baseline? \n- RF-DETR is positioned as a specialist detector, but how does it compare to fine-tuned VLMs in terms of accuracy-latency trade-offs?\n- The buffering method reduces throttling, but did the authors consider other techniques? Please explain why buffering was preferred over alternatives.\n- The authors may consider the weakness above and address these key concerns.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The novelty of this work is acceptable. It applys end-to-end weight-sharing NAS to DETR-based detectors, which has been underexplored. Unlike prior NAS methods focused on image classification or backbones, RF-DETR optimizes full detection pipelines, including segmentation heads.\n+ The paper includes extensive experiments on COCO and RF100-VL. Ablations validate design choices, such as backbone replacements and NAS components. Results show consistent improvements.\n+ The authors have clarified their motivations. Some figures can help understanding, and the language is precise. Some method details have been provided.\n+ This method outperforms some famous baselines like YOLO and D-FINE, and the NAS framework allows customization for diverse hardware.", "weaknesses": "- The generalization beyond COCO and RF100-VL cannot be confirmed.​​ The experiments focus on two benchmarks, but the claim of generalizability to \"any target dataset\" is not fully validated. Testing on diverse domains would demonstrate broader applicability. The paper notes that hyperparameters may overfit to COCO-like data and more cross-dataset results would alleviate this concern.\n- The paper lacks a theoretical analysis of why weight-sharing NAS generalizes well to unseen architectures. For example, it does not provide robustness analysis for the NAS mechanism.\n- The authors may further figure out the specific gaps in existing NAS methods for object detection in the introduction. While it mentions overfitting to COCO, it does not thoroughly explain why current NAS approaches fail in detection tasks or how RF-DETR uniquely addresses these issues, especially NAS is not a new concept or tool.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)