D-FINE: Redefine Regression Task of DETRs as Fine-grained Distribution Refinement
Authors: Yansong Peng
Hebei Li
Peixi Wu
Yueyi Zhang
Xiaoyan Sun
Feng Wu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Object Detection
Real-Time
Detection Transformer
Knowledge Distillation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We introduce D-FINE, a powerful real-time object detector that achieves outstanding localization precision by redefining the bounding box regression task in DETR models. D-FINE comprises two key components: Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). FDR transforms the regression process from predicting fixed coordinates to iteratively refining probability distributions, providing a fine-grained intermediate representation that significantly enhances localization accuracy. GO-LSD is a bidirectional optimization strategy that transfers localization knowledge from refined distributions to shallower layers through self-distillation, while also simplifying the residual prediction tasks for deeper layers. Additionally, D-FINE incorporates lightweight optimizations in computationally intensive modules and operations, achieving a better balance between speed and accuracy. Specifically, D-FINE-L / X achieves 54.0% / 55.8% AP on the COCO dataset at 124 / 78 FPS on an NVIDIA T4 GPU. When pretrained on Objects365, D-FINE-L / X attains 57.1% / 59.3% AP, surpassing all existing real-time detectors. Furthermore, our method significantly enhances the performance of a wide range of DETR models by up to 5.3% AP with negligible extra parameters and training costs. Our code and models: https://github.com/Peterande/D-FINE.

Published as a conference paper at ICLR 2025
D-FINE: REDEFINE REGRESSION TASK IN DETRS AS FINE-GRAINED DISTRIBUTION REFINEMENT
Yansong Peng', Hebei Lit, Peixi Wu', Yueyi Zhang'; Xiaoyan Sun Feng Wu1.2 Uniivrrsity of Science and Technology of China
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center (pengyansong, lihebei, wupeixi @mail. ustcc. edu en 2zhyuey, sunxiaoyan, fengwu)@ustc. edu.c...eccc...
COCOAP
Ph
30
Lateace
Pacare (M)
F1.05 (G)
Figure 1: Comparisons with other detectors in terms of latency (left), model size (mid), and compu- tational cost (right). We measure end-to-end latency using TensorRT FP16 on an NVIDIA GPU.
ABSTRACT
We introduce D-FINE, powerful real-time object detector that achieves out- standing localization precision by redefining the bounding box regression task DETR models. D-FINE comprises two key components: Fine-grained Distri- bution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). FDR transforms the regression process from predicting fixed coor- dinates to iteratively refining probability distributions, providing a fine-grained intermediate representation that significantly enhances localization accuracy. GO- LSD is bidirectional optimization strategy that transfers localization knowledge from refined distributions to shallower layers through self-distillation, while also simplifying the residual prediction tasks for deeper layers. Additionally, D-FINE incorporates lightweight optimizations in computationally intensive modules and operations, achieving better balance between speed and accuracy. Specifically, D-FINE-L/X achieves 54.0% 55.8% AP on the COCO dataset at 124/78 FPS NVIDIA T4 GPU. When pretrained Objects365, D-FINE-L/X attains 57.1% 59.3% AP, surpassing all existing real-time detectors. Furthermore, our method significantly enhances the performance of a wide range of DETR models by up to 5.3% AP with negligible extra parameters and training costs. Our code and models: https: /gggtttthhb....g com.Peeterande/-EIIN
INTRODUCTION
The demand for real-time object detection has been increasing across various applications (Arani
al. 2022). Among the most influential real-time detectors are the YOLO series (Redmon al.
2016a Wang et al. 2023a b) Glenn. 2023, Wang & Liao 2024 Wang et al.) 2024a Glenn. (024).
widely recognized for their efficiency and robust community ecosystem. As strong competitor, the
Detection Transformer (DETR) (Carion et al., 2020; Zhu et al. 2020; Liu et al.
2021 Li et al., [022]
Zhang et al. 2022 offers distinct advantages due to its transformer-bat architecture, which allows
*Corresoondigg authors

---

Published as a conference paper at ICLR 2025
for global context modeling and direct set prediction without reliance Non-Maximum Suppres- sion (NMS) and anchor boxes. However, they are often hindered by high latency and computational demands (Zhu et al. 2020; Liu et al. 2021: Li et al., 2022] Zhang et al. 2022). RT-DETR (Zaa et al. 2024) addresses these limitations by developing a real-time variant, offering an end-to-end alternative to YOLO detectors. Moreover, LW-DETR (Chen et al. (2024) has shown that DETR can achieve higher performance ceilings than YOLO, especially when trained on large-scale datasets like Objects365 (Shao et al. 2019).
Despite the substantial progress made in real-time object detection, there remain potential aspects for improvement detector performance. One key aspect is the formulation of bounding box re- gression. Most detectors predict bounding boxes by regressing fixed coordinates, treating edges as precise values modeled by Dirac delta distributions (Liu et al., 2016 Ren et al.) 2015 Tian et al. 2019; Lyu et al. 2022). While this approach has shown significant success in maintaining efficiency, using fixed coordinates inherently comes with limitations in modeling localization uncer- tainty. Consequently, models are constrained to use LI loss and IoU loss, which provide insufficient guidance for adjusting each edge independently (Girshick 2013). This makes the optimization pro- cess sensitive to small coordinate changes, potentially leading to slow convergence and suboptimal performance. Although methods like GFocal (Li et al. 2020; 202I) address uncertainty through probability distributions, they remain limited by anchor dependency, coarse localization, and lack of iterative refinement. Another aspect lies in maximizing the efficiency of real-time detectors, which are constrained by limited computation and parameter budgets to maintain speed. Knowledge distil- lation (KD) is a promising solution, transferring knowledge from larger teachers to smaller students to improve performance without increasing costs (Hinton et al., 2015). However, traditional KD methods like Logit Mimicking and Feature Imitation have proven inefficient for detection tasks and can even cause performance drops in state-of of models (Zheng et al., 2022). In contrast, local- ization distillation (LD) has shown better results for detection. Nevertheless, integrating LD remains challenging due to its substantial training overhead and incompatibility with anchor-free detectors.
To address these issues, we propose D-FINE, a novel real-time object detector that redefines bound- ing box regression and introduces an effective self-distillation strategy. Our approach tackles the problems of difficult optimization in fixed-coordinate regression, the inability to model localiza- tion uncertainty, and the need for effective distillation with less training cost. We introduce Fine- grained Distribution Refinement (FDR) to transform bounding box regression from predicting fixed coordinates to modeling probability distributions, providing a more fine-gres grained intermediate representation. FDR refines these distributions iteratively in a residual manner, allowing for progres- sively finer adjustments and improving localization precision. Recognizing that deeper layers pro- duce more accurate predictions by capturing richer localization information within their probability distributions. we introduce Global Optimal Localization Self- Dolf-Distillation (GO-LSD). GO-LSD transfers localization knowledge from deeper layers to shallower ones with negligible extra training cost. By aligning shallower layers' predictions with refined outputs from later layers, the model learns to produce better early adjustments, accelerating convergence and improving overall perfor- mance. Furthermore, we streamline computationally intensive modules and operations in existing real-time DETR architectures (Zhao et al., 2024] Chen et al. 2024), making D-FINE faster and more lightweight. While such modifications typically result in performance loss, FDR and GO-LSD effectively mitigate this degradation, achieving a better balance between speed and accuracy.
Experimental results on the COCO dataset (Lin et al. 2014a) demonstrate that D-FINE achieves state-of-the--at performance in real-time object detection, surpassing existing models in accuracy and efficiency. D-FINE-L and D-FINE-X achieve 54.0% and 55.8% AP, respectively va12017, running at 124 FPS and 78 FPS on an NVIDIA T4 GPU. After pretraining on larger datasets like Objects365 (Shao al. 2019), the D-FINE series attains up to 59.3% AP, surpassing all existing real-time detectors, showcasing both scalability and robustness. Moreover, our method enhances a variety of DETR models by up to 5.3% AP with negligible extra parameters and training costs, demonstrating its flcxibility and generalizability. In conclusion, D-FINE pushes the perfor- mance boundaries of real-time detectors. By addressing key challenges in bounding box regression and distillation efficiency through FDR and GO-LSD, we offer a meaningful step forward in object detection, inspiring further exploration in the field.
COCO
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Real-Time End-to-End Object Detectors. The YOLO series has led the way in real-time ob- ject detection, evolving through innovations in architecture, data augmentation, and training tech- niques (Redmon et al. 2016a Wang et al. 2023a Glenn. 2023; Wang & Liao 2024 Glenn. 2024]. While efficient YOLOs typically rely on Kon-Maximum Suppression (NMS), which intro- duces latency and instability between speed and accuracy. DETR (Carion et al. 2020) revolutionizes object detection by removing the need for hand-crafted components like NMS and anchors. Tradi- tional DETRs (Zhu al. 2020, Meng et al. 2021 Zhhnn et al. 2022 Wang et al. 2022 Liu al.) 2021 Li et al., 2022 Chen et al. 2022a c have achieved excelling performance but at the cost of high computational demands, making them unsuitable for real-time applications. Recently, RT- DETR (Zhao et al. 2024) and LW-DETR (Chen et al.
2024) have successfully adapted DETR for 2024a) also eliminates the need for NMS,
real-time use. Concurrently, YOLOv10 (Wang et al.
marking a significant shift towards end-to-end detection within the YOLO series.
Distribution-Base Object Detection. Traditional bounding box regression methods (Redmon et al. 2016b Liu et al. 2016 Ren et al. 2015) rely on Dirac delta distributions, treating bound- ing box edges as precise and fixed, which makes modeling localization uncertainty challenging. To address this, recent models have employed Gaussian discrete distributions represent bounding
boxes (Choi et al. 2019 Li et al. 2020 Qiu et al. 2020: Li et al. 2021), enhancing the modeling of uncertainty. However, these methods all rely on based frameworks, which limits their compatibility with modern anchor-free detectors like YOLOX (Ge et al., 2021) and DETR (Carion et al. )020). Furthermore, their distribution representations are often formulated in a coars--rraied manner and lack effective refinement, hindering their ability to achieve more accurate predictions.
Knowledge Distillation. Knowledge distillation (KD) (Hinton et al. 2015) is powerful model compression technique. Traditional KD typically focuses on transferring knowledge through Logit Mimicking (Zagoruyko & Komodakis 2017, Mirzadeh al.. 2020; [Son et al. (021). Fit- Nets (Romero et al. 2015) initially propose Feature Imitation, which has inspired a series of subse-
quent works that further expand upon this idea (Chen et al., 2017: Dai et al. 2021 Guo al. 2021
Li et al. 2017] Wang et al., 2. Most approaches for DETR (Chang et al. 2023 Wang et al.. 2024b) incorporate hybrid distillations of both logit and various intermediate representations. Re- cently, localization distillation (I.D) (Zeng et al. (002)) demonstrates that transferring localization knowledge is more effective for detection tasks. delf-distillation (Zhang et al., 2019 202T) is spe- cial case of KD which enables earlier layers to learn from the model's own refined outputs, requiring far fewer additional training costs since there's 's no need to separately train a teacher model.
3 PRELIMINARIES
Bounding box regression in object detection has traditionally relied modeling Dirac delta distri- butions, either using centroid-hased {ær, y. wr, h) or edge-distance {c, d} forms, where the distances d= {t, 6, I, r are measured from the anchor point tion, which treats bounding box edges as precise and fixed, makes it difficult to model localization uncertainty, especially in ambiguous cases. This rigid representation may limit optimization and potentially lead to localization errors with small prediction shifts.
{æe, ye). However, the Dirac delta assump-
To address these problems, GFocal (Li al., 2020 2021) regresses the distances from anchor points to the four edges using discretized probability distributions, offering a more flexible modeling of the bounding box. In practice, bounding box distances = (, is modeled as:
d = dmax
n
P(n).
N
1==
where dmax is a scalar that limits the maximum distance from the anchor center, and P(n) denotes the probability of each candidate distance of four edges. While GFocal introduces step forward in handling ambiguity and uncertainty through probability distributions, specific challenges in its regression approach persist: (1) Anchor Dependency: Regression is tied to the anchor box center, limiting prediction diversity and compatibility with anchor-free frameworks. (2) No Iterative Refine- ment: Predictions made in one shot without iterative refinements, reducing regression robustness.
3

---

potential edge offset. The weighted sum of the Published as a conference paper at ICLR 2025
(3) Coarse Localization: Fixed distance ranges and uniform bin intervals can lead to coarse local- ization, especially for small objects, because each bin represents a wide range of possible values.
Localization Distillation (LD) is promising approach. demonstrating that transferring localiza- tion knowledge is more effective for detection tasks (Zheng et al. 2022). Built upon GFocal, enhances student models by distilling valuable localization knowledge from teacher models, rather than simply mimicking classification logits or feature maps. Despite its advantages, the method still relies on anchor-based architectures and incurs additional training costs.
4 METHOD
We propose D-FINE, a powerful real-time object detector that excels in speed, size, computational cost, and accuracy. D-FINE addresses the shortcomings of existing bounding box regression ap- proaches by leveraging two key components: Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Seff-Distillation (GO-LSD), which work in tandem to significantly enhance performance with negligible additional parameters and training time cost.
(1) FDR iteratively optimizes probability distributions that act as corrections to the bounding box predictions. providing more fine-grained intermediate representation. This approach captures and optimizes the uncertainty of each edge independently. By leveraging the non-uniform weighting function, FDR allows for more precise and incremental adjustments at each decoder layer, improving localization accuracy and reducing prediction errors. FDR operates within an anchor-free, end-to- end framework, enabling a more flexible and robust optimization process.
(2) GO-LSD distill localization knowledge from refined distributions into shallower layers. As training progresses, the final layer produces increasingly precise soft labels. Shallower layers align their predictions with these labels through GO-LSD, leading to more accurate predictions. As early- stage predictions improve, the subsequent layers can focus on refining smaller residuals. This mutual reinforcement creates synergistic effect, leading progressively more accurate localization.
To further enhance the efficiency of D-FINE, we streamline computationally intensive modules and operations in existing real-time DETR architectures (Zhao et al. 2024), making D-FINE faster and more lightweight. Although these modifications typically result in some performance loss, FDR and GO-LSD effectively mitigate this degradation. The detailed modifications are listed in Table
4.1 FINE-GRAINED DISTRIBUTION REFINEMENT
Fine-grained Distribution Refinement (FDR) iteratively optimizes a fine-grained distribution gen- erated by the decoder layers, shown in Figure 2 In the first decoder layer, preliminary bounding boxes are predicted by traditional bounding box regression head, while the D-FINE head gener- ates the initial probability distributions for the four edges (top, bottom, left, right). The preliminary bounding boxes serve as reference boxes, while subsequent layers focus on refining them by ad- residual manner. The refined distributions are then applied adjust the
justing distributions four edges of the corresponding initial bounding box, progressively improving its accuracy with each iteration. Mathematically, let Ddd {ar, y, w, h) denote the initial bounding box prediction, where {ar, y} represents the predicted center of the bounding box, and (w. h) represent the width and height of the box. We can then convert b" into the center coordinates c" = {x. y and the edge distances d = {t, 6, r., which represent the distances from the center to the top, bottom, left, and right edges. For the 1-th layer, the refined edge distances d' = {t', b' 1,, [") are computed as:
d' d'=do d (h, b, W. w Σ (())Pr((m, le (1,2...., L),
where (n) = (P))(n), Pr(()) (n)) represents four separate distributions, one for each edge. Each distribution predicts the likelihood of candidate offset values for the corresponding edge. These candidates are determined by the weighting function (n), where n indexes the dis- crete bins out of N, with each bin corresponding distributions produces the edge offsets. These edge offsets are then scaled by the height h and width of the initial bounding box, ensuring the adjustments proportional to the box size.
4

---

Published as a conference paper at ICLR 2025
Refired Ebdeeiiationss
Weigfter Eistriaatioonn Wosttrer
Drooke Layer
Trittl Hilox
Ineial Biss
1
E
Weighting
Edge WoURrOG
Setianl
Figure 2: Overview of D-FINE with FDR. The probability distributions that act as a more fine- grained intermediate representation iteratively refined by the decoder layers in residual manner. Non-uniform weighting functions are applied to allow for finer localization.
The refined distributions updated using residual adjustments, defined as follows:
(r((m) Softmax (logits' (n)) Softmax (llgiiss (n) + logits logits""1(n)) (n)).
(3)
where logits from the previous layer logits' (n) reflect the confidence in each bin's offset value for the four edges. The current layer predicts the residual logits Alogits (n), which are added the previous logits to form updated logits logits' (n). These updated logits are then normalized using the softmax function, producing the refined probability distributions.
To facilitate precise and flexible adjustments, the weighting function W(n) is defined as:
2 ((1) = 2a
n 0
2-20 pー
( +
15 n <
((n)=
(4)
-442n N-2
- -c ( +1)
2 $ n n
B(W 1) 2a
n= N,
where a and e are syyper-parameter. controlling the upper bounds and curvature of the function. As shown in Figure 2 the shape of (n)) ensures that when bounding box prediction is near accurate, small curvature W ((n) allows for finer adjustments. Conversely, the bounding box prediction is far from accurate, the larger curvature near the edges and the sharp changes at the boundaries of (nn) ensure sufficient flexibility for substantial corrections.
To further improve the accuracy of our distribution predictions and align them with ground truth values, inspired by Distribution Focal Loss (DFL) (Li et al. 2020), we propose new function, Fine-Grained Localization (FGL) Loss, which is computed as:
1.
K
LPGL
Σ
IoUR (w- -CE CE (pr'( + +w CE (Pr' (Pr((r)):
1=1
1=
16 W(n+)l W(n+) W (n-))l W [W ((n)]
I ((n,))
W+
where Pr'(n)x represents the probability distributions corresponding to the k-th prediction. ø is the relative offset calculated as (d - dh, h, w). dc represents the ground truth edge- distance and n.-, n. are the bin indices adjacent to ó. The cross-entrope (CE) losses with weights wi and W ensure that the interpolation between bins aligns precisely with the ground truth offset. By incorporating loU Uased weighting, FGL loss encourages distributions with lower uncertainty to become concentrated, resulting in more precise and reliable bounding box regression.
5

---

Published as a conference paper at ICLR 2025
Derathed Prekites Hatchedd Prodetic
Toeeheo
Lociliation Kaowledge
Sad- latee
Krowlaige
DDFT Lass Weight
Deooder Layer
Dieoeerr Lavve
Decoder Lagrer
Baskonee
Figure 3: Overview of GO-LSD process. Localization knowledge from the final layer's refined
distributions is distilled into shallower layers through DDF loss with decoupled weighting strategies.
4.2 GLOBAL OPTIMAL LOCALIZATION SELF-DISTILLATION
Global Optimal Localization Distiiiitilltion (GO-LSD) utilizes the final layer's refined distribu- tion predictions distill localization knowledge into the shallower layers, shown in Figure This process begins by applying Hungarian Matching (Carion et al.] (020)) to the predictions from each layer, identifying the local bounding box matches at every stage of the model. To perform global optimization, GO-LSD aggregates the matching indices from all layers into unified union set. This union set combines most accurate candidate predictions across layers, ensuring that they all benefit from the distillation process. In addition to refining the global matches, GO-LSD also optimizes unmatched predictions during training to improve overall stability, leading to improved overall performance. Although the localization is optimized through this union set, the classification task still follows one-to-one matching principle, ensuring that there are no redundant boxes. This strict matching means that some predictions in the union set well-localized but have low confi- dence scores. These oo-cconfidence predictions often represent candidates with precise localization, which still need to be distilled effectively.
To address this, we introduce Decoupled Distillation Focal (DDF) Loss, which applies decoupled weighting strategies to ensure that high-IoU but low-confidence predictions given appropriate weight. The DDF Loss also weights matched and unmatched predictions according to their quantity, balancing their overall contribution and individual losses. This approach results in more stable and effective distillation. The Decoupled Distillation Focal Loss LDDF is then formulated as:
L-1
Km
Ka
CDDF = 2
KL (n)x. Pr Pr)(n))) +
Σ
3x KL (Pr(()),, Pr'(m))) Pr
a
1=1
VKm
Kg
loU
= Confs VKn + K
VKm + VK,
where KL represents the Kullback-Leinn divergence (Hinton et al. 2015), and T the temperature parameter used for smoothing logits. The distillation loss for the k-th matched prediction is weighted by agg, where Kn, and K2 are the numbers of matched and unmatched predictions, respectively. For the k-thh unmatched prediction, the weight is Sr, with Confg denoting the classification confidence.
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
To validate the effectiveness of our proposed methods, we conduct experiments on the COCO (Lin jet al. (2114a) and Objects365 (Shao et al., 2019) datasets. We evaluate our D-FINE using the stan- dard COCO metries, including Average Precision (AP) averaged over IoU thresholds from 0.50 to 0.95, as well as AP at specific thresholds (AP30 and APTs) and AP across different object scales: small (APs), medium (APa) and large (APz). Additionally, we provide model efficiency metrics by reporting the number of parameters (#Params.), computational cost (GFLOPs), and end-to-end latency. The latency is measured using TensorRT FP16 NVIDIA T4 GPU.

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison of various real-time object detectors COCO va12017.
Model
Paramss
GFLOPs
Lateney (ms)
Aprai
AP
AP
AP
AP
APL.
Non-end-to-eed Real-time Object Detectors
YOLOv6-L
59M
150
9.04
52.8
70.3
57.7
34.4
58.1
70.1
YOLOv7-L
36M
104
16.81
51.2
69.7
55.5
35.2
55.9
66.7
YOLOv7-X
71M
189
21.57
52.9
71.1
57.4
36,9
57.7
68.6
YOLOv8-L
43M
165
12.31
52.9
69.8
57.5
35.3
58.3
69.8
YOLOv8-X
68M
257
16.59
53.9
71.0
58.7
35.7
59.3
70.7
YOLOv9-C
25M
102
10.66
53.0
70.2
$7.8
36.2
58.5
69.3
YOLOv9-E
57M
189
20.53
55.6
72.8
60.6
40.2
61.0
71.4
Gold-YOLO4
75M
152
9.21
53.3
70.9
33.8
58,9
69.9
RTMDet-L
52M
80
14.23
51.3
68.9
55.9
33.0
55.9
68.4
RTMDet-X
95M
142
21.59
52.8
70.4
57.2
35.9
57.3
69.1
YOLO11-L
25M
87
10.28
53.4
70.1
58.2
35.6
59.1
69.2
YOL011-X
57M
195
14.39
54.7
71.6
59.5
37.7
59.7
70.2
YOLO11-L"
25M
87
6.31
52.9
69.4
57.7
35,2
58.7
68.8
YOL011-X*
57M
195
10.52
54.1
70.8
58.9
37.0
59.2
69.7
End-to-end Real-time Object Detectors
YOLOv10-L.
24M
120
7.66
53.2
70.1
58.1
35.8
58.5
69.4
YOLOv10-X
30M
160
10.74
54.4
71.3
59.3
37.0
59.8
70.9
RT-DETR --00
42M
136
9.12
53.1
71.3
57.7
34,8
58.0
70.0
RT-DETR-RIIE
76M
259
13.61
54.3
72.7
58.6
36.0
58.8
72.1
RT-DETR-HCG
32M
107
9,25
53.0
71.7
57.3
34.6
57.4
71.2
TT-DETR-H--A
67M
234
14.01
54.8
73.1
59.4
35.7
59.6
72.9
XT-DETRv2-L
42M
136
9.15
53.4
71.6
57.4
36.1
57.9
70.8
RT-DETRRv-X
76M
259
13.66
54.3
72.8
58.8
35.8
58.8
72.1
RT-DETRR3-L
42M
136
9.12
53.4
71.7
RT-DETRv3-X
76M
259
13.61
54.6
73.1
LW-DETR-L
47M
72
8.21
49.5
LW-DETR-X
118M
174
16.06
53.0
D-FINE-L (Ours)
31M
91
8.07
54.0
71.6
58.4
36.5
58.0
71.9
D-FINE-X (Ours)
62M
202
12.89
55.8
73.7
60.2
37.3
60.5
73.4
End-to-end Real-time Object Detectors (Pretrained Objects366)
10LOv10-L
24M
120
$4.0
71.0
58.9
36.5
59.2
70.5
YOLOv10-X
30M
160
10.74
54.9
71.9
37.6
71.7
RT-DETR-
42M
136
9.12
55.3
73.4
37.9
71.8
RT-DETR- R101
76M
259
13.61
61.3
38.3
73.5
LW-DETR-L
47M
72
8.21
LW-DETR-X
118M
174
76.9
63.3
40.9
D-FINE-L (Ours)
31M
8.07
57.1
D-FINE-X (Ours)
62M
202
12.89
59.3
42.3
64.2
NMS is tuned with confidence threshold of 0.01.
5.2 COMPARISON WITH REAL-TIME DETECTORS
Table provides a comprehensive comparison between D-FINE and various real-time object detec- tors COCO va 1 2017. D-FINE achieves excellent balance between efficiency and accuracy across multiple metrics. Specifically, D-FINE-L attains an AP of 54.0% with 31M parameters and 91 GFLOPs, maintaining low latency of 8.07 ms. Additionally, D-FINE-X achieves AP of 55.8% with 62M parameters and 202 GFLOPs, operating with latency of 12.89 ms. As depicted in Figure [ which shows scatter plots latency vs. AP, parameter count vs. AP, and FLOPS vs. AP, D-FINE consistently outperforms other state-of-ther of-tth--rtt models across all key dimensions. D-FINE- achieves higher AP (54.0%) compared Y0LOv10-L (53.2%), RT-DETR-RS0 (53.1%), and LW-DETR-X (53,0%), while requiring fewer computational resources (91 GFLOPs 120, 136, and 174). Similarly, D-FINE-X surpasses YOLOv 10-X and RT-DETR- R101 by achieving superior performance (55.8% AP 54.4% and 54.3%) and demonstrating greater efficiency in terms of lower parameter count, GFLOPs, and latency.
We further pretrain D-FINE and YOLOv10 the Objects365 dataset (hao al. 2019). before finetuning them on COCO. After pretraining, both D-FINE-L and D-FINE-X exhibit significant performance improvements, achieving AP of 57.1% 1% and 59.3%, respectively. These enhancements

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison of various real-time object detectors CrowdHuman.
Model
#Params.
GFLOPs
Latency (ms)
Ap
AP
AP
AP
AP
API.
YOLOv8-S
11
29
6.96
50.4
78.1
YOLOv9-S
7M
26
8.02
50.5
77.1
YOLOvI0-S
7M
22
2.65
50.6
78.6
RT-DETR-RIE
20M
61
4.63
54.1
82.3
RT-DETRV2-S
20M
60
4.59
54.2
82.6
52.2
19.4
36.8
47.2
D-FINE-S (Ours)
10M
25
3.49
55.5
87.1
60.1
28.4
44.5
52.7
Table 3: Effectiveness of FDR and GO-LSD across various DETR models on COCO va12017.
Model
#Params.
#Epochs
Apeal
AP
AP
APg
AP
AP
Deformable-DETi
40M
12
43.7
62.2
46.9
26.4
46.4
57.9
+ FDR & GO-LSD
40M
12
47.1 (+3.4)
64.7
50.8
29.0
50.3
62.8
DAB-DETR
48M
12
44.2
62.5
47.3
27.5
47.1
58.6
+ FDR & GO-LSD
48M
12
49.5 (45.3)
67.2
54.1
31.8
53.2
63.3
DN-DETR
48M
12
46.0
64,8
49.9
27.7
49.1
62.3
FDR GO-LSD
48M
12
49.7 (+3.7)
67.5
54.4
31.8
53.4
63.8
DINO
47M
12
49.0
66.6
53.5
32.0
52.3
63.0
+ FDR GO-LSD
47M
12
51.6 (+2.6)
68.6
56.3
33.8
55.6
65.3
DINO
47M
24
50.4
68.3
54.8
33.3
53.7
64.8
+ FDR GO-LSD
47M
24
52.4 (+2.0)
69.5
56.9
34.6
55.7
66.2
enable them to outperform YOLOv10-L and YOLOv10-X by 3.1% and 4.4% AP, thereby position- ing them as the nop-performing models in this comparison. What's more. following the pretraining protocol of YOLOv8 (Glenn. 2023). YOLOv10 pretrained on Objects365 for 300 epochs. In contrast, D-FINE requires only 21 epochs to achieve its substantial performance gains. These find- ings corroborate the conclusions of LW-DETR (Chen et al., (2024), demonstrating that DETR-based models benefit substantially more from pretraining compared other detectors like YOLOs.
Table compares D-FINE-S with state-of-the-art real-time detectors on the CrowdHuman dataset (Shao al. 2018). D-FINE-S achieves the highest AP (55.5%) and AP (87.1%) among all models, with only 10M parameters, 25 GFLOPs, and latency of 3.49ms.
5.3 EFFECTIVENESS ON VARIOUS DETR MODELS
Table demonstrates the effectiveness of FDR and GO-LSD across multiple DETR-based object detectors COCO va 2017. These methods designed for flexibility and can be seamlessly integrated into Deformable DETR, DAD-DETR, DN-DETR, and DINO, consistently boosting de- tection accuracy by 2.0% 5.3% without increasing the number of parameters and computational burden. These results highlight the effectiveness of FDR and GO-LSD in enhancing localization precision and maximizing efficiency, demonstrating their adaptability and substantial impact across various end-to-end detection frameworks.
5.4
ABLATION STUDY
5.4.1
THE ROADMAP D-FINE
Table 4 showcases the stepwise progression from the baseline model (RT-DETR-HGNaww.. Zhaoo et al. 2024)) to proposed D-FINE framework. Starting with the baseline metrics of 53.0% AP, 32M parameters, 110 GFLOPs, and 9.25 ms latency, we first remove all the decoder projection lay- This modification reduces GFLOPs to 97 and cuts the latency 8.02 ms, although it decreases AP to 52.4%. To address this drop, we introduce the Target Gating Layer, which recovers the AP to 52.8% with only marginal increase in computational cost.
8

---

Published as a conference paper at ICLR 2025
Table 4: Step-by-step modifications from baseline model D-FINE. Each step shows changes in
AP, the number of parameters, latency, and FLOPs.
Model
Apeal
#Params.
Latency (ms)
GFLOPs
bascline:
(Zhao ct al. 2024
53,0
32M
9.25
110
Remove Decoder Projection Layers
52.4
32M
8.02
97
+ Target Gating Layers
52.8
33M
8.15
Encoder CSP layers GELAN WWag & Liao 2024
53.5
46M
10.69
167
Reduce Hidden Dimension in GELAN by half
52.8
31M
8.01
91
Uneven Sampling Points 3, M: L:
52.9
31M
7.90
RT-DETRy2 Training Strategy (Lv et al. 2024
53.0
31M
7.90
91
+ FDR
53.5
31M
8.07
91
+ GO-LSD
54.0
(+1.0)
31M
(-3%)
8.07(-13%)
91
(-17%)
Table 5: Distillation methods comparison in terms of performance, training time, and GPU memory usage. GO-LSD achieves the highest Ap with minimal additional training cost.
Methods
Apeal
Time/Epoch Memory
FDR-based Methods
Apraí
Time/Epoch Memory
baseline
53.0
29min
8552M
baseline + FDR
53.8
30min
8730M
Logit Mimicking
52.6
31min
8554M
Localization Distill.
53.7
31min
8734M
Feature Imitation
52.9
31min
8554M
GO-LSD
54.5
31min
8734M
The Target Gating Layer is strategically placed after the decoder's cross-attention module, replacing the residual connection. It allows queries to dynamically switch their focus on different targets across layers, effectively preventing information entanglement. The mechanism operates as follows:
x = ([x1: X2 W + bb, + ([x1, X2] W + +b),
(7)
where X1 represents the previous queries and is the cross-attention result. is the sigmoid acti- vation function applied to the concatenated outputs, and [-] represents the concatenation operation.
Next, we replace the encoder's CSP layers with GELAN layers (Wang & Liao 2024). This substi- tution increases AP to 53.5% but also raises the parameter count, GFLOPs, and latency. To mitigate the increased complexity, we reduce the hidden dimension of GELAN, which balances the model's complexity and maintains AP at 52.8% while improving efficiency. We further optimize the sam- pling points by implementing uneven sampling across different scales (S: M: 6, L: 3), which slightly increases AP to 52.9%. However, alternative sampling combinations such as (S: 6, M: 3, L: 3) and (S: 3, M: 3, 6) result a minor performance drop of 0. 0.1% AP. Adopting the RT-DETRv2 training strategy (Lv et al. 2024) (sce Appendix A.I.I for details) enhances AP to 53.0% without affecting the number of parameters or latency. Finally, the integration of FDR and GO-LSD mod- ules elevates AP to 54.0%, achieving 13% reduction in latency and 17% reduction in GFLOPs compared to the baseline model. These incremental modifications demonstrate the robustmess and effectiveness of D-FINE framework.
5.4.2 COMPARISON OF DISTILLATION METHODS
cabperss different distillation methods based on performance, training time. and GPU mem- usage. The bascline model achieves an AP of 53.0%, with training time of 29 minutes per epoch and memory usage of 8552 MB on four NVIDIA RTX 4090 GPUs. Due to the instability of one-to-one matching in DETR, traditional distillation techniques like Logit Mimicking and Feature Imitation do not improve performance; Logit Mimicking reduces AP 52.6%, while Feature Imita- tion achieves 52.9%. Incorporating our FDR module increases AP to 53. 8% with minimal additional training cost. Applying vanilla Localization Distillation (Zheng et al. 2022) further increases AP to 53.7%. Our GO-LSD method achieves the highest AP of 54.5%. with only 6% increase in training time and a 2% rise in memory usage compared to the baseline. Notably, no lightweight optimizations applied in this comparison, focusing purely distillation performance.

---

Published as a conference paper at ICLR 2025
nesa fibec and Druiberce Laper tifer Chamberrns
Inaul Bioe Decri aloud Lace Derrt
Insul lilos and Denii plons
lage files Domttaae
Weatee
Westerd
Lasa niles and Diathesoa
lezul and Darst zince
Issul Bioe Dient
Westme Danaaked
Exnt
Tap
Widtee Bheaiiked
1at
Right
Ratan
lat
Eight
Button
Figure 4: Visualization of FDR across detection scenarios with initial and refined bounding boxes, along with unweighted and weighted distributions, highlighting improved localization accuracy.
5.5 VISUALIZATION ANALYSIS
Figure illustrates the process of FDR across various detection scenarios. We display the filtered detection results with two bounding boxes overlaid on the images. The red boxes represent the initial predictions from the first decoder layer, while the green boxes denote the refined predictions from the final decoder layer. The final predictions align more closely with the target objects. The first row under the images shows the unweighted probability distributions for the four edges (left, top, right, bottom). The second row shows the weighted distributions, where the weighting function ((n) has been applied. The red curves represent the initial distributions, while the green curves show the final, refined distributions. The weighted distributions emphasize finer adjustments near accurate predictions and allow for enabling rapid changes for larger adjustments, further illustrating how FDR refines the offsets of initial bounding boxes, leading increasingly precise localization.
CONCLUSION
In this paper, introduce D-FINE, powerful real-time object detector that redefines the bound- ing box regression task in DETR models through Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). Experimental results on the COCO dataset demonstrate that D-FINE achieves state-of- ofite-of-thh-art accuracy and efficiency, surpassing all existing real-time detectors. Limitation and Future Work: However, the performance gap between lighter D-FINE models and other compact models remains small. One possible reason is that shallow de- coder layers may yield less accurate final-layer predictions, limiting the effectiveness of distilling localization knowledge into earlier layers. Addressing this challenge necessitates enhancing the lo- calization capabilities of lighter models without increasing inference latency. Future research could investigate advanced architectural designs novel training paradigms that allow for the inclusion of additional sophisticated decoder layers during training while maintaining lightweight inference by simply discarding them test time. We hope D-FINE inspires further advancements in this area.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grants 62472399, 62021001, and 62032006.
REFERENCES
Elahe Arani, Shruthi Gowda, Ratnajit Mukherjee, Omar Magdy, Senthilkumar Kathiresan, and Bahram Zonooz. A comprehensive study of real-time object detection networks across multi- ple domains: A survey. arXiv preprint arXiv:2208. 10895, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Jiahao Chang, Shuo Wang. Hai-Ming Xu, Zehui Chen, Chenhongyi Yang, and Feng Zhao. De- trdistill: A universal knowledge distillation framework for detr-families. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. -998,, 2023.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In NeurlPS, 2017.
Qiang Chen, Xiaokang Chen, Gang Zeng, and Jingdong Wang. Group detr: Fast training conver- gence with decoupled one-to-many label assignment. arXiv preprint arXiv:2207.13005 2022a.
Qiang Chen, Jian Wang, Chuchu Han, Shan Zhang, Zexian Li, Xiaokang Chen, Jiahui Chen, Xi- aodi Wang, Shuming Han, Gang Zhang, Haocheng Feng. Kun Yao, Junyu Han, Errui Ding, and Jingdong Wang. Group detr v2: Strong object detector with encoder-decoder pretraining, 2022b.
Qiang Chen, Jian Wang, Chuchu Han, Shan Zhang. Zexian Li, Xiaokang Chen, Jiahui Chen, Xiaodi Wang, Shuming Han, Gang Zhang, et al. Group detr v2: Strong object detector with encoder- decoder pretraining. arXiv preprint arXiv:2211.03355
2022c.
Qiang Chen, Xiangbo Su, Xinyu Zhang, Jian Wang. Jiahui Chen, Yunpeng Shen, Chuchu Han, Ziliang Chen, Weixiang Xu, Fanrong Li, et al. Lw-detr: A transformer replacement to yolo for real-time detection. arXiv preprint urXiv:2406.0344 2024.
Jiwoong Choi, Dayoung Chun, Hyun Kim, and Hyuk-Jac Lee. Gaussian YOLOv3:
An accurate and fast object detector using localization uncertainty for autonomous driving. In ICCV, 2019.
Cheng Cui, Ruoyu Guo, Yuning Du, Dongliang He, Fu Li, Zewu Wu, Qiwen Liu, Shilei Wen, Jizhou Huang, Xiaoguang Hu, Dianhai Yu, Errui Ding, and Yanjun Ma. Beyond self-supervision: A simple yet effective network distillation alternative to improve backbones, 2021.
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General instance distillation for object detection. In CVPR, 2021.
Zheng Ge, Songtao Liu, Feng Wang. Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.
Ross Girshick. Fast R-CNN. In 2015.
Jocher Glenn. Yolov8, httppr//bbssuiitr.rrr....oooooor 2023.
2023.
Jocher Glenn. Yo http::///ocs.lltaaaare.eoooooood
2024.
2024.
Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Dis- tilling object detectors via decoupled features. In CVPR, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint / 002331 2015.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13619-13627, 2022.

---

Published as a conference paper at ICLR 2025
Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In CVPR, 2017.
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized Focal Loss: learning qualified and distributed bounding boxes for dense object de- tection. In NeurlPS, 2020.
Xiang Li, Wenhai Wang. Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. In CVPR, 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014a.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014b.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang. Xianbiao Qi. Hang Su, Jun Zhu, and Lei Zhang. Dab- detr: Dynamic anchor boxes are better queries for detr. In International Conference on Learning Representations, 2021.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yane Fu, and Alexander C. Berg. Ssd: Single shot multibox detector. In ECCV, 2016.
Wenyu Lv, Yian Zhao, Qinyao Chang, Kui Huang, Guanzhong Wang, and Yi Liu. RI-detrv2: Im- proved baseline with bag-of-freebies for real-time detection transformer, 2024.
Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, and Kai Chen. Rimdet: An empirical study of designing real-time object detectors. ArXiv, abs/2212.07788, 2022.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3651-3660 -6600, 2021.
Seyed Iman Mirzadeh, Mchrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020.
Heqian Qiu, Hongliang Li, Qingbo Wu, and Hengean Shi. Offset bin classification network for accurate object detection. In CVPR, 2020.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 779-788, 2016a.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In CVPR, 2016b.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-253. 2015.
Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhu- man: A benchmark for detecting human in a crowd. arXiv preprint arXiv: / /805.00/23 2018.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A lcage-scale, high-quality dataset for object detection. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 8430- 8439, 2019.
12

---

Published as a conference paper at ICLR 2025
Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distilla- tion using multiple teacher assistants. In ICCV, 2021.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. detection. In ICCV, 2019.
FCOS: Fully convolutional one-stage object
Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Yolov 10: Real-time end-to-end object detection. In The European Conference on Computer Vision (ECCV), 2024a.
Chengcheng Wang. Wei He, Ying Nic, Jianyuan Guo, Chuanjian Liu, Yunhe Wang, and Kai Han. Gold-yolo: Efficient object detector via anthbrr-nnddistribute mechanism. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2023a.
Chien- Yao Wang and Hong-Yune Yuan Mark Liao. YOLOv9: Learning what you want to learn using programmable gradient information. arXiv preprint arXiv:2402. 2024.
Chien- Yao Wang, Alexey Bochkovskiy, and Hong-Yur Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 7464-7475, 2023b.
Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-graud mne-grained feature imitation. In CVPR, 2019.
Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 2567-2575, 2022.
Yu Wang. Xin Li. Shengzhao Weng, Gang Zhang, Haixiao Yue, Haocheng Feng, Junyu Han, and Errui Ding. Kd-detr: Knowledge distillation for detection transformer with consistent distillation points sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16016-16025, 2024b.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations, 2022.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher; Improve the performance of convolutional neural networks via self distillation. In ICCV, pp. 3713-3722 2019.
Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8):4388- 4403, 2021.
Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Liu, and Jie Chen. Detrs beat yolos on real-time object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Zhaohui Zheng, Rongguang Ye, Ping Wang. Dongwei Ren, Wangmeng Zuo, Qibin Hou, and Ming- Ming Cheng. Localization distillation for dense object detection. In Proceedings the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 9407-9416 9416, 2022.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: De- formable transformers for end-to-end object detection. In International Conference on Learning Representations, 2020.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1
IMPLEMENTATION DETAILS
A.1.1 HYPERPARAMETEE CONFIGURATIONS
Table summarizes the hyperparameter configurations for the D-FINE models. All variants HGNetV2 backbones pretrained on ImageNet (Cui et al. 2021 Russakovsky et al. 2015) and the AdamW optimizer. D-FINE-X set with embedding dimension of 384 and feedforward dimen- sion of 2048, while the other models use 256 and 1024, respectively. The D-FINE-X and D-FINE-L have 6 decoder layers, while D-FINE-M and D-FINE-S have 4 and 3 decoder layers. The GELAN module progressively reduces hidden dimension and depth from D-FINE-X with 192 dimensions and layers to D-FINE-S with dimensions and layer. The base learning rate and weight decay for D-FINE-X and D-FINE-L. are 2.5 x 10-1 and 1.25 x 10-1 respectively, while D-FINE-M and D-FINE-S use 2 x 10-4 and 1 x 10-1 Smaller models also have higher backbone learning rates than larger models. The total batch size is 32 across all variants. Training schedules include 72 epochs with advanced augmentation (RandomP www.emmohoottteeeeeeeeeoo www.ttttttiiiiiis..oeeoeei.e.iee BandomIoUCrop. and RMulti Scale Input) followed by 2 epochs without advanced augmen- tation for D-FINE-X and D-FINE-L, and 120 epochs with advanced augmentation followed by epochs without advanced augmentation for D-FINE-M and D-FINE-S (RT-DETRv2 Training Strat- egy (Ly et al., 2024) in Table 4). The number of pretraining epochs is 21 for D-FINE-X and D- FINE-L models, while for D-FINE-M and D-FINE-S models, ranges from 28 to 29 epochs.
RandomZoomOus
Table 6: Hyperparameter configurations for different D-FINE models.
Setting
D-FINE-X
D-FINE-L
D-FINE-M
D-FINE-S
Backbone Name
HGNetv2-B5
HGNetv2-B4 AdamW
HGNetv2-B2 B22
HGNetv2-1B0
Optimizer
AdamW
AdamW
AdamW
Embedding Dimension
384
256
256
256
Feedforward Dimension
2048
1024
1024
1024
GELAN Hidden Dimension
192
64
GELAN Depth
3
2
1
Decoder Layers
6
3
Queries
300
300
300
300
in (n))
0.5, 0.5
0.5, 0.25
0.5, 0.25
0.5, 0.25
Bin Number N
32
32
32
32
Sampling Point Number
(S: 3, M: 6, L)
(S: 3, M: 6, L:
(S: 3,
M: L:
(S: 3, M: 6, L:
Temporature T
5
Base LR
2.5c-4
2.5e-4
2e-4
2e-4
Backbone LR
2.5e-6
1,25e-5
2e-5
le-4
Weight Decay
1.25e-4
1.25e-4
le-4
le-4
Weight of LVFL.
I
1
Weight of LyBox
S
5
5
Weight of Laou
2
2
2
Weight of LAGI.
0.15
0.15
0.15
0.15
Weight of LYDP
1.5
1.5
1.5
1.5
Total Batch Size
32
32
EMA Decay
0.9999
0.9999
0.9999
0.9999
Epochs (w/ w/o Adv. Aug.)
+2+2
72+2
+20+4
120+4
Epochs (Pretrain Finetune)
+1+31
+1
29
28 58
A.1.2 DATASETS SETTINGS
For pretraining, following the approach in (Chen et al. 2022b Zhang al. 2022 Chen al. 2024). we combine the images from the Objects365 (Shao et al. 20199) train set with the validate set, exclud- ing the first 5k images. To further improve training efficiency, exceeding 640 x 640 down to 640 x 640 beforehand. We use the standard COC02017 (Lin et al, 2014b) data splitting policy, training COCO train2011 and evaluating COCO val 12017.
resize all images with resolutions
14

---

Published as a conference paper at ICLR 2025
A.2 VISUALIZATION OF D-FINE PREDICTIONS
Figure demonstrates the robustness of the D-FINE-X model, visualizing its predictions in various challenging scenarios. These include occlusion, low-light conditions, motion blur, depth of field effects, rotation, and densely populated scenes with numerous objects in close proximity, Despite these difficulties, the model accurately identifies and localizes objects, such as animals, vehicles, and people. This visualization highlights the model's ability to handle complex real-world conditions while maintaining robust detection performance.
KIL
Hivenia
Figure 5: Visualization of D-FINE-X (without pre-training Objects365) predictions under chal- lenging conditions, including occlusion, low light, motion blur, depth of field effects, rotation, and densely populated scenes (confidence threshold=0.5).
15

---

Published as a conference paper at ICLR 2025
A.3 COMPARISON WITH LIGHTER DETECTORS
7 a comprehensive comparison of D-FINE models with various lightweight real-time object detectors in the S and M model sizes on the COCO va 2017. D-FINE-S achieves im- pressive AP of 48.5%, surpassing other lightweight models such as YOLO-S (46.4%) and RI-DETRy2-S (48.1%), while maintaining low latency of 3.49 with only 10.2M parameters and 25.2 GFLOPs. Pretraining on Objects365 further boosts D-FINE-S to 50.7%, marking an im- provement of +2.2%. Similarly, D-FINE-M attains AP of 52.3% with 19.2M parameters and 56.6 GFLOPs at 5.62 ms, outperforming YOLOv 10-00v0-MM (51.1%) and RT-DETRRV-N (49.9%). Pretraining 0bjects365 consistently enhances D-FINE-M, yielding +2.8% gain. These results demonstrate that D-FINE models strike an excellent balance between accuracy and efficiency, consistently sur- passing other state-of-the-an lightweight detectors while preserving real-time performance.
Table 7: Performance comparison of S and M sized real-time object detectors on COCO va 12017.
Model
#Params. GFLOPs
Latency (ms)
Ap
Ap
Ap
AP
AP
Appa
Non-end-to-end Real-time Object Detectors
YOLOv6-S
7M
45.0
YOLOv6-M
35M
5.48
50.0
66.9
54.6
YOLOv8-S
11M
6.96
44.9
48.6
49.9
YOLOv8-M
26M
79
9.66
50.2
67.2
54.6
32.0
55.7
66.4
YOLOx9-S
YOLOv9-M
20M
50.2
54.6
32.0
Gold-YOLO-S
22M
46.4
51.3
Gold-YOLO-M
41M
3.21
51.1
68.5
32.3
56.1
RTMDet-S
9M
7.77
44.6
61.9
48.1
24.9
48.5
62.5
RTMDet-M
25M
39
10.62
49.4
66.8
53.7
30.3
53.9
66.2
YOLO11-S
9M
46.6
51.3
YOLO11-M
20M
8.79
51.2
67.9
Y
9M
2.86
47.0
63.9
50.7
29.0
51.7
64.4
YOLO11-M*
20M
68
4.95
51.5
68.5
55.7
33.4
57.1
67.9
End-to-ent Real-time Object Detectors
YOLOv10-S
7M
22
2.65
46.3
63.0
50.4
26.8
51.0
63.8
YOLOv10-M
15M
59
4.97
51.1
68.1
55.8
33.8
56.5
67.0
RT-DETR-RI8
20M
61
4.63
46.5
63.8
50.4
28.4
49.8
63.0
RT-DETR-R34
31M
93
6.43
48.9
66.8
52.9
30.6
52.4
66.3
RT-DETRv2-S
20M
60
4.59
48.1
65.1
57.4
36.1
57.9
70.8
RT-DETRv2-M
31M
92
6.40
49,9
67.5
58.6
35.8
58.6
72.1
RI-DETRv3-R18
20M
61
4.63
48.1
66.2
RT-DETRv3-R34
31M
93
6.43
49.9
67.7
LW-DETR-S
15M
17
3.02
43.6
LW-DETR-M
28M
43
5.23
47.2
D-FINE-N (Ours)
4M
7.2
2.12
42.8
60.2
45.4
22.8
46.8
61.9
D-FINE-S (Ours)
10M
25
3.49
48.5
65.6
52.6
29.1
52.2
65.4
D-FINE-M (Ours)
19M
57
5.55
52.3
69.8
56.4
33.2
56.5
70.2
End-to-end Real-tine time Object Detectors (Pretrained on 3655cts365)
RT-DETR-RIE R18
20M
61
4.63
49.2
66.6
53.5
33.2
52.3
64.8
LW-DETR-S
15M
48.0
66.9
51.7
26.8
52.5
65.5
LW-DETR-M
28M
52.6
72.0
56.7
32.6
57.7
70.7
D-FINE-S (Ours)
10M
3.49
50.7
67.6
55.1
32.7
54.6
66.5
D-FINE-M (Ours)
19M
57
5.62
55.1
72.6
59.7
37.9
59.4
71.7
NMS is tuned with confidence threshold of 0.01.
Table presents a concise performance comparison on Objects365. Our D-FINE models consis- tently deliver high AP scores while maintaining low latency and minimal computational overhead. In particular, D-FINE-X achieves an impressive 49.5% AP with 62M parameters at 12.89 ms, and the lightweight D-FINE-S registers competitive 31.0% AP with only 10M parameters. These results underscore the excellent balance between accuracy and efficiency offered by the D-FINE series the complex, large-scale dataset, outperforming both RT-DETR and YOLOv10 models.
16

---

Published as a conference paper at ICLR 2025
Table 8: Performance comparison of real-time object detectors Objects365.
Model
#Params.
GFLOPs
Latency (ms)
AP
AP
RT-DETR-RID
20M
61
4.63
22.9
31.2
RT-DETR-REE -R50
42M
136
9.12
35.1
46.2
RT-DETR-RIOS
76M
259
13.61
36.8
48.3
YOLOv10-L
24M
120
7.66
34.2
44.9
YOLOvI0-X
30M
160
10.74
34.9
45.8
D-FINE-S (Ours)
1OM
25
3,49
31,0
41.0
D-FINE-M (Ours)
19M
57
5.55
38.6
49.7
D-FINE-L (Ours)
31M
91
8.07
44.7
56.9
D-FINE-X (Ours)
62M
202
12.89
49.5
62.4
Table Hyperparameter ablation studies on D-FINE-L. is a very small value. a, indicate that and are learnable parameters.
a.
1+
2
2+
8
Apoal
52.7
53.0
53.3
53.2
53.2
53.1
N
4
8
16
32
64
128
A
53.3
53.4
53.5
53.7
53.6
53.6
T
2.5
7.5
10
20
Ap
53.2 53.7 54.0
53.8
53.7
53.5
A.4 HYPERPARAMETER SENSITIVITY ANALYSIS
Appendix A.3 presents subset of hyperparameter ablation studies evaluating the sensitivity of our model to key parameters in the FDR and GO-LSD modules. We examine the weighting function parameters and c, the number of distribution bins N, and the temperature T used for smoothing logits in the KL divergence.
(1) Setting 1/2 and
yields the highest AP of 53.3%. Notably, treating and learnable parameters (a, 2) slightly decreases AP to 53.1 %, suggesting that fixed values simplify the optimiza- tion process. When is extremely large, the weighting function approximates the linear function with equal intervals, resulting in a suboptimal AP of 53.0%. Additionally, values of a that are too large or too small can reduce fineness limit flexibility, adversely affecting localization precision. (2) Increasing the number of distribution bins improves performance, with maximum AP of 53.7% achieved at = 32. Beyond N 32, no significant gain observed. (3) The temperature con- trols the smoothing of logits during distillation. An optimal AP of 54.0% is achieved at T = 5, indicating a balance between softening the distribution and preserving effective knowledge transfer.
A.5 CLARIFICATION ON THE INITIAL LAYER REFINEMENT
In the main text, we define the refined distributions at layer as:
Pr' (m) Softmax Alogits (m) + logits'-1(n))
where Alogits' (n) are the residual logits predicted by layer 1, and logits ((xiss""((x)))) are the logits from the previous layer. For the initial layer (1 1). there is no previous layer, so the formula simplifies to:
Pr (n) = Softmax (logits (n)).
(9)
Here, logits (n) are the logits predicted by the first layer.
This clarification ensures the formulation is consistent and mathematically rigorous for all layers.
17


Reviews:
[{"review_id": "r70mnKDXrW", "reviewer": "Reviewer_oAYL", "summary": "This paper introduces a real-time object detector based on DETR through two main components: fine-grained distribution refinement, which refines probability distributions iteratively for improved accuracy, and global optimal localization self-distillation, which optimizes localization through a bidirectional self-distillation approach.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to understand and well-presented. A set of experiments are conducted on the COCO and Objects365 datasets.", "weaknesses": "1. I do not fully agree with the claim that using fixed coordinates results in poor performance due to inadequate modelling of localization uncertainty. While the statement shows potential limitations of fixed-coordinate regression, many SOTA object detectors (Faster R-CNN and RetinaNet), represent significant advances in both speed and detection. Therefore, to say that all methods using fixed coordinates were fundamentally limited overlooks these advancements. The authors may clarify their claim by acknowledging the success of fixed-coordinate methods while explaining how the proposed approach can address these limitations\n\n2. It has been stated, \" As early predictions improve, the subsequent layers can focus on refining smaller residuals.\", what do you mean by  \"refining smaller residuals\"?\n\n3. In page 5, Eq. (3).  The normalizing updated logits can result in very small gradients when logits are large, making it harder for the model to learn effectively in deeper layers. This can reduce the model’s precision in refining bounding box edges. Moreover, softmax forces the output to sum to one, which can limit its ability to model complex relationships between bins and reduce its performance for handling localization uncertainty. Have you tried any other normalization methods such as Entmax normalization which well fit into DERT methods? and why softmax has been chosen over other options?\n\n4. In Figure 4 what those sub-plots (right, left, ...) mean and represent? There is no discussion about these plots. It is not clear why these plots are shown and what they are representing. Moreover, it is unclear, what the pick value means and what different lines (red, green) are supposed to be for the best performance. The authors need to provide a more detailed caption or explanation for this figure, specifically describing what each subplot represents and the meaning of the different colored lines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nLJAIKs7gu", "reviewer": "Reviewer_hveE", "summary": "The paper proposes D-FINE, which consists of two methods, Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD) to improve the performance of real-time object detectors. Based on the probability distribution representation of bbox, the paper uses multiple D-FINE head to predict the prob. distributions of bboxes on $L$ layers in one decoder. FDR uses a hand-crafted weight function to weight the prob. distribution of bbox and then iteratively refine the bboxes. GO-LSD is inspired from localization distillation, which distills the localization knowledge from the prob. distribution of bboxes of the last layer to the ones of the shallow layers. The paper also designs two losses to work with the proposed FDR and GO-LSD. Experiments on COCO benchmark show the effectiveness of the proposed method.", "questions": "1. How many D-FINE heads are used in the 1-st decoder, e.g., the value of $L$? Is FDR applied within each decoder layer or across decoder layers? \n\n2. If FDR is applied differently across different decoders, whether the number of D-FINE heads is consistent across all decoder layers or if it varies? How and why it differs? I suggest that the authors add this information to provide a clearer picture of the model's architecture and operation would enhance the reproducibility of the proposed method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is technically sound, which is supported by sufficient experiments.\n\n2. The paper shows the roadmap from the baseline model to the proposed D-FINE framework, making the technical contribution clear and transparent.\n\n3. The proposed D-FINE achieves state-of-the-art performance in real-time object detection.", "weaknesses": "1. It is confusing to me that in Fig. 2, it seems that D-FINE with FDR is applied to the 1-st decoder layer of the object detector. Then, within 1 decoder layer, multiple D-FINE head is used to generate the prob. distribution of bboxes. However, on lines 207-208, you mention \"the first decoder layer predicts preliminary bounding boxes and preliminary probability distributions through a traditional bounding box regression head and **a D-FINE head**\". In Sec. 4.2,  GO-LSD utilizes the final layer’s refined distribution predictions to distill localization knowledge into the earlier layers. One can see in Fig. 3 that the self-distillation is conducted between different decoders. Thus, is that the meaning of the word \"layer\" different in FDR and GO-LSD? What is the exact meaning of \"layer\" in the context of FDR versus GO-LSD? \n\n2. In Fig. 3, the matched prediction and unmatched prediction are colored by green squares and yellow squares. What does the gray squares stand for? I suggest that the authors could explicitly state what the gray squares represent in the caption or legend of Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y8V7nWkQVv", "reviewer": "Reviewer_5Uux", "summary": "The paper introduces D-FINE, a real-time object detection model that enhances DETR by redefining bounding box regression with Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). The FDR transforms bounding box predictions into refined probability distributions, while GO-LSD uses refined localization knowledge to improve earlier layers' predictions. Experiments on the COCO dataset show that D-FINE achieves good performance and high speed, positioning it as a strong competitor to existing real-time detectors.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents D-FINE, an enhancement to the DETR framework that tackles bounding box regression with a novel fine-grained distribution refinement and self-distillation mechanism. \n\n2. The experimental results on the COCO dataset demonstrate competitive performance, showing that D-FINE outperforms many existing real-time object detectors, with a favorable trade-off between speed and accuracy. Additionally, the paper’s ablation studies and hyperparameter tuning offer insightful explanations of the model’s design choices.\n\n3. The experiments and comparisons with various DETR-based models are thorough. The inclusion of both small and large models, alongside real-world visualizations, strengthens the validity of the claims about D-FINE's performance across diverse conditions.", "weaknesses": "The experiments focus primarily on the COCO dataset. Evaluating D-FINE across a broader range of datasets could strengthen its generalizability claims and confirm its robustness across different object detection contexts, such as crowded scene CrowdHuman and long tail scene LVIS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DNUmpPPjmZ", "reviewer": "Reviewer_759J", "summary": "D-FINE is a real-time object detection model that refines bounding box regression in Detection Transformers using Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). These techniques enhance localization accuracy and efficiency by iteratively refining bounding boxes and transferring refined knowledge across layers. D-FINE outperforms other real-time detectors on the COCO dataset, achieving a better balance of speed and precision with minimal added costs.", "questions": "1.\tIn Table 3, FDR and GO-LSD modules have a 1% mAP improvement in total for D-FINE. However, according to table 2, these two modules improve accuracy by 2% at least. Can the author explain this phenomenon?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tD-FINE surpasses all existing real-time detectors with negligible extra parameters and training costs. Source code is provided. This method appears to be highly solid and reproducible.\n2.\tTable 2 illuminates that the proposed FDR and GO-LSD are efficient for a series of DETR models, demonstrating the robustness of the module.\n3.\tThe proposed method is both innovative and systematic.", "weaknesses": "1.\tIn Table 3, FDR and GO-LSD modules have a 1% mAP improvement in total for D-FINE, which is noticeably less than the improvement shown in Table 2.\n2.\tThe language in the article needs refinement, as some phrases may lead to ambiguity. For example, line 206-208, “Initially, the first …” and line 209, “…one for each edge”.\n3.\tVariable names in the Method need to be consistent. For example, line 213, {W, H} and line 139 {w,h}. The lowercase ‘w’ is preferable, as the uppercase ‘W’ repeats the notation used for the weighting function below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)