D-FINE: Redefine Regression Task of DETRs as Fine-grained Distribution Refinement

## Abstract

Abstract We introduce D-FINE , a powerful real-time object detector that achieves outstanding localization precision by redefining the bounding box regression task in DETR models. D-FINE comprises two key components: Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD) . FDR transforms the regression process from predicting fixed coordinates to iteratively refining probability distributions, providing a fine-grained intermediate representation that significantly enhances localization accuracy. GO-LSD is a bidirectional optimization strategy that transfers localization knowledge from refined distributions to shallower layers through self-distillation, while also simplifying the residual prediction tasks for deeper layers. Additionally, D-FINE incorporates lightweight optimizations in computationally intensive modules and operations, achieving a better balance between speed and accuracy. Specifically, D-FINE-L / X achieves 54.0% / 55.8% AP on the COCO dataset at 124 / 78 FPS on an NVIDIA T4 GPU. When pretrained on Objects365, D-FINE-L / X attains 57.1% / 59.3% AP, surpassing all existing real-time detectors. Furthermore, our method significantly enhances the performance of a wide range of DETR models by up to 5.3% AP with negligible extra parameters and training costs. Our code and pretrained models: https://github.com/Peterande/D-FINE .

## 1 Introduction

The demand for real-time object detection has been increasing across various applications (Arani et al., ). Among the most influential real-time detectors are the YOLO series (Redmon et al., ; Wang et al., ; ; Glenn., ; Wang & Liao, ; Wang et al., ; Glenn., ), widely recognized for their efficiency and robust community ecosystem. As a strong competitor, the Detection Transformer (DETR) (Carion et al., ; Zhu et al., ; Liu et al., ; Li et al., ; Zhang et al., ) offers distinct advantages due to its transformer-based architecture, which allows for global context modeling and direct set prediction without reliance on Non-Maximum Suppression (NMS) and anchor boxes. However, they are often hindered by high latency and computational demands (Zhu et al., ; Liu et al., ; Li et al., ; Zhang et al., ). RT-DETR (Zhao et al., ) addresses these limitations by developing a real-time variant, offering an end-to-end alternative to YOLO detectors. Moreover, LW-DETR (Chen et al., ) has shown that DETR can achieve higher performance ceilings than YOLO, especially when trained on large-scale datasets like Objects365 (Shao et al., ).

Despite the substantial progress made in real-time object detection, several unresolved issues continue to limit the performance of detectors. One key challenge is the formulation of bounding box regression. Most detectors predict bounding boxes by regressing fixed coordinates, treating edges as precise values modeled by Dirac delta distributions (Liu et al., ; Ren et al., ; Tian et al., ; Lyu et al., ). While straightforward, this approach fails to model localization uncertainty. Consequently, models are constrained to use L1 loss and IoU loss, which provide insufficient guidance for adjusting each edge independently (Girshick, ). This makes the optimization process sensitive to small coordinate changes, leading to slow convergence and suboptimal performance. Although methods like GFocal (Li et al., ; ) address uncertainty through probability distributions, they remain limited by anchor dependency, coarse localization, and lack of iterative refinement. Another challenge lies in maximizing the efficiency of real-time detectors, which are constrained by limited computation and parameter budgets to maintain speed. Knowledge distillation (KD) is a promising solution, transferring knowledge from larger teachers to smaller students to improve performance without increasing costs (Hinton et al., ). However, traditional KD methods like Logit Mimicking and Feature Imitation have proven inefficient for detection tasks and can even cause performance drops in state-of-the-art models (Zheng et al., ). In contrast, localization distillation (LD) has shown better results for detection. Nevertheless, integrating LD remains challenging due to its substantial training overhead and incompatibility with anchor-free detectors.

To address these issues, we propose D-FINE, a novel real-time object detector that redefines bounding box regression and introduces an effective self-distillation strategy. Our approach tackles the problems of difficult optimization in fixed-coordinate regression, the inability to model localization uncertainty, and the need for effective distillation with less training cost. We introduce Fine-grained Distribution Refinement (FDR) to transform bounding box regression from predicting fixed coordinates to modeling probability distributions, providing a more fine-grained intermediate representation. FDR refines these distributions iteratively in a residual manner, allowing for progressively finer adjustments and improving localization precision. Recognizing that deeper layers produce more accurate predictions by capturing richer localization information within their probability distributions, we introduce Global Optimal Localization Self-Distillation (GO-LSD). GO-LSD transfers localization knowledge from deeper layers to shallower ones with negligible extra training cost. By aligning shallower layers’ predictions with refined outputs from later layers, the model learns to produce better early adjustments, accelerating convergence and improving overall performance. Furthermore, we streamline computationally intensive modules and operations in existing real-time DETR architectures (Zhao et al., ; Chen et al., ), making D-FINE faster and more lightweight. While such modifications typically result in performance loss, FDR and GO-LSD effectively mitigate this degradation, achieving a better balance between speed and accuracy.

Experimental results on the COCO dataset (Lin et al., ) demonstrate that D-FINE achieves state-of-the-art performance in real-time object detection, surpassing existing models in accuracy and efficiency. D-FINE-L and D-FINE-X achieve 54.0% and 55.8% AP, respectively on COCO val2017, running at 124 FPS and 78 FPS on an NVIDIA T4 GPU. After pretraining on larger datasets like Objects365 (Shao et al., ), the D-FINE series attains up to 59.3% AP, surpassing all existing real-time detectors, showcasing both scalability and robustness. Moreover, our method enhances a variety of DETR models by up to 5.3% AP with negligible extra parameters and training costs, demonstrating its flexibility and generalizability. In conclusion, D-FINE pushes the performance boundaries of real-time detectors. By addressing key challenges in bounding box regression and distillation efficiency through FDR and GO-LSD, we offer a meaningful step forward in object detection, inspiring further exploration in the field.

## 2 Related Work

Real-Time / End-to-End Object Detectors. The YOLO series has led the way in real-time object detection, evolving through innovations in architecture, data augmentation, and training techniques (Redmon et al., ; Wang et al., ; ; Glenn., ; Wang & Liao, ; Glenn., ). While efficient, YOLOs typically rely on Non-Maximum Suppression (NMS), which introduces latency and instability between speed and accuracy. DETR (Carion et al., ) revolutionizes object detection by removing the need for hand-crafted components like NMS and anchors. Traditional DETRs (Zhu et al., ; Meng et al., ; Zhang et al., ; Wang et al., ; Liu et al., ; Li et al., ; Chen et al., ; ) have achieved excelling performance but at the cost of high computational demands, making them unsuitable for real-time applications. Recently, RT-DETR (Zhao et al., ) and LW-DETR (Chen et al., ) have successfully adapted DETR for real-time use. Concurrently, YOLOv10 (Wang et al., ) also eliminates the need for NMS, marking a significant shift towards end-to-end detection within the YOLO series.

Distribution-Based Object Detection.
Traditional bounding box regression methods (Redmon et al., ; Liu et al., ; Ren et al., ) rely on Dirac delta distributions, treating bounding box edges as precise and fixed, which makes modeling localization uncertainty challenging. To address this, recent models have employed Gaussian or discrete distributions to represent bounding boxes (Choi et al., ; Li et al., ; Qiu et al., ; Li et al., ), enhancing the modeling of uncertainty. However, these methods all rely on anchor-based frameworks, which limits their compatibility with modern anchor-free detectors like YOLOX (Ge et al., ) and DETR (Carion et al., ). Furthermore, their distribution representations are often formulated in a coarse-grained manner and lack effective refinement, hindering their ability to achieve more accurate predictions.

Knowledge Distillation. Knowledge distillation (KD) (Hinton et al., ) is a powerful model compression technique. Traditional KD typically focuses on transferring knowledge through Logit Mimicking (Zagoruyko & Komodakis, ; Mirzadeh et al., ; Son et al., ). FitNets (Romero et al., ) initially propose Feature Imitation, which has inspired a series of subsequent works that further expand upon this idea (Chen et al., ; Dai et al., ; Guo et al., ; Li et al., ; Wang et al., ). Most approaches for DETR (Chang et al., ; Wang et al., ) incorporate hybrid distillations of both logit and various intermediate representations. Recently, localization distillation (LD) (Zheng et al., ) demonstrates that transferring localization knowledge is more effective for detection tasks. Self-distillation (Zhang et al., ; ) is a special case of KD which enables earlier layers to learn from the model’s own refined outputs, requiring far fewer additional training costs since there’s no need to separately train a teacher model.

## 3 Preliminaries

Bounding box regression in object detection has traditionally relied on modeling Dirac delta distributions, either using centroid-based $\{x,y,w,h\}$ or edge-distance $\{\mathbf{c},\mathbf{d}\}$ forms, where the distances $\mathbf{d}=\{t,b,l,r\}$ are measured from the anchor point $\mathbf{c}=\{x_{c},y_{c}\}$. However, the Dirac delta assumption, which treats bounding box edges as precise and fixed, makes it difficult to model localization uncertainty, especially in ambiguous cases. This rigid representation not only limits optimization but also leads to significant localization errors with small prediction shifts.

To address these problems, GFocal (Li et al., ; ) regresses the distances from anchor points to the four edges using discretized probability distributions, offering a more flexible modeling of the bounding box. In practice, bounding box distances $\mathbf{d}=\{t,b,l,r\}$ is modeled as:

$$ $\mathbf{d}=d_{\max}\sum_{n=0}^{N}\frac{n}{N}\mathbf{P}(n),$ (1) $$

where $d_{\max}$ is a scalar that limits the maximum distance from the anchor center, and $\mathbf{P}(n)$ denotes the probability of each candidate distance of four edges.
While GFocal introduces a step forward in handling ambiguity and uncertainty through probability distributions, specific challenges in its regression approach persist:
(1) Anchor Dependency: Regression is tied to the anchor box center, limiting prediction diversity and compatibility with anchor-free frameworks.
(2) No Iterative Refinement: Predictions are made in one shot without iterative refinements, reducing regression robustness.
(3) Coarse Localization: Fixed distance ranges and uniform bin intervals can lead to coarse localization, especially for small objects, because each bin represents a wide range of possible values.

Localization Distillation (LD) is a promising approach, demonstrating that transferring localization knowledge is more effective for detection tasks (Zheng et al., ). Built upon GFocal, it enhances student models by distilling valuable localization knowledge from teacher models, rather than simply mimicking classification logits or feature maps. Despite its advantages, the method still relies on anchor-based architectures and incurs additional training costs.

## 4 Method

We propose D-FINE, a powerful real-time object detector that excels in speed, size, computational cost, and accuracy. D-FINE addresses the shortcomings of existing bounding box regression approaches by leveraging two key components: Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD), which work in tandem to significantly enhance performance with negligible additional parameters and training time cost.

(1) FDR iteratively optimizes probability distributions that act as corrections to the bounding box predictions, providing a more fine-grained intermediate representation. This approach captures and optimizes the uncertainty of each edge independently. By leveraging the non-uniform weighting function, FDR allows for more precise and incremental adjustments at each decoder layer, improving localization accuracy and reducing prediction errors. FDR operates within an anchor-free, end-to-end framework, enabling a more flexible and robust optimization process.

(2) GO-LSD distill localization knowledge from refined distributions into shallower layers. As training progresses, the final layer produces increasingly precise soft labels. Shallower layers align their predictions with these labels through GO-LSD, leading to more accurate predictions. As early-stage predictions improve, the subsequent layers can focus on refining smaller residuals. This mutual reinforcement creates a synergistic effect, leading to progressively more accurate localization.

To further enhance the efficiency of D-FINE, we streamline computationally intensive modules and operations in existing real-time DETR architectures (Zhao et al., ), making D-FINE faster and more lightweight. Although these modifications typically result in some performance loss, FDR and GO-LSD effectively mitigate this degradation. The detailed modifications are listed in Table 3.

### 4.1 Fine-grained Distribution Refinement

Fine-grained Distribution Refinement (FDR) iteratively optimizes a fine-grained distribution generated by the decoder layers, as shown in Figure 2. Initially, the first decoder layer predicts preliminary bounding boxes and preliminary probability distributions through a traditional bounding box regression head and a D-FINE head (Both heads are MLP, only the output dimensions are different). Each bounding box is associated with four distributions, one for each edge. The initial bounding boxes serve as reference boxes, while subsequent layers focus on refining them by adjusting distributions in a residual manner. The refined distributions are then applied to adjust the four edges of the corresponding initial bounding box, progressively improving its accuracy with each iteration.

Figure: Figure 2: Overview of D-FINE with FDR. The probability distributions that act as a more fine-grained intermediate representation are iteratively refined by the decoder layers in a residual manner. Non-uniform weighting functions are applied to allow for finer localization.
Refer to caption: x1.png

Mathematically, let $\mathbf{b}^{0}=\{x,y,W,H\}$ denote the initial bounding box prediction, where $\{x,y\}$ represents the predicted center of the bounding box, and $\{W,H\}$ represent the width and height of the box. We can then convert $\mathbf{b}^{0}$ into the center coordinates $\mathbf{c}^{0}=\{x,y\}$ and the edge distances $\mathbf{d}^{0}=\{t,b,l,r\}$, which represent the distances from the center to the top, bottom, left, and right edges. For the $l$-th layer, the refined edge distances $\mathbf{d}^{l}=\{t^{l},b^{l},l^{l},r^{l}\}$ are computed as:

$$ $\mathbf{d}^{l}=\mathbf{d}^{0}+\{H,H,W,W\}\cdot\sum_{n=0}^{N}W(n)\mathbf{Pr}^{l }(n),\quad l\in\{1,2,\dots,L\},$ (2) $$

where $\mathbf{Pr}^{l}(n)=\{\Pr_{t}^{l}(n),\Pr_{b}^{l}(n),\Pr_{l}^{l}(n),\Pr_{r}^{l}(
n)\}$ represents four separate distributions, one for each edge. Each distribution predicts the likelihood of candidate offset values for the corresponding edge. These candidates are determined by the weighting function $W(n)$, where $n$ indexes the discrete bins out of $N$, with each bin corresponding to a potential edge offset. The weighted sum of the distributions produces the edge offsets. These edge offsets are then scaled by the height $H$ and width $W$ of the initial bounding box, ensuring the adjustments are proportional to the box size.

The refined distributions are updated using residual adjustments, defined as follows:

$$ $\mathbf{Pr}^{l}(n)=\text{Softmax}\left(\textit{logits}^{l}(n)\right)=\text{ Softmax}\left(\Delta\textit{logits}^{l}(n)+\textit{logits}^{l-1}(n)\right),$ (3) $$

where logits from the previous layer $\textit{logits}^{l-1}(n)$ reflect the confidence in each bin’s offset value for the four edges. The current layer predicts the residual logits $\Delta\textit{logits}^{l}(n)$, which are added to the previous logits to form updated logits $\textit{logits}^{l}(n)$. These updated logits are then normalized using the softmax function, producing the refined probability distributions.

To facilitate precise and flexible adjustments, the weighting function $W(n)$ is defined as:

$$ $W(n)=\begin{cases}2\cdot W(1)=-2a&\quad n=0\vspace{5pt}\\ c-c\left(\frac{a}{c}+1\right)^{\frac{N-2n}{N-2}}\quad&1\leq n<\frac{N}{2} \vspace{5pt}\\ -c+c\left(\frac{a}{c}+1\right)^{\frac{-N+2n}{N-2}}\quad&\frac{N}{2}\leq n\leq N -1\vspace{5pt}\\ 2\cdot W(N-1)=2a&\quad n=N,\end{cases}$ (4) $$

where $a$ and $c$ are hyper-parameters controlling the upper bounds and curvature of the function. As shown in Figure 2, the shape of $W(n)$ ensures that when bounding box prediction is near accurate, small curvature in $W(n)$ allows for finer adjustments. Conversely, if the bounding box prediction is far from accurate, the larger curvature near the edges and the sharp changes at the boundaries of $W(n)$ ensure sufficient flexibility for substantial corrections.

To further improve the accuracy of our distribution predictions and align them with ground truth values, inspired by Distribution Focal Loss (DFL) (Li et al., ), we propose a new loss function, Fine-Grained Localization (FGL) Loss, which is computed as:

$$ $\displaystyle\mathcal{L}_{\text{FGL}}=\sum_{l=1}^{L}\left(\sum_{k=1}^{K}\text{ IoU}_{k}\left(\omega_{\leftarrow}\cdot\textbf{CE}\left(\mathbf{Pr}^{l}(n)_{k}, n_{\leftarrow}\right)+\omega_{\rightarrow}\cdot\textbf{CE}\left(\mathbf{Pr}^{l }(n)_{k},n_{\rightarrow}\right)\right)\right)$ $\displaystyle\omega_{\leftarrow}=\frac{|\phi-W(n_{\rightarrow})|}{|W(n_{ \leftarrow})-W(n_{\rightarrow})|},\quad\omega_{\rightarrow}=\frac{|\phi-W(n_{ \leftarrow})|}{|W(n_{\leftarrow})-W(n_{\rightarrow})|},\quad\quad\quad\quad\quad$ (5) $$

where $\mathbf{Pr}^{l}(n)_{k}$ represents the probability distributions corresponding to the $k$-th prediction. $\phi$ is the relative offset calculated as $\phi=(\mathbf{d}^{GT}-\mathbf{d}^{0})/\{H,H,W,W\}$. $\mathbf{d}^{GT}$ represents the ground truth edge-distance and $n_{\leftarrow},n_{\rightarrow}$ are the bin indices adjacent to $\phi$. The cross-entropy (CE) losses with weights $\omega_{\leftarrow}$ and $\omega_{\rightarrow}$ ensure that the interpolation between bins aligns precisely with the ground truth offset. By incorporating IoU-based weighting, FGL loss encourages distributions with lower uncertainty to become more concentrated, resulting in more precise and reliable bounding box regression.

Figure: Figure 3: Overview of GO-LSD process. Localization knowledge from the final layer’s refined distributions is distilled into shallower layers through DDF loss with decoupled weighting strategies.
Refer to caption: x2.png

### 4.2 Global Optimal Localization Self-Distillation

Global Optimal Localization Self-Distillation (GO-LSD) utilizes the final layer’s refined distribution predictions to distill localization knowledge into the shallower layers, as shown in Figure 3.
This process begins by applying Hungarian Matching (Carion et al., ) to the predictions from each layer, identifying the local bounding box matches at every stage of the model. To perform a global optimization, GO-LSD aggregates the matching indices from all layers into a unified union set. This union set combines the most accurate candidate predictions across layers, ensuring that they all benefit from the distillation process. In addition to refining the global matches, GO-LSD also optimizes unmatched predictions during training to improve overall stability, leading to improved overall performance. Although the localization is optimized through this union set, the classification task still follows a one-to-one matching principle, ensuring that there are no redundant boxes. This strict matching means that some predictions in the union set are well-localized but have low confidence scores. These low-confidence predictions often represent candidates with precise localization, which still need to be distilled effectively.

To address this, we introduce Decoupled Distillation Focal (DDF) Loss, which applies decoupled weighting strategies to ensure that high-IoU but low-confidence predictions are given appropriate weight. The DDF Loss also weights matched and unmatched predictions according to their quantity, balancing their overall contribution and individual losses. This approach results in more stable and effective distillation.
The Decoupled Distillation Focal Loss $\mathcal{L}_{\text{DDF}}$ is then formulated as:

$$ $\displaystyle\mathcal{L}_{\text{DDF}}=T^{2}\sum_{l=1}^{L-1}\left({\sum_{k=1}^{ K_{m}}\alpha_{k}\cdot\textbf{KL}\left(\mathbf{Pr}^{l}(n)_{k},\mathbf{Pr}^{L}(n )_{k}\right)+\sum_{k=1}^{K_{u}}\beta_{k}\cdot\textbf{KL}\left(\mathbf{Pr}^{l}( n)_{k},\mathbf{Pr}^{L}(n)_{k}\right)}\right)$ $\displaystyle\alpha_{k}=\text{IoU}_{k}\cdot\frac{\sqrt{K_{m}}}{\sqrt{K_{m}}+ \sqrt{K_{u}}},\quad\beta_{k}=\text{Conf}_{k}\cdot\frac{\sqrt{K_{u}}}{\sqrt{K_{ m}}+\sqrt{K_{u}}},\quad\quad\quad\quad\quad\quad\quad$ (6) $$

where KL represents the Kullback-Leibler divergence (Hinton et al., ), and $T$ is the temperature parameter used for smoothing logits. The distillation loss for the $k$-th matched prediction is weighted by $\alpha_{k}$, where $K_{m}$ and $K_{u}$ are the numbers of matched and unmatched predictions, respectively. For the $k$-th unmatched prediction, the weight is $\beta_{k}$, with $\text{Conf}_{k}$ denoting the classification confidence.

## 5 Experiments

### 5.1 Experiment Setup

To validate the effectiveness of our proposed methods, we conduct experiments on the COCO (Lin et al., ) and Objects365 (Shao et al., ) datasets. We evaluate our D-FINE using the standard COCO metrics, including Average Precision (AP) averaged over IoU thresholds from 0.50 to 0.95, as well as AP at specific thresholds (AP50 and AP75) and AP across different object scales: small (APS), medium (APM), and large (APL). Additionally, we provide model efficiency metrics by reporting the number of parameters (#Params.), computational cost (GFLOPs), and end-to-end latency. The latency is measured using TensorRT FP16 on an NVIDIA T4 GPU.

Table: Table 1: Performance comparison of various real-time object detectors on COCO val2017.

### 5.2 Comparison with Real-Time Detectors

Table 1 provides a comprehensive comparison between D-FINE and various real-time object detectors on COCO val2017. D-FINE achieves an excellent balance between efficiency and accuracy across multiple metrics. Specifically, D-FINE-L attains an AP of 54.0% with 31M parameters and 91 GFLOPs, maintaining a low latency of 8.07 ms. Additionally, D-FINE-X achieves an AP of 55.8% with 62M parameters and 202 GFLOPs, operating with a latency of 12.89 ms.

As depicted in Figure 1, which shows scatter plots of latency vs. AP, parameter count vs. AP, and FLOPs vs. AP, D-FINE consistently outperforms other state-of-the-art models across all key dimensions. D-FINE-L achieves a higher AP (54.0%) compared to YOLOv10-L (53.2%), RT-DETR-R50 (53.1%), and LW-DETR-X (53.0%), while requiring fewer computational resources (91 GFLOPs vs. 120, 136, and 174). Similarly, D-FINE-X surpasses YOLOv10-X and RT-DETR-R101 by achieving superior performance (55.8% AP vs. 54.4% and 54.3%) and demonstrating greater efficiency in terms of lower parameter count, GFLOPs, and latency.

We further pretrain D-FINE and YOLOv10 on the Objects365 dataset (Shao et al., ), before finetuning them on COCO. After pretraining, both D-FINE-L and D-FINE-X exhibit significant performance improvements, achieving AP of 57.1% and 59.3%, respectively. These enhancements enable them to outperform YOLOv10-L and YOLOv10-X by 3.1% and 4.4% AP, thereby positioning them as the top-performing models in this comparison. What’s more, following the pretraining protocol of YOLOv8 (Glenn., ), YOLOv10 is pretrained on Objects365 for 300 epochs. In contrast, D-FINE requires only 21 epochs to achieve its substantial performance gains. These findings corroborate the conclusions of LW-DETR (Chen et al., ), demonstrating that DETR-based models benefit substantially more from pretraining compared to other detectors like YOLOs.

### 5.3 Effectiveness on various DETR models

Table 2 demonstrates the effectiveness of our proposed FDR and GO-LSD methods across multiple DETR-based object detectors on COCO val2017. Our methods are designed for flexibility and can be seamlessly integrated into any DETR architecture, significantly enhancing performance without increasing the number of parameters and computational burden. Incorporating FDR and GO-LSD into Deformable DETR, DAD-DETR, DN-DETR, and DINO consistently improves detection accuracy, with gains ranging from 2.0% to 5.3%. These results highlight the effectiveness of FDR and GO-LSD in enhancing localization precision and maximizing efficiency, demonstrating their adaptability and substantial impact across various end-to-end detection frameworks.

Table: Table 2: Effectiveness of FDR and GO-LSD across various DETR models on COCO val2017.

### 5.4 Ablation Study

#### 5.4.1 The Roadmap to D-FINE

Table 3 showcases the stepwise progression from the baseline model (RT-DETR-HGNetv2-L (Zhao et al., )) to our proposed D-FINE framework. Starting with the baseline metrics of 53.0% AP, 32M parameters, 110 GFLOPs, and 9.25 ms latency, we first remove all the decoder projection layers. This modification reduces GFLOPs to 97 and cuts the latency to 8.02 ms, although it decreases AP to 52.4%. To address this drop, we introduce the Target Gating Layer, which recovers the AP to 52.8% with only a marginal increase in computational cost.

The Target Gating Layer is strategically placed after the decoder’s cross-attention module, replacing the residual connection. It allows queries to dynamically switch their focus on different targets across layers, effectively preventing information entanglement. The mechanism operates as follows:

$$ $\mathbf{x}=\sigma\left(\begin{bmatrix}\mathbf{x_{1}},\mathbf{x_{2}}\end{ bmatrix}\mathbf{W}^{T}+\mathbf{b}\right)_{1}\cdot\mathbf{x_{1}}+\sigma\left( \begin{bmatrix}\mathbf{x_{1}},\mathbf{x_{2}}\end{bmatrix}\mathbf{W}^{T}+ \mathbf{b}\right)_{2}\cdot\mathbf{x_{2}}$ (7) $$

where $\mathbf{x_{1}}$ represents the previous queries and $\mathbf{x_{2}}$ is the cross-attention result. $\sigma$ is the sigmoid activation function applied to the concatenated outputs, and $[.]$ represents the concatenation operation.

Next, we replace the encoder’s CSP layers with GELAN layers (Wang & Liao, ). This substitution increases AP to 53.5% but also raises the parameter count, GFLOPs, and latency. To mitigate the increased complexity, we reduce the hidden dimension of GELAN, which balances the model’s complexity and maintains AP at 52.8% while improving efficiency. We further optimize the sampling points by implementing uneven sampling across different scales (S: 3, M: 6, L: 3), which slightly increases AP to 52.9%. However, alternative sampling combinations such as (S: 6, M: 3, L: 3) and (S: 3, M: 3, L: 6) result in a minor performance drop of 0.1% AP. Adopting the RT-DETRv2 training strategy (Lv et al., ) (see Section A.1.1 for details) enhances AP to 53.0% without affecting the number of parameters or latency. Finally, the integration of FDR and GO-LSD modules elevates AP to 54.0%, achieving a 13% reduction in latency and a 17% reduction in GFLOPs compared to the baseline model. These incremental modifications demonstrate the robustness and effectiveness of our D-FINE framework.

Table: Table 3: Step-by-step modifications from baseline model to D-FINE. Each step shows changes in AP, the number of parameters, latency, and FLOPs.

**Table 4: Hyperparameter ablation studies on D-FINE-L. $\epsilon$ is a very small value. $\widetilde{a}$, $\widetilde{c}$ indicate that $a$ and $c$ are learnable parameters.**
| $a$, $c$ | $\tfrac{1}{4}$, $\tfrac{1}{4}$ | $\tfrac{1}{2}$, $\tfrac{1}{\epsilon}$ | $\mathbf{\tfrac{1}{2}}$, $\mathbf{\tfrac{1}{4}}$ | $\tfrac{1}{2}$,  $\tfrac{1}{8}$ | $1$,  $\tfrac{1}{4}$ | $\widetilde{a}$, $\widetilde{c}$ |
| --- | --- | --- | --- | --- | --- | --- |
| AP^val | 52.7 | 53.0 | 53.3 | 53.2 | 53.2 | 53.1 |
| $N$ | 4 | 8 | 16 | 32 | 64 | 128 |
| AP^val | 53.3 | 53.4 | 53.5 | 53.7 | 53.6 | 53.6 |
| $T$ | 1 | 2.5 | 5 | 7.5 | 10 | 20 |
| AP^val | 53.2 | 53.7 | 54.0 | 53.8 | 53.7 | 53.5 |

#### 5.4.2 Hyperparameter Sensitivity Analysis

Table 5 presents a subset of hyperparameter ablation studies evaluating the sensitivity of our model to key parameters in the FDR and GO-LSD modules. We examine the weighting function parameters $a$ and $c$, the number of distribution bins $N$, and the temperature $T$ used for smoothing logits in the KL divergence.

(1) Setting $a=\tfrac{1}{2}$ and $c=\tfrac{1}{4}$ yields the highest AP of 53.3%. Notably, treating $a$ and $c$ as learnable parameters ($\tilde{a},\tilde{c}$) slightly decreases AP to 53.1%, suggesting that fixed values simplify the optimization process. When $c$ is extremely large, the weighting function approximates the linear function with equal intervals, resulting in a suboptimal AP of 53.0%. Additionally, values of $a$ that are too large or too small can reduce fineness or limit flexibility, adversely affecting localization precision. (2) Increasing the number of distribution bins improves performance, with a maximum AP of 53.7% achieved at $N=32$. Beyond $N=32$, no significant gain is observed.
(3) The temperature $T$ controls the smoothing of logits during distillation. An optimal AP of 54.0% is achieved at $T=5$, indicating a balance between softening the distribution and preserving effective knowledge transfer.

#### 5.4.3 Comparison of Distillation Methods

Table 5 compares different distillation methods based on performance, training time, and GPU memory usage. The baseline model achieves an AP of 53.0%, with a training time of 29 minutes per epoch and memory usage of 8552 MB on four NVIDIA RTX 4090 GPUs. Due to the instability of one-to-one matching in DETR, traditional distillation techniques like Logit Mimicking and Feature Imitation do not improve performance; Logit Mimicking reduces AP to 52.6%, while Feature Imitation achieves 52.9%. Incorporating our FDR module increases AP to 53.8% with minimal additional training cost. Applying vanilla Localization Distillation (Zheng et al., ) further increases AP to 53.7%. Our GO-LSD method achieves the highest AP of 54.5%, with only a 6% increase in training time and a 2% rise in memory usage compared to the baseline. Notably, no lightweight optimizations are applied in this comparison, focusing purely on distillation performance.

Figure: Figure 4: Visualization of FDR across detection scenarios with initial and refined bounding boxes, along with unweighted and weighted distributions, highlighting improved localization accuracy.
Refer to caption: extracted/5934706/fig/Pred_000000089648.jpg

### 5.5 Visualization Analysis

Figure 4 illustrates the process of FDR across various detection scenarios. We display the filtered detection results with two bounding boxes overlaid on the images. The red boxes represent the initial predictions from the first decoder layer, while the green boxes denote the refined predictions from the final decoder layer. The final predictions align more closely with the target objects. The first row under the images shows the unweighted probability distributions for the four edges (left, top, right, bottom). The second row shows the weighted distributions, where the weighting function $W(n)$ has been applied. The red curves represent the initial distributions, while the green curves show the final, refined distributions. The weighted distributions emphasize finer adjustments near accurate predictions and allow for enabling rapid changes for larger adjustments, further illustrating how FDR refines the offsets of initial bounding boxes, leading to increasingly precise localization.

## 6 Conclusion

In this paper, we introduce D-FINE, a powerful real-time object detector that redefines the bounding box regression task in DETR models through Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). Experimental results on the COCO dataset demonstrate that D-FINE achieves state-of-the-art accuracy and efficiency, surpassing all existing real-time detectors.
Limitation and Future Work: However, the performance gap between lighter D-FINE models and other compact models remains small. One possible reason is that shallow decoder layers may yield less accurate final-layer predictions, limiting the effectiveness of distilling localization knowledge into earlier layers. Addressing this challenge necessitates enhancing the localization capabilities of lighter models without increasing inference latency. Future research could investigate advanced architectural designs or novel training paradigms that allow for the inclusion of additional sophisticated decoder layers during training while maintaining lightweight inference by simply discarding them at test time. We hope D-FINE inspires further advancements in this area.

## Appendix A Appendix

### A.1 Implementation Details

#### A.1.1 Hyperparameter Configurations

Table 6 summarizes the hyperparameter configurations for the D-FINE models. All variants use HGNetV2 backbones pretrained on ImageNet (Cui et al., ; Russakovsky et al., ) and the AdamW optimizer. D-FINE-X is set with an embedding dimension of 384 and a feedforward dimension of 2048, while the other models use 256 and 1024, respectively. The D-FINE-X and D-FINE-L have 6 decoder layers, while D-FINE-M and D-FINE-S have 4 and 3 decoder layers. The GELAN module progressively reduces hidden dimension and depth from D-FINE-X with 192 dimensions and 3 layers to D-FINE-S with 64 dimensions and 1 layer. The base learning rate and weight decay for D-FINE-X and D-FINE-L are $2.5\times 10^{-4}$ and $1.25\times 10^{-4}$, respectively, while D-FINE-M and D-FINE-S use $2\times 10^{-4}$ and $1\times 10^{-4}$. Smaller models also have higher backbone learning rates than larger models. The total batch size is 32 across all variants. Training schedules include 72 epochs with advanced augmentation (RandomPhotometricDistort, RandomZoomOut, RandomIoUCrop, and RMultiScaleInput) followed by 2 epochs without advanced augmentation for D-FINE-X and D-FINE-L, and 120 epochs with advanced augmentation followed by 4 epochs without advanced augmentation for D-FINE-M and D-FINE-S (RT-DETRv2 Training Strategy (Lv et al., ) in Table 3). The number of pretraining epochs is 21 for D-FINE-X and D-FINE-L models, while for D-FINE-M and D-FINE-S models, it ranges from 28 to 29 epochs.

Table: Table 6: Hyperparameter configurations for different D-FINE models.

#### A.1.2 Datasets Settings

For pretraining, following the approach in (Chen et al., ; Zhang et al., ; Chen et al., ), we combine the images from the Objects365 (Shao et al., ) train set with the validate set, excluding the first 5k images. To further improve training efficiency, we resize all images with resolutions exceeding $640\times 640$ down to $640\times 640$ beforehand. We use the standard COCO2017 (Lin et al., ) data splitting policy, training on COCO train2017, and evaluating on COCO val2017.

### A.2 Visualization of D-FINE Predictions

Figure 5 demonstrates the robustness of the D-FINE-X model, visualizing its predictions in various challenging scenarios. These include occlusion, low-light conditions, motion blur, depth of field effects, rotation, and densely populated scenes with numerous objects in close proximity. Despite these difficulties, the model accurately identifies and localizes objects, such as animals, vehicles, and people. This visualization highlights the model’s ability to handle complex real-world conditions while maintaining robust detection performance.

Figure: Figure 5: Visualization of D-FINE-X (without pre-training on Objects365) predictions under challenging conditions, including occlusion, low light, motion blur, depth of field effects, rotation, and densely populated scenes (confidence threshold=0.5).
Refer to caption: x3.png

### A.3 Comparison with Lighter Detectors

Table 7 presents a comprehensive comparison of D-FINE models with various lightweight real-time object detectors in the S and M model sizes on the COCO val2017. D-FINE-S achieves an impressive AP of 48.5%, surpassing other lightweight models such as Gold-YOLO-S (46.4%) and RT-DETRv2-S (48.1%), while maintaining a low latency of 3.49 ms with only 10.2M parameters and 25.2 GFLOPs. Pretraining on Objects365 further boosts D-FINE-S to 50.7%, marking an improvement of +2.2%. Similarly, D-FINE-M attains an AP of 52.3% with 19.2M parameters and 56.6 GFLOPs at 5.62 ms, outperforming YOLOv10-M (51.1%) and RT-DETRv2-M (49.9%). Pretraining on Objects365 consistently enhances D-FINE-M, yielding a +2.8% gain. These results demonstrate that D-FINE models strike an excellent balance between accuracy and efficiency, consistently surpassing other state-of-the-art lightweight detectors while preserving real-time performance.

Table: Table 7: Performance comparison of S and M sized real-time object detectors on COCO val2017.

### A.4 Clarification on the Initial Layer Refinement

In the main text, we define the refined distributions at layer $l$ as:

$$ $\mathbf{Pr}^{l}(n)=\text{Softmax}\left(\Delta\text{logits}^{l}(n)+\text{logits }^{l-1}(n)\right),$ (8) $$

where $\Delta\text{logits}^{l}(n)$ are the residual logits predicted by layer $l$, and $\text{logits}^{l-1}(n)$ are the logits from the previous layer.

For the initial layer ($l=1$), there is no previous layer, so the formula simplifies to:

$$ $\mathbf{Pr}^{1}(n)=\text{Softmax}\left(\text{logits}^{1}(n)\right).$ (9) $$

Here, $\text{logits}^{1}(n)$ are the logits predicted by the first layer.

This clarification ensures the formulation is consistent and mathematically rigorous for all layers.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 2
strengths: The paper is easy to understand and well-presented. A set of experiments are conducted on the COCO and Objects365 datasets.
summary: This paper introduces a real-time object detector based on DETR through two main components: fine-grained distribution refinement, which refines probability distributions iteratively for improved accuracy, and global optimal localization self-distillation, which optimizes localization through a bidirectional self-distillation approach.
weaknesses: 1. I do not fully agree with the claim that using fixed coordinates results in poor performance due to inadequate modelling of localization uncertainty. While the statement shows potential limitations of fixed-coordinate regression, many SOTA object detectors (Faster R-CNN and RetinaNet), represent significant advances in both speed and detection. Therefore, to say that all methods using fixed coordinates were fundamentally limited overlooks these advancements. The authors may clarify their claim by acknowledging the success of fixed-coordinate methods while explaining how the proposed approach can address these limitations

2. It has been stated, " As early predictions improve, the subsequent layers can focus on refining smaller residuals.", what do you mean by  "refining smaller residuals"?

3. In page 5, Eq. (3).  The normalizing updated logits can result in very small gradients when logits are large, making it harder for the model to learn effectively in deeper layers. This can reduce the model’s precision in refining bounding box edges. Moreover, softmax forces the output to sum to one, which can limit its ability to model complex relationships between bins and reduce its performance for handling localization uncertainty. Have you tried any other normalization methods such as Entmax normalization which well fit into DERT methods? and why softmax has been chosen over other options?

4. In Figure 4 what those sub-plots (right, left, ...) mean and represent? There is no discussion about these plots. It is not clear why these plots are shown and what they are representing. Moreover, it is unclear, what the pick value means and what different lines (red, green) are supposed to be for the best performance. The authors need to provide a more detailed caption or explanation for this figure, specifically describing what each subplot represents and the meaning of the different colored lines.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How many D-FINE heads are used in the 1-st decoder, e.g., the value of $L$? Is FDR applied within each decoder layer or across decoder layers? 

2. If FDR is applied differently across different decoders, whether the number of D-FINE heads is consistent across all decoder layers or if it varies? How and why it differs? I suggest that the authors add this information to provide a clearer picture of the model's architecture and operation would enhance the reproducibility of the proposed method.
soundness: 3
strengths: 1. The proposed method is technically sound, which is supported by sufficient experiments.

2. The paper shows the roadmap from the baseline model to the proposed D-FINE framework, making the technical contribution clear and transparent.

3. The proposed D-FINE achieves state-of-the-art performance in real-time object detection.
summary: The paper proposes D-FINE, which consists of two methods, Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD) to improve the performance of real-time object detectors. Based on the probability distribution representation of bbox, the paper uses multiple D-FINE head to predict the prob. distributions of bboxes on $L$ layers in one decoder. FDR uses a hand-crafted weight function to weight the prob. distribution of bbox and then iteratively refine the bboxes. GO-LSD is inspired from localization distillation, which distills the localization knowledge from the prob. distribution of bboxes of the last layer to the ones of the shallow layers. The paper also designs two losses to work with the proposed FDR and GO-LSD. Experiments on COCO benchmark show the effectiveness of the proposed method.
weaknesses: 1. It is confusing to me that in Fig. 2, it seems that D-FINE with FDR is applied to the 1-st decoder layer of the object detector. Then, within 1 decoder layer, multiple D-FINE head is used to generate the prob. distribution of bboxes. However, on lines 207-208, you mention "the first decoder layer predicts preliminary bounding boxes and preliminary probability distributions through a traditional bounding box regression head and **a D-FINE head**". In Sec. 4.2,  GO-LSD utilizes the final layer’s refined distribution predictions to distill localization knowledge into the earlier layers. One can see in Fig. 3 that the self-distillation is conducted between different decoders. Thus, is that the meaning of the word "layer" different in FDR and GO-LSD? What is the exact meaning of "layer" in the context of FDR versus GO-LSD? 

2. In Fig. 3, the matched prediction and unmatched prediction are colored by green squares and yellow squares. What does the gray squares stand for? I suggest that the authors could explicitly state what the gray squares represent in the caption or legend of Figure 3.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. The paper presents D-FINE, an enhancement to the DETR framework that tackles bounding box regression with a novel fine-grained distribution refinement and self-distillation mechanism. 

2. The experimental results on the COCO dataset demonstrate competitive performance, showing that D-FINE outperforms many existing real-time object detectors, with a favorable trade-off between speed and accuracy. Additionally, the paper’s ablation studies and hyperparameter tuning offer insightful explanations of the model’s design choices.

3. The experiments and comparisons with various DETR-based models are thorough. The inclusion of both small and large models, alongside real-world visualizations, strengthens the validity of the claims about D-FINE's performance across diverse conditions.
summary: The paper introduces D-FINE, a real-time object detection model that enhances DETR by redefining bounding box regression with Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). The FDR transforms bounding box predictions into refined probability distributions, while GO-LSD uses refined localization knowledge to improve earlier layers' predictions. Experiments on the COCO dataset show that D-FINE achieves good performance and high speed, positioning it as a strong competitor to existing real-time detectors.
weaknesses: The experiments focus primarily on the COCO dataset. Evaluating D-FINE across a broader range of datasets could strengthen its generalizability claims and confirm its robustness across different object detection contexts, such as crowded scene CrowdHuman and long tail scene LVIS.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1.	In Table 3, FDR and GO-LSD modules have a 1% mAP improvement in total for D-FINE. However, according to table 2, these two modules improve accuracy by 2% at least. Can the author explain this phenomenon?
soundness: 4
strengths: 1.	D-FINE surpasses all existing real-time detectors with negligible extra parameters and training costs. Source code is provided. This method appears to be highly solid and reproducible.
2.	Table 2 illuminates that the proposed FDR and GO-LSD are efficient for a series of DETR models, demonstrating the robustness of the module.
3.	The proposed method is both innovative and systematic.
summary: D-FINE is a real-time object detection model that refines bounding box regression in Detection Transformers using Fine-grained Distribution Refinement (FDR) and Global Optimal Localization Self-Distillation (GO-LSD). These techniques enhance localization accuracy and efficiency by iteratively refining bounding boxes and transferring refined knowledge across layers. D-FINE outperforms other real-time detectors on the COCO dataset, achieving a better balance of speed and precision with minimal added costs.
weaknesses: 1.	In Table 3, FDR and GO-LSD modules have a 1% mAP improvement in total for D-FINE, which is noticeably less than the improvement shown in Table 2.
2.	The language in the article needs refinement, as some phrases may lead to ambiguity. For example, line 206-208, “Initially, the first …” and line 209, “…one for each edge”.
3.	Variable names in the Method need to be consistent. For example, line 213, {W, H} and line 139 {w,h}. The lowercase ‘w’ is preferable, as the uppercase ‘W’ repeats the notation used for the weighting function below.

Meta-review: (a) The paper presents D-FINE, a real-time object detection model improving DETR. It introduces Fine-grained Distribution Refinement (FDR) for refined bounding box distributions and Global Optimal Localization Self-Distillation (GO-LSD) to enhance earlier predictions. Experiments on COCO show D-FINE is fast and competitive with current detectors.

(b) The paper is clear and well-structured, supported by thorough experiments on COCO and Objects365. D-FINE outperforms many real-time detectors, balancing speed and accuracy effectively. Ablation studies and hyperparameter tuning provide valuable insights. Comprehensive comparisons with DETR-based models and real-world visualizations validate its robustness. 

(c) The main weaknesses of the first version included insufficient explanations and visualizations, ambiguous phrasing, inconsistent variable names, and a lack of experiments on additional object detection datasets. These issues were all resolved during the rebuttal.

(d) The primary reason for acceptance is the solid, practical performance of the model for real-time detection and its innovative approaches, including distribution refinement and localization self-distillation. This work has the potential to significantly impact the object detection community.

Decision: Accept (Spotlight)