Fractional Correspondence Framework in Detection Transformer

Authors: masoumeh  zareapoor
pourya  shamsolmoali
huiyu  zhou
yue  lu
salvador  garcía
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 19:50:56

Abstract: The Detection Transformer (DETR), by incorporating the Hungarian algorithm, has significantly simplified the matching process in object detection tasks. This algorithm facilitates optimal one-to-one matching of predicted bounding boxes to ground-truth annotations during training. While effective, this strict matching process does not inherently account for the varying densities and distributions of objects, leading to suboptimal correspondences such as failing to handle multiple detections of the same object or missing small objects. To address this, we propose the Regularized Transport Plan (RTP). RTP introduces a flexible matching strategy that captures the cost of aligning predictions with ground truths to find the most accurate correspondences between these sets. By utilizing the differentiable Sinkhorn algorithm, RTP allows for soft, fractional matching rather than strict one-to-one assignments. This approach enhances the model's capability to manage varying object densities and distributions effectively. Our extensive evaluations on the MS-COCO and VOC benchmarks demonstrate the effectiveness of our approach. RTP-DETR, surpassing the performance of the Deform-DETR and the recently introduced DINO-DETR, achieving absolute gains in mAP of {\bf{+3.8\%}} and {\bf{+1.7\%}}, respectively.

Main paper: 
 introduction: 
 Object detection aims to identify and localize objects within images across various categories. The advent of deep learning has significantly enhanced object detection, enabling models to achieve high accuracy and robustness across complex settings [41] . Central to the efficacy of these models is the matching process-how predictions are accurately aligned with ground-truth objects. Accurately pairing each predicted object with its ground-truth is a crucial and challenging task [2] . Conventional matching strategies in object detection, such as those used by two-stage detectors (e.g., Faster R-CNN) [27] and one-stage detectors (e.g., YOLO, SSD) [20, 26] , rely on predefined anchor boxes and intricate overlap metrics (e.g., IoU) for aligning predictions with ground-truths [8, 41] . Despite their widespread use, these heuristic-based methods limit the model's ability to learn optimal matching from data directly [2, 13, 18] , and add complexity due to manual anchor and threshold adjustments.The Detection Transformer (DETR) [2] emerges as a promising solution, introducing an end-to-end framework that simplifies the object detection pipeline by using the Hungarian matching algorithm [15] . This algorithm provides unique one-to-one correspondences between predicted and ground-truth objects, optimizing the matching cost under the assumption of equal set sizes. In cases, where the two sets do not have the same size, significant preprocessing is required to construct a square cost matrix [13, 16, 40] . DETR addresses this challenge by generating a fixed number of bounding box predictions for each image, with the model learning to classify excess predictions as "no object". However, this way can not fully handle densely packed objects or significantly small objects, as the algorithm's cost function primarily guides these matches without directly considering the overall spatial and class distribution of objects [1, 7, 19, 25, 40] . It also requires a long training schedule to converge, as illustrated in [7, 42] , and Figure 1 . Several approaches are proposed to address the prediction-ground truth matching in the DETR, each aiming to navigate the complexities of diverse scenes with greater efficacy. Zhao et al. [40] showed that one-to-one matching does not provide direct supervision for generating predicted objects. They introduced MS-DETR which combines one-to-one and one-to-many strategies, enriching training by considering both individual predicted objects and their context. [39] also argues that the one-to-one assignment often falls short due to the variable distributions and densities of the objects.Instead of relying on a fixed threshold to discard unlikely pairs, this work computes a marginal probability for each object pair, and provides a more stable basis for object pairing across diverse conditions. Hou et al. [13] improves the matching technique by introducing a salience score to evaluate the relationship between detected and actual objects. This score is designed to ensure an accurate matching, addressing the Hungarian algorithm's limitations in discriminating between closely situated or similar objects. Rank-DETR [25] addresses the mismatch between confidence scores and localization accuracy of predicted bounding boxes. By prioritizing predictions that have more accurate localization, it seeks to improve the quality of matches between detection and actual objects. DETR-like detectors, despite their remarkable performance, rely on the Hungarian algorithm for one-to-one correspondences between detected objects and ground truths, and they integrate additional strategies to handle one-to-many matching scenarios.Our work goes beyond this traditional method by looking for a reliable way to match sets of predictions with ground-truths of potentially diverse sizes. We preserve the core property of the Hungarian algorithm-minimizing the assignment cost-while extending its capabilities to address fractional assignments and distribution discrepancies efficiently. Our matching technique is based on optimal transport (OT) theory [24] , optimizing the transport plan to minimize the cost across the entire distribution rather than focusing on individual matches. Central to our model is the computation of a transport plan, Γ, where each element Γ 𝑖 𝑗 represents the weight (degrees of matching) between predicted object 𝑖 and ground-truth 𝑗. This plan derives from an optimization process aimed at reducing the overall cost 𝐶 between prediction-ground truth pairs. Unlike the Hungarian algorithm, which treats matching as a binary decision, the transport plan assigns weights to each pair of prediction-ground truth, capturing the degree of match. Then, through these weighted assessments, our model finds the best correspondence between prediction and actual objects. OT, by considering the distribution of all predictions and ground-truths, allowing for a more nuanced, probabilistic matching that accurately detects small objects that can be overlooked under a strict one-to-one matching scheme.In summary: 1 We verify that DETR with the Hungarian algorithm suffers from slow convergence and accurate matching for complex senses. 2 We propose a regularized transport plan to find the best alignment between predictions and ground-truths, showing how regularization can improve the convergence (see Figure 3 ). 3 Our experimental results show that our model outperforms existing DETR-based methods, including Deform-DETR [42] , DN-DETR [16] , Rank-DETR [25] , and other training-efficient variants DINO-DETR [38] and Stable-DETR [19] . 
matching flexibility: 
 In any matching process, it is essential to establish a matching cost that quantifies the degree of alignment between two sets. Figure 2 compares various matching strategies on an image from the VOC training dataset. The matching cost for each pair of prediction and ground-truth bounding boxes is calculated using the Generalized Intersection over Union (GIoU). In the input image-(a) groundtruth objects are color-coded and the prediction boxes are in black. (b) displays a cost matrix that measures the matching between predictions and ground-truths, where rows (1-5) represent predicted bounding boxes and columns (A, B) denote ground-truth boxes. Column C is the background cost used in the DETR models for cases of excess predictions over ground-truth objects, guiding the model on which predictions to discard or classify as background. The color intensity is the magnitude of the cost, where a dense color denotes a higher cost (poor matching), and a lighter color indicates a lower cost (better matches). With a background cost set at 0.75, any prediction-ground truth pair exceeding this threshold is considered a match with the background and it is a false positive.(c) shows the RTP with regularization 𝜖 = 0 and 𝜅 2 = 0. This allows some ground-truths to receive multiple matches, while others get none. This is effective when the goal is to reduce false positives, ensuring predictions closely align with actual objects. (d) illustrates the one-to-one matching by the Hungarian algorithm, ensuring each ground-truth pairs with a single prediction. This is ideal for cases requiring strict correspondence between predictions and ground-truths without duplicates. However, it may not capture the full complexity of object interactions or the presence of multiple objects in close proximity [7, 13, 25, 40] . It only enforces a match based on the lowest cost function but misses complex details due to the constraints of choosing a single best match for each prediction or ground-truth. Moreover, the Hungarian algorithm does not account for the gradual improvement of predictions during training, slowing the model convergence [1, 13, 19, 42] . (e) RTP with 𝜖 = 0 and 𝜅 1 = 0 demonstrates that each ground-truth is matched with the best prediction, ensuring all actual objects are detected. As a result, a single prediction can match with multiple ground-truths, which is ideal for cases where missing any object is a greater concern than duplicate detections. (f-h) present different settings of RTP by adjusting the regularization parameters: (f) sets 𝜖 = 0.05, 𝜅 1 = 100, and 𝜅 2 = 0.01; (g) uses 𝜖 = 0.05 with both 𝜅 1 and 𝜅 2 set to zero; (h) uses 𝜖 = 0.05, 𝜅 1 = 0.01, and 𝜅 2 = 100. We can consider these results optimal when there is a need for a balance between precision and recall. This analysis can highlight the advantages of regularized OT over the Hungarian algorithm, especially to manage discrepancies in the number of predictions and ground-truths, and to provide a nuanced matching technique for complex object detection tasks. Our insights align with the findings in [1, 7, 38, 42] . 
related work 3.1 detr with different matching frameworks: 
 DETR [2] revolutionized object detection by presenting it as a set prediction problem, using one-to-one matching supervised by the Hungarian algorithm for end-to-end training. Several subsequent works have proposed to address the slow convergence of DETR from different perspectives. [33] stated that the cross-attention A, B ) and prediction boxes (1-5). We selected an input image with highly overlapped objects. Subfigures (c-h) illustrate the matching through color density, with black indicating a higher match and lighter colors (approaching white) signifying lower or no matches. This visualization helps in understanding the effectiveness of the matching process. See Section 2 for a detailed explanation. mechanism in the decoder is the bottleneck to training efficacy, and suggested an encoder-only architecture as a solution. Gao et al. [10] aimed to streamline the cross-attention process by integrating a Gaussian before regulating attention within the model. Another direction to enhance DETR is to refine its matching strategy, which is more relevant to our work. This focus stems from the critical role of matching in object detection [8, 13] , where accurately pairing each predicted object with its ground-truth during training (as illustrated in Figure 2 ). Deform-DETR [42] enhances the DETR's efficiency and performance, particularly when dealing with small objects or objects that require finer spatial resolution. This model uses Hungarian matching via a new optimization technique to find the best one-to-one correspondence between predicted and ground-truth objects during training. [13] proposed Salient-DETR, a salient score between detected and actual objects to ensure that only the most relevant objects are matched. Stable-DETR [8] reveals that DETR's slow convergence and performance issues stem from an unstable matching problem. It addresses this by introducing a stable matching technique that adjusts matching costs to prioritize positional metrics over semantic scores. DN-DETR [16] uses parallel decoders with shared weights to process multiple sets of noisy queries. These queries are crafted by introducing noise to the ground-truth object, aiming to improve the model's accuracy. Group DETR [4] enhances DETR by adding multiple decoder groups, each designed to handle specific subsets of object queries. Both DN-DETR and Group DETR use one-to-one matching strategy for every group of object queries to ensure accurate alignments. DINO [38] advances this by incorporating dynamic anchors and denoising training, achieving the first state-of-the-art performance on the COCO benchmark among DETR variants. Meanwhile, DETA [23] explores one-to-many assignment strategies by adopting additional decoders, which significantly increase computational demands, as noted by [40] . MS-DERT [40] refines the Hungarian algorithm by implementing mixed supervision, combining one-to-one and oneto-many matching to boost the training efficiency without raising the cost during inference. Different from these approaches, our model seeks optimal alignment by computing transport plans between prediction and ground-truth sets via Sinkhorn's algorithm [6] , to find the best correspondence.Originally, optimal transport (OT) [24] tackles the problem of finding the most cost-effective way to align two sets of points (or distributions). It looks for an optimal coupling (transport plan) between distributions 𝜇 and 𝜈, representing it as a joint probability distribution. In other words, if we define 𝑈 (𝜇, 𝜈) as the space of probability distributions over 𝑅 𝑑 with marginals 𝜇 and 𝜈, the optimal transport is the coupling Γ ∈ 𝑈 (𝜇, 𝜈), which minimizes the following quantity: min⟨Γ, 𝐶 𝑖 𝑗 ⟩, where 𝐶 𝑖 𝑗 is the cost of moving 𝑖 (from 𝜇) to 𝑗 (from 𝜈), respectively. The use of OT has gained popularity in generative modeling [30] , adversarial training [35] , and domain adaptation [3] , and many other disciplines [5, 32, 34] , since the introduction of Sinkhorn's algorithm [6] . Recent efforts in computer vision include matching predicted classes through Wasserstein distance [11] , developing a specialized loss function for handling rotated bounding boxes [36] , and introducing new metrics to evaluate model performance [22] . OT and the Hungarian algorithm, while both aimed at solving the assignment problem by finding the best matches between elements of two sets, differ significantly in their methodological execution. The Hungarian algorithm is designed for one-to-one matching, while, OT uses a probabilistic coupling, allowing the mass of a predicted object to be distributed across multiple ground-truths and vice versa, which is more flexible in capturing challenges in the assignments [3, 5, 7, 31] . 
method: 
 We first revisit important details on the Hungarian algorithm and OT, which will be useful to describe our proposed model. DETR uses the Hungarian algorithm [15] to establish an one-toone pairing between predicted and actual objects by minimizing matching costs. The objective is to find an optimal permutation 𝜎 of the 𝑀 predictions that minimizes the total matching cost:EQUATIONwhere 𝐶 (O * , Ô) is the cost of matching the 𝑖-th ground-truth object to a prediction indexed by 𝜎 (𝑖). 𝐶 is a weighted sum of a classification loss and a localization loss (bounding boxes), defined asEQUATIONin which 𝑝 𝑗 (𝑐 𝑖 ) represents the probability that the 𝑗-th prediction correctly classifies the 𝑖-th ground-truth. 𝐿 𝑏𝑏𝑜𝑥 and GIoU measure the localization error between predicted and actual bounding boxes, with 𝜆 parameters tuning the importance of each component. While the Hungarian algorithm ensures unique pairings to minimize the matching cost, it cannot always align with the complex realties of object detection, where multiple predictions correspond to a single object due to overlaps or visual ambiguities [4, 13, 40] . Moreover, this matching approach does not consider the overall distribution of predictions or ground-truths, which means it evaluates matches individually, ignoring the broader context of how all predictions and ground-truths should ideally be distributed or matched [13, 25] .In the quest to enhance DETR performance, we use OT [24] , a mathematical formulation that has been widely applied in various alignment problems. OT aims to minimize the cost of transporting 'mass' from one distribution to another. In the context of object detection, OT can align a set of predicted objects to a set of groundtruth objects. Each 'mass' corresponds to an object, whether a prediction or a ground-truth. Consider the ground-truth objects {𝑜 * 𝑖 } 𝑁 𝑖=1 and the predicted objects { ô 𝑗 } 𝑀 𝑗=1 , with their respective distribution weights 𝜇 ∈ Δ 𝑁 and 𝜈 ∈ Δ 𝑀 . Here, 𝜇 and 𝜈 denote the importance or confidence of each ground-truth and prediction, located at 𝑜 * 𝑖 and ô 𝑗 , respectively. We then use a transportation cost 𝐶 ∈ R 𝑀 ×𝑁 to compute a plan Γ that minimizes the total cost of matching each prediction 𝑗 to a ground-truth 𝑖 minEQUATION⟨Γ, C⟩ 𝐹 is the Frobenius dot product between the transport plan and cost matrix. The cost C ≥ 0 represents how well (or poorly) a predicted object matches with a ground-truth. The polytope 𝑈 (𝜇, 𝜈) is a set of transportation plans of dimension 𝑀 × 𝑁 to match predictions with ground truths, ensuring an optimal alignment under specified constraints. The cost matrix C, with its elements C 𝑖 𝑗 is computed based on the Wasserstein distance (details are provided in the Suppl. file). Unlike OT, the Hungarian algorithm does not account for the distributions 𝜇 and 𝜈 of ground truths and predictions, treating all matches with equal importance [1, 7, 25, 40] . It enforces a strict one-to-one matching, leading to situations where some predictions may not match any ground-truth, thus deemed to match the background. OT, on the other hand, allows the distribution of each ground-truth object to be shared among several predictions. Similarly, a single prediction can correspond to multiple ground truths, providing a flexible matching process.It is well-known that OT generates fully-dense transportation plans, meaning that every prediction is (fractionally) matched with all ground-truths [3, 5, 7] . This matching flexibility, however, introduces the problem of over-splitting. This occurs when the mass is spread too thinly across multiple predictions or ground truths, hence reducing the accuracy and reliability of the matching process. Indeed, OT alone cannot provide an effective alignment [29] . We incorporate a Kullback-Leibler (KL) divergence to relax the strict conservation of marginal constraints in OT by implementing soft penalties [3] . This adjustment helps to control the distribution of probabilities, preventing over-splitting and ensuring that the matching process remains both accurate and fair. Given distributions 𝜇 ∈ R 𝑁 for ground-truths and 𝜈 ∈ R 𝑀 for predictions, along with a cost matrix 𝐶 where 𝐶 𝑖 𝑗 reflects the cost of matching prediction 𝑗 to ground-truth 𝑖, our goal is to find a transportation plan Γ ∈ R 𝑀 ×𝑁 that minimizes the total transportation cost while ensuring that the resulting distribution of predictions and ground-truths (after transportation) closely aligns with their original distributions:EQUATIONwhere 𝐷 KL is KL divergence. ⟨𝐶, Γ⟩ = 𝑁 𝑖=1 𝑀 𝑗=1 𝐶 𝑖 𝑗 Γ 𝑖 𝑗 is the total transport cost, and 𝜅 1 , 𝜅 2 are regularization parameters that balance the KL divergence terms. These terms align the transported mass distributions (Γ 𝑇 1 𝑀 and Γ1 𝑁 ) with the original distributions (𝜈 and 𝜇), allowing the model to address discrepancies in the total distribution between predictions and ground-truths efficiently. The KL divergence measures the difference between the transported distribution (1 𝑇 𝑀 , 1 𝑁 ) and the original distribution (𝜇, 𝜈), as followEQUATIONfor predictions, ensuring alignment with 𝜈, andEQUATIONfor ground truths, ensuring alignment with 𝜇. By embedding KL divergence, the finalized transport plan adheres closely to the original distributions of both predictions and ground-truths. This ensures that the matching process respects the inherent probabilistic nature of detections, accommodating scenarios where the number of predictions exceeds or falls short of the number of ground-truth.Constraints with Entropy Regularization. We now consider an entropic regularization of transport plan Γ that controls the smoothness of the coupling and is particularly useful for reducing the computational complexity [3, 6, 29] . Computation of regularized OT relies on Sinkhorn's algorithm [6] which is notable for its efficiency and the ability to differentiate with respect to inputs. The resulting transport plan is easier to interpret because it provides a probabilistic view of the relationships between predicted and actual objects (Figure 2(f-h )). The regularized version of Eq. ( 4) is minEQUATIONwhere 𝐻 (Γ) = -𝑖 𝑗 Γ 𝑖 𝑗 log(Γ 𝑖 𝑗 ) is the entropy of the plan Γ with 𝜖 as the regularization parameter. Sinkhorn's algorithm is well-suited for solving the regularized OT because it iteratively adjusts Γ to satisfy the mass constraints while maximizing entropy, leading to a more balanced and flexible assignment [5] and this smoothing effect shown in Figure 3 . However, reducing it to (𝜖 → 0) and KL = 0, leads to numerical instability because the benefits of the sinkhorn algorithm and parallelization are no longer applicable.Require: 𝐶: cost matrix (𝑀 × 𝑁 ), 𝜇: distribution of ground-truths (𝑁 -vector), 𝜈: distribution of predictions (𝑀-vector), 𝜅: regularization parameter for KL divergence, 𝜖: regularization parameter for entropy Ensure: Γ: Optimal Assignment (𝑀 × 𝑁 matrix) In this case, we recover the exact OT (Eq. ( 3)) which has shown one-to-one mapping as discussed in [5] and shown in Figure 3 .In classical DETR, predictions are matched to ground-truth bounding boxes using the Hungarian algorithm, through a cost function 2). This approach requires an equal number of predictions and ground-truth boxes for effective matching. However, our procedure is as follows: for each image, generate predictions (ĉ 𝑗 , b 𝑗 ) and compare these to the ground-truths (𝑐 * 𝑖 , 𝑏 * 𝑖 ) using the cost function 𝐶. The cost matrix integrates both classification and localization losses and defined as:𝐶 match ( ô 𝑗 , 𝑜 * 𝑖 ) = 𝜆 prob (1 -⟨ĉ 𝑗 , 𝑐 * 𝑖 ⟩) + 𝜆 𝑙 ∥ b 𝑗 -𝑏 * 𝑖 ∥ + 𝜆 𝐺𝐼𝑜𝑈 (1 - 𝐺𝐼𝑜𝑈 ( b 𝑗 , 𝑏 * 𝑖 )) in Eq. (𝐶 ( ô 𝑗 , 𝑜 * 𝑖 ) = 𝐿 cls ( ĉ 𝑗 , 𝑐 * 𝑖 ) + 𝐿 loc ( b 𝑗 , 𝑏 * 𝑖 ).The first component ensures that similar object classes are matched together by evaluating the probability that the 𝑗-th prediction correctly classifies the 𝑖-th ground-truth. The second component addresses the spatial alignment by measuring the discrepancy between the predicted bounding box and the ground-truth bounding box. To further enhance the matching process, we incorporate entropic regularization into the cost function (Eq. ( 7)) and employ Sinkhorn's algorithm to compute the optimal transport plan Γ. The proposed matching strategy is given in Algorithm 1. We also train the model by cross-entropy (CE) loss, Table 2 : Enhancing object detection performance with DINO-DETR on the COCO val2017 dataset, using ResNet50 [12] and Swin-Large [21] backbones. RTP-DETR is presented as a complementary model to existing methods and achieves consistent enhancements in performance. We also include the result of DINO-DETR trained for 36 epochs as a reference for comparison.𝑁 𝑖=1 𝑀 𝑗=1 Γ 𝑖 𝑗 •𝐿 train ( ô 𝑗 , 𝑜 * 𝑖 ), where 𝐿 train ( ô 𝑗 , 𝑜 * 𝑖 ) = 𝜆 CrossE ( ĉ 𝑗 , 𝑐 * 𝑖 ) + 𝜆 𝑙 ∥ b 𝑗 -𝑏 * 𝑖 ∥ + 𝜆 𝐺𝐼𝑜𝑈 (1 -𝐺𝐼𝑜𝑈 ( b 𝑗 , 𝑏 
training settings: 
 Our evaluation was conducted on two widely recognized datasets to validate the effectiveness of our matching strategy, which has a time complexity of 𝑂 (𝑁 𝑀 log(𝑁 𝑀)), where 𝑁 is the number of ground truth objects and 𝑀 is the number of predicted objects. The first dataset, COCO object detection [17] , includes 118,287 training images and a validation set of 5,000 images. The second dataset, PASCAL-VOC object detection [9] , is fine-tuned on VOC train-val07+12 with approximately 16.5k images and evaluated on the test2007 set. We use mean Average Precision (AP) and mean Average Recall (AR) as primary metrics for evaluation. We compare the performance of our model (RTP-DETR) against several DETR variants, including DN-DETR [16] , Salience-DETR [13] , Rank-DETR [25] , DINO [38] , and Group-DETR [4] . All models are trained using the same set of hyperparameters: 𝜆 𝑝𝑟𝑜𝑏 = 𝜆 𝐶𝑟𝑜𝑠𝑠𝐸 = 2, 𝜆 𝐺𝐼𝑜𝑈 = 2 and 𝜆 𝑙 = 5. Regarding our model-specific parameters, the entropic regularization is set to 𝜖 = 𝜖 0 /(log(2𝑀) + 1) with 𝜖 0 = 0.17. This adjusts the entropic regularization based on the number of predictions, enhancing the stability and convergence of the model. We used multiple values of 𝜅 2 but 𝜅 1 is fixed to a large value 𝜅 1 = 100.Table 1 shows the performance of RTP-DETR compared with other high-performing DETR-based methods on the COCO object detection val2017 set using ResNet-20 [12] . With only 12 training epochs, RTP-DETR achieves an impressive AP of 50.7%, which suppresses H-DETR by +2.0% and exceeds the most recent state-of-the-art Salience-DETR and Rank-DETR, by +1.3% and +0.5%. Interestingly, we observe notable improvements in AP 75 , demonstrating the advantage of our approach at higher thresholds. Additionally, Table 2 illustrates the effectiveness of our model in enhancing DINO-DETR, a training-efficient DETR variant that has received significant attention in the object detection task. The results are obtained using two different backbones, ResNet-50 [12] and Swin-Large [21] . We also compare our model with other integrations such as Rank-DINO and Stable-DINO. The improvements with our model are notable: there's an increase of 1.6% in AP performance when using the ResNet-50 backbone (49% vs. 50.6%) and 1.1% with Swin-L (56.8% vs. 59.7%). At a higher IoU threshold (AP 75 ), the enhancements become even more pronounced, achieving +1.7% improvement with ResNet-50 and +1.3% with Swin-L. These results indicate our model's robustness and generalizability across various DETR-based models. Table 3 summarizes the integration of our RTP with several DETR variations. As can be seen, when combined with Deform-DETR, the AP has increased by +1.8, from 46.9% to 48.7%. Similarly, the performance of Group-DETR has improved by +0.7, moving from 48% to 48.7%, while using only 12 epochs schedule. Additionally, our RTP has enhanced DINO-DETR's performance, boosting its AP by +1.4 over a 24-epoch and by an additional +1.6 AP when the training is extended to 36 epochs. It's important to note that many of these methods rely on a one-to-one matching strategy, with modifications mainly focused on optimizing the Hungarian algorithm to better accommodate one-to-many relationships between predictions and ground truths during training. Unlike these advances, RTP utilizes a different matching strategy, providing additional advantages of regularization that contribute to the performance gains observed.Adding entropy regularization into the transport plan (Eq. ( 7)), yields an efficient and seamless matching process. Figure 3 provides a clear illustration of this improvement by showing the effectiveness and speed of the RTP matching process. Furthermore, Figure 1 details the convergence behavior of our model, demonstrating that RTP enables significantly faster convergence compared to standard DETR. The slower convergence rates of conventional DETR can be attributed to the discrete and unstable nature of the Hungarian algorithm, particularly during the early stages of training [19, 25, 42] . We also conducted a comparative analysis of our model alongside two prominent DETR variants, DINO [38] and DN [16] , known for their effective one-to-many matching capabilities. The models use a ResNet-50 backbone and are trained over 12 epochs, as shown in Figure 4 . This comparison demonstrates that RTP not only refines the matching process but also significantly accelerates model convergence during training, thereby providing a robust solution to the inherent limitations of the Hungarian matching algorithm. .9 (+0.7) 62.9 (+0.5)Table 4 : The effect of combining RTP with Align-DETR demonstrates a significant improvement in performance.5.5.1 Computation Time. Figure 5 compares the time complexity of different matching models, focusing on the Hungarian algorithm, RTP without KL divergence (𝜅 = 0), and the full implementation of RTP. We observe that both RTP variants are relatively stable, even with increasing numbers of predictions. This consistency is due to the use of entropy regularization (𝐻 (Γ)) within the OT framework, which effectively controls computational complexity and ensures only minimal increases in processing time as predictions scale. On the other hand, the Hungarian algorithm shows a noticeable increase in computation time as the number of predictions grows (beyond 900 predictions). This behavior points to a limitation of the Hungarian algorithm when dealing with high-volume predictions [1, 19] . The performance of our models is particularly notable when handling large numbers of predictions. Through GPU parallelization of the Sinkhorn algorithm, we achieve more than 30× speedup. These computations were performed on an Nvidia TITAN X GPU. We also used SSD300 which makes 8, 732 predictions, as a reference point. The implementation of SSD is provided in the Suppl. file.IoU-Optimized Loss. We explore the combination of RTP with recent DETR variants, particularly those leveraging IoU-optimized loss to enhance DETR performance significantly [1, 19, 25] . By integrating RTP with Align-DETR [1] , we achieve complementary effects between RTP and IoU-aware loss. As illustrated in We illustrate the influence of two hyperparameters, 𝜖 0 and 𝜅 2 , in our matching technique. As defined in (5.1), the regularization parameter 𝜖 is set to 𝜖 0 /(log(2𝑀) + 1), which scales based on the number of predictions 𝑀. As 𝑀 increases, log(2𝑀) enlarges, causing 𝜖 to decrease. Figure 7 details the impact of the regularization 𝜖 0 on the matching process. When 𝜖 0 is a very small value, the effect of 𝜖 becomes minimal, and at 𝜖 0 = 0, entropy regularization no longer plays a role in the matching process. Conversely, larger 𝜖 0 values increase the corresponding 𝜖, potentially risking overfitting or loss of essential details in detection. Empirical results indicate that our model performs best when 𝜖 0 is within the range [0.15 -0.25]. Additionally, our analysis on the hyperparameter 𝜅 2 is presented in Figure 7 . Our model achieves optimal results when 𝜅 2 = 0.01. We maintain 𝜅 1 = 100, as changing this value leads to an object imbalance problem, emphasizing its importance in preserving the accuracy of ground-truth alignment. Indeed, 𝜅 1 ensures accurate alignment of ground-truths to predictions, while 𝜅 2 allows flexibility through probabilistic matching. In contrast, the optimized model (bottom row) exhibits well-defined and focused attention, accurately capturing the shape and position of each object, even in densely packed scenes. This demonstrates that regularization helps the model to better concentrate on relevant parts of the image, thereby improving object detection accuracy. 
conclusion: 
 In this paper, we have examined the comparative effectiveness of the Hungarian algorithm and transportation plan in object detection, focusing on reducing the matching costs between predicted and actual data. The Hungarian algorithm, with its strict one-to-one matching, is effective when the number of predictions matches the number of ground truths exactly. However, this rigid strategy limits its application and fails to capture the complexities and variations present in real-world data. In contrast, regularized optimal transport, through a probabilistic coupling, offers a flexible solution that accounts for the entire distribution of matches. This strategy not only facilitates a more comprehensive understanding of the relationships between predictions and ground-truths objects but also handles the discrepancies in set sizes. Our findings demonstrate that the transportation plan with entropic regularization consistently outperforms the Hungarian method by providing a more accurate and flexible alignment without relying on predefined thresholds. In future, we extend the regularized OT framework to zero-shot detection scenarios, where the model must detect objects not seen during training. This requires developing transferable transport plans that can generalize across different object categories and domains. 
 
 Appendix:  


Decision: The paper introduces the Regularized Transport Plan (RTP) for object detection, enhancing DETR adaptability in complex scenarios, as demonstrated on the MS-COCO benchmarks. Initial reviews were mixed, with concerns about the paper's focus and clarity. Post-rebuttal, reviewers acknowledged improvements made by the authors in addressing these concerns, leading to an improved average rating leaning towards acceptance.

The recommendation is to accept the paper for publication, as the paper presents a novel and meaningful contribution to the field of object detection.

Reviews: review: paper_summary: paper_summary: This paper analyzes the drawbacks of the matching strategy in DETR-based detection methods. It proposes a regularized transport method for the alignment between predictions and ground truths, which accelerates convergence during the training process. In experiments, this paper compares the proposed method with several DETR-based detection methods and achieves competitive results. Additionally, as a plug-and-play matching strategy, it has been combined and compared with multiple baselines, resulting in consistent performance improvements.
strength_weakness: strength_weakness: 1. The matching strategy in DETR-based detection methods has been deeply analyzed.
2. The paper's writing and algorithm descriptions are clear and easy to read.
3. This paper offers a novel and unique perspective and approach to DETR-based detection methods.
limitations: limitations: 1. I think that a time complexity analysis for Algorithm 1 is very necessary, not just a comparison of average computation times.
2. In the visualization in Figure 6, the authors mention that “the regularized model appears to be better at discerning the exact shape and position of the objects.” Is this enhancement in the model’s ability to perceive the shape of the objects due to smoother and faster convergence? I hope the authors can provide a more detailed explanation to help readers understand.
3. Using color density to compare the degree of matching in Figure 2 is very clear, but I think it would be more intuitive to compare the Hungarian matching and the proposed matching strategy based on visual results in the images. For example, compare the different prediction boxes matched to the same target in the ground truth under the two matching strategies, and explain the drawbacks of the Hungarian matching method and the advantages of the proposed method.
4. According to Table 3 and Table 4, RTP does not show much performance improvement on object detection datasets. On the other hand, based on Figure 3 and Figure 4, the model’s convergence speed and results improve after using RTP. I wonder if the authors could analyze their thoughts on balancing these two aspects, i.e., the significance of better convergence and minor performance gains in this field?
paper_summary: paper_summary: This paper is motivated by the fact that one-to-one Hungarian Matching does not effectively model the varying densities and distributions
 of objects in DETRs. The author propose a new matching algorithm Regularized Transport Plan (RTP) to handle this problem. 
Experiment results on COCO show it advantages by overpass the state-of-the-art DETRs (Rank-DETR, Stable-DINO).
strength_weakness: strength_weakness: 1. The ground truth - model prediction problem is modeled as a optimal transport problem in this paper, which is novel in the DETR research area.
2. This paper is well motivation and find a very interesting research question. The authors thinks the matching procedure from a new perspective.
3. The experiment results is very solid, validating its effectiveness.
limitations: limitations: 1. Figure 1 is not very fair. The original DETR is very naive, and compare the proposed method (with contains many advances in previous works) with it is not fair. The results on VOC is also not persuasive, it should be replaced with some large dataset.
2. The reason for why the proposed method leading to a more balanced and flexible assignment  is not very well clarified. Please detailed discuss it.
3.Minors:
1) spell error (e.g. DET --> DETR in L588)
2) spell error (e.g. DERTR --> DETR in L762)
3) the font  size in images should match that in the paper.
paper_summary: paper_summary: This paper proposes a regularized transport plan to find the best alignment between predictions and ground-truths. It enhances DETR adaptability to complex object detection scenarios.
strength_weakness: strength_weakness: The method is formulated in detail and clearly. 

Extensive experiments are conducted to prove its effectiveness.
limitations: limitations: The paper claims that DETR is unsuitable for complex scenarios, such as detecting small or dense objects. Concrete results are needed to demonstrate the effectiveness of RTP in these scenarios.

Some parts of the writing and organization need improvement. For example, in Section 2, the significance of the hyperparameters (what large and small values represent) is not clearly explained in advance.
paper_summary: paper_summary: This paper proposes the Regularized Transport Plan (RTP) for object detection. The RTP is computed using the differentiable Sinkhorn algorithm to allow for soft, fractional matching rather than strict one-to-one assignments. This approach enhances DETR adaptability to complex object detection scenarios, providing a nuanced and precise assessment of disparities between prediction and ground-truth distributions. Evaluations on the MS-COCO benchmarks demonstrate the effectiveness of the approach.
strength_weakness: strength_weakness: 1. RTP can make DETR converge faster and improve detection performance.
2. RTP have a lower computational cost than Hungarian algorithm when the number of prediction is larger than 300.
3. RTP can be combined with many variants of DETR for further improvements.
limitations: limitations: 1. The paper only shows evaluation of 2D object detection on COCO dataset. How about the performance on 2D panoptic segmentation, 2D pose estimation, 3D object detection, and multi-object tracking?
2. More explanation on why RTP improve the matching. Like visualization of features and comparing results with DETR.
3. The paper focuses on improving a component of DETR. I think the paper is suit for computer vision and machine learning conferences.