Union-over-Intersections: Object Detection beyond Winner-Takes-All
Authors: Aritra Bhowmik
Pascal Mettes
Martin R. Oswald
Cees G. M. Snoek
Venue: ICLR 2025 Spotlight (2025)
Keywords: localization based feature representation
intersection over union
object detection.
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: This paper revisits the problem of predicting box locations in object detection architectures. Typically, each box proposal or box query aims to directly maximize the intersection-over-union score with the ground truth, followed by a winner-takes-all non-maximum suppression where only the highest scoring box in each region is retained. We observe that both steps are sub-optimal: the first involves regressing proposals to the entire ground truth, which is a difficult task even with large receptive fields, and the second neglects valuable information from boxes other than the top candidate. Instead of regressing proposals to the whole ground truth, we propose a simpler approach—regress only to the area of intersection between the proposal and the ground truth. This avoids the need for proposals to extrapolate beyond their visual scope, improving localization accuracy. Rather than adopting a winner-takes-all strategy, we take the union over the regressed intersections of all boxes in a region to generate the final box outputs. Our plug-and-play method integrates seamlessly into proposal-based, grid-based, and query-based detection architectures with minimal modifications, consistently improving object localization and instance segmentation. We demonstrate its broad applicability and versatility across various detection and segmentation tasks.

Published as a conference paper at ICLR 2025
UNION-OVER-IIVEESSECIIOSS OВJЕCT DETECTION BEYOND WINNER-TAKES-ALL
Aritra Bhowmik
Pascal Mettes
Martin R. Oswald
Cess G. M. Snoek
Atlas Lab, University of Amsterdam (a ...bhowkkk, p. mettes, oswaadd, sn.ee
ABSTRACT
This paper revisits the problem of predicting box locations in object detection architectures. Typically, each box proposal or box query aims to directly maximize the intersection-over-miioon score with the ground truth, followed by a winner-takes- all non-maximum suppression where only the highest scoring box in each region is retained, We observe that both steps are sub-optimal: the first involves regressing proposals to the entire ground truth, which is difficult task even with large receptive fields, and the second neglects valuable information from boxes other than the top candidate. Instead of regressing proposals to the whole ground truth, we propose a simpler approach: regress only to the area of intersection between the proposal and the ground truth. This avoids the need for proposals to extrapolate beyond their visual scope, improving localization accuracy. Rather than adopting winner-takes-all strategy, we take the union over the regressed intersections of all boxes in a region to generate the final box outputs. Our plug-and-play method integrates seamlessly into proposal-basedd
grid-based, and query- based detection architectures with minimal modifications, consistently improving object localization and instance segmentation. We demonstrate its broad applicability and versatility across various detection and segmentation tasks.
INTRODUCTION
Object detection is a long-standing challenge computer vision, with the goal of spatially localizing and classifying objects in images by their bounding box (Burl et al., 1998; Viola & Jones, 2004; Felzenszwalb al., 2010). Over the past decade, significant progress has been made, driven by advances in various stages of the detection pipeline. From the foundational R-CNN (Girshick et al., 2014) Faster R-CNN (Ren al., 2015), and paradigm-shiie shifting YOLO architectures (Redmon al., 2016; Ge et al., 2021; Bochkovskiy et al., 2020; Redmon & Farhadi, 2018; Huang et al., 201 8), to the recent innovations by transformer networks (Zhu et al., 2021; Carion et al., 2020; Dai et al., 2021a; Nguyen et al., 2022), these developments have enhanced feature extraction and detection. Advances in loss functions, such as focal loss (Lin et al., 2017) and IOU-aware loss (Wu et al., 2020a; Zhang et al., 2021), have further addressed class imbalance and precise localization. Regardless of backhone or optimization choices, most modern methods generate a set of proposal boxes, grid-boxes or queries. These are classified and spatially regressed, followed by ranking non-maximum suppression to retain only the highest scoring box per location. In essence, each proposal solves the intersection--vve--uiioo alignment with the ground truth, followed by winner-takes-all approach. We revisit this fundamental problem of object localization and candidate selection.
We identify two issues with the common setup object detection and propose a simple solution. First, the goal of proposal-basee (Ren et al., 2015; He et al., 2017), grid-based (Redmon & Farhadi, 2018; Bochkovskiy et al., 2020) and query-based (Carion et al., 2020; Zhu et al., 2021) detectors, is to learn proposal boxes that independently represent ground truth objects. As shown in Figure 1, this often becomes an ill-posed problem. During the forward pass, proposal boxes typically capture only portion of the ground truth, leading an extrapolation challenge when regressing for perfect alignment. Second, there are multiple proposals or queries for each ground truth object, offering complementary views. While existing methods use box voting select the top candidate, we focus merging information from multiple proposals, leveraging their complementary strengths for improved detection.

---

Published as a conference paper at ICLR 2025
Regression Stage
Post- Dost-Processing Stage
Tracitional
Final on
Ground Truth
Proposals
Competition
Regress Outtide Scope (hars)
Input image
Feature Encoder
Combi
Collaboratio
Visual Scope
Regross Ins Scope (easy)
Union Interscctons
Figure I: Union-over-Interrsctioon vs. Winner-takes-all. We introduce two simple modifications to the traditional object detection pipeline. First, the regression stage, rather than requiring proposals to align with the entire ground truth, we adjust the targets to focus solely on their intersection with the ground truth. Second, in the post-processing stage, we perform Union-over-Interscciion over the traditional practice of discarding less optimal bounding boxes. Our approach underscores the advantage of cooperative interaction among proposals, demonstrating that collaboration yields superior results over competitive exclusion.
Our solution is straightforward: we decompose the problems of regressing to the entire ground truth from single proposal and winner-takes-all candidate selection into easier solve intersection and union problems. We do so by altering two steps of the object detection pipeline: (i) We set new target for the regression of proposals: rather than predicting the entire ground truth, we only predict the region of intersection. Thus, we only regress towards the ground truth within the visual scope of the proposal. (ii) Given a group proposals with predicted ground truth intersections, we form the final prediction by taking the union over the intersection regions. i.e., instead of selecting only the most confident proposal a region, we use the wisdom the crowd to form our final predictions.
The two stages barely affect the existing object detection pipelines. In the current regression heads, we only need to change the target coordinates. the current non-maximum suppression, we use the exact same way of grouping proposals, but instead of winner-takes-all, intersections (Uol) of all proposals group as the final prediction. Our approach can therefore grid-based, and query-basee object detectors, be it convolutional
take the union-over-
be plugged into proposal-baseee or transformer architecture. Despite its technical simplicity, Uol directly improves the detection performance. We show how revisited approach improves canonical detection and instance segmentation methods across multiple datasets. We also demonstrate through a controlled experiment that our approach is ready benefit from future improvements in classification and regression performance, and hope to establish it as the preferred formulation for the next generation of object detectors.
2 RELATED WORKS
Object detection architectures. Object detection architectures can be broadly categorized into: single-stage, two-stage, and transformer-based detectors, each with their own characteristics and merits. Single-stage detectors such as SSD (Liu et al., 2016), the YOLO (Redmon et al., 2016) series (including YOLOv3 (Redmon & Farhadi, 2018), YOLOv4 (Bochkovskiy et al., 2020), YOLOX (Ge et al., 2021), YOLO-Lite (Huang et al., (018)). RetinaNet (Lin et al., 2017), EfficientDet (Tan al., 2020), CornerNet (Law & Deng. 2018), CenterNet (Duan et al., 2019). and FCOS (Tian et al., 2019) prioritize speed, directly predicting object classes and bounding boxes in one pass. RetinaNet introduces Focal Loss for class imbalance and small object detection, while EfficientDet leverages EfficientNet (Tan & Le, 2019) backbones for scalable efficiency, Innovations in CornerNet, CenterNet, and FCOS focus on anchor-free methods, further enhancing real-time efficiency.
In the realm of two-stage object detectors, the R-CNN family has played pivotal role. This family includes R-CNN (Girshick et al., 2014), Fast R-CNN (Girshick, 2015), and Faster R-CNN (Ren et al., 2015), which introduced end-to-end training with Region Proposal Networks. Additionally, Mask R-CNN (He et al., 2017) added segmentation capabilities, Cascade R-CNN (Cai & Vascon- celos, 2018) improved accuracy via multi-stage framework, and Libra R-CNN (Pang al., 2019)
2

---

Published as a conference paper at ICLR 2025
addressed training imbalances. Enhancements include TridentNet (Paz et al., 2021), which handles scale variation with parallel branches, and methods like Grid R-CNN (Lu et al., 2019) for precise localization and Double-Heal Head R-CNN (Wu et al., 2020b). which differentiates between the classi- fication and bounding box regression, further enriching the field. Transformer-base
models like DETR (Carion et al., 2020), Deformable DETR (Zhu et al., 2021), TSP-FCOS (Sun al., 2021), Swin Transformer (Liu et al., 2021), and ViT-FRCNN (Beal et al., 2020) have revolutionized object detection. DETR views detection as set prediction, Deformable DETR adds deformable attention for focused image analysis, TSP-FCOS integrates transformers with FCOS (Tian et al., 2019) for scale variance, Swin Transformer employs a hierarchical approach, and ViT-FRCNN combines Vision Transformer (Dosovitskiy et al., 2021) with Faster R-CNN (Ren et al., 2015) for feature extraction. Follow-up work include Boxer (Nguyen et al., 2022), DINO (Zhang et al., 2022), DAB-DETR (Liu et al., 2022), UP-DETR (Dai et al., 2021b), Dynamic DETR (Dai et al., 2021a), and Conditional DETR (Meng et al., 2021), enhancing various aspects like training efficiency and dynamic predictions.
Despite the diversity in approaches, common aspect across these three paradigms is the presence of a classification and bounding box regression stage, in which each proposal needs to individually align with the entire ground truth object. We show that proposal-baseds methods benefit from our approach of first regressing to intersections, followed by a union-over- intersection grouping.
and query- -aaeed
Object detection post-processing. After the regression stage, selection of the best regressed proposal from set of candidate detections ensures accuracy, with Non-Maximum Suppression (Girshick, 2015; Ren et al., 2015; Redmon et al., 2016; Liu et al., 2016) (NMS) being key technique for selecting the best proposals by eliminating redundant detections. Traditional NMS, while effective, has been surpassed by variants such as Soft-NMS (Bodla et al., 2017), which adjusts confidence scores instead of discarding detections. improving performance in crowded scenes. Learning NMS (Hosang et al., 2017) introduces adaptability by incorporating suppression criteria into the network training. Further refinements like IoU-aware NMS (Wu et al., 2020a; Zhang et al., 2021) and Distance- (DloU) NMS (Zheng et al., 2020) consider both overlap and spatial relationships between boxes for more precise detection. Additional post-processing techniques include Matrix NMS (Wang et al., 2020), which employs matrix operation for efficient suppression, and Adaptive NMS (Liu et al., 2019), which dynamically adjusts thresholds based on object densities. Furthermore, ASAP- NMS (Tripathi et al., 2020) optimizes processing speed, while Fitness NMS (Tychsen- -mith & Petersson, 2018) considers both detection scores and spatial fitness for more accurate suppression to enrich post-processing in object detection. Bounding box regression losses, crucial for refining predicted bounding boxes, have evolved to better align predictions with ground truth boxes. loU loss (Yu et al., 2016; Jiang et al., 2018) directly optimizes the overlap between predicted and ground truth boxes, while GloU loss (Rezatofighi et al., 2019) addresses scenarios where predicted boxes fall outside the ground truth penalizing such deviations. DIoU (Zheng et al., 2020) and a- (He et al., 2021) losses further enhance regression quality by incorporating factors like the distance between box centers and aspect ratio consistency. These advanced losses have become benchmarks for evaluating the quality of bounding box regressions and are integral to improving detection accuracy in modern object detection pipelines. The improved post-processing techniques address the limitation of selecting single winner from each cluster of proposals at a location. However, all techniques still assume that each individual box is already fully aligned with the corresponding ground truth box and serve primarily remove duplicate detections. Instead, seek obtain wisdom from the crowd by using all proposals for the same object instance to form the final object detection output. We do so by making minimal adjustments to the mon-maximum suppression pipeline.
3 UNION-OVER-IITEERSECII IN OBJECT DETECTION
Our Union-over- Interssntesstttions (Uol) approach improves the object detection pipeline with minimal adjustments, without adding new methods or architectures. We first outline the standard pipeline, fo- cusing on bounding box regression and top candidate selection, followed by two simple modifications, as shown in Figure 2.
Problem statement. In traditional object detection, bounding box regression learns mapping function of that transforms a proposal box P=(Pa,, a. Py. Pzz. Py,) ) to approximate a ground truth box G=(Gz,: Gys: Gz5y Gy2). This is typically achieved by minimizing the I.1 or 7.2 loss between the
3

---

Published as a conference paper at ICLR 2025
Regression Stage
Post- Post-Processing Stage
Input:
begin
Set of Proposals P= T=(P..... [H]
Corresponding ground truths G (Bu -n Re)
while
empty do
Corresponning detection scores $= (5...., lou threshold k
m
argmax
M-
B - Pari C- (If)
P P|H
for in range (len(P)) do
Loss Targets Regression:
B1. where (b).... bu) are the set of box predictions.
Loss
ioulM, kthen end
5
[Eu .. Bx)
Traditional targets
loutH, pl then
T= [p: 0gu Pa O Bs)
Our targets
P\p;5 51s,; M top (C) combined
end
Main Difference: regress only the intersections, the whole ground truth.
Main Difference: Union over discording.
end
D
M
return 0,
end
Figure 2: Pseudo code demonstrating our minimal changes in the object detection pipeline. During regression, we adjust the target of the proposals from the entire ground truth to only the intersection with ground truth. In proccssss,,,n we group boxes by proposal rather than regressed outcomes and merge regressed intersections. avoiding the discard of non-maximum boxes.
predicted box and the ground truth. However, this approach presents a challenge: proposals P often capture only part of the ground truth G, requiring f to extend P beyond its visible scope. As must infer parts of G not visible within P, this becomes a difficult task. While large receptive fields help, refining proposals through a union-based approach yields better results.
Additionally, the set of proposals for ground truth object, P= P,). provides comple- mentary views, but non-maximum suppression or ranking selects only one candidate, discarding others. This assumes the intersection-over-uiii problem is already solved, merely excluding duplicates the same region.
To address these issues, redefine box regression an intersection learning task, aligning proposals with the intersecting parts of the ground truth boxes. Instead of discarding non-top proposals, we replace non-maximum suppression with union operator, defining the final object union-over- intersection of all proposals in a region. Figure illustrates how these tasks lead to final object detection.
Intersection- regression. We redefine the regression task as an intersection learning problem. Instead of regressing to the entire ground truth, each proposal regresses only the visible part the ground truth, i.e., the intersection between the proposal and ground truth. Let /=(Iz, I Izz. Ign) be the intersection of boxes P and G. The task is learn a mapping function f such that (P) The intersection is computed as:
Lx (P, Gz
I, max( P Gan)
(1)
1z = min( Pays Ga2)
1n = min(Py Gyz)
(2)
The loss for this task is:
Lintersection
Σ
((())) 1
(3)
iE(21.91=22322
where can be 1 or 2 depending on whether the L, or L2 loss is used. Intersection-basee grouping. Let P= Pe...,
......,Pyy P. be the set of all proposals for ground truth object. The corresponding intersections with the ground truth are I= , I2..... In}. Our goal is to combine these intersections into bounding box using function - TR. minimizing the
4

---

Published as a conference paper at ICLR 2025
difference between the combined box and the ground truth:
L Σ
Ie((I) G.}
VE( (22
During inference, we refine the union intersections to ensure a tight fit. Unlike traditional NMS, which groups regressed boxes, our method groups proposals, using the highest scoring proposal as a seed. This approach leverages the original proposals, as intersections often lack sufficient IoU for effective grouping, increasing the likelihood of fully encompassing the ground truth. Instead of discarding non-maxima, we take the minimum [st, vil and maximum [ab v of all intersection regressions the group. This forms set of combined boxes (= B2, ...... Bm Each combined box B3 is candidate bounding box for the object represented by group gj. By design, our method yields the same number of predictions current detectors and compatible with any NMS variant.
Finally, we perform regression step to refine the union-over intersection box predictions. The regression function which represents the refined predictions:
maps each combined box to a final regressed box, denoted as R.
Leetieement
Σ Σ
Ir(B,s) -Tyil
(5)
1= LE (P..).m..))
where B3, is the i-th coordinate of the combined box By. and TJ: & is the corresponding ground truth coordinate.
The overall loss function is combination of the intersection loss and refinement loss:
Laoial Lintersection + X Trefinement
where A, set to 0,5 in our experiments, controls the weight of the refinement loss.
4 EXPERIMENTS
4.1 DATASETS, EVALUATION, AND BASELINE DETECTOR IMPLEMENTATIONS
Datasets. We conduct evaluations on COCO (Lin al., 2014), covering two tasks: object detection and instance segmentation. The MS-COCO 2017 dataset is key benchmark for object detection and instance segmentation, comprising 80 categories with 118k training and evaluation images. features maximum of 93 object instances per image, with average of 7 objects. Further results on PASCAL VOC (Everingham et al., 2010) are provided in the supplemental. For Pascal VOC. we leverage the 2007 and 2012 editions. It spans 20 object categories, with 5,011 training and 4,952 testing images in VOC2007, and an additional 11,540 training images in VOC2012.
Evaluation eriteria. We adhere the standard evaluation protocols. For MS-COCO (Lin et al., 2014), we report the mean Average Precision (mAP) across loU thresholds from 0.5 to 0.95 mAP, specific IoU thresholds (0.5 and 0.75), multiple scales (small, medium and large), and Average Recall (AR) metrics. For PASCAL VOC (Everingham et al., 2010), we report the mAp metric at IoU thresholds of 0.5, 0.6, 0.7, 0.8 and 0.9. For instance segmentation, we report our results on MS-COCO with the mAP across IoU thresholds from 0.5 to 0.95 and across multiple scales (small, medium, large).
Baseline detector implementations. We evaluate our Unionn-over-Intersectoons (Uol) approach across five diverse object detection architectures: Faster R-CNN (Ren et al., 2015), Mask R- CNN (He et al., 2017), Cascade R-CNN (Cai & Vasconcelos, 2018), YOLOv3 (Redmon & Farhadi, 2018), and Deformable DETR (Zhu et al., 2021), representing single-stage, two-stage, and wansformer-baseer detectors. This comprehensive selection highlights the versatility of our method across different detection paradigms.
We trained Faster R-CNN, Mask R-CNN, and Cascade R-CNN on COCO using standard configura- tions such as random horizontal flip. 512 proposals, and SGD optimization over 12 epochs, with a learning rate decay at epochs 8 and 11 YOLOv3 trained with the Darknet-53 backbone for 273 epochs using common augmentation strategies and SGD. Deformable DETR is trained for 50 epochs with the AdamW optimizer.
5

---

Published as a conference paper at ICLR 2025
Table I: Detection comparison on MS-COCO. No matter the object detection method and back- bone tested, when we replace the traditional box proposal selection process with our Union-over- Intersections (Uol), it results in improved accuracy for all metrics.
Method
Backbone
MAPf
APsat
AP
APst
APx
APL
Faster R-CNN (Ren et al, 2015)
R-50-fpn
37.4
58.1
40.4
21.2
41.0
48.1
Faster R-CNN w/ Col
R-50-fpn
38.1
58.7
40.9
21.7
41.8
49.5
Faster R-CNN (Ren et al. 2015)
R-101-fpn
39.4
60.1
43.1
22.4
43.7
51.1
Faster R- CNN w/ Uol
R-101-fph
40.3
60.8
43.5
23.1
44.5
52.8
Mask R-CNN (He et al., 2017)
R-50-fpn
38.2
58.8
41.4
21.9
40.9
49.5
Mask R- R-CNN Uol
R-50-fpn
38.8
59.6
41.8
22.2
41.6
50.9
Mask R-CNN (He et al., 2017)
R-101-fpn
39.8
60.3
43.4
23.1
43.8
52.5
Mask R-CNN w/ Uol
R-101-fpn
40.9
61.1
44.0
23.5
44.7
54.6
Cascade R-CNN (Cai & Vasconcelos, 2018)
R-101-fpn
42.5
60.7
46.4
23.5
46.5
56.4
Cascade R-CNN w/ Uol
R-101-fpn
43.1
61.9
46.8
24.0
47.2
57.3
YOLOv3 (Redmon & Farhadi, 2018)
DarkNet-53
33.7
56.6
35.3
19.4
36.8
44.3
YOLOv3 w/ Uo!
DarkNet-53
34.5
57.5
35.9
19.9
37.3
45.2
Def-DETR (Zhu 2021)
R-50-fpn
44.3
63.2
48.6
26.8
47.7
58.8
Def-DETR w/ Uol
R-50-fpn
44.8
63.9
49.1
27.2
48.3
59.8
Inference cost. Our method adds minimal overhead. On Faster R-CNN, we achieve 14.1 fps (compared to 14.4 fis baseline) due to the extra regression stage. In Deformable DETR, switching to part-based regression leads to slightly longer training times: 23h for 50 epochs vs. 21h for baseline.
4.2 BENEFIT OF
To showcase the effectiveness of Union-over www-ooveeee--ettionnn in object detection pipelines, we perform a series of one-to-one comparisons. For Faster R-CNN, Mask R- R-CNN and Cascade R-CNN, we modify the regression targets focus on object parts. For YOLO, we modify its object-to porect-to-gee to-grid assignment strategy by using IoU-based criterion instead of the traditional center-based approach, assigning objects to multiple grid cells if they overlap sufficiently. Each anchor is tasked with regressing the part of the object it overlaps with best. In Deformable DETR. each query predicts a bounding box through the regression head after the decoder. Instead of matching these boxes to full ground truth boxes, divide the ground truth into quadrants, assign queries to specific parts for part-based regression. Finally we apply our grouping and merging to all methods. While our method integrates with most architectures, adapting it to keypoint- methods like FCOS requires more modifications, as they lack bounding box regions. Code is provided at https: //github. com/ wewarrrrabhaeee trabhowmi /UoI.
Detection comparison on MS-COCO. Table 1 presents the results of our Union-over--ntersection approach on MS-COCO (Lin al., 2014), demonstrating consistent improvements across Faster R-CNN, Mask R-CNN, Cascade R-CNN, YOLOv3 and Deformable DETR. Our method enhances detection performance across Re and ResNet-101 backbones, as well different architectures, highlighting its broad applicability.
In particular, integrated our approach into Cascade R-CNN, which refines object proposals over multiple stages with increasing loU thresholds. Unlike Cascade R-CNN's stage-by-stage refinement of individual proposals, our method refines the combined boxes after grouping within single stage. We applied intersection-bes based regression and aniion-ver-intersecce CNN's final stage, improving performance without altering the earlier stages. This demonstrates how method can complement even complex, multi-stage architectures.
grouping in Cascade R-
Moreover, our approach adapts seamlessly to different detection paradigms: working with proposals in two-stage detectors like Faster R-CNN grid cells in one-stage detectors like YOLOv3, and queries in transformer- based detectors like Deformable DETR, with minimal modifications. This flexibility allows method enhance performance across various architectures with minimal changes.

---

Published as a conference paper at ICLR 2025
Table 2: Integration of IoU-based losses with Uol. Replacing the L1 loss with IoU-based losses such as GloU, DIoU, and Alpha-loU in our method for Faster R-CNN with a ResNet-50 backbone demonstrates consistent performance improvements, highlighting the flexibility and complementary nature of our approach.
IoU Loss Type
Base (mAP1)
Base w/ Uol (mAP1)
Base (APxst)
Base w/ Uol (APsst)
L
37.4
38.1
40.4
40.9
GloU
38.0
38.6
41.1
42.0
DIoU
38.1
38.8
41.1
41.9
Alpha-IoU
38.9
39,4
41.8
42.6
Table 3: Instance segmentation comparison on MS-COCO. Our Union-overr-nterssciion (Uol) when applied Mask R-CNN with ResNet50 and ResNet101 backbones improves the segmentation performance, Uol is especially effective when segmenting large objects.
Method
Backbone
mAPt
APst
APMt
APt
Mask R-CNN
R-50
34.7
15.8
36.9
51.1
Mask R-CNN w/ Uol
R-50-fpn
35.3
16.2
37.5
51.7
Mask R-CNN
R-101
36.0
16.7
39.1
53.0
Mask R-CNN w/ Uol
R-101-fpn
36.8
17.5
39.8
53.8
Table Uol is independent of the grouping method. Comparison across NMS variants with and without Uol Faster R-CNN using ResNet-50. Results are reported on the COCO validation set.
NMS Type
Base (mAPf)
Base w/ Uol (mAP1)
Base (APıst)
Base w/ Uol (APıst)
NMS
37.4
38.1
40.4
40.9
Cluster-NMS
37.6
38.4
40.4
41.0
Soft-NMS
38.2
38.8
40.9
41.7
Integration of loU-based losses with Uol. To evaluate the flexibility of our framework, we replaced the L1 regression loss in method with loU-based losses, including GloU (Rezatofighi al., 2019), DIoU (Zheng et al., 2020), and Alpha-IoU (He et al., 2021). The results, shown in Table confirm that loss functions but also enhances their performance. These findings underscore the versatility of our method, which seamlessly integrates with various loss functions to achieve consistent improvements.
Interseettntssstttinnss (Uol) approach not only complements these advanced
(Uol) approach the task of instance segmentation using Mask R-CNN with ResNet50 and ResNet101 backbones (Table 3). Our method consistently improves segmentation performance across various object sizes, with notable gains in mean Average Precision (mAP). Specifically, we observe an increase of 0.7 points for ResNet50 and 0.9 points for ResNet101 backbones. These improvements can be attributed to the effective merging multiple proposals, which leads more precise segmentation masks. The benefits are especially evident for larger objects, where tighter and more accurate localization crucial. This demonstrates the versatility of our method, not only object detection but also in enhancing instance segmentation tasks across different backbone architectures.
Instance segmentation comparison on MS-COCO. We extended our
4.3 ABLATION STUDIES
Next we perform a series of ablation studies to pinpoint the source of our improvements.
UoI is independent of the grouping method. While we use traditional NMS as the canonical grouping method in our experiments, our approach evaluate this, we conducted experiments using Cluster-NMS (Zheng et al., 2021), NMS, and Soft- NMS (Bodla et al., 2017). From Table we see that across all variants, our (Uol) approach delivers consistent improvements, demonstrating its robustness and flexibility. This highlights that our method can take advantage of advancements in grouping strategies, making adaptable to different detection frameworks and improving performance across the board.
tied to any specific grouping strategy. To
Our improvements come from better localization only. We first investigate where our improved results come from. The final mAP depends both accurate object localization and correctly classifying objects. In Table 5, we dissect the classification and localization performance on Faster R-CNN with and without our approach. We find that the classification accuracy remains nearly
7

---

Published as a conference paper at ICLR 2025
Table 5: Classification versus localization ablation MS-COCO. Our approach improves the quality the object localization without hampering classification accuracy, highlighting that union- over-intersection (Uol) is more viable for object detection than winner-takes-all.
Method
Classification Accuracy [%]]
Localization mloU [%it
Faster R-CNN
76.4
53.7
Faster R R-CNN w/ Uo!
76.5
64.4
Table 6: Localization Recall Precision (LRP) comparison for Faster R-CNN on various metrics. Replacing the traditional box proposal selection with Union-over-Interseciion (Uol) reduces LRP emrors across all metrics.
Method
LRP (Error))
LRPL
LRP
LRP
Faster R-CNN
67.6
17.2
24.2
44.3
Faster R-CNN w/ Uol
65.3
12.7
23.9
43.8
identical, while the localization performance obtains clear leap in mean IoU. Hence, our changes positively affect the localization quality without hampering the classification performance.
We also assess our method's performance using the Localization Recall Precision (LRP) metric, introduced by Oksuz et al. (2021) in Table 6. The LRP metric provides comprehensive evaluation by combining localization error, false positives, and false negatives into a single error measure:
LRP (Error): Overall error that incorporates all aspects.
LRP Specifically measures localization error, reflecting how accurately bounding boxes match ground truth objects.
LRPpp: Measures false positive errors.
LRPPN: Measures false negative errors.
Our Union-overr-Ineesscttinn (Uol) approach delivers a notable reduction in localization error (LRPLec). reducing it from 17.2 12.7, demonstrating improved bounding box precision. Importantly, this improvement in localization is achieved without too many changes in false positive and false negative rates, where we observe only slight reductions in LRPpp (from 24.2 to 23.9) and LRP (from 44.3 to 43.8). The stability in false positives and false negatives suggests that the more accurate the model's class predictions become, the greater the overall improvement we can achieve with our method.
Addressing NMS limitations without architectural change. While methods like one-to-few label assignment (Li et al., 2023) address NMS limitations by introducing entirely new architectures, our approach provides a simple, plug-and-play solution that can be applied to existing detection frameworks without architectural changes. For example, one-to-few achieves 40.9 mAP on the COCO validation set using ResNer101, through modification of the Faster R-CNN baseline (39.4 mAP) with additional architectural changes. In contrast, our method can be seamlessly integrated into stronger architecture like Cascade R-CNN (Cai & Vasconcelos, 2018) with the same ResNet backbone, achieving 4 mAP. This demonstrates that our approach not only avoids the complexity of redesigning architectures but also leverages existing frameworks to achieve superior results, which one-to-few cannot.
Robustness to poor quality proposals. One might question whether directly regressing to the ground truth from box proposals problematic, given the large receptive fields in modern networks. In Figure 3 (a), we ablate detection performance based on the quality of the top box proposals. When proposals have high overlap with the ground truth, both methods perform similarly. However, as proposal quality drops, standard pipelines struggle. Direct regression works well only with high- quality proposals. Our approach, by looking beyond winner-takes-all more robust to lower-quality proposals, improving object localization.
Regressing to intersections is simply an easier task. One the key hypotheses in our work is that regressing to intersection areas simplify the task compared to regressing to the entire ground truth. As shown in Figure 3(b), this simplification results in lower regression loss during training. Notably, despite dynamic target assignment to object parts, the training remains stable and
8

---

Published as a conference paper at ICLR 2025
3.5
hodee ei USte
PalC ALAB
30
Fatar 4-44
24
50
20009 225 00900
COXB
(b)
Figure 3: Overview of four ablation studies on MS-COCO with Faster R-CNN. (a) traditional object detection struggles when the best proposal has a low initial overlap with the ground truth, whereas our iinon--vvr-Interseciin variations proposal quality.
which looks beyond the winner-takes-all, is more robust to (b) regressing to intersections is a simpler task to optimize as is evident from the lower loss convergence. (c) the optimal number of top proposals per group for the piion-over-nteessction intersections is five. (d) as object detector classification performance improves, there are more correctly classified proposals that each cover parts of the ground truth. Instead of selecting a single proposal and discarding the rest, it is more effective to use all proposals to create a more comprehensive representation of the object.
Table Test-time augmentation benefits us further. Both box voting and Uol involve grouping predictions but differ in approach: box voting merges regressed boxes, while Uol groups original proposals before merging intersections. Test-time augmentation enhances both methods, but Uol remains effective without it, highlighting its robustness compared to box voting' reliance on test-time augmentation. Single-scaling uses an image scale of 666x400, while Multi-scaling incorporates both 666x400 and 2000x 1 200 as image scales.
Test- time adaptation setting
Method
None
Flip
Flip Single-scaling Flip Multi-scaling
Faster R- R-CNN w/ Box voting
37.5
37.7
38.0
38.8
Faster R-CNN w/ Uol
38.1
38.2
38.6
39.3
achieves improved localization accuracy during inference, as evidenced by Table 5. We conclude that regressing to intersections not only simplifies object detection but also enhances final detection performance through the union-over-innersectios strategy.
Figure 3 (c) shows the object detection performance as a function of the number of proposals used per group when taking the union over intersections. Our approach consistently uses maximum of five top proposals per region across all datasets and architectures, with all available proposals used if fewer than five are present. Performance improves from 2 to 5 proposals, then drops due to noisy, low-scoring proposals, achieving the best results with 5 proposals. This setting balances sufficient information for box merging while avoiding overly large boxes.
Intermediate group size is best for union-over-interseecie
Test-time augmentation benefits us further. We analyze the impact of test-time augmentation (TTA) on our iiiion-over-nneessettoon (Uol) strategy and compare it with box voting (Roman et al., 2019), which involves confidence-weighted box merging. TTA involves applying multi-scale testing and horizontal flips during inference, generating additional predictions for aggregation. Box voting relies heavily on the increased diversity of predictions from TTA, while Uol proves effective even without TTA and further benefits when combined with
With better classification, Uol becomes more powerful. We examine whether improvements in object classification will reduce or expand the gap between traditional methods and our approach. In an oracle experiment, we simulate varying proportions of known ground truth labels (20% 40%, 50%, 60%, 80%, and 100%) while the remaining predictions come from the network. As shown in Figure (d), as the accuracy of classification labels improves, our cinron--vve--nteeeetttioe increasingly outperforms traditional techniques. This is because correctly classified proposals, each covering parts of the ground truth, combine to form more comprehensive representation. In contrast, traditional methods, which rely on a single proposal, often miss portions of the object. These results highlight that as classification improves, the advantages of Uol become pronounced.
method

---

Published as a conference paper at ICLR 2025
Table 8: Effectiveness of the second regression stage. Uol inherently involves a second regression stage, where the combined intersections are further refined for improved performance. Adding just a second regression stage to the Faster R-CNN bascline alone does not yield similar benefits.
Method
MAPt
AP
Faster R-CNN
37.4
40.4
Faster R-CNN w/ 2nd regression
37.4
40.5
Faster R-CNN w/ Uol
38.1
40.9
Detection Results
Limitations
Faster CLCNNN
Faster R-CNN w/Ull
Figure 4: Qualitative analysis showcasing the effectiveness of our method applied to Faster R-CNN for object detection. Starting from the left, our method successfully removes the incorrect 'person" prediction, precisely localizes the entire bird, and identifies the third cat as distinet entity. However, in scenarios with clutter lot of overlap, observed in the right image, approach also consolidates multiple parrots into a single detection.
Effectiveness of the second regression stage. We include an ablation study to analyze the necessity of the second regression stage in method. Unlike baseline methods, where an additional regression head does not provide benefits due to redundant tasks, our approach assigns distinct roles to the regression stages. The first head focuses parts of the object, while the second refines the union of these parts into a cohesive bounding box. As shown in Table 8, this distinction makes the second regression stage effective, further improving performance.
Limitations. Our method benefits from multiple proposals, allowing the Union-overrIIterseeee strategy to unify their information effectively. However, for small objects, if only one proposal is available, our approach defaults to the baseline. Figure 4 highlights another limitation with crowded same-class objects, where closely positioned instances may merge into single box ("3 birds' example). Advanced grouping strategies could mitigate this issue.
CONCLUSION
In this paper, we introduced plug-and-play approach to object detection that enhances localization by focusing the intersection between proposals and ground truth and applying union-over-intersseioons instead of a winner-takes-ae strategy. These simple changes, requiring minimal modifications to existing architectures, consistently improve performance across two-stage, single-stage, and transformer-based detectors for object detection as well instance segmentation. Our method proves especially robust for stricter overlap thresholds and lower-quality proposals, making highly flexible across various frameworks. By improving localization, approach offers practical solution that strengthens object detection and segmentation without architectural dependencies.
Acknowledgement. This work has been financially supported by TomTom, the University of Amsterdam and the allowance of Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy. We also extend our gratitude the anonymous reviewers for their valuable feedback and insightful suggestions during the rebuttal stage, which considerably improved this work.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer-based object detection. arXiv preprint arXiv:2012.09955, 2020.
Alexey Bochkovskiy, Yai Yao Wang, and Hong-Yuan accuracy of object detection. arXiv preprint 2Xiv:2004.109344 2020.
Yuan Mark Liao. Yolov4: Optimal speed and
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. poft-nms-improwwe object detection with one line of code. In Proceedings of the IEEE international conference on computer vision, pp. 5561-5569, 2017.
Michael C. Burl, Markus Weber, and Pictro Perona. A probabilistic approach to object recognition using local photometry and global geometry. In Proceedings of the European conference computer vision, 1998.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6154-6162, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European conference on computer vision, pp. 213-229. Springer, 2020.
Xiyang Dai, Yinpeng Chen, Jianwei Yang, Pengchuan Zhang, Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2988-2997, 2021a.
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 1601-1610. 2021b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International conference learning representations, 2021.
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6569-6578, 2019.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 303-338 2010.
Pedro F. Felzenszwalb, Ross B. Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained past- models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021.
arXiv preprint arXiv:2107.088: 2021.
Ross Girshick. Fast r-cnn. Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 580-587, 2014.
Jiabo He, Sarah Erfani, Xingjun Ma, James Bailey, Ying Chi, and Xian-Sheng Hua. Alpha-IoU: A family of power intersection over union losses for bounding box regression. In Advances in neural information processing systems, volume 34, pp. 20230 20242, 2021.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Jan Hosang, Rodrigo Benenson, and Bernt Schiele. Learning non-maximum suppression. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 4507-4515 2017.
Rachel Huang, Jonathan Pedoeem, and Cuixian Chen. Yolo-lite: a real-time object detection algorithm optimized for non-gpu computers. In Proceedings of the IEEE international conference on big data, pp. 2503-2510 10, 2018.
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In Proceedings of the European conference on computer vision (ECCV), pp. 784-799, 2018.
Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In Proceedings of the European conference on computer vision, pp. 734-750, 2018.
Shuai Li, Minghan Li, Ruihuang Li, Chenhang He, and Lei Zhang. One-to-few label assignment for end-to-end dense detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7350-7359, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. In Proceedings of the international conference on learning representations, 2022.
Songtao Liu, Di Huang, and Yunhong Wang. Adaptive nms: Refining pedestrian detection in crowd. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6459-6468, 2019.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander Berg. Ssd: Single shot multibox detector. In Proceedings of the European conference computer vision, pp. 21-37. Springer, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 10012-10022, 2021.
Xin Lu, Buyu Li, Yuxin Yue, Quanquan Li, and Junjie Yan. Grid r-cnn. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7363-7372, 2019.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. Proceedings of the IEEE/CVF international conference on computer vision, pp. 3651-3660, 2021.
Bien Nguyen, Jihong Ju, Olaf Booij, Martin R Oswald, and Cees GM Snoek. Boxer: Box- attention for 2d and 3d transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4773-4782, 2022.
Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. One metric to measure them all: Localisation recall precision (Irp) for evaluating visual detection tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12))94469444 94663,666,,, 2021.
Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. Libra r-enn: Towards balanced learning for object detection. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 821-830. 2019.
12

---

Published as a conference paper at ICLR 2025
David Paz, Hengyuan Zhang, and Henrik Christensen. Tridentnet: A conditional generative model for dynamic trajectory generation. In Proceedings of the international conference on intelligent autonomous systems, pp. 403-416. Springer, 2021.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, volume 28, 2015.
Hamid Rezatofighi, Nathan Tsoi, Jun Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 658-666, 2019.
Solovyev Roman, Weimin Wang, and Tatiana Gabruseva. Weighted boxes fusion: ensembling boxes for object detection models. arXiv preprint vrXiv:1910.133000 2019.
Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris M Kitani. Rethinking transformer-based set prediction for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp. 361 1-3620, 2021.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 1105-6115 14. PMLR, 2019.
Mingxing Tan, Ruoming Pang. and Quoc V Le. EfficientDet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 10781-10790, 2020.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9627-9636, 2019.
Rohun Tripathi, Vasu Singla, Mahyar Najibi, Bharat Singh, Abhishek Sharma, and Larry Davis. Asap-nms: Accelerating mon-maximum suppression using spatially aware priors. arXiv preprint arXiv:2007.09785. 2020.
Lachlan Tychsen-Smin and Lars Petersson. Improving object localization with fitness nms and bounded iou loss. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6877-6885, 2018.
Paul Viola and Michael J. Jones. Robust real-time face detection. International Journal of Computer Vision, 2004.
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. In Advances in neural information processing systems, volume 33, pp. 17721-17732, 2020.
Shengkai Wu, Xiaoping Li, and Xinggang Wang. lou-aware single-stage object detector for accurate localization. Image and Vision Computing, 97:103911, 2020a.
Yue Wu, Yinpeng Chen, Lu Yuan, Zicheng Liu, Lijuan Wang, Hongzhi Li, and Yun Fu. Rethinking classification and localization for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10186-10195, 2020b.
Jiahui Yu, Yuning Jiang, Zhangyang Wang. Zhimin Cao, and Thomas Huang. Unitbox: An advanced object detection network. In Proceedings of the ACM international conference on multimedia, pp. 516-520, 2016.
13

---

Published as a conference paper at ICLR 2025
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03666 2022.
Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sunderhauf. Varifocalnet: An iou-aware dense object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8514-8523, 2021.
Zhaohui Zheng, Ping Wang. Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren. Distance-iou loss: Faster and better learning for bounding box regression. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 12993-13000, 2020.
Zhaohui Zheng, Ping Wang, Dongwei Ren, Wei Liu, Rongguang Ye, Qinghua Hu, and Wangmeng Zuo. Enhancing geometric factors in model learning and inference for object detection and instance segmentation. IEEE Trunsactions on Cybernetics, 52(8):8574-886, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In Proceedings of the international conference on learning representations, 2021.
14

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 ADDITIONAL RESULTS ON PASCAL VOC
To provide a more comprehensive evaluation, we include results for Faster R-CNN, Mask R-CNN, Cascade R- E-CNN, YOLÓv3 and Deformable DETR on the PASCAL VOC dataset. As shown in Table our Thioo-overrInterseetiio across all methods on PASCAL VOC, with again stronger gains observed at higher overlap thresholds.
(Uol) strategy also consistently improves detection performance
Table 9: Comparison on PASCAL VOC with additional methods. Adding Uol improves perfor-
mance across all methods, particularly for higher overlap thresholds.
Method
Backbone
APsof
AP
APsot
Faster R-CNN
R-50-fpn
71.2
51.3
6.8
Faster R-CNN w/ Uol
R-50-fpm
72.9
55.3
11.3
Mask R-CNN
R-50-fpn
73.1
52.7
7.9
Mask R-CNN w/ Uo!
R-50-fpn
74.0
56.9
13.1
Cascade R-CNN
R-50-fpn
77.5
54.1
10.5
Cascade R-CNN w/ Uol
R-50-fpn
78.8
57.9
14.8
YOLÓv3
DarNet-53
56.1
41.7
5.3
YOLOv3 w/ Uol
DarNet-53
57.0
44.2
8.8
Deformable DETR
R-50-fpn
83.2
68.5
22.1
Deformable DETR Uol
R-50-fpn
84.1
26.5
A.2 EFFECT OF IOU TIRESHOLDD ON GROUPING STRATEGY
To evaluate the impact of the IoU threshold (k) on our Uol strategy, we conducted an ablation study using Faster R-CNN with ResNer-101 backbone. The IoU threshold influences group sizes during the grouping stage: lowering the threshold (c.g.. 0.3) increases group sizes, while raising it (c.g.. 0.7) decreases group sizes. As shown Table 10, the optimal number of proposals shifts to 4 for k=0.7 but remains at for k00.3. Importantly, the highest mAP is achieved with k=0.5, reaffirming the effectiveness of the default setting. These results further validate the robustness of our approach to the IoU threshold choice.
Table 10: Effect of loU threshold (k) grouping strategy. Lowering raising the IoU threshold shifts the optimal number of proposals, but the highest mAP is consistently achieved with 0.5.
Number of Proposals
IoU Threshold &
4
5
7
0.3
38.1
39.0
39.5
39.2
38.4
0.5
39.3
39.8
40.3
39.2
38.5
0.7
38.5
39.9
39.4
38.1
37.7
A.3 IMPACT OF ADDITIONAL CLASSIFICATION HEAD ON CONFIDENCE SCORES
To assess the effect adding a classification head for the combined boxes, we conducted an ablation using the Faster R-CNN baseline. As shown in Table 11, this modification resulted in marginal improvements in mAP and AP However, since the gains are minimal and the primary improvements in our approach from better localization (see Table 5 in the main paper), did not include the additional classification stage in our main method. These results are provided here for completeness.
A.4 EVALUATING FULL-OBIECT REGRESSION VERSUS PART-BASED REGRESSION
To evaluate the implications of applying the Union of Proposals with the original regression objective, we conducted an experiment using full-object regression, where the model predicts the complete ground truth bounding boxes instead of focusing part-based regions. As shown in the table
15

---

Published as a conference paper at ICLR 2025
Table 11: Effect of adding a classification head for combined boxes. Adding classification stage offers minimal improvements, reaffirming that the primary gains of our approach come from improved localization
Method
mAPf
APıst
Faster R-CNN w/ Uol
38.1
40.9
Faster R-CNN Uol and 2nd classification stage
38.2
41.1
below, full-object regression results in looser bounding boxes after combining proposals, even with second-stage regression. This leads to a considerable drop in performance compared our part- based regression strategy (Uol). The results confirm that focusing on parts of objects enables finer localization and better overall accuracy.
Table 12:
Comparison of full-object regression versus part-based regression (Uol). Part-based regression achieves better performance by enabling finer localization.
Method
mAPt
APıst
Full-object regression
35.8
37.5
Part-based regression (Uol)
38.1
40.9
A.5 ADAPTATION OF Uol FOR DEFORMABLE DETR AND YOLO
Here we provide more details on how our approach integrates with Deformable DETR and YOLO. Code is provided at https: : / / github.. com/ ari trabhowiik //oI.
In the standard Deformable DETR, each query predicts bounding box via the regression head after the decoder. These bounding boxes are then matched to full ground truth boxes using Hungarian matching, with the best query assigned to each ground truth box. Our method modifies this matching process. Specifically, we divide each ground truth box into four quadrants (top-left, top-right, bottom- left, bottom-right), treating these quadrants as pert-level targets. During training, each query is assigned to a specific quadrant based its IoU overlap, and the regression head is tasked with predicting the corresponding part of the ground truth. At inference, these part-level predictions are grouped using Intersection- based Grouping strategy and merged into complete bounding boxes.
Similarly, for YOLO, we adapt its object-to-gut assignment strategy. Instead of assigning an object single grid cell based on the object's 's center, to multiple grid cells it overlaps sufficiently with their anchors. Each anchor predicts the part of the object it best overlaps with, and during inference, our grouping strategy merges these predictions into final bounding boxes. These modifications retain the original architecture of YOLO while enabling part-based regression and grouping.
loU-based criterion. An object is assigned
A.6 MORE QUALITATIVE RESULTS
16

---

Published as a conference paper at ICLR 2025
Detection Results
Limitations
Faster R- CNN
Faster R. CNN
Figure 5: Qualitative analysis showcasing the effectiveness of our method applied to Faster R-CNN for object detection. Starting from the left, our Uol approach diminishes false positives and achieves tighter localization for both the bird and the cow, outperforming the baseline method in detecting large objects. However, in scenarios with clutter or significant overlap, our method may also aggregate objects into a single group, as seen in the rightmost image with the two cars grouped as one in the right corner.
Figure 6: Qualitative analysis of method for instance segmentation the COCO 2017 dataset. Our method merges information from multiple proposals, naturally leading to precise localization and segmentation masks.
17


Reviews:
[{"review_id": "QWrsfeqjrN", "reviewer": "Reviewer_zZMj", "summary": "This paper revisits the problem of predicting box locations in object detection architectures. Traditional approaches typically regress box proposals to maximize the intersection-over-union (IoU) score with the ground truth and then apply non-maximum suppression (NMS) to retain only the highest scoring box in each region. The authors argue that both steps are suboptimal: regressing proposals to the entire ground truth is a challenging task, and NMS ignores potentially useful information from other boxes. To address these issues, the authors propose a simpler method—regressing proposals only to the intersection area between the proposal and the ground truth, rather than the entire ground truth, thus avoiding the need for proposals to extrapolate beyond their visual scope and improving localization accuracy. Additionally, they suggest generating the final box outputs by taking the union of the regressed intersections from all boxes in a region, rather than using a winner-takes-all approach, thus preserving valuable information from other boxes. This plug-and-play method integrates seamlessly into any detection architecture with minimal modifications and significantly enhances object localization and instance segmentation. The experimental results demonstrate its broad applicability and impressive performance across a variety of detection and segmentation tasks.", "questions": "1.\tTo enhance the overall aesthetic consistency of the paper's layout, a few minor formatting adjustments are recommended:\n\n1）\tSome bolded paragraph titles (e.g., the bold text followed by a period, such as \"Problem Statement.\" in line 150) are set on a new line rather than immediately followed by the paragraph content. This may be intentional due to title length, or it might be a formatting issue. It is suggested to align these paragraph titles directly with the content following them. For example, \"Intersection-based Grouping.\" in line 203 and \"Regressing to intersections is simply an easier task.\" in line 448 should follow this format.\n\n2）\tThe title \"Inference cost\" in line 267 lacks a period and should be updated to \"Inference Cost.\" Additionally, the caption of Figure 4 (\"Qualitative results…\") is not bolded like the caption in Figure 5; please consider ensuring consistency with Figure 5's formatting.\n\n3）\tThe paragraph titles throughout the paper vary between Title Case and Sentence Case. Although I will not list each instance here, please consider standardizing these for consistency.\n\n2.\tTo ensure accuracy and clarity, a few content-related questions have also arisen:\n\n1）\tIn line 220, the text states, \"The regression function r : B → R maps each combined box to a final target box.\" Here, does R refer to the set of all ground truths? Previous sections define G or T as representing all ground truths, so it is unclear why R is used here instead of G or T. Could this be clarified?\n\n2）\tTo guarantee the precision of the pseudocode in the Post-Processing Stage on the right side of Figure 2, could you confirm if it would be necessary to include \"P ← P \\ p_i\" in the conditional statement within the red dashed box to ensure the \"while P ≠ empty do\" loop functions correctly (assuming the pseudocode within the green solid box is disregarded)?\n\n3.\tCould you clarify why the conditional statement in the pseudocode within the green solid box on the right side of Figure 2 uses \"iou(H, pi) ≥ k\" rather than \"iou(M, bi) ≥ k\" as in the red dashed box? Has there been any experimentation to validate the effectiveness of \"iou(H, pi) ≥ k\" over \"iou(M, bi) ≥ k\"? If not, please consider including such justification.\n\n4.\tIn Table 3, there is a Comparison of instance segmentation on MS-COCO data, showing considerable improvements in AP for various object sizes (i.e., small, medium, and large objects as defined by COCO). The COCO definition of \"small objects\" encompasses those with an area under 32×32 pixels. Compared with small-object detection tasks in infrared imaging (where small objects typically occupy less than 10×10 pixels and may even be as small as a few pixels), COCO's small objects are relatively large. Could the authors clarify whether their proposed UoI method would also be effective for these considerably smaller targets?\n\n5.\tFigure 3(c) demonstrates that on the MS-COCO dataset, Faster R-CNN achieves optimal performance with the UoI strategy when the intermediate group size is 5. Have the authors considered studying the effect of varying the IoU threshold k in the UoI strategy (i.e., the IoU threshold used in the conditional statement in the pseudocode within the green solid box on the right side of Figure 2)?\n\n6.\tFigures 4 and 5 highlight certain limitations of the proposed UoI approach, specifically in handling crowded scenes. Could the authors specify if these crowded scenes refer to instances of multiple objects or instances of the same class, or do they also include multiple instances of different classes? Additionally, might the size of objects or instances in these crowded scenes significantly restrict UoI’s performance? Please provide a detailed discussion on these aspects.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The \"UoI\" strategy proposed in this paper is highly innovative and practical, as it can seamlessly integrate into existing two-stage object detection networks, significantly improving detection performance. The authors creatively introduce the idea of regressing proposals to the intersection area with the ground truth, rather than the entire ground truth, addressing the challenges of regressing to the full target region. This not only improves localization accuracy but also offers an elegant and practical solution that can be easily adopted within current detection architectures. The paper is clearly written, with a logical structure that makes the proposed UoI method easy to understand. The figures and pseudocode are well-presented, offering an intuitive explanation of the approach. Furthermore, the extensive ablation experiments validate the effectiveness of UoI across various detectors, further strengthening the paper’s contribution. Overall, the paper excels in originality, clarity, and practical applicability.", "weaknesses": "While the paper excels in terms of methodological innovation, clarity of writing, and the presentation of figures and pseudocode, there is room for improvement in the design of comparative and ablation experiments. Specifically, the validation of the proposed UoI strategy is limited to a few methods, which may constrain its generalizability. For example, Table 1 only compares Faster R-CNN and Def-DETR’s detection performance on the PASCAL VOC dataset, while Table 2 provides comparisons for five methods (Faster R-CNN, Mask R-CNN, Cascade R-CNN, YOLOv3, and Def-DETR) on the MS-COCO dataset. However, why are methods like Mask R-CNN, Cascade R-CNN, and YOLOv3 not included in the PASCAL VOC comparison? Including these methods would strengthen the results and provide a more comprehensive evaluation.\n\nMoreover, Tables 4-7 and Figure 3 mainly show results for Faster R-CNN, without testing the UoI strategy on other classic detection architectures. To further validate the effectiveness of UoI, it would be beneficial to conduct experiments on additional well-established methods, demonstrating its applicability and advantages across a broader range of architectures. This would enhance the paper’s persuasiveness and provide stronger evidence for the practical utility of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSpKRwGM2S", "reviewer": "Reviewer_GWg2", "summary": "This paper propose Union-over-Intersections method for object detection, which predict intersection box between proposal box and ground truth box and combine overlapped boxes into the final instance.", "questions": "- Intuitively, this method seems more beneficial for detecting larger objects, as the \"combine box\" approach tends to union the box when merging bounding boxes within a group, potentially biasing towards larger boxes. Is combining multiple predictions still an ideal detection strategy for small objects? The smaller targets are often adequately covered by a single proposal, which means the approach sometimes defaults to the standard paradigm. Although the main experimental results indicate an improvement in mAP across object sizes, I would encourage the authors to further analyze how this paradigm performs for targets of different sizes.\n\n- It seems that the target of the regression stage is dynamic. I would encourage the authors to further analyze about the concerned instability.\n\n- The confidence scores of the combined boxes are not introduced in method descriptions. It is important because score is essential for mAP. (good score also benifits mAP)\n\n- Isn't the combined boxes the new boxes? Why the number of predictions remain? Does the mAP evaluator obtain more predictions than baseline?\n> By design, our method yields the same number of predictions as current detectors and is compatible with any NMS variant.\n\n- Is regression function r a learnable regression head? Why an additional regression refinement module is required? Isn't the combined boxes the prediction boxes? (We nevel refine NMS outputs with an additional regression module. The baseline method seems lack one refinement module)\n\n- How does the detectors with other types of proposal (e.g. DETR with non-two-stage-initialized object queries, FCOS with point based dense head) be integrated seamlessly by UoI? It seems that this method can only be a box refinement module for these non-box-proposal methods.\n> Our plug-and-play method integrates seamlessly into any detection architecture with minimal modifications", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The authors provide a thorough explanation of the methodology, making it easy to understand. \n- The authors present a novel insight and have correctly implemented the methodology, achieving promising results.\n- The experimental content is comprehensive, covering key results across popular detection algorithms and both detection and segmentation tasks. The ablation study effectively discusses the strengths of the approach and includes an analysis of its limitations.", "weaknesses": "- While this paper attempts to challenge the conventional box refinement paradigm (predict box directly and Winner-Takes-All for post-processing) in object detection models, it does not sufficiently demonstrate its viability as a true alternative to the original paradigm. (NOTE that although I mention this as a weakness, I still consider the approach to be novel and appreciate their effort to explore new insights. However, from an intuitive standpoint, the method does not strongly compel me to adopt it.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WlsZzwmWkG", "reviewer": "Reviewer_fZZc", "summary": "The authors propose the Union-over-Intersections (UoI) method, which modifies the object detection pipeline in two key ways:\n\n1. **Intersection-Focused Regression** : Instead of regressing proposals to the entire ground truth, the method focuses on the intersection area, simplifying the regression task and improving localization accuracy.\n2. **Union of Proposals** : In the post-processing stage, rather than using a winner-takes-all strategy, the UoI method combines information from all proposals by taking the union of their regressed intersections, enhancing the final detection output.\n\nThe paper demonstrates decent improvements in object localization and instance segmentation across various architectures, including Faster R-CNN and YOLOv3. The UoI method is adaptable and can be integrated into existing detection frameworks with minimal changes, making it a promising advancement in the field of object detection.", "questions": "1. What would be the implications of applying the Union of Proposals to learn the complete ground truth bounding boxes without altering the learning objectives?\n2. Please provide a rationale for the weaknesses summarized above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The method is straightforward, requiring only modifications to the training objectives and post-processing steps, making it easy to implement within existing proposal-based detectors.\n* Experimental results demonstrate a solid improvement in accuracy while introducing only a minimal additional computational overhead", "weaknesses": "My primary concern lies in the novelty and significance of the method. The paper emphasizes the use of intersections, but in my view, intersections are a subset of Intersection-over-Union (IoU). Learning IoU inherently involves understanding intersections, and there are already existing works focused on learning IoU, such as IoU loss. This diminishes the overall importance of the paper. Additionally, learning the complete ground truth bounding box implies the need to understand both intersections and unions. With proper tuning, I believe that incorporating more information would benefit neural network learning rather than focusing solely on a partial representation.\n\nThe concept of the Union of Proposals appears to function similarly to a voting mechanism. Box voting for accuracy improvement has already been extensively validated through test-time augmentation techniques in challenges like VOC and COCO. Moreover, learning the complete ground truth bounding box can naturally facilitate a voting process as well.\n\nLastly, there are concerns regarding generalizability. The method relies on proposals, which limits its applicability in proposal-free approaches, such as YOLO and FCOS. This reliance on a specific number of proposals necessitates tuning, making integration with proposal-free methods less convenient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QE6DOTmONn", "reviewer": "Reviewer_JFjT", "summary": "The paper proposes to predict a set of intersection boxes for one given ground-truth box and then take the union of them to form the final detection box. The method is simple yet effective and is verified on 5 classic detectors.", "questions": "Does the difference between the middle group of Table 1 (Faster R-CNN) and Table 4 lie in whether Soft-NMS is applied or not?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple yet effective and easy to be re-produced.\n\n2. The discussion of the problem is quite clear; the paper is well-written.\n\n3. The experiments and analyses are sufficient to support the soundness of the proposed method over the vanilla regression method. The results also verify its good versatility for a variety of object detectors.", "weaknesses": "1. As a box regression method, the paper did not compare the proposed method with IoU-based loss functions, e.g., GIoU [r1], CIoU [r2], Alpha-IoU [r3], EIoU [r4], etc, making the superiority of the methods less convincing.\n\n2. It is mentioned by line 216, the proposed Intersection-based Grouping is compatible with any NMS variant. In the experiment, only Soft-NMS is tested. I suggest the authors provide more ablation on this.\n\n3. In Fig.3(c), it shows that the performance may be sensitive to the number of proposals on Faster R-CNN and MS COCO. This characteristic may limit its robustness in different scenarios, e.g., different detectors or different datasets.\n\n4. Why the datasets of ablation studies switch frequently between COCO and VOC? Not consistent.\n \n5. The font size of Fig.3 is too small.\n\n[r1] Rezatofighi H, Tsoi N, Gwak J Y, et al. Generalized intersection over union: A metric and a loss for bounding box regression. CVPR 2019.\n\n[r2] Zheng Z, Wang P, Liu W, et al. Distance-IoU loss: Faster and better learning for bounding box regression. AAAI 2020.\n\n[r3] He J, Erfani S, Ma X, et al. $\\alpha $-IoU: A family of power intersection over union losses for bounding box regression. NeurIPS 2021.\n\n[r4] Zhang Y F, Ren W, Zhang Z, et al. Focal and efficient IOU loss for accurate bounding box regression[J]. Neurocomputing, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)