Union-over-Intersections: Object Detection beyond Winner-Takes-All

## Abstract

Abstract This paper revisits the pipeline for detecting objects in images with proposals. For any object detector, the obtained box proposals or queries need to be classified and regressed towards ground truth boxes. The common solution for the final predictions is to directly maximize the overlap between each proposal and the ground truth box, followed by a winner-takes-all ranking or non-maximum suppression. In this work, we propose a simple yet effective alternative. For proposal regression, we solve a simpler problem where we regress to the area of intersection between proposal and ground truth. In this way, each proposal only specifies which part contains the object, avoiding a blind inpainting problem where proposals need to be regressed beyond their visual scope. In turn, we replace the winner-takes-all strategy and obtain the final prediction by taking the union over the regressed intersections of a proposal group surrounding an object. Our revisited approach comes with minimal changes to the detection pipeline and can be plugged into any existing method. We show that our approach directly improves canonical object detection and instance segmentation architectures, highlighting the utility of intersection-based regression and grouping.

## 1 Introduction

Figure: Figure 1: Proposed modifications to traditional object detection. The object detection process begins with a set of box proposals. Traditional methods regress each proposal towards the ground truth and select the best candidate as the final prediction. Our approach, however, regresses each proposal only to its intersection with the ground truth. We then unite these intersections, refine the resulting proposal, and obtain our final prediction. We utilize the complementary information from multiple proposals, leading to a more accurate prediction.
Refer to caption: /html/2311.18512/assets/x1.png

Object detection forms a long-standing challenge in computer vision, with the goal of spatially localizing and classifying object boxes in image collections. In the past decade, object detection has witnessed astonishing progress due to many advances in various stages of the detection pipeline. The evolution of object detection backbones, from the foundational shifts in R-CNN [] to Faster R-CNN [], and the paradigm-changing YOLO architectures [, , , , ], to the recent integration of transformer networks [, , , ], represents a significant trajectory of progress in enhancing feature extraction and detection capabilities. In addition, the evolution of object detection loss functions, including the focal loss [] and IOU-aware loss [, ], has addressed challenges like class imbalance and precise localization, further advancing automated detection ability. Regardless of the choice of the backbone, the number of stages, or the optimization, all these works obtain a set of proposal boxes, grid-boxes or queries. To arrive at their final detection, the proposal boxes are classified and regressed for spatial alignment, after which (optionally) non-maximum suppression is performed to remove all but the highest scoring box per location. Put differently, the consensus is to have each proposal box solve the intersection-over-union alignment with ground truth object locations, followed by a winner-takes-all approach to obtain the best box within a spatial region. In this paper, we revisit this fundamental problem of accurate object localization and optimal candidate selection.

We identify two issues with the common setup of object detection and propose a simple solution.
First, the goal of any object detector, *e.g*., [, , ], regardless of its architecture, is to accurately learn proposal boxes that independently represent ground truth objects in an image. As highlighted in [Fig. 1](#S1.F1), this is often an ill-posed problem. During the forward pass of detectors, proposal boxes are generated, typically capturing only a portion of the ground truth within their scope. Learning to regress for perfect ground truth alignment leads to a blind in-painting challenge. Second, during the forward pass of all detectors, one common thing that occurs is that there are always multiple proposals, or queries, relating to one ground truth object. Although they provide complementary views to the ground truth, non-maximum suppression or ranking only serves to exclude duplicate detections. By doing so, these methods overlook and ignore the valuable information in discarded proposals.

Our solution is straightforward: we decompose the problems of proposal-to-ground truth regression and proposal-candidate selection into easier to solve intersection and union problems. We make the following contributions:

- (1)
Intersection-based Regression. We set a new target for the regression of proposals: rather than predicting the overlap with the ground truth, we only predict the region of intersection.
Thus, we only regress towards the ground truth within the visual scope of the proposal.
- (2)
Intersection-based Grouping. Given a group of proposals with predicted ground truth intersections, we form the final prediction by taking the union over the intersection regions. In other words, instead of selecting only the most confident proposal in a region, we use the wisdom of the crowd to form our final predictions.

The two stages come with minimal changes to existing object detection pipelines. We only change the regression head target towards intersections and modify the post-processing with a grouping process over a winner-takes-all. Our approach can therefore be directly plugged into any object detector. Despite its technical simplicity, the proposed decomposition directly improves the detection performance. We show how our revisited approach improves canonical detection and instance segmentation methods across multiple dataset, especially at high overlap thresholds during evaluation. As classification and regression performance continues to improve, we lastly perform a controlled experiment where we show that our approach benefits more from future advances, making it the preferred formulation for the next generation of object detectors.

## 2 Related Work

### 2.1 Object Detection Architectures

Object detection has seen various advancements with the development of deep learning architectures. These architectures can be broadly categorized into: single-stage, two-stage, and transformer-based detectors, each with their own characteristics and merits.

Single-stage detectors such as SSD [], YOLO [] series (including YOLOv3 [], YOLOv4 [], YOLOX [], YOLO-Lite []), RetinaNet [], EfficientDet [], CornerNet [], CenterNet [], and FCOS [] prioritize speed, directly predicting object classes and bounding boxes in one pass. RetinaNet introduces Focal Loss for class imbalance and small object detection, while EfficientDet leverages EfficientNet [] backbones for scalable efficiency. Innovations in CornerNet, CenterNet, and FCOS focus on anchor-free methods, enhancing real-time efficiency.

In the realm of two-stage object detectors, the R-CNN family, comprising R-CNN [], Fast R-CNN [], Faster R-CNN [] (introducing end-to-end training with Region Proposal Networks), Mask R-CNN [] (adding segmentation capabilities), Cascade R-CNN [] (improving accuracy via a multi-stage framework), and Libra R-CNN [] (addressing training imbalances), has been crucial. Enhancements include TridentNet [], which handles scale variation with parallel branches, and additional methods like Grid R-CNN [] for precise localization and Double-Head R-CNN [], which differentiates between classification and bounding box regression, further enriching the field.

Transformer-based models like DETR [], Deformable DETR [], TSP-FCOS [], Swin Transformer [], and ViT-FRCNN [] have revolutionized object detection. DETR views detection as set prediction, Deformable DETR adds deformable attention for focused image analysis, TSP-FCOS integrates transformers with FCOS [] for scale variance, Swin Transformer employs a hierarchical approach, and ViT-FRCNN combines Vision Transformer [] with Faster R-CNN [] for feature extraction. Further advancements include Boxer transformer [], DINO [], DAB-DETR [], UP-DETR [], Dynamic DETR [], and Conditional DETR [], enhancing aspects like training efficiency and dynamic prediction, underscoring transformers’ adaptability in object detection.

Despite the diversity in approaches, a common aspect across these three paradigms is the presence of a classification and bounding box regression stage following the proposal stage. In this stage, each proposal box is tasked with independently solving the intersection-over-union problem, a challenge that is inherently ill-posed. Proposal boxes often suffer from misalignment, capturing only a portion of the ground truth within their scope. Moreover, non-maximum suppression or ranking stages throw away all proposal boxes except for the top scoring one, even though the proposal boxes provide complementary views about the ground truth. We tackle these limitations by decomposing the problem into separate intersection and union stages, resulting in an easier and better performing detection pipeline.

### 2.2 Post-processing for Object Detection

After the regression stage, selection of the best regressed proposal from a multitude of candidate detections ensures accuracy, with Non-Maximum Suppression [, , , ] (NMS) being a key technique for selecting the best proposals by eliminating redundant detections. Traditional NMS, while effective, has been surpassed by variants such as Soft-NMS [], which adjusts confidence scores instead of discarding detections, improving performance in crowded scenes. Learning NMS [] introduces adaptability by incorporating suppression criteria into the neural network training. Further refinements like IoU-aware NMS [, ] and Distance-IoU (DIoU) NMS [] consider both overlap and spatial relationships between bounding boxes for more precise detection. Additional post-processing techniques include Matrix NMS [], which employs a matrix operation for efficient suppression, and Adaptive NMS [], which dynamically adjusts thresholds based on object densities. Furthermore, ASAP-NMS [] optimizes processing speed, while Fitness NMS [] considers both detection scores and spatial fitness for more accurate suppression to further enrich post-processing in object detection.

The improved post processing techniques address the limitation of selecting a single winner from each cluster of proposals at a location. However, all techniques still assume that each individual box is already fully aligned with the corresponding ground truth box and serve primarily to remove duplicate detections. In this work, we seek to get wisdom from the crowd by using all proposals at the same location to form the final object detection output. We can do so by making minimal adjustments to the non-maximum suppression pipeline.

Figure: Figure 2: Revisiting Proposal-based Object Detection. We propose a novel strategy for proposal-based object detection. The first part of the detection pipeline that encodes features for each proposal remains unchanged, but the subsequent bounding box regression is different. Top row: Existing detectors regress each proposal towards the ground truth bounding box. This compels the regression network to extend the proposal to regions beyond its initial coverage, needing to predict the ground truth bounding box size potentially far beyond its visible scope. This is illustrated with proposals around the seal’s head that need to be enlarged far *outside* the network input (outside is grayed out). The subsequent non-maximum suppression (NMS) selects the locally best rated regressed proposal as the detection result and all other regressed proposals are discarded. Bottom row: Our Intersection-based Regression approach aims to predict only the *intersection* of the proposal box with the ground truth bounding box. Hence, it only reasons about data which is *inside* the network input, thus solving a more well-posed task. Another key difference of our approach to traditional detectors is that we do not discard all but one proposals, but exploit the Intersection-based Regression output from *multiple* proposals and merge them. In particular, we merge the top-k predicted intersections in an Intersection-based Grouping procedure.
Refer to caption: /html/2311.18512/assets/x6.png

## 3 Method

Problem statement. In traditional object detection, bounding box regression involves learning a mapping function $f$ that transforms a proposal box $P{=}(P_{x1},P_{y1},P_{x2},P_{y2})$ to closely approximate a ground truth box $G{=}(G_{x1},G_{y1},G_{x2},G_{y2})$, where $(x1,y1,x2,y2)$ refers to the top-left $x$ and $y$ coordinates and bottom-right $x$ and $y$ coordinates respectively. Typically, this is achieved by minimizing the $L1$ or $L2$ loss between the predicted box and the ground truth. However, this approach often poses a challenge: the proposal boxes $P$ are commonly badly aligned, with only a part of the ground truth $G$ visible in the scope of the proposal.
Consequently, the function $f$ is compelled to extend $P$ to regions beyond its initial coverage, essentially requiring it to predict beyond its visible scope. This scenario leads to an ill-posed problem, as $f$ is forced to infer parts of $G$ not contained within $P$.
Then, the set of all proposals for a ground truth object $\mathcal{P}{=}\{P_{1},P_{2},\ldots,P_{n}\}$, providing complementary views, are sorted and only one candidate is picked via non-maximum suppression or ranking. This assumes that the intersection-over-union problem is already solved and only serves to exclude duplicate detections in the same region.

To address these issues, first, we redefine the task as an intersection learning problem in Section [3.1](#S3.SS1). For individual proposal boxes, we only require solving the intersection alignment with ground truth boxes, *i.e*. we only need to regress to the part of the proposal that is shared with the ground truth.
Then, instead of discarding complementary informations from proposals, our method strategically learns to leverage these fragments in Section [3.2](#S3.SS2). Given a collection of proposal boxes and their identified intersection, we can replace the non-maximum suppression with a union operator, where we identify the final object as the *union-over-intersection* of all the proposals in the same region. Figure [2](#S2.F2) shows how these two simpler tasks lead to a final object detection.

### 3.1 Intersection-based Regression

We redefine the regression task as an intersection learning problem. Instead of regressing to the entire ground truth box, each proposal is tasked to regress only to the visible part of the ground truth object, i.e., the intersection between the proposal and the ground truth. This makes the mapping function $f$ to be well-defined, making it easier to learn such a transformation.

Formulation. Let $I{=}(I_{x1},I_{y1},I_{x2},I_{y2})$ be the intersection of boxes $P$ and $G$. The new task is to learn a mapping function $f^{\prime}$ such that $f^{\prime}(P)\approx I$.
For supervision purposes, the intersection is computed as:

$$ $\displaystyle I_{x1}$ $\displaystyle=\max(P_{x1},G_{x1})\enspace,$ (1) $\displaystyle I_{y1}$ $\displaystyle=\max(P_{y1},G_{y1})\enspace,$ (2) $\displaystyle I_{x2}$ $\displaystyle=\min(P_{x2},G_{x2})\enspace,$ (3) $\displaystyle I_{y2}$ $\displaystyle=\min(P_{y2},G_{y2})\enspace.$ (4) $$

The loss function for this task is defined as:

$$ $L_{t}=\sum_{i\in\{x1,y1,x2,y2\}}\big{|}f^{\prime}(P_{i})-I_{i}\big{|}^{t}\enspace,$ (5) $$

where $t$ takes the value of either 1 or 2, corresponding to the application of $L_{1}$ or $L_{2}$ loss, respectively, during regression training.

The second aspect of our approach involves strategically leveraging the partial information contained within the multiple proposals generated for a single ground truth object.

### 3.2 Intersection-based Grouping

In conventional object detection methods, a single proposal is selected to represent the final detection, while the remaining proposals are discarded. Instead of discarding these proposals, which contain valuable but fragmented information, our method learns to merge these segments effectively. As a result, our approach generates a more comprehensive and accurate representation of the ground truth object.

Formulation. Let us denote the set of all proposals for a ground truth object as $\mathcal{P}{=}\{P_{1},P_{2},\ldots,P_{n}\}$, and the corresponding intersections with the ground truth as $\mathcal{I}{=}\{I_{1},I_{2},\ldots,I_{n}\}$. Our goal is to find a combination of these intersections that best represents the ground truth object. We define a combination function $c:\mathcal{I}\rightarrow\mathbb{R}^{4}$ that takes a set of intersections and outputs a bounding box. The task is to learn this function such that it minimizes the loss between the combined box and the ground truth box $G$:

$$ $L=\sum_{i\in\{x1,y1,x2,y2\}}|c_{i}(\mathcal{I})-G_{i}|\enspace.$ (6) $$

Our model is designed to comprehend the partial nature of proposals via intersection regression. The combination of intersection is done in two stages: grouping and regression, as described below.

Grouping. To perform the grouping, we introduce a grouping function $g:\mathcal{P}\rightarrow\mathcal{G}$ that maps the set of proposals $\mathcal{P}$ to a set of groups $\mathcal{G}{=}\{g_{1},g_{2},\ldots,g_{m}\}$. Each group $g_{j}$ is a subset of $\mathcal{P}$ and represents a potential object in the image. The grouping function is designed to group together proposals that are likely to belong to the same object. This is achieved by considering the spatial overlap and semantic similarity between proposals. Once the proposals are grouped, we combine the intersections corresponding to the proposals of each group to obtain a set of combined boxes $\mathcal{B}{=}\{B_{1},B_{2},\ldots,B_{m}\}$. Each combined box $B_{j}$ is a candidate bounding box for the object represented by group $g_{j}$.

Refinement. Finally, we perform a regression step to refine the combined boxes. We define a regression function $r:\mathcal{B}\rightarrow\mathcal{R}$ that maps each combined box to a final target box. The regression function is learned to minimize the loss between the target boxes and the ground truth boxes:

$$ $L=\sum_{j=1}^{m}\sum_{i\in\{x,y,w,h\}}|r(B_{ji})-T_{ji}|\enspace,$ (7) $$

where $B_{ji}$ is the $i$-th coordinate of the combined box $B_{j}$ and $T_{ji}$ is the $i$-th coordinate of the ground truth box corresponding to $B_{j}$.
This approach enables the integration of valuable information from multiple proposals into a single, enhanced proposal. Rather than selecting a single optimal candidate and discarding the others, our method extracts and amalgamates the most relevant aspects from each proposal, thereby constructing a superior candidate that more accurately represents the target ground truth object.

Integrating the decomposition into existing object detectors requires only a few alterations. For the Intersection-based Regression, we only need to change the target coordinates in the regression head from the ground truth box to the intersection area between proposal and ground truth. To obtain final object detection outputs via Intersection-based Grouping, we sort and cluster proposals akin to non-maximum suppression. Rather than keeping only the top box, we take the union of the regressed intersections in the same cluster as the output. Despite the simplicity of the approach, we show that decomposing the intersection-over-union alignment in object detection directly impacts performance.

## 4 Experiments

### 4.1 Datasets, Evaluation and Implementation

Datasets.
In our experimental setup, we conduct comprehensive evaluations on two widely recognized datasets, COCO [] and PASCAL VOC [], covering two key tasks in computer vision: object detection and instance segmentation. These datasets are chosen for their diversity and complexity, offering a robust platform to assess the effectiveness of our proposed method.

$\triangleright$ MS-COCO 2017: This dataset is a standard benchmark in object detection and instance segmentation. It includes 80 object categories with 118k images for training and 5k for evaluation. The dataset is known for its diversity, containing a wide range of objects and scenes. It features a maximum of 93 object instances per image, with an average of 7 objects.

$\triangleright$ PASCAL VOC: We leverage the 2007 and 2012 editions. The VOC2007 dataset includes 5,011 images for training and 4,952 images for testing. In addition, the VOC2012 dataset provides an additional 11,540 images for training. This dataset covers 20 object categories and is notable for its detailed annotations for object detection. The 4,952 images from VOC2007 are used for testing.

Evaluation Criteria. We adhere to the standard evaluation protocols established in the literature for both datasets for the task of object detection. For COCO [], we use the mean Average Precision (mAP) across IoU thresholds from 0.5 to 0.95, mAP at specific IoU thresholds (0.5 and 0.75), multiple scales (small, medium and large) and Average Recall (AR) metrics. For PASCAL VOC [], we use the mAP metric at IoU thresholds of 0.5, 0.6, 0.7, 0.8 and 0.9. For instance segmentation, we use the mean Average Precision (mAP) across IoU thresholds from 0.5 to 0.95 and across multiple scales (small, medium and large). These provide a comprehensive assessment of our model’s performance.

Implementation with Three Base Detectors.
We test our approach with three base detectors: Faster R-CNN [], Mask R-CNN [], and YOLO v3 []. This selection is aimed at exploring a broader range of architectures encompassing both single and two stage detectors.

For the COCO dataset, we utilize Faster R-CNN and Mask R-CNN with Feature Pyramid Networks [] and ResNet101 [] and ResNet50 backbones. The models are configured with an image size of 1333$\times$800, random horizontal flip augmentation, and 512 proposals. Training is conducted using Stochastic Gradient Descent [] with a mini-batch size of 2 per GPU, an initial learning rate of 0.02 (reduced by a factor of 10 at the 8th and 11th epochs), a weight decay of $10^{-4}$, and a momentum of 0.9. We limit the training to 12 epochs for these models.
YOLO v3, implemented with a Darknet-53 backbone, is another key component of our experiments on COCO. The model is trained with image size of 608$\times$608 and employs random flip augmentation. We use a batch size of 64, a Stochastic Gradient Descent optimizer with a learning rate of 0.001, a momentum of 0.9, and a weight decay of 0.0005. The training duration for YOLO v3 is set to 273 epochs.

For the Pascal VOC dataset, we exclusively train Faster R-CNN with ResNet50 and ResNet101 backbones. The training configuration mirrors that of the COCO setup, with adjustments made to accommodate the dataset’s characteristics. The training duration is set to 4 epochs, with a learning rate reduction after the 3rd epoch.

### 4.2 Ablations

Importance of Intersection-based Regression.

Figure: Figure 3: Regression Loss during training for Faster R-CNN with our Intersection-based Regression and the traditional approach.
Refer to caption: /html/2311.18512/assets/x7.png

**Table 1: Ablation to show importance of Intersection-based Regression. Comparison between traditional regression and our intersection regression method atop Faster R-CNN [] reveals that intersection regression enhances detection performance for all metrics.**
| Method | AP$\uparrow$ | $\text{AP}_{\text{50}}\!\!\uparrow$ | $\text{AP}_{\text{75}}\!\!\uparrow$ |
| --- | --- | --- | --- |
| Faster R-CNN [] | 39.4 | 60.1 | 43.1 |
| Faster R-CNN w traditional grouping | 39.6 | 60.4 | 43.2 |
| Faster R-CNN w intersection grouping | 40.3 | 60.8 | 43.5 |

In [Tab. 1](#S4.T1), we compare the efficacy of our intersection-focused regression against conventional regression methods. For this experiment, we train two networks: One with our approach aka Intersection-based Regression followed by Intersection-based Grouping; the other with traditional bounding box regression followed by similar grouping but with the regressed proposals.

When applying our intersection and grouping technique on top of Faster R-CNN [], we achieved a mean average precision (mAP) of 40.3, a notable improvement over Faster R-CNN’s standalone mAP of 39.4. In contrast, using conventional regression methods in combination with the grouping process yielded an mAP of only 39.6, underscoring the importance of our intersection and grouping approach. [Fig. 3](#S4.F3) indicates the lower regression loss of our formulation against the traditional approach during training.

Effects of Intersection-based Grouping
To clarify the impact of the Intersection-based Grouping stage in object detection, we conducted an ablation study on two baseline models: Faster R-CNN [] and Mask R-CNN []. This experiment aimed to understand how varying the number of proposals combined at this stage affects the detection performance in terms of mean average precision (mAP).
In our methodology, after the Intersection Regression stage, we sort and group proposals akin to non-maximum suppression. We then varied the number of grouped proposals to be combined, ranging from 2 to a higher count, performed refinement, and observed the corresponding changes in mAP.

Figure: Figure 4: Effects of Intersection-based Grouping atop Faster R-CNN [] and Mask R-CNN []. We plot the mean average precision (mAP) outputs for both against the number of proposals combined in this stage. We observe that the mAP increases as the number of proposals combined rises from 2 to 5, reaching a peak at 5 proposals. Beyond this point, further increasing the number of proposals results in a decrease in mAP.
Refer to caption: /html/2311.18512/assets/x8.png

The results, as depicted in Figure [4](#S4.F4), show a consistent pattern for both Faster R-CNN and Mask R-CNN. We noted an increase in mAP as the number of combined proposals rose from 2 to 5, with the peak performance occurring at 5 proposals. However, beyond this point, the mAP begins to decline with the increase in the number of proposals combined. This trend suggests there is an optimal number of proposals to combine for maximizing detection accuracy, with 5 being the most effective in our experiment.

### 4.3 Comparison with Others

Detection Comparison on PASCAL VOC.
Our method was tested on the PASCAL VOC [] dataset using Faster R-CNN [] with ResNet50 [] and ResNet101 backbones, demonstrating significant performance improvements as shown in [Sec. 4.3](#S4.SS3). Notably, it excelled at high IOU thresholds for average precision, achieving 4.5 AP and 5.8 AP increases over the baseline Faster R-CNN models with ResNet50 and ResNet101, respectively, at a 0.9 IOU threshold. High IOU threshold performance indicates our method’s proficiency in accurate localization, as it measures the necessary overlap between the proposal and ground truth for a correct prediction. This aligns with our approach of using complementary information from object proposals, enhancing precision in object detection.

**Table 2: Comparison on PASCAL VOC of our method atop Faster R-CNN, using ResNet 50 and ResNet 101 backbones. We outperform Faster R-CNN across backbones on all metrics, with significant improvements on higher IOU thresholds.**
| Method | Backbone | $\text{AP}_{\text{50}}\!\!\uparrow$ | $\text{AP}_{\text{60}}\!\!\uparrow$ | $\text{AP}_{\text{70}}\!\!\uparrow$ | $\text{AP}_{\text{80}}\!\!\uparrow$ | $\text{AP}_{\text{90}}\!\!\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Faster R-CNN [] | R-50 | 71.2 | 64.8 | 51.3 | 29.3 | 6.8 |
| This paper | R-50 | 72.9 | 66.7 | 55.3 | 36.4 | 11.3 |
| \cdashline1-7<br>Faster R-CNN [] | R-101 | 81.4 | 76.5 | 66.9 | 48.5 | 15.6 |
| This paper | R-101 | 82.7 | 77.4 | 68.9 | 53.3 | 21.4 |

#### Results and Discussion.

The results in LABEL:tab:oracle reveal a significant improvement in object localization accuracy when using our approach compared to the traditional method. From [Sec. 4.3](#S4.SS3), we see that as the accuracy of classification labels for each proposal improves, our method demonstrates a marked increase in detection performance compared to traditional techniques. This improvement is attributed to the fact that correctly classified proposals, each covering parts of the ground truth, collectively contribute to a more comprehensive representation when grouped. Traditional methods, which select a single proposal, often miss significant portions of the ground truth. In contrast, our approach, by combining these segmented parts, achieves substantial performance gains. The results clearly show that with further advances in the field, it becomes even more advantageous to use our revisited point of view over the traditional one.

## 6 Conclusion

This paper revisits the object detection pipeline by focusing on intersection of proposal boxes with ground truth and better utilization of discarded proposal information. Our key contributions, Intersection-based Regression and Intersection-based Grouping, have been demonstrated to significantly enhance object localization accuracy and proposal comprehensiveness. The effectiveness of our method is validated through experiments on the COCO dataset, showing improvements in various detectors and instance segmentation. Notably, our approach also achieves higher average precision on the PASCAL VOC dataset.

## Acknowledgements

This work has been financially supported by TomTom, the University of Amsterdam and the allowance of Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.

## Peer Reviews

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1.	To enhance the overall aesthetic consistency of the paper's layout, a few minor formatting adjustments are recommended:

1）	Some bolded paragraph titles (e.g., the bold text followed by a period, such as "Problem Statement." in line 150) are set on a new line rather than immediately followed by the paragraph content. This may be intentional due to title length, or it might be a formatting issue. It is suggested to align these paragraph titles directly with the content following them. For example, "Intersection-based Grouping." in line 203 and "Regressing to intersections is simply an easier task." in line 448 should follow this format.

2）	The title "Inference cost" in line 267 lacks a period and should be updated to "Inference Cost." Additionally, the caption of Figure 4 ("Qualitative results…") is not bolded like the caption in Figure 5; please consider ensuring consistency with Figure 5's formatting.

3）	The paragraph titles throughout the paper vary between Title Case and Sentence Case. Although I will not list each instance here, please consider standardizing these for consistency.

2.	To ensure accuracy and clarity, a few content-related questions have also arisen:

1）	In line 220, the text states, "The regression function r : B → R maps each combined box to a final target box." Here, does R refer to the set of all ground truths? Previous sections define G or T as representing all ground truths, so it is unclear why R is used here instead of G or T. Could this be clarified?

2）	To guarantee the precision of the pseudocode in the Post-Processing Stage on the right side of Figure 2, could you confirm if it would be necessary to include "P ← P \ p_i" in the conditional statement within the red dashed box to ensure the "while P ≠ empty do" loop functions correctly (assuming the pseudocode within the green solid box is disregarded)?

3.	Could you clarify why the conditional statement in the pseudocode within the green solid box on the right side of Figure 2 uses "iou(H, pi) ≥ k" rather than "iou(M, bi) ≥ k" as in the red dashed box? Has there been any experimentation to validate the effectiveness of "iou(H, pi) ≥ k" over "iou(M, bi) ≥ k"? If not, please consider including such justification.

4.	In Table 3, there is a Comparison of instance segmentation on MS-COCO data, showing considerable improvements in AP for various object sizes (i.e., small, medium, and large objects as defined by COCO). The COCO definition of "small objects" encompasses those with an area under 32×32 pixels. Compared with small-object detection tasks in infrared imaging (where small objects typically occupy less than 10×10 pixels and may even be as small as a few pixels), COCO's small objects are relatively large. Could the authors clarify whether their proposed UoI method would also be effective for these considerably smaller targets?

5.	Figure 3(c) demonstrates that on the MS-COCO dataset, Faster R-CNN achieves optimal performance with the UoI strategy when the intermediate group size is 5. Have the authors considered studying the effect of varying the IoU threshold k in the UoI strategy (i.e., the IoU threshold used in the conditional statement in the pseudocode within the green solid box on the right side of Figure 2)?

6.	Figures 4 and 5 highlight certain limitations of the proposed UoI approach, specifically in handling crowded scenes. Could the authors specify if these crowded scenes refer to instances of multiple objects or instances of the same class, or do they also include multiple instances of different classes? Additionally, might the size of objects or instances in these crowded scenes significantly restrict UoI’s performance? Please provide a detailed discussion on these aspects.
soundness: 4
strengths: The "UoI" strategy proposed in this paper is highly innovative and practical, as it can seamlessly integrate into existing two-stage object detection networks, significantly improving detection performance. The authors creatively introduce the idea of regressing proposals to the intersection area with the ground truth, rather than the entire ground truth, addressing the challenges of regressing to the full target region. This not only improves localization accuracy but also offers an elegant and practical solution that can be easily adopted within current detection architectures. The paper is clearly written, with a logical structure that makes the proposed UoI method easy to understand. The figures and pseudocode are well-presented, offering an intuitive explanation of the approach. Furthermore, the extensive ablation experiments validate the effectiveness of UoI across various detectors, further strengthening the paper’s contribution. Overall, the paper excels in originality, clarity, and practical applicability.
summary: This paper revisits the problem of predicting box locations in object detection architectures. Traditional approaches typically regress box proposals to maximize the intersection-over-union (IoU) score with the ground truth and then apply non-maximum suppression (NMS) to retain only the highest scoring box in each region. The authors argue that both steps are suboptimal: regressing proposals to the entire ground truth is a challenging task, and NMS ignores potentially useful information from other boxes. To address these issues, the authors propose a simpler method—regressing proposals only to the intersection area between the proposal and the ground truth, rather than the entire ground truth, thus avoiding the need for proposals to extrapolate beyond their visual scope and improving localization accuracy. Additionally, they suggest generating the final box outputs by taking the union of the regressed intersections from all boxes in a region, rather than using a winner-takes-all approach, thus preserving valuable information from other boxes. This plug-and-play method integrates seamlessly into any detection architecture with minimal modifications and significantly enhances object localization and instance segmentation. The experimental results demonstrate its broad applicability and impressive performance across a variety of detection and segmentation tasks.
weaknesses: While the paper excels in terms of methodological innovation, clarity of writing, and the presentation of figures and pseudocode, there is room for improvement in the design of comparative and ablation experiments. Specifically, the validation of the proposed UoI strategy is limited to a few methods, which may constrain its generalizability. For example, Table 1 only compares Faster R-CNN and Def-DETR’s detection performance on the PASCAL VOC dataset, while Table 2 provides comparisons for five methods (Faster R-CNN, Mask R-CNN, Cascade R-CNN, YOLOv3, and Def-DETR) on the MS-COCO dataset. However, why are methods like Mask R-CNN, Cascade R-CNN, and YOLOv3 not included in the PASCAL VOC comparison? Including these methods would strengthen the results and provide a more comprehensive evaluation.

Moreover, Tables 4-7 and Figure 3 mainly show results for Faster R-CNN, without testing the UoI strategy on other classic detection architectures. To further validate the effectiveness of UoI, it would be beneficial to conduct experiments on additional well-established methods, demonstrating its applicability and advantages across a broader range of architectures. This would enhance the paper’s persuasiveness and provide stronger evidence for the practical utility of the proposed approach.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Intuitively, this method seems more beneficial for detecting larger objects, as the "combine box" approach tends to union the box when merging bounding boxes within a group, potentially biasing towards larger boxes. Is combining multiple predictions still an ideal detection strategy for small objects? The smaller targets are often adequately covered by a single proposal, which means the approach sometimes defaults to the standard paradigm. Although the main experimental results indicate an improvement in mAP across object sizes, I would encourage the authors to further analyze how this paradigm performs for targets of different sizes.

- It seems that the target of the regression stage is dynamic. I would encourage the authors to further analyze about the concerned instability.

- The confidence scores of the combined boxes are not introduced in method descriptions. It is important because score is essential for mAP. (good score also benifits mAP)

- Isn't the combined boxes the new boxes? Why the number of predictions remain? Does the mAP evaluator obtain more predictions than baseline?
> By design, our method yields the same number of predictions as current detectors and is compatible with any NMS variant.

- Is regression function r a learnable regression head? Why an additional regression refinement module is required? Isn't the combined boxes the prediction boxes? (We nevel refine NMS outputs with an additional regression module. The baseline method seems lack one refinement module)

- How does the detectors with other types of proposal (e.g. DETR with non-two-stage-initialized object queries, FCOS with point based dense head) be integrated seamlessly by UoI? It seems that this method can only be a box refinement module for these non-box-proposal methods.
> Our plug-and-play method integrates seamlessly into any detection architecture with minimal modifications
soundness: 4
strengths: - The authors provide a thorough explanation of the methodology, making it easy to understand. 
- The authors present a novel insight and have correctly implemented the methodology, achieving promising results.
- The experimental content is comprehensive, covering key results across popular detection algorithms and both detection and segmentation tasks. The ablation study effectively discusses the strengths of the approach and includes an analysis of its limitations.
summary: This paper propose Union-over-Intersections method for object detection, which predict intersection box between proposal box and ground truth box and combine overlapped boxes into the final instance.
weaknesses: - While this paper attempts to challenge the conventional box refinement paradigm (predict box directly and Winner-Takes-All for post-processing) in object detection models, it does not sufficiently demonstrate its viability as a true alternative to the original paradigm. (NOTE that although I mention this as a weakness, I still consider the approach to be novel and appreciate their effort to explore new insights. However, from an intuitive standpoint, the method does not strongly compel me to adopt it.)

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. What would be the implications of applying the Union of Proposals to learn the complete ground truth bounding boxes without altering the learning objectives?
2. Please provide a rationale for the weaknesses summarized above.
soundness: 3
strengths: * The method is straightforward, requiring only modifications to the training objectives and post-processing steps, making it easy to implement within existing proposal-based detectors.
* Experimental results demonstrate a solid improvement in accuracy while introducing only a minimal additional computational overhead
summary: The authors propose the Union-over-Intersections (UoI) method, which modifies the object detection pipeline in two key ways:

1. **Intersection-Focused Regression** : Instead of regressing proposals to the entire ground truth, the method focuses on the intersection area, simplifying the regression task and improving localization accuracy.
2. **Union of Proposals** : In the post-processing stage, rather than using a winner-takes-all strategy, the UoI method combines information from all proposals by taking the union of their regressed intersections, enhancing the final detection output.

The paper demonstrates decent improvements in object localization and instance segmentation across various architectures, including Faster R-CNN and YOLOv3. The UoI method is adaptable and can be integrated into existing detection frameworks with minimal changes, making it a promising advancement in the field of object detection.
weaknesses: My primary concern lies in the novelty and significance of the method. The paper emphasizes the use of intersections, but in my view, intersections are a subset of Intersection-over-Union (IoU). Learning IoU inherently involves understanding intersections, and there are already existing works focused on learning IoU, such as IoU loss. This diminishes the overall importance of the paper. Additionally, learning the complete ground truth bounding box implies the need to understand both intersections and unions. With proper tuning, I believe that incorporating more information would benefit neural network learning rather than focusing solely on a partial representation.

The concept of the Union of Proposals appears to function similarly to a voting mechanism. Box voting for accuracy improvement has already been extensively validated through test-time augmentation techniques in challenges like VOC and COCO. Moreover, learning the complete ground truth bounding box can naturally facilitate a voting process as well.

Lastly, there are concerns regarding generalizability. The method relies on proposals, which limits its applicability in proposal-free approaches, such as YOLO and FCOS. This reliance on a specific number of proposals necessitates tuning, making integration with proposal-free methods less convenient.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: Does the difference between the middle group of Table 1 (Faster R-CNN) and Table 4 lie in whether Soft-NMS is applied or not?
soundness: 3
strengths: 1. The proposed method is simple yet effective and easy to be re-produced.

2. The discussion of the problem is quite clear; the paper is well-written.

3. The experiments and analyses are sufficient to support the soundness of the proposed method over the vanilla regression method. The results also verify its good versatility for a variety of object detectors.
summary: The paper proposes to predict a set of intersection boxes for one given ground-truth box and then take the union of them to form the final detection box. The method is simple yet effective and is verified on 5 classic detectors.
weaknesses: 1. As a box regression method, the paper did not compare the proposed method with IoU-based loss functions, e.g., GIoU [r1], CIoU [r2], Alpha-IoU [r3], EIoU [r4], etc, making the superiority of the methods less convincing.

2. It is mentioned by line 216, the proposed Intersection-based Grouping is compatible with any NMS variant. In the experiment, only Soft-NMS is tested. I suggest the authors provide more ablation on this.

3. In Fig.3(c), it shows that the performance may be sensitive to the number of proposals on Faster R-CNN and MS COCO. This characteristic may limit its robustness in different scenarios, e.g., different detectors or different datasets.

4. Why the datasets of ablation studies switch frequently between COCO and VOC? Not consistent.
 
5. The font size of Fig.3 is too small.

[r1] Rezatofighi H, Tsoi N, Gwak J Y, et al. Generalized intersection over union: A metric and a loss for bounding box regression. CVPR 2019.

[r2] Zheng Z, Wang P, Liu W, et al. Distance-IoU loss: Faster and better learning for bounding box regression. AAAI 2020.

[r3] He J, Erfani S, Ma X, et al. $\alpha $-IoU: A family of power intersection over union losses for bounding box regression. NeurIPS 2021.

[r4] Zhang Y F, Ren W, Zhang Z, et al. Focal and efficient IOU loss for accurate bounding box regression[J]. Neurocomputing, 2022.

Meta-review: (a) The authors propose the Union-over-Intersections (UoI) method, which improves object detection by focusing regression on intersection areas for better localization and combining proposal information through union in post-processing.

(b) The strengths of the paper include a simple, clear, and easily reproducible method, a well-structured and well-written presentation, sufficient analyses to demonstrate effectiveness, and a promising advancement in the field of object detection.

(c) The major weaknesses are the limited novelty of the method, as intersections are already addressed by IoU and IoU loss, and the Union of Proposals resembles established voting mechanisms. Additionally, the method's reliance on proposals raises concerns about generalizability to proposal-free approaches like FCOS. 

(d) The most important reasons for acceptance are that this work changes the original detection pipeline, which regresses the full bounding box and uses the winner-takes-all NMS paradigm, to one that regresses parts of the box and merges proposals to obtain the final result. This is a very novel approach in the field of object detection, and the authors have demonstrated its effectiveness.

Decision: Accept (Spotlight)