MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection
Authors: Kemal Oksuz
Selim Kuzucu
Tom Joy
Puneet K. Dokania
Venue: Accepted by TMLR ()

Abstract: Combining the strengths of many existing predictors to obtain a Mixture of Experts which is superior to its individual components is an effective way to improve the performance without having to develop new architectures or train a model from scratch. However, surprisingly, we find that naively combining off-the-shelf object detectors in a similar way to Deep Ensembles, can often lead to degraded performance. We identify that the primary cause of this issue is that the predictions of the experts do not match their performance, a term referred to as miscalibration. Consequently, the most confident detector dominates the final predictions, preventing the mixture from leveraging all the predictions from the experts appropriately. To address this, when constructing the Mixture of Experts for object detection, we propose to combine their predictions in a manner which reflects the individual performance of the experts; an objective we achieve by first calibrating the predictions before filtering and refining them. We term this approach the Mixture of Calibrated Experts (MoCaE) and demonstrate its effectiveness through extensive experiments on 5 different detection tasks, showing that it: (i) improves object detectors on COCO and instance segmentation methods on LVIS by up to $\sim 2.5$ AP; (ii) reaches state-of-the-art on COCO test-dev with $65.1$ AP and on DOTA with $82.62$ $\mathrm{AP_{50}}$; (iii) outperforms single models consistently on recent detection tasks such as Open Vocabulary Object Detection. Code is available at: https://github.com/fiveai/MoCaE

Published in Transactions on Machine Learning Research (10/2024)
MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detection
Kemal Oksuz
kemal. okw.zoofuueeeeeek
Five AT Ltd., United Kingdom
Selim Kuzucu
sebim.knuncu220fuee..o
Five AT Ltd., United Kingdom
Tom Joy
tom.joy@ftue.ot
Five AT Ltd., United Kingdom
Puneet K. Dokania
punect.dokaniaefive..
Five AI Ltd., United Kingdom
Reviewed on OpenReview: /ttps://openreee opwreevee......ee.em net/ forun? id- 2-=ffBa fJEs 128J
Abstract
Combining the strengths of many existing predictors to obtain a Mixture of Experts which superior to its individual components is effective way to improve the performance without having to develop new architectures train model from scratch. However, surprisingly, find that naïvely combining afftth--hhelf object detectors in similar way to Deep Ensembles, can often lead to degraded performance. We identify that the primary cause of this issue is that the predictions of the experts do not. match their performance, a term referred to as miscalibration. Consequently, the most confident detector dominates the final predictions, preventing the mixture from leveraging all the predictions from the experts appropriatcly. To address this, when constructing the Mixture of Experts for object detection, we propose to combine their predictions in manner which reflects the individual performance of the experts; an objective achieve by first calibrating the predictions before filtering and refining them. We term this approach the Mixture of Calibrated Experts (MoCaE) and demonstrate its effectiveness through extensive experiments on 5 different detection tasks, showing that it: (i) improves object detectors on COCO and instance segmentation methods on LVIS by up to ~ 2.5 AP; (ii) reaches state-of-the-art on COCO lest-dev with 65.1 AP and on DOTA with 82.62 APs0; (iii) outperforms single models consistently recent detection tasks such Open Vocabulary Object Detection. Code is available at: httt::://gitubb..om///ieeeaa
1 Introduction
Deep Ensembles (DEs) Lakshminarayaaan et al. 2017 is an effective method for obtaining improved performance by simply training multiple models before combining their predictions at inference time. Providing
that compute is accessible, and inference time is not a significant issue, this approach provides a significant. boost in performance at minimal cost. Another variant of this approach, the Mixture of Experts (MoE) is practice achieved by combining different. predictors (Jacobs et al. 1991 Jordan & Jacobs 1994 Xu al.
1994 Yuksel al. 2012 Sukhbaatar et al. 2024 Given that these experts will typically behave diffcrently for different data samples, one would thus expect that the model is able to leverage the benefits of one whilst ignoring the contributions of the other poorer models. Interestingly, when considering object detectors, we
"SK and TJ contributed this project during their employment Five AT Oxford team.

---

Published in Transactions on Machine Learning Research (10/2024)
RS R-CNN
100
ATSS
RS R-CNN
34.42
PAA
% of detections
80
ATSS
PAA
60
96.51
3.42
39.97
40
25.61
20
0
0.0
0.2
0.4
0.6
0.8
1.0
confidence score
(a) Mixture of uncalibrated Experts
(b) Mixture of calibrated Experts (Ours)
(c) Uncalibrated confidence scores
Figure 1: Piecharts showing % of detections from three similarly performing detectors in their resulting MoEs on COCO dataset. (a) MoE of uncalibrated detectors, (b) MoE of calibrated detectors, and (c) histogram of confidence scores.
observe that naïvely combining experts in the standard way often leads to a degradation in performance, resulting in an MoE that is completely unable to leverage the strengths of the individual experts in certain situations.
We identify that the primary reason for this due to a failure when combining the predictions, such that the final output does not respect the individual performance of the experts, an issue known as miscalibration (Guo et. al. 2017) implying that the predicted confidences do not match the accuracy. This inconsistency results in the most confident detector dominating the final predictions, regardless of its accuracy. To illustrate, we combine RS R-CNN, ATSS and PAA with different characteristics making them non-trivial to combine. Specifically, RS R-CNN (Oksuz et al., 2021a) is a two-stage detector optimizing a ranking-based loss function, whereas ATSS /Zhang et al. 2020b) and PAA (Kim & Lee 2020) are both one-stage detectors both trained by focal loss (Lin et al. (2020) using different anchor-to-object assignment approaches. As can be seen Fig. [a), RS R- R-CNN dominates the predictions due to its high level of confidence, which are shown in Fig. [c).
It is natural to ask why is this specifie to MoE? and not present in DE? For DE, the main source of variation stems from the initialisation and other stochastic processes present in the optimisation, leading to similar histograms of predictive confidences. However, for an MoE, despite the fact that the experts perform similarly, there is a vast diversity in the mechanisms to arrive at the predictions in object detection: such as the use of an additional auxiliary localisation head (Tian et al. 2019 Zhang et al. 2020b, Jiang et al., 2018 Huang 2020) or the choice classifier, which commonly vary between a soft max (Ren el al.
et al. 2019 Kim & Lee
2017
Carion al. 2020
Dai al.) 2016
Bolya et al.
2019) or sigmoid classifiers for each class (Zhang et al,
2020b
Lin et al. 2020
Kim & Lee 2020
Zhu al.
2021 Furthermore, different backbones (Pinto et al.
2022),
loss functions (Mukhoti et al. 2020 and the training length (Mukhoti et al. 2020 Oksuz et al. 2023
can drastically affect the confidence of the model.
Consequently, given the vast diversity of detectors, and the corresponding differences in their associated confidences for the predictions, it is imperative that for an effective MoE to be constructed, their confidences must match their performance; that is, they are said be calibrated (Guo et al.) 2017 Oksuz et al., 2023 To address this, we propose Mixture of Calibrated Experts (MoCAE), which calibrates the individual experts and combines the predictions using our refinement strategy, approach term Refining Non-Maximun Suppression (NMS). As post-hoc calibrators
more effective compared to the existing training- training-time time approaches Munir et al. 2022 Pathiraja et al. 2023) Munir et al., 2023a b) for object detection (Kuzucu
et al. 2024 Oksuz el. al.) 2023 we employ Isotonic Regression (IR) Zadrozny & Elkan (2002) or Linear Regression (L) as post-hoc calibrators while calibrating the experts. This makes our method an example of a practical use-case as well as a benefit of calibrating of object detectors. Furthermore, utilizing the benefits of post-hoc calibrators, MoCaE extremely simple to implement and leverage while combining
off-the-shelf detectors. The resulting effects of our method can be seen in Fig. [bb and in Fig. 2 in which
2

---

Published in Transactions on Machine Learning Research (10/2024)
surfboard 0.29
sur feraborrd 0.55
person: H0b6
person
erson
person 0:22
erson 0.35
person |6,61
pe person 0.55
(a) Expert 1: RS R-CNN
(b) Expert 2: ATSS
(c) Expert 3: PAA
surfboard 0.40
surfboard
person: 45:65 person 0.65
person: 1-052
person
1son 0..69
person 0.51
person
(d) Uncalibrated Experts
(e) Calibrated Experts (Ours)
(I) Ground Truth Objects
Figure 2: Detections are color-coded. red: RS R-CNN, blue: ATSS, green: PAA. (a-c) Outputs of the detectors on an example image. RS R-CNN misses the "surfboard", ATSS misses a "person", PAA has a notable localisation error for the "person" in front seat. (d-f) The detections from MoE of uncalibrated detectors; MoE of calibrated detectors; and the ground truth. (d) is dominated by the most confident RS R-CNN and misses the "surfboard", After calibration in (e), all objects are detected accurately by improving each expert.
MOCAE is able to detect all objects in the scene with good localisation quality and avoids the false- possiios (FP) of the third person picked up by RS R-CNN and ATSS, but not by PAA. Overall, our contributions can be summarized as:
We show that due to the diversity in training regimes for different detectors they consequently bocome miscalibrated in vastly different ways, resulting in MoEs where the most confident expert dominates.
To address this, we propose MOCAE which first calibrates the experts and then combines their predictions through Refining NMS as our refinement mechanism while making the prediction.
We show that MOCAE yields significant gain over the single models and DEs on different real world challenging detection tasks: such as (i) improving object detectors by up to ~ 2.5 AP; (ii) reaching state-of-the-art (SOTA) on COCO test-dev with 65.1 AP and on DOTA for rotated object detection with 82.62 AP; (iii) outperforming single models consistently on recent detection tasks such as Open Vocabulary Object Detection (OVOD).
2 Background and Notation
Given that the sot of M objects in an image is represented by (b),c;) M where b, E RR is bounding c,) box and e E (1,...,)) K} its class; the goal of an object detector is to predict the bounding boxes and the class labels for the objects in X, f(X) (c,, b, p.N., where é, b,.P. P. represent the class, bounding box and confidence score of the ith detection respectively and N the mumber of predictions. In general, the detections are obtained in two steps, f(X) = (h (X) (Ren et al.) 2017 Lin et al.) 2020 Carion al.) 2020 Sun et al. 2018): where (xX) = (8: f pr No is a noural network predicting raw detections with bounding boxes bram and predicted class distribution prew. Then, ht()) consists of post-processing steps to obtain the final detections from raw detections. Commonly, h(-) includes discarding the detections predicted as background; NMS to remove the duplicates; and keeping useful detections, normally achieved via top-k survival, where typically k = 100 for COCO. Further discussion is provided in App. A
3

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Localisation-wware Expected Calibration Error (LAECE) of different detectors on COCO mini- test before and after calibration using Isotonic Regres- sion as the post-hoc calibrator. Isotonic Regression improves the calibration of different detectors.
Table 2: Recall@loU (RôIoU where IoU is 0.50 or 0.75) and Average Recall (AR) using 100 detections for the setting in Tab.
Method
R@0.50
R@0075
AR
RS R-CNN (1)
83.2
62.7
58.3
Calibration RS R-CNN
ATSS
PAA
ATSS (2)
83.1
65.9
60.8
PAA (3)
83.4
65.8
61.1
36.45
5.01
11.23
Uncal. MoE (1+-2-3)
83.6
6-1.1
59.7
3.19
4.54
1.09
Cal. MoE (1+2+33
85.1
67.7
62.3
3 Enabling Accurate MoEs via MoCaE
We seek to examine the inconsistency of calibration errors for different detectors before proceeding to propose MOCAE. Specifically, in Sec. 3.1 we highlight the many reasons why detectors differ significantly in their confidence, and the consequences of this when constructing MoF. To address this, Sec. 3.2 proposes MOCAE, which calibrates the individual detectors and refine their predictions.
3.1 Why do different detectors produce vastly different confidences?
Among different factors causing this difference, one major factor is related to the paramcterisation of the predictive function. For example, some recent detectors employ an additional auxiliary head to predict
localisation confidence (Tian et al. 2019 Zhang et al. 2020b Jiang et al. 2018 Huang et al. 2019 Kim
& Lee 2020). Consequently, the auxiliary head choice such as centerness
Tian et al. 2019 Zhang et al.
2020bb or loU (Jiang et al.) 2018 Kim & Lee 2020) as well as the aggregation function such as multiplication
Tian et al. 2019 ZZhang et al.) 2020b) or geometric mean (Kim Lee
2020) provides significant variation the confidence scores. Architectural difference can also manifest itself in the type of the detector, which
can be fully convolutional one-stage Zhang al. 2020b
two-stage (Ren et al. 2017 transformer-basee CCarion et al. 2020 Zhu et al. 2021 J. Another major factor causing the confidence incompatibility across the
Tian al.)
2019
Cai & Vasconcelos, 2018), bottom-up [Law & Deng 2018 Duan et al.,
2019 as well
detectors is the used classifier, which is commonly vary between a softmax (Ren et al.,
2017 Carion et al.)
2020:
Dai et al.
2016
Bolya et. al. 2019 or sigmoid classifiers for each class (Zhang el. al.
2020b Lin el al.
2020
Kim & Lee
2020
Zhu et al.
2021]. Besides, different backbones (Pinto et al. 2022), training objectives
Mukkoot al.]
2020
Chen ct al.;
2020 Oksuz al. 2020 Wang ct al. 2020; Lin al.,
2020
Lin al.
2017b: Li al.
2020
2019
Kahraman al. 2023 Oksuz et al.] 2021b) and the training length (Mukhoti
et al.
2020 Oksuz et al. 2023 affect the confidence of the model.
How different are the predicted confidences? To evaluate this, we leverage the recent work of Oksuz 2023) requiring the confidence of the predictions to match the product of classification (cls.) and localisation (loc.) performance (perf.) using the following criterion:
et al.
- P(a==cllp)
BLL [ooU [ bo())) = P.. Vp E [0, 1).
(1)
Precission cls. perf. Expected IoU as loc. perf-
where B, (f:)) is the set of true-positive (TP) boxes with the confidence score of fi, and by()) is the ground-truth box that 6, matches with. In practice, the confidence space is split into J bins to compute the calibration error based on equation ! Then, LAECE (Oksuz al. 2023) measures the absolute difference between the accuracy and the associated confidence:
D Dej
Σ 1=1
pj
precision x 1oU)(1)
(2)
where Dr and Dr denote the set of all detections and those in the bin for class c respectively. Similarly, the average confidence, precision and average Intersection-over-UUne (IoU) of Dj are denoted by 77. precision
4

---

Published in Transactions on Machine Learning Research (10/2024)
1.0
1.0
mean loU
mean loU
0.8
% Samples
0.8
% of Samples
LaECE= 36.45%
LaECE= 5.01%
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0.0
0.0
0.2 0.4
0.6
0.8
1.0
0.2 0.4 0.6
0.8
1.0
Confidence
Confidence
(a) RS R-CNN
(b) ATSS
Figure 3: Reliability diagrams of (a) overconfident RS R-CNN and (b) underconfident ATSS.
and IoU'(). While, we employ LAECE, we slightly modify it due to the difference of our calibration criterion in Eq. + which will be introduced shortly. Specifically, we measure the difference between the confidence of the detection and its loU with the best overlapping ground-truth as LAECE:
D De Dj
LAECE Σ
Lo b6
33)
We report LAECE in Tab. where we see that RS R-CNN, PAA and ATSS, three similar performing detectors in terms of AP, have vastly different LAFCE, implying different confidence predictions. Moreover, in Fig. we display the reliability plots for RS R-CNN and ATSS, which shows that RS R-CNN is siguificantly more confident than ATSS.
An uncalibrated Mixture of Experts For the purposes of exposition, first show that naïvely combining these uncalibrated detectors results in poor MoE, where the most confident detector dominates the MoB regardless of its accuracy. To show this, similar to (Casado- García & Heras 2020), we construct a "Vanilla MoE" which aggregates the uncalibrated predictions from RS R-CNN (Oksuz et al. 2021a), ATSS (Zhang ict al., 2020b) and PAA (Kim & Lee 2020) using NMS. Fig. T shows that Vanilla MoE dominated by the most confident RS R-CNN with a small contribution from less confident PAA and almost no delections from the least confident ATSS. Furthermore, we see Fig. d), that it is unable identify the surfboard and produces FP for person. As a result, while would expect MoE to detect more objects than individual detectors and obtain better recall, Tab. shows the opposite; Vanilla MoE yields lower Average Recall (AR) compored to ATSS and PAA. This clearly indicates that naïvely obtaining MoF. will normally be biased and lead to an ineffective mixture.
We have now highlighted that a fundamental issue with constructing an MoE, that a situation can often arise when one of the detector dominates the predictions. However, this not necessarily deficiency, as one would expect an accurate detector to dominate the predictions when combined with inaccurate ones. Conceptually, we want the MoE to combine predictions based their performance, which can be inferred through the confidence estimates provided at test time. However, shown above, it is imperative that these predictions are calibrated. Therefore, to appropriately construct the MoE, we calibrate the experts individually, before filtering the predictions in our refinement strategy. As we show in Sec. 4 this enables reliable contributions from each detector and an effective MoE.
3.2 Constructing an Effective Mixture of Experts
Here we highlight the two main components for constructing an effective MoE, obtaining similarly performing calibrated experts; and aggregating their detections in the best way possible. Overall pipeline of MOCAE is presented in Fig.
5

---

Published in Transactions on Machine Learning Research (10/2024)
3.2.1 Calibrating Individual Experts
Having identified the issue with the Vanilla MoE, the question naturally arises as to how we calibrate the single detectors to address its deficiency. As opposed to the classification lask, object detection jointly solves both classification and regression tasks; and also involves post-processing steps that influence the accuracy of the detector. Therefore, it is not straightforward as to what objective the calibrator should have and at which stage of the pipeline it should be applied. A natural choice would be to calibrate the scores such that it helps the crucial aggregation stage (e.g., NMS). This stage does not require training and has significant impact on the accuracy of a detector.
For simplicity, let's consider the standard NMS, which groups the detections that have an ToU with the maximum-scoring detection larger than a predefined IoU threshold. Then, within that group, NMS survives the detection with the largest score and removes the remaining detections from the detection set. In such setting, also discussed by the recent works (Li et al., 2020 2019 Kahraman et al.] 2023 Jiang et al. 2018: Zhang et al. 2020b), the ideal confidence that should be transferred to the NMS is the ToU of the detection with the object. This will guide NMS to pick accurately-localiser detections for the objects detected by multiple detectors. Furthermore, if an object is detected by a single less confident detector, aligning the confidence with IoU implies that the scores of the TPs are to be promoted. Thus, the TPs of a less confident detector will not be dominated by the FPs of more confident ones unlike the case in Fig. [ a). Following this inluition, we call detector calibrated if it yields a confidence that matches the IoU, implying
( (..)). [bU, bo(a)) = p.. V& E [0. 1].
(4)
where (f.) is the set of detection boxes with the confidence score of p. and by(s) is the ground-truth box that 8, has the highest IoU with.
From an optimization perspective, calibrating each expert to meet the criterion in Eq. 4 requires us to design objective that maps the output confidence of each bounding box to calibrated one. Though there can be several ways to design such an objective, we take a rather simple approach where we learn a post-hoc calibrator Ço [0, 1] + [0, 1] using the input-target pairs (fp,, LoU b)) obtained on a held-out validation set. Specifically, we parameterise (o(.) as a simple LRR model, containing only two Iearnable parameters or an IR model (Zadrozny & Elkan 2002; Oksuz et al. 2023 Kuzucu et al. 2024). Thereby being easily applicable to any off- off-the-shelf detector without adding any notable overhead.
Tab. 1 and Tab. show an example case using IR, where the LAECE improves and the resulting MoE has higher recall than the uncalibrated MoE and single models. Please refer to App. B and App. D for more details. Furthermore, Theorem I indicates that Eq. 4 is the optimal choice in terms of Average Precision (AP) in obtaining an MoE for object detection.
Theorem 1. Assume that F. different experts are combined in the form of an MoE and the detection set of e-th expert for class e is denoted by Be. Given the union of these detections (before aggregation) is denoted by Braw = F. = Be, the MoE using perfectly colibrated experts in terms of Eq. 4 yields optimal AP for class over any possible MoEs following Lemma 1 Accordingly, denoting the number of TPs in Brow by (Br( and the number of ground-truid objects for class by M > O, the resulting AP is NTP(B""") (Please refer to M
App. C for Lemma i and the proofs.)
3.2.2 Refining NMS for Aggregating Detections
Another critical component of the MoE is aggregating the combined detections. There is a very high chance that more than one detector produces the same detection; therefore we aim to suppress these duplicate detections targeting the same object and obtain detections with high localisation quality. As aforementioned, NMS is method that fits for this purpose and, as we observe experimentally, docs provide highly competitive results. However, it is rigid in nature when removing overlapping detections, thereby not utilizing the rich information provided by multiple MoEs. To address this, we present Refining NMS that simply combines Soft NMS (Bodla et al.) 2017) with Score Voting (Kim & Loo. 2020). Before we present the details, we note that we use Refining NMS only for object detection as extending score voting to instance segmentation and rotated object detection is not trivial, hence left as future work.
6

---

Published in Transactions on Machine Learning Research (10/2024)
(t. 6}
Detector 1
Postprocess
Calibrate
{6.6}
X
Detector 2
Postprocess
Calibrate
Refining NMS
M
(ő. b}
Detector 3
Postprocess
Calibrate
Figure 4: MoCAE pipeline. Given image X, each detector follows its own pipeline including postprocessing (in orange) and outputs (f.b). Without any modification to the pipeline of each detector, we calibrate the confidence scores of each detector and aggregate them via Refining NMS providing the detections of MOCAE.
Given the set of detections from all experts as B (()).)))) from a class é, NMS survives the one with highest confidence (say (ba. Pa)) and removes all boxes that have loU with ba more than a pre-defined threshold IoUNMS from B. This process takes place until there is no remaining detection cither to be survivod removed. Differently, instead of removing the detections completely, Soft. NMS decreases the confidence score of highly overlapping detection as function of their overlap with Pa). In particular, we use linear decay while decreasing the confidence of these boxes, such that
P& px x (1 (8,,6),).))
if ToU b.) < ToUNMS
Ps =
(5)
elsc.
This less rigid removal mechanism results in higher recall. Following Soft NMS, we employ Score Voting (Kim & Lee 2020) to obtain a bounding box with better localisation for each surviving box after Soft NMS. Specifically, the refined box &, is obtained by considering B as the set of detections from all experts before Soft NMS (i.e., including the ones removed by Soft NMS):
b, EyeBDTTOU where I0U, = (1-1 6,3)2
(6)
Ejes PÚ TOU,
where asv is the hyper-parameter, which set to 0.04. As result, combining these two approaches leads to much effective aggregator to combine the predictions from different experts.
4 Experiments
In this section we seek to first outline the criticality of calibrating individual object detectors when constructing an MoF. (Sec. 4.1) Second, we present the effectiveness of MoCAE on many standard detection benchmarks and show that reaches state-of-the-art on COCO for object detection and DOTA for rotated bounding box detection (Sec. 1.2). Then, we highlight the reliability MoCAE under domain shift (Sec. 4.3). Finally, outline the limitations (Sec. [14) of MOCAE, Our extensive experiments with 19 different detectors on datasets show that MOCAE is consistently superior to the single models, Vanilla MoE and Deep Ensembles.
4.1 Effect of Calibration on Obtaining MoEs
To demonstrate the effect of calibration, we use the common COCO dataset (Lin et al., 2014). Similar to [Kuppers et al.) 2022), randomly split COCO val set with 5K images into two, and use 2.5K images for testing COCO minitest and the remaining 2.5K images as COCO minival to analyse calibration. Specifically, we find it sufficient to use 500 images for calibrating the detectors. In our experiments, we mainly use COCO-style AP and also report (i) AP%,, AP75 as the APs measured at loU thresholds 0.50 and 0.75; as well as (ii) APs, APM and AP, to present the accuracy on small, medium and large objects. In terms of models, here we combine RS R-CNN, ATSS and PAA with ResNct-50 (He al., 2016) with FPN [Lin et al. (2177a) backbone. These detectors have different characteristies making them non-trivial combine. Specifically, RS R-CNN (Oksuz et al. 2021a is two-stage detector optimizing ranking-based

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Effect calibration. All MoE's except the last row use standard NMS.
Model
Combined Detectors
Detection Performance
Calibration
Type
RS R-CNN
ATSS
PAA
AP
AP50
APr5
Single Models
N/A
42.4
62.1
46.2
N/A
43.1
61.5
47.1
N/A
/
43.2
60.8
47.1
X
/
42.4
62.1
46.3
V
44.1
63.0
48.4
X
43.4
62.5
47.1
/
44.0
62.7
47.9
MoEs
X
43.3
60.9
47.2
44.4
62.5
48.5
x
43.4
62.5
47.1
/
/
44.7
63.1
48.9
/+ Relining NMS
45.5
63.2
50.0
Table 5: Effect of calibration on transformer-basce detectors. All MoEs except the last row use NMS.
Model
Combined Detectors
Detection Performance
Calibration
Type
Co-DETR
BRS-DETR.
DINO
AP
APso
AP75
Single
N/A
49.8
67.6
54.5
Models
N/A
50.3
67.4
54.6
N/A
50.4
67.9
54.6
X
49.8
68.4
54.1
51.0
69.3
55.1
51.0
69.4
55.2
50.9
69.4
55.0
MoEs
50.1
68.3
53.8
51.5
69.6
55.3
51.0
69.5
55.2
51.5
70.0
55.6
+ Refining NMS
51.9
70.0
56.5
loss function, whereas ATSS (Zhang al. 2020b) and PAA Kim & Lee 2020 are both onc-stage detectors
trained by focal loss (Ln al. 2020). Also, different from ATSS with centerness head, PAA employs ToU prediction head and they differ in obtaining the confidence score.
Calibration is crucial for accurate MoEs In order to highlight the effect of calibration in different settings, we construct three MoEs from pair-wise combinations of RS R-CNN, ATSS and PAA well one MoE that combines all three. In order to focus only calibration, here we the standard NMS with an loU threshold of 0.65 as in Wang et al. (2022a). Tab. 3 presents the results of MoEs with uncalibrated and calibrated detectors. The striking observation is that without calibration, the MoEs perform similar the single models and calibration enables accurate MoEs for all four settings. Specifically, using calibrated MoEs yields 1AP gain, and using three improves AP by 1.5 compared single models; showing the effectiveness of calibration.
Table 4: LAECE of transformer- baased detectors on COCO mini-lest before and after calibration using Isotonic Regression as the post-hoc calibrator.
Calibration Co-DETR
BRS-DETR
DINO
7.63
13.12
5.06
6.52
6.38
4.33
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Object detection performance on COCO minitest. MoEs obtained by our MOCAE outperforms DEs significantly even with less detectors. Our gains in green are obtained compared to the best single model for each performance measure, represented underlined.
Model Type
Detector
AP
AI'so
AP's
AP's
APM
AP'L
RS R-CNN
42.4
62.1
46.2
26.8
46.3
56,9
Single Models
ATSS
43.1
61.5
47.1
27.8
47.5
54.2
PAA
43.2
60.8
17.1
27.0
47.0
57.6
RS R-CNN X
43.4
63.0
47.7
28.0
47.5
57.0
Deep Ensembles
ATSS
44.1
62.3
48.4
29.4 4
49.0
56.3
PAA X 5
444
62.0
18.4
28.9
49.0
59.2
Vanilla MoE (RS R-CNN, ATSS, PAA)
43.4
62.5
47.1
27.3
47.3
58.0
Mixtures of Experts
MOCAE (ATSS and PAA) Ours
44.8
62.4
49.2
29.4 4
49.1
57.6
MOCAE (RS R-CNN, ATSS, PAA) Ours
45.5
63.2
50.0
29.7
49.7
59.3
-2.
+1.1
+2.9
+1.9
+2.2
+1.7
To further show the effect of calibration, we also combine three different transformer-basse detectors:
Co-DETR Zong et al.
2023 utilizes conventional detectors during training Deformable DETR Zhu
eet al. 2021
BRS-D DETR Yavuz et al
2024) trains Co-DETR with a ranking- loss, and
DINO (Zhang al. 2023 is essentially an improved version of Deformable DETR.
We obtain IR. calibrator for each detector and improve their calibration shown in Tab. Then, we combine them all and in pairs in Tab. 5 which demonstrates that calibrated MoEs perform notably better in almost all combinations. For example, the gap between calibrated and uncalibrated MoEs is 1.4 AP once BRS-DETR and DINO are combined. We note that, in the single exception of combining Co-DETR and DINO, the calibrated MoE performs similar to the uncalibrated onc. This is expected as the uncalibrated confidence distribution of these two detectors are very similar (as shown in Fig. (a,)), in which case calibration does not make a difference, Finally, both Tab. 3 and Tab. 5hshow that Refining NMS has a positive effect the resulting mixture. Resulting MoCAE (i.c., using calibrated and Refining NMS) outperforms the best uncalibrated MoE (i) by 2 AP in Tab. 3 and (ii) by ~ 1 AP in Tab. 5
Even with fewer models, MoCaE is superior to DEs Next we compare MOCAE, Vanilla MoE and DFs. We calibrate the components of DEs while combining them, though we do not observe a significant effect from the calibration in their performance (see App. D for details); which is an expected outcome. Tab. 6 shows that the DEs perform consistently better than the single models; validating them as strong baselines. The main observation in Tab. 6 is that combining different types of few detectors into an MoE performs significantly beller Ihan DEs. Specifically, MOCAE with only two detectors, ATSS and PAA, outperforms all DEs, each with five components. Also, combining three detectors by MOCAE performs 1.1 AP better than its closest counterpart DE. This is because the same type of detectors make similar errors, which yiclds less gain once they are combined together. However, different types of detectors complement each other thanks to their diversity. Finally, MOCAE outperforms Vanilla MoE by 2 AP in this setting. Our final model obtains 45.5 AP and outperforms the best single model in all AP variants significantly.
4.2 Benchmarking MoCaE on Various Tasks
In this section, demonstrate that combining if-ttheshhelf detectors via. MOCAE improves single detectors various detection tasks up to 2.5 AP, which is a significant performance improvement. Our MOCAE reaches SOTA results on COCO dataset among public models and on DOTA dataset for rotated object delection. Specifically, we evaluate on four different tasks: object detection (COCO (Lin et al. 2014)), ), rotated object detection (DOTA (Xia et al., 2018)), open vocabulary object detection (COCO and ODinW35 (Li al., 2022a)) and instance segmentation (LVIS (Gupta et al. 2019)). For these tasks, we use total of 15 different. detectors include one-stage and two-stage, convolutional, transformer-basee ones and foundation models.

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: Detection performance on COCO test-dev using strong detectors. Green: Gain against best single model (underlined).
Table 8: Performance COCO lest-dev using state- of-the-art detectors. MOCAE improves the most accurate publicly available model by 0.7 AP and reaches SOTA.
Method
AP
APma
AP7s
APs
APM
APL
YOLOvv
Wang et ol. 2022a
55.5
73.0
60.6
37.99
58.8
67.7
Method
AP
so
APrs
AP,
APx
APL.
Query Inst.
Fang et nl.
2021
55.7
75.7
61.4
36.2
58.4
70.9
EVA Fann et al. 2029
64.4
82.3
70.9
48.2
67.6
77.5
DyHead
Dai et al.
2021
56.6
75.5
61.8
39.4
59.8
68.7
Co-DETR Zong
61.3
71.0
67.1
77.5
Vanilla MoE
57.6
76.6
63.2
40.0
60.9
70.8
Vanilla MoE
64.6
71.0
48.5
67.6
77.5
+1.0
+0.9
+1.4
4-0.6
+-1.1
0.1
+0.2
+0.4
0.0
+0.3
0.0
0.0
MOCAE (Ours)
59.0
77.2
64.7
41.1
62.6
72.4
MoCAE (Ours)
65.1
82.7
71.9
49.2
68.1
78.1
12.4
1.5
22.9
11.
1.28
1..5
+0.7
+0.4
0.99
+1.0
+0.5
+0.66
Table 9: Rotated object detection performance on DOTA test set. APso is used following DOTA. See App. D for all classes.
Table 10: Comparison open vocabulary object detection. Green: Gain against best single model (underlined).
5 Clasers with Lowrst Performance Bridge Soccer Houndab. Harbor Helico.
Method
COCO APto APr
ODinW-35
Method
AP
APaig
RTMDet LSKN
Lyv
2022
81.32
58.50
72.12
70.85
81.16
77.24
Grounding DINO-T Lau al 2024
48.5
64.5
52.9
22.7
13.8
La al. 2023
$1.85
61.47
71.67
71.35
79.19
80.85
MQ-GLIP-T T [X et at 2023
46.3
62.7
50.6
21.4
8.2
Vanilla MoF
80.00
61.77
70.98
65.92
84.28
77.57
Vanilla MoF.
48.1
64.8
52.5
22.4
11.8
10.30
-1.44
+3.12
3.28
+0.3
MoCAF (Ouns)
82.62
61.38
74.12
84.49
$1.93
MOCAF (Ours)
49.5
66.4
54.3
23.2
14.9
10.77
0.09
+3.38
-2.77
+3.33
11.08
+1.0
+2.1
+1.4
+.05
+1.1
Object Detection on COCO Here, we evaluate on COCO test-dev by submitting our result to the evaluation server. In the first setting, we combine the following well-known and effective detectors:
YOLOv7 (Wang et al. 2022a) with a large convolutional backbone following its original setting, QueryInst (Fang et al.) 2021 as a wansformer-basee detector with Swin-L (Liu et al. 2021)
backbone,
ATSS with transformer- based dynamic head DDi al. 2021 and again Swin-L backbone.
These detectors differ from each other in terms of the pretraining data, backbone or architecture as summarized in App. D Tab. 7 shows that our MOCAE reaches 59.0 AP with gain of 2.4 AP on this challenging setting as well. As our gain here is similar to that of Tab. 6 we can casily say that the gain of our MOCAE has not saturated this stronger setting, which commonly the opposite in the literature.
Finally, we also evaluate MOCAE on the two most recent SOTA publicly available detector:
EVA (Fang et al. 2018) for detection,
2023
foundation model for vision using Cascade Mask R-CNN CCai & Vasconcelos
Co-DETR (Zong et al. 2023 a transformer-based detector.
Tab. 3 shows that MOCAE reaches SOTA with 65.1 AP on COCO test-deu and outperforms all eristing public detectors by 0.7 AP. This further shows the effectiveness of MOCAE.
Rotated Object Detection on DOTA We now investigate MOCAE for rotated object detection on DOTA v1.0 dataset (Xia et al. 2018) with 15 classes. DOTA is also challenging dataset comprising of aerial images that are very dense in terms of objects. Specifically, DOTA dataset has 67.1 objects on average per image. We use all 458 images in the validation set to calibrate the detectors and report AP30 on the test set by submitting our results to the evaluation server. We combine LSKN (Li et al. 2023) and RTMDet [Lyu et al., 2022) as two recent SOTA detectors. Following the literature, we use NMS with an loU threshold of 0.35 as Soft NMS and Score Voting are not straightforward use in this task. Tab. g suggests that we establish new SOTA with 82.62 50 DOTA; improving the previous SOTA by 0.77. Having examined the classes,
'Published CVPR 2023 and ICCV 2023
10

---

Published in Transactions on Machine Learning Research (10/2024)
Table 11: Instance segmentation results on LVIS val
Table 12: Instance segmentation results on
set. AP denotes detection AP.
COCO minitest combining transformer-base
Mask2Former and VitDet with Mask R-CNN.
Method
AP
50
AP
AP,
AP.
AP;
AP box
Method
AP
Al's
AP
AP,
APn.
AP'
Mask R-CNN
25.4
39.2
27.3
15.7
24.7
30.4
26.6
m
RS Loss
25.1
38.2
26.8
16.5
24.3
29.9
25.8
Mask2Former
46.6
69.5
50.4
26.7
50.1
70.5
Seesaw Loss
25.4
39.5
26.9
15.8
24.7
30.4
25.6
ViT
46.2
69.5
50.5
28.1
49.9
65.5
Vanilla MoE
25.2
38.3
26.8
16.5
24.3
29.9
25.9
Vanilla MaF.
47.3
71.4
51.6
28.9
50.8
68.7
0.2
-1.
00.5
0.0
0.4
00.5
-0.7
+00.7
+1.9
+0.88
+-2.2
4-0.7
-1.8
MoCAF
27.7
42.8
29.4
18.2
27.3
82.4
29.1
MoCAE
47.7
71.7
52.1
29.2
51.5
68.7
(Ours)
+2.3
+3.3
+2.1
1.7
22.4
+20
+2.5
(Ours)
+1.1
+2.2
+1.7
+2.5
+1.4
-1.8
we note that our improvement originates mostly from the classes with relatively lower performance; with the exception of the class 'bridge' which performs marginally worse. For example, on 'soccer-field', 'roundabout', 'harbor' classes where the single detectors have between 70 80 AP the improvement is around 3 AP These gains enable to demonstrate the ability of MOCAE to set new SOTA in rotated object detection.
Open Vocabulary Object Detection (OVOD) We now investigate the effect of MOCAE on OVOD task Li et al. 2022b OVOD task is a recently proposed challenging task in which the aim is to detect the objects pertaining to the classes a given text prompt. This requires the models to be able to interpret the given text prompt as well the image and yicld the detection results. Furthermore, the text prompt can contain phrases that not necessarily in the training data. For this challenging lask, combine two strong and recent models:
Grounding DINO (Liu et al. 2024 transformer-based detector, and
MQ-Det (Xu et al. 2023). ), an anchor-based OVOD relying on GLIP (Li et. al. 2022b
We obtain the calibrators on a subset of the Objects365 dataset (Shao et al. the pretraining data for both of these models. Therefore, the calibrators are also limited the pretraining data only. In our evaluation, we evaluate the models on COCO and DDDinW-35(2
2019 which is included in
al. 2022a datasets following the common convention (Li et al., 2022b Liu et al., 2024 Xu et al. 2023). Note that ODinW-35 is a challenging dataset with 35 different subdatasets, some of which are substantially different from the pretraining dataset. To illustrate, ODinW-33 includes subdatasets specifically for 'potholes' on the road and infrared images of 'dogs' and 'people'. Tab. I0 shows that MOCAE improves the single models on both of these datasets on all performance measures notably. As an example, the median AP of the best single model on ODinW-35 increases from 13.8 to 14.9, which suggests an 8% relative gain.
Instance Segmentation Given that MOCAE is beneficial for object detection, one would expect it to improve performance on the instance segmentation task. To verify this, first use LVIS (Gupta al. 2019 as long-tailed dataset for instance segmentation with more than 1K classes. Following its standard evaluation, we also report the AP on rare (APr), common (APz) and frequent (AP;) classes. Similar to COCO, we reserve 500 images from val set to calibrate the detectors, and test our models on the remaining 19.5K images of the val set. We combine three diverse off- shel Mask R-CNN variants a MoE:
The vamilla Mask R-CNN HHe et al. 2017) with ResNeXt-101 Xic et al. 2016 backbone, softmax classifier and using Repeat Factor Sampling (RFS) used for the long-tailed nature of LVIS,
Mask R-CNN with ResNet-50, sigmoid classifier, trained with RS Loss Oksuz et al. 2021a and
RFS,
Mask R-CNN with ResNet-50, softmax classifier, trained with Seesaw Loss Wang et al. RFS.
2020 but no
Tab. 11 shows that while the Vanilla MoE performs worse than the best single model, MOCAE boosts the segmentation AP by 2.3, an improvement of 10% over the best single model. Also, the detection AP (APbox) improves by 2.5 aligned with our previous findings.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Table 13:
Comparison on domain-shifted datasets using COCO mini-test with ImageNet-CC style corruptions and Objects45K, natural shift from COCO. Green: Gain against best single model (underlined).
Corrupted COCO mini-test
Objects45K
Model Type
Detector
Severity of the Corruption
mcan AP
1
2
3
4
AP
AP 30
AP75
Single Models
RS R-CNN
33.2
27.7
21.9
16.2
11.8
22.2
28.6
41.2
31.2
ATSS
33.8
28.2
22.2
16.3
11.8
22.5
28.7
39.8
31.2
PAA
34.4
29.2
23.1
16.9
12.1
23.1
28.7
39.5
31.0
RS R-CNN X 5
34.6
29.3
23.5
17.5
12.8
23.5
29.8
42.6
32.7
DEs
ATSS X 5
35.0
29.5
23.4
17.4
12.7
23.6
29.3
40.4
31.8
PAA x 5
35.7
30.4
24.4
18.2
13.2
24.4
29.4
40.1
31.8
Vanilla MoE
34.5
29.1
23.1
17.2
12.5
23.3
29.3
41.4
31.8
MoEs
MOCAF (Ours) +1.9
36.3
30.8
24.6
18.2
13.2
24.6
30.6
41.6
33.3
+1.6
+1.5
+1.3
+1.1
+1.5
+1.9
+0.4
+2.1
Secondly, we combine two recent highly accurate approaches on COCO dataset to further support our claims:
Mask2Former (Chong et al. 2022) with Swin-S Liu et al. (2021 backbone and
Vit et al. 2022c) that uses a Mask R-CNN prediction head with a transformer-based Vit-B
backbone (Dosovitskiy al. 2021
Tab. 12 shows that MOCAE improves the best detector by 1.1 mask AP, outperforms Vanilla MoE well, thereby showing the effectiveness of MoCAE for combining instance segmentation models.
4.3 How Reliable MoCaE?
As we are ensembling detectors in the form of MoE, it is naturally to evaluate how reliable MOCAE is. To evaluate this, investigate how MOCAE performs under domain shift. (Michaelis et al. 2019 Hendrycks & Dictterich 2019), and also test MOCAE on the recently proposed Self-aware Object Detection (SAOD) task [Oksuz et al.] 2023). Here, we use our setting in Sec. 4.1 in which we combine RS R-CNN, ATSS and PAA. That is, we use the calibrators trained on clean COCO and do not train a new calibrator.
Domain Shift (Synthetic and Natural) Following the convention (Michaclis et al. 2019 Oksuz et al. 2023 Munir et al. 2022), we apply 15 ImageNet-C style corruptions (Hendrycks & Dietterich 2019 under different severities for synthetic domain shift on COCO, Similar the clean data, we observe Tab. 13 that combining only three models (RS R-CNN, ATSS and PAA) outperforms DEs with five components thanks to the diversity of the detectors. Here see, that the performance over the best single model improves by 1.5 AP. Next, we evaluate the performance of MoCAF on Objects45K (Oksuz et al.) 2023). Please note that this dataset has the same set of classes as in COCO, but collected and annotated separately, thereby implying a natural domain shift and a similar setting was used in (Harakeh & Waslander, 2021). Tab. 13 shows that the DFs do not provide notable gains this setting. For example, while PAA x 5 only improves single PAA A only by 0.7 AP, MOCAE improves the best single model by 2 AP, outperforming all DEs and Vanilla MoE.
Self-aware Object Detection (SAOD) Finally, evaluate MOCAE on the recently proposed SAOD task Oksuz et al., 2023), which requires the object detectors to be self-aware. This task requires detectors to provide reliable uncertainty estimates along with accurate and calibrated detections in holistic manner; which is eva evaluated using the Detection Awareness Quality (DAQ). To evaluate the models on this task, we convert RS R-CNN, ATSS, PAA, Vanilla MoE and MOCAE to a self-aware detector following (Oksuz et al,, 2023). MOCAE improves the DAQ of the best single model from 40.9 to 42.9 and outperforms Vanilla MoE. by 0.5 DAQ. Overall, this suggest that MOCAE is more reliable than the single detectors. Details are provided in App. due to space limitation.
12

---

Published in Transactions on Machine Learning Research (10/2024)
4.4 Ablation Analysis, Discussion and Limitations
Ablation Analysis We provide further ablation of MoCAF in Tab. 14 in which the calibration appears to be the major factor of the performance gain. Soft NMS yields small gain of 0.1 AP and Score Voling, combining boxes from different detectors to extract new bounding box, improves AP notably from 44.8 to 45.5. Please refer to App D for further details.
Table 14: Ablation analysis of MoCAE. We use the MoE combining RS R-CNN, ATSS and PAA on COCO mini-test.
Refnnnng NMS
Calibration
AP
Soft NMS
Score Voting
X
X
43.3
Using Recent MoE Approaches to Combine Off- MoE methods have been very effective to improve the performance of the transformer-based models (Jawahar et al., 2023 Dai
x
X
14.7
/
/
X
44.8
the-shelf Detectors Recently-propott
x
/
X
43.4
x
/
44.4
/
45.5
it al.
2024; Lin et al., 2024 Jiang et al.) 2024 Shen et al.
2023
Pióro et al.) 2024 Zhou et al.; 2022] Fedus et al.)
Ruiz et al., 2021). Conventionally, these methods replace the fully connected layer in the transformer by a softmax layer to route the input followed by multiple experts of fully connected layers. They are commonly applied to the classification--asedd problems and supervised during training in an end-to-end manner. Considering their success, we now aim to see whether such methods can be useful in combining multiple off-the-shelf detectors leveraging small val. set, as did. To do so, we design Transformer layer that routes the detections from different experts. Specifically, given detections from all experts, this layer consists of a sell-attention block in which all detections attend to each other followed by a sigmoid for each detection to predict its confidence. Similar to our approach, the predicted confidence is trained using COCO mini-val set to match the IoU of the dctection with the ground truth and to select the set of detections yielding the highest AP. Though we perform proper hyper-parameter search and experimented with different. loss functions, this model does not generalize well enough to the test set. Essentially, the AP on the data that it is trained is very high, but the test AP is significantly low. This suggests that non-trivial changes are required in order to make these MoEs effective for combining off-the-shelf object detectors.
2022
Limitations In cases where the performance gap is large between the experts, we observe that both MOCAE and Vanilla MoE tend to perform worse than the best single model. For example, combining RS R-CNN, ATSS, PAA (APs x 40) with EVA and Co-DETR (APs so 65) does not result in a stronger MoE than EVA or Co-DETR (refer to App. D for the details). However, once we use an Oracle MoE obtained by directly assigning the calibration targets as the confidence, MOCAE reaches 86.7AP achieving more than 20 AP improvement compared to the best single model; an observation which aligns with our theoretical insights. This suggests that calibration is critical for effective MoEs and more effort is required from the community in calibrating object detectors.
5 Conclusions
A direct result of the vastly different training regimes employed in training object detectors is that their predictions are miscalibrated such that some are more confident than the others. This lack of consistency makes constructing an MoF. in a naïve approach futile, as the most confident detector dominates the predictions, even though its performance may not warrant this weighting. Consequently, to address this we introduced MoCAE as a simple, principled and effective approach, which first calibrates the individual detectors before combining them appropriately through our refinement strategy. Specifically, in the calibration stage we aligned the confidence with the IoU of the detection with the object that it overlaps the most with. We showed that this is an effective calibration (arget, resulting in accurate MoEs with consistent gains across different detection tasks, reaching SOTA on many challenging detection benchmarks such as COCO and DOTA. Whilst our choice of calibration function performed well on the settings demonstrated here, we further observe that increased gains can be achieved the community develops more sophisticated calibration methods, objective we leave to future work.
13

---

Published in Transactions on Machine Learning Research (10/2024)
References
Yu Bai, Song Mei, Huan Wang, and Caiming Xiong. Don't just blame over-parametrization for over-confidence: Theoretical analysis of calibration in binary classification. In ICML Workshop on Distribution Free Uncertainty Quantification, 2022.
Navanccth Bodla, Bharat Singh, Rama Chellappa, and Larry S. Davis. Soft-nms immprovin object detection with one line of code. In IEEE/ /CV International Conference on Computer Vision (ICCV), 2017.
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Daniel Bolya, Scan Foley, James Hays, and Judy Hoffman. Tide: A general toolbox for identifying object detection errors. In The IEEE European Conference on Computer Vision (ECCV), 2020.
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang, Ace: Ally complementary experts for solving long-tailes recognition in one-shot, 2021.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. In TEEE/CVF Conference on Computer Vision and Pattern Recograttion (CVPR), 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), 2020.
Ángela Casado- Gareia and Jónathan Heras. Ensemble methods for object detection. In European Conference on Artificial Intelligence, 2020.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv, 1906.07155, 2019.
Kean Chen, Weiyao Lin, Jianguo li, John See, Ji Wang, and Junni Zou. Ap-loss for accurate one-stage object detection. IEFE Transactions on Pattern Analysis and Machine Intelligence (TPAMT), pp. 1-1, 2020.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar, Masked-attention mask transformer for universal image segmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Jiacheng Cheng and Nuno Vasconcclos. Calibrating deep neural networks by pairwise constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition (CVPR), 2022.
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024.
Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-FCN: Object detection via region-based fully convolutional networks. In Advonces in Neural Information Processing Systems (NeurIPS), 2016.
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang, Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7373 7382, June 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkorcit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL httpss://pperreviert wttp://wwww...eteooo..oeeoeeoooo
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In IEEE/CVF International Conference Computer Vision (ICCV), 2019.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. In Proceedings of the IEEE/CVE International Conference on Compuler Vision (ICCV), pp. 6910 6919, October 2021.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In IEEE/CVI Conference on Computer Vision and Pattern Recognition CeaiiiionnC// (CVPR), 2023.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning, 2022.
Vitaly Feldman. Does learning require memorization? A short tale about long tail. CoRR, abs/ 1906.05271, 2019. URL http::/arxiv.. org/ http://abs/19...022222
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2881 28911. Curran Associates, Inc., 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330, PMLR, 2017.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Twis: A dataset for large vocabulary instance segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ali Harakch and Steven L. Waslander. Estimating and ovaluating regression prodictive uncertainty in deep object delectors. In International Conference on Learning Representations (ICLR), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IRFE/CVF Conference on Computer Vision and Pattern Recográttion (CVPR), 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. In IEEE/CVE International Conference on Computer Vision (ICCV), 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Zhaojin Huang, Lichao Huang, Yongchao Gong, Chang Huang, and Xinggang Wang. Mask scoring r-cmn. Tn IEEE/CVF Conference on Computer Vision and Pattern Recogrition (CVPR), 2019.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural compulation, 3(1):79 87, 1991.
Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul-Mageed, Laks
V. S. Lakshmanan, Ahmed Hassan Awadallah, Sebastien Bubeck, and Jianfeng Gao. Automoe: Neural architecture search for efficient sparsoly activated transformers, 2023. URL https://openreveew.noot forummid=3y fo33m
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaune Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Ante Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Sxymon Antoniak, Teven Le Scao, Théophile Gervel, Thibaut. Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In The European Conference on Computer Vision (ECCV), 2018
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural compudation, 6(2):181-214 214, 1994.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Tom Joy, Yuge Shi, Philip H. S. Torr, Tom Rainforth, Sebastian M. Schmon, and N. Siddharth. Learning multimodal vaes through mutual supervision, 2022.
Fehmi Kahraman, Kemal Oksuz, Sinan Kalkan, and Emre Akbas. Correlation loss: Enforcing correlation between classification and localization. In Association for the Advancement of Artificial Intelligence (AAAI), 2023.
Kang Kim and Hee Seok Lee. Probabilistic anchor assigument with iou prediction for object delection. In The European Conference Computer Vision (ECCV), 2020.
Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. In Advances in Newral Information Processing Systems (NewrIPS), volume 32, 2019.
Fabian Kuppers, Jan Kronenberger, Amirhossein Shantia, and Anselm Haselhoff. Multivariate confidence calibration for object detection. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2020.
Fabian Kuppers, Jonas Schneider, and Anselm Haselhoff. Parametric and multivariate uncertainty calibration for regression and object detection. In Safe Artificial Intelligence for Automated Driving Workshop in The European Conference on Computer Vision, 2022.
Selim Kuzucu, Kemal Oksuz, Jonathan Sadeghi, and Puneet K. Dokania. On calibration of object detectors: Pitfalls, evaluation and baselines. In The European Conference on Computer Vision (ECCV), 2024.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty ostimation using deep ensembles. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newrul Information Processing Systems, volume 30, 2017.
Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In The European Conference on Computer Vision (ECCV), 2018.
Hyungtae Lee, Sungmin Eum, and Heesung Kwon. Me r-cmn: Multi-expert r-enn for object detection. IEEE Transactions Image Processing, 29:1030-104 2020. doi: 11110//IIP.019..2201
Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao. Elevater: A benchmark and toolkit for evaluating language-augmente visual models, 2022a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language- image pre-training. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022b.
Xiang Li, Wenhai Wang, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss v2: Learning reliable localization quality estimation for dense object detection. In IEEE/CVF Conference on Compuler Vision and Pottern Recognition (CVPR), 2019.
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision (ECCV), 2022c.
Yuxuan Li, Qibin Hou, Zhaohui Zheng, Ming-Ming Cheng, Jian Yang, and Xiang Li. Large seloctive kernel network for remote sensing object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16794-1 16805, October 2023.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models, 2024.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In The European Conference Computer Vision (ECCV), 2014.
Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. Feature pyramid networks for object detection. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In JEEE/CVF International Conference on Computer Vision (ICCV), 2017b.
Tsung-Y: Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection.
IEEE Transaclions on Pallern Analysis and Machine Intelligence (TPAMI), 42(2):318 327, 2020.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.
In The European Conference on Computer Vision (ECCV), 2024.
Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Chongqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, and Kai Chen. Rumdet: An empirical study of designing real-time object detectors. ArXiv, 2022.
C. Michaelis, B. Mitzkus, R. Geirhos, E. Rusak, O. Bringmann, A. S. Ecker, M. Bethge, and W. Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming, In NewrIPS Workshop on Machine Learning for Autonomous Driving, 2019.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Doka- nia. Calibrating deep neural networks using focal loss. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. Curran Associates, Inc., 2020. URL httpps://proceedings pww.:ppp.cc//ppp///////////////0 pdff
7587536ee70002477777220400000000 15288 1008-15299..00
Muhammad Akhtar Munir, Muhammad Haris Khan, M. Saquib Sarfraz, and Mohsen Ali. Towards improving calibration in object detection under domain shift. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Muhammad Akhtar Munir, Muhammad Haris Khan, Salman Khan, and Fahad Khan. Bridging precision and confidence: A train-time loss for calibrating object detection. In IEEE Conference on Computer Vision and Pallern Recognition (CVPR), 2023a.
Muhammad Akhtar Munir, Salman Khan, Muhammad Haris Khan, Mohsen Ali, and Fahad Khan. Cal-DETR: Calibrated detection transformer. In Thirty- Conference on Neural Information Processing Systems, 2023b. URL http://pppnreeeeeeeeeettrrrr/aaa
Lukás Neumann, Andrew Zisserman, and Andrea Vedaldi. Relaxed softmax: Efficient confidence auto- calibration for safe pedestrian detection. In NIPS MLITS Workshop on Machine Learning for Intelligent Transporlation System, 2018.
Jeremy Nixon, Michael W. Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mcasuring calibration in deep learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. Localization recall precision (LRP): A new performance metric for object detection. In The European Conference on Computer Vision (ECCV), 2018.
Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. A ranking-based, balanced loss function unifying classification and localisation in object detection. In Advances in Newral Information Processing Systems (NewrIPS), 2020.
Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. Rank & sort loss for object detection and instance segmentation. In The International Conference on Computer Vision (ICCV), 2021a.
Kemal Oksuz, Baris Can Cam, Fehmi Kahraman, Zeynep Sonat Baltaci, Sinan Kalkan, and Emre Akbas. Mask-aware ion for anchor assignment in real-time instance segmentation. In The British Machine Vision Conference (BMVC), 2021b.
Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. One metric measure them all: Localisation recall precision (Irp) for evaluating visual detection tasks. IEEE Transactions Pattern Anolysis ond Machine Intelligence, pp. 1-1, 2021c.
Kemal Oksuz, Tom Joy, and Puneet K. Dokania. Towards building self-aware object detectors via reliable uncertainty quantification and calibration. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Bimsara Pathiraja, Malitha Gunawardhana, and Muhammad Haris Khan. Multiclass confidence and localization calibration for object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Francesco Pinto, Philip H. S. Torr, and Puncet K. Dokania. An impartial take to the enn vs transformer robustness contest. In The European Conference on Computer Vision (ECCV), 2022.
Maciej Pióro, Kamil Ciebiera, Krystian Król, Jan Ludziejewski, Michal Krutul, Jakub Krajewski, Szymon Antoniak, Piotr Miloš, Marck Cygan, and Sebastian Jaszczur. Moo-mamba: Efficient selective state space models with mixture of experts, 2024.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 39(6))1137 1149, 2017.
Towards real-time object detection
Carlos Riquelme Ruiz, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jemation, André Susano Pinto, Daniel Keysers, and Neil Houlsby, Scaling vision with sparse mixture of experts. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In IEFE/CVF International Conference Computer Vision (ICCV), 2019.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Mixture-of-experts meets instruction tuninga winning combination for large language models, 2023.
Yuge Shi, N. Siddharth, Brooks Paige, and Philip H. S. Torr. Variational mixture-of-exxerts autoencoders for multi-modal deep generative models, 2019.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen tau Yih, Jason Weston, and Xian Li. Branch mix:-mix: Mixing expert 1lms into a mixture-of-experts 11m, 2024.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, and Ping Luo. SparseR-CNN: End-to-end object detection with learnable proposals. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
7hi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. Tn IEEE/CVF International Conference Computer Vision (ICCV), 2019.
Yhien-Yao Wang, Alexcy Bochkovskiy, and Hong-Yuan Mark Liao. YOLOv7: Trainable bag-of-frreebes sets new state-of-the-art for real-time object detectors. arXiv preprint (xXiv:227.002690 2022a.
Deng-Bao Wang, Lei Fong, and Min-Ling Zhang. Rethinking calibration of deep ncural networks: Do not be afraid of overconfidence. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. 2021 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 9690-9699, 2020.
Xudong Wang, Long Lian, Zhonggi Miao, Ziwei Liu, and Stella X. Yu. Long-tailed recognition by routing diverse distribution-aware experts, 2022b.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-supervised learning, 2018.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection acrial images. In The IEFE Conference on Computer Vision and Pottern Recognition (CVPR), June 2018
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. arv, 1611.05431, 2016.
Lei Xu, Michael Jordan, and Geoffrey E Hinton. An alternative model for mixtures of experts. In G. Tesauro, D. Tourelsky, and T. Leen (eds.), Advances in Neural Information Processing Systems, volume 7. MIT Press, 1994. URL http:://prroeeeeuggeee www.:eeediinn...niss..e.. (B0b666aee66656662200000000000-0
/ille//paapx///99///iel/
pdf
Yifan Xu, Mengdan Zhang, Chaoyou Fu, Peixian Chen, Xiaoshan Yang, Ke Li, and Changsheng Xu. Multi- modal quoried object dctection in the wild. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Feyza Yavuz, Baris Can Cam, Adnan Harun Dogan, Kemal Oksuz, Emre Akbas, and Sinan Kalkan. Bucketed ranking-based losses for elficient training of object detectors. In European Conference on Computer Vision (ECCV), 2024.
Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. Twenty years of mixture of experts. IEFE transactions on neural networks and learning systems, 23(8):1177- -19119771111,,,,, 2012.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 694-699, 2002.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In The Eleventh International Conference on Learning Representations, 2023. URL |nttps://openrevieee http://ppaaa.eieeeeeeeooraaaaaaa
Hongkai Zhang, Hong Chang, Bingpeng Ma, Naiyan Wang, and Xilin Chen. Dynamic r-cnn: Towards high quality object detection via dynamic training. In The European Conference on Computer Vision (ECCV), 2020a.
Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. Bridging the gap between anchor-basod and anchor-free free detection via adaptive training sample selection. In IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2020b.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for lest-agnostic long-tailed recognition. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neurol Information Processing Systems, volume 35, pp. 34077 34090. Cur- ran Associates, Inc., 2022. URL https::/pproceedings neurips................ cc/paper httpps/eecpppprrreeeeppppeeeee _ftlee//paaeee/22//111aa aec1saaceeeea22222000aaaeeeeeeee
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cunulative learning for long-tailed visual recognition, 2020.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, zhifeng Chen, Quoc V Le, and James Laudon. Mixture-o-cxperts with expert choice routing. In Advances in Newral Information Processing Systems, volume 35. Curran Associates, Inc., 2022.
Xizhou Zhu, Weijic Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable {detr}: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2021.
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. IEFE/CVF International Conference on Computer Vision (ICCV), 2023.
20

---

Published in Transactions on Machine Learning Research (10/2024)
APPENDICES
A Related Work
Mixture of Experts (MoEs) The main aim of combining multiple experts in the form of an MoE is to leverage the expertise of each expert, which ideally specialises in different subpopulations of the input data (Jacobs et al. 1991; Jordan & Jacobs 1994 Xu et al., [994 Yuksel et al. 2012). In the literature, this aim is achieved by variety of techniques. Some of the methods aggregates the predictions of individual experts Carcao-García & Heras 2020). Broadly speaking, from this perspective, DEs (Lakshminarayanal et al.) 2017], in which multiple models are trained and then aggregated, can also be considered as an example in this group. Another set of methods (Zhou et al. 2020 Lee et al. 2020 Wang et al., 2022b Zhang et al.) 2022 Cai et al., 2021 Ruiz et al.) 2021). ) as the majority of the techniques proposed the deep learning era, allows the experts to share certain network components among experts, which are then combined (or selected) by galing (a.k.a. routing) mechanism for the sake of efficiency. Essentially, this gating mechanism decides which expert to rely on conditioned on particular subpspace of the input space. One particular and intuitive use-case of this group of methods is long-tailed classification (Zhou et al. 2020 Wang et al. 2022b Cai et al. 2021 Zhang et al. 2022), in which different experts specialise on different classes with various cardinalities, i.e. the classes with few or many training examples. The shared features are then routed by a gating mechanism to leverage experts' specialisation in an efficient manner. Another effective application of MoEs is within the domain of variational autoencoders (Wu & Goodman 2018 Shi et al. 2019 Joy et al. where different experts are generally utilised for different modalities.
2022
Ensemble Methods in Object Detection Despite their aforementioned success in the classification literature, ensembling detectors either in the form of a DE by using the same model with different initialisation or as an MoE by combining different type of models has received very little attention (Lee et al.) 2020 Gasado-Garcia & Heras, 2020). Among the few existing works, Casado-García & Heras (020) combine a set of detectors through various aggregation strategies, such as unanimous agreement between the detectors. Therefore, this method combines off-the-shelf detectors, and accordingly, we refer as Vanilla MoE in the paper baseline. However, we note that it does not consider either calibrating the detector advanced aggregation techniques unlike our work. As the second work in this domain, Multi-Expert R-CNN (Lee et al. 2020 is designed to exploit multiple experts in the R-CNN family, in which different R-CNN models correspond to different experts for a specific Rol. Consequently, it is not applied to a wide range of different detectors such as the common one-stage detectors or transformer- based detectors.
Calibration in Object Detection As extensively studied for classification, calibration refers to the alignment. of accuracy and confidence of model (Guo el al. 2017 Nixon el al. 2019 Kumar et al. 2019 Wang et al. 2021 Mukhoti et al. 2020 Cheng & Vasconcelos, 2022). Specifically, a classifier is said to be calibrated if it yields an accuracy of on its predictions with a confidence of for all E [0, 1]. Earlier definitions for the calibration of detectors (Kuppers et al. 2020 Neumann et al. 2018 extend this definition
with an objective to align the confidence of detector with its precision,
P(q = cl..) = p., Vp, E [0,
(A.7)
where P(c.- = cglpt) denotes the precision as the ratio of correctly classified predictions among all detections. Extending from this definition, Oksuz et al. (2023) take into account that object detection is a joint task of classification and localisation. Thereby defining the accuracy as the product of precision and average loU of TPs, calibration of object detectors requires the following to be true
P(Q =
bw(r)] = P.. e [0,1],
(A.S)
where (f.))) is the set of TPs with the confidence of P. and bot) is the object that i, matches with. Then, LAECE obtained by discretizing the confidence score space into J bins for each class. Specifically for class e, denoting the set of detections by De and those in the jth bin by Dj as well the average confidence,
21

---

Published in Transactions on Machine Learning Research (10/2024)
(praw, graw
Detector
Calibrate
Postprocess
pMoE
{praw, Brawy
Bg removal
X
Detector
Calibrate
NMS
MoE
(praw braw
Top-k
Detector 3
Calibrate
Figure A.5: Early calibration. Raw confidence scores pram are calibrated, and the standard post- past-processing steps handle aggregation in which NMS (in blue) removes the duplicates from multiple detectors.
precision and average loU of D3 by Pj, precision (() and IoU respectivcly, LAECE for class is defined as
LaECE
pj
precision (() x ToU (i)
(A.9)
De
Finally, the detector LAECE is the average of LaECE's over classes in the dataset, measuring the calibration error for a detector as a lower-belter measure.
Comparative Summary Different from the existing few works on the ensemble methods of object detectors, we comprehensively investigate how to obtain MoEs object detection using off-the-shelf detectors. While doing so, unlike existing work, (i) we identify that miscalibration of different detectors prevents them to be combined properly due to the peculiarities of the detectors; (ii) we introduce a strong aggregation technique which refer to Refining NMS combining Soft NMS and Score Voting from previous work; and (iii) we employ very diverse set of detectors from a wide range of detection tasks. As for the miscalibration of the delectors, we rely on LaECE in Eq. A.S. Consequently, MOCAE follows the main aim of the MoE as the predictions from each expert are aggregated in a way that the strength each detector is leveraged as we comprehensively demonstrate in our experiments. And besides, from the MoE perspective Refining NMS can be considered as form of gating (or routing) mechanism, which decides the best way combine the detections from different experts.
B Further Details on Calibration
Throughout our paper, we calibrate the final confidence scores. One can also consider object detection that, the raw confidence scores (before postproccssing) can be calibrated as well. For this reason, we first present early calibration in App. B.1 Then, for the sake of completeness,
discuss why we prefer post-hoc calibration methods in App. B.2 well as how measure the calibration error of MOCAE based on Eq. 4
in App. B.3
B.1
Early Calibration of Object Detectors for Obtaining MoEs
Up to now, discussed calibrating final confidence scores ù; similar to Kuppers et al. (0200 Oksuz et al. (2023), which we show as late calibration in Fig. Besides, we also investigate the effectiveness of early calibration by calibrating the raw probabilities pr of the delectors as illustrated in Fig. A.5 While we find both approaches to perform similar in MoEs, we use late calibration as it is simpler owing to less number of final detections. Still, as the first to investigate carly calibration, we present an additional use-case of early calibration in App. D in which we show that reduces the sensitivity of the model to the background removal threshold in post-processing.
22

---

Published in Transactions on Machine Learning Research (10/2024)
B.2 Choice of the Calibration Method
There are multiple calibralion methods used for object detection including training-lime (Munir et al. 2022
2023a; Pathiraja et al. 2023) and post-hoc calibration methods (Oksuz et al. 2023 Zadrozny & Elkan 2002
Guo et al.) 2017). Idcally, we would expect a calibration method to casily generalize to a wide range of detectors as well as detection tasks. However, training-tine time calibration methods are typically incorporated into only a small set of object detectors and not tested on a variety of relevant tasks such as the rotated bounding box detection, which makes them unfit for purposes. On the other hand, post-hoe calibration methods formalize calibration a regression task from the predicted final confidence to target confidence, making them easily applicable to any detectors and any detection task. Throughout this work, we investigate Linear Regression (LR) and Isotonic Regression (TR) ZZddrozny & Elkan 2002) considering the criterion in Eq. We further observe that a Class-agnostic (CA) IR calibrotor for each delector oblained on 500 images
is sufficient for MoCAE. App D present extensive experimental analyses.
B.3 Measuring the Calibration Error for MoE
We introduce our calibration criterion in Eq. 4 This criterion requires the confidence of a detection to align with its loU with the ground truth box that the detection overlaps the most. As stated carlier, computing the calibration error based on this criterion corresponds to using an IoU threshold of to validate TPs. This is equivalent to using precision"()) for class c Eq. 2 which then reduces to
D De
LaRCE
pj - TOU ToU
6/|
(A.10)
where De denotes the set of detections for class c; Dj is the set of detections in the jth bin for class c; pj is the average confidence of the detections in Dj; and Io)"(j) is the average IoU of the detections in Dy. Following Oksuz et al. 2023). we use
and average over LaECE' of classes for the detector LaECE.
In addition to LaECE, here we define Localisation-awall aware Average Calibration Error (LAACE) and Localisation- aware Maximum Calibration Error (LAMCE) similar to the way how Expected Calibration Error (ECE) is extended to Average Calibration and Maximum Calibration Errors. We find LAACE and LAMCE useful as they reduce the dominance of certain bins on the calibration error as in the case of LAECE. This is especially important for early calibration from which thousands of confidence scores are obtained from single image, most of which have a confidence close O. Specifically, in our case, we define LaACE" for class as
J - 18--OU((1 1)) LaACE
- j=1
(A.11)
and LaMCES for class as
LaMCE max JE(1,.....)
-
(A.12)
Following LAECE, we obtain LAACE and LAMCE for the detector by averaging over the classes.
C Theoretical Discussion on the Optimality of the Calibration Criterion in Eq. 4
Lemma I discusses the conditions under which an optimal AP can be achieved for a given set of pre-NMS detections.
Lemma 1. Given set of detection bores for class c, denoted by Braw = (6paw 8rew ,.... Br.aw) we first assume that the post-prucessing (NMS in this case) does not remove TPs and can remove duplicales in Brau Let
A TP is detoction that has at least IoU with ground-truth of where is the loU threshold validate TPs. In the case of more than detection satisfy this criterion, the common convention accept the detection with the highest score as TP and the remaining ones duplicates, which are counted as FPs while computing the AP.
23

---

Published in Transactions on Machine Learning Research (10/2024)
denote the kth truth boz by by and the detection set post NMS by B ((b1,1)), (62, (22,12).., (bN. PN)) where Pi) correspond a tuple with the i-th bounding box and associated confidence score. If the detections in ensures that [oU(b, by) > [oū by) Df > Dj for all i + j and k is the ground trulh that each detection has marimum IoU with, then B provides the optimal AP for class e given Braw The volue of the optimal AP in this case Nrrib""", where Nrr(Brau) is the number of TPs in Braw and M > 0 is the number of ground-truth objects from class c.
M
Proof. Below we show that B satisfies all the three necessary conditions required to maximize the AP for any IoU threshold T (to identify TPs):
1. B is to have Nr that is, no TP is to be removed in Braw by postprocessing. Note that this is handled by post-processing as an assumption in the Lemma.
The minimum confidence score of TPs in B is to be higher than the marimum confidence score of FPs. This is also ensured considering (i) the assumption that is ToU bz) > ToU[(y by) if p. > Pj for all i + j and k is the ground truth that each detection has maximum loU with; and (ii) the duplicates are removed by NMS. As a result, all TPs are ranked higher than all FPs.
3. B is to include the delection with the largest ToU with each ground truth k k NMS selects the detection with the highest confidence score among a group of overlapping detections. Considering that NMS does not remove any TPs and lb.(b, bx) > [oū bu) for all i and j if pÚ > P3 and i + j holds, NMS survives the detections with the best localisation quality as they have the highest scores in their groups. As a result, this condition also satisfied.
To compute the area under the precision-recall curve, we need the precision and recall pairs. Please note that once the aforementioned criteria are satisfied, the precision will be 1 when recall interval is between Nre(B==))
M I; and will be zero beyond this. Therefore, the area under the precision-recal curve trivially turns out to be Nr
[0,
On the Optimality of Eq. 4 Based on Lemma we now present our theorem and its proof.
Theorem 1. Assume that F. different experts are combined the form of an MoE and the detection set of e-th expert for class e is denoted by B., Given the union of these detections (before aggregation) is denoted by Braw = _ Be, the MoE using perfectly colibrated experts in terms of Eq. 4 gields optimal AP for class c over any possible MoEs following Lemma 1 Accordingly, denoting the number of TPs in Brow by NFr(Braw) and the number of ground-trute objects for class e by M > O, the resulting AP is M
Proof. Please note that it is trivial to show that the detoctions in B of the perfectly calibrated detectors ensures the requirement in Lemma [ that IoU(b, by) > (o, by) if 0; > Pú for all + j and is the ground truth that each detection has maximum IoU with. This is because the calibration target ensures that P. ToU by) for each detection As result, following from Lemma 1 the theorem holds.
Please note that it is trivial to show that Theorem 1 generalizes to the cases in which the dalaset involves multiple classes and COCO-style AP is used. This is because the former case is estimated as the average of the APs over different classes and the latter is simply the average of the APs over different loU thresholds (please see footnote for further discussion). Consequently, as the class-wise APs are optimal, so do the dataset AP and the COCO-style AP. Please refer App. D.7 for the experiment presenting the effect of Theorem D using an Oracle MoE.
When consider the AP for single r, this criteria is not mandatory to be satisfied as the conventional AP consider localisation performance loosely Oksuz al. (2021e) However, the localisation performance
important aspect of object detector and it is cousidered by various performance measures such COCO-style AP, which corresponds the average over APs with 10 different thresholds LRP Error et al. 2021ch. As result, while consider the AP for a single 7 in this section, in order for our theoretical justifications be applicable to other performance measures, also take this criterion in account.
24

---

Published in Transactions on Machine Learning Research (10/2024)
D Further Experiments and Analyses
Here, we present further experiments and analyses that are not included in the main text due to space limilation.
D.1 Further Details on Used Models
We provide the details of the used models as follows. We again note that we haven't trained any model but used off-the-shelf detectors with the exception of DEs. Here we provide further details on the used detectors. Still, as it is not feasible to provide all of the details, we also present the papers and repositories that we borrow these off-the-shelf models in order to ensure the reproducibility of our results.
Object Detection We use two different configurations. In the first one, employ three detectors with backbone. These detectors are:
ResNet-50 (He et al. 2016) with FPN [Li et al. 2017a.
Rank & Sort R-CNN (RS R-CNN) (Oksuz et al.
2021a is a recent representative of the two-stage
R-CNN family (Ren et al.
2017 Dai et al. 2016
Zhang et al. 2020a) optimizing a ranking-basee
loss function,
2020b a common one stage bascline, relies on the one-stage ATSS architecture but with different anchor assignment mechanism and postprocessing of the confidence scores.
Adaptive Training Sample Selection (ATSS) (Zhang ct al.
Probabilistic Anchor Assignment (PAA) (Kim & Lee 2020)
We obtain RS R- R-CNN and ATSS from (Oksuz et al., 2023) and PAA from (Chen et al. 2019 All these detectors are trained for 36 epochs using multi-scale training data augmentation in which the shorter side of the image is resized within the range of [480, 800] for RS R-CNN and ATSS and [640, 800] for PAA. We do not use Soft NMS and Score Voting for the single detectors.
In our second setting, we the following detectors:
YOLOv7 (Wang et al. 2022a with a large convolutional backbone following its original setting, QueryInst (Fang et al. 2021 as transformer-based detector with Swin-L (Liu et al. 2021 backbone,
ATSS with transformer-baser dynamic head Dai ct al. 2021 and again Swin-L backbone.
Again, we obtain YOLOv7 and dynamic head from mmdetection (Chen al. 2019 and use the official repository of QueryInst (Fang al. 2021).
Furthermore, to improve the SOTA on COCO test-dev, we two of the most recent and strong public detectors, EVA (Fang et al. 2023) and Co-DETR (Zong et al. 2023). EVA (Fang et al. 2023) is foundation model for computer vision that utilises Cascade Mask R-CNN (Cai & Vasconcelos 2018) to perform object detection wheras Co-DETR (Zong et al. 2023) a recent transformer-basee detector. For both of them, we directly consider the official reposílories and do not change any settings, including the Soft-NMS for EVA Fang et al. 2023)
Rotated Object Detection For rotated object detection, we use RTMDet and LSKN as two different. detectors. We obtain RTMDet again from mmdetection (which is also the official repository for RTMDet) and LSKN from its official repository
(Li et al. 2023
Open-Vocabult Vocabulary Object Detection We two of the most recent works on vision-language foundation models literature, which can be listed as follows:
Grounding DINO (Liu et al.] 2024 is a transformer-based detector.
MQ-GLIP from the recent [Xu et al. 2023).
an anchor-based detector that introduces multi-modal
queries on top of GLIP [Li et al. 2022b).
For both Grounding Dino (Liu et al., 2024 and MQ-GLIP (uu et al. 2023). we consider the versions employing Swin-T (Liu et al.) 2021) as the backbone. We do not perform any prompt engineering and do not change any settings. Furthermore, we directly utilise the official GitHub repositories for both of the models.
25

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.15: Accuracy and calibration performance uncalibrated, early calibrated and late calibrated models on COCO mini-test. A red cell indicates a notable AP drop compared the uncalibrated detector while green cell implies consistency. Please note that the calibration performance of early and late calibration should not be compared they differ in their train and test sets (pre-NMS vs. post-NMS). Still for the interested reader, example is that early calibration yields 8.3, 17.1 and 9.7 LaECE for ATSS, PAA and RS R-CNN respectively once evaluated on post-NMS detections, CW: wlses-wise, CA: class-agnostic, bold: best calibration performance, underlined: second best. Calibrator is not available (N/A) for uncalibrated models. CA IR provides a good balance of AP and calibration performance. The results are presented on COCO minilesl.
Cal. Calsbratr
LaECE
LaACE
LaMCE
Type Type
RS R-CNNATSSERR ATSSPAA RS R-CNN ATSS PAA
RS R-CINN ATSS PAA R-CNN ATSS
PAA
N/A
N/A
42.4
43.1
43.2
13.79
0.20
3.39
5.18
25.22 9.73
35.37
42.53
10.40
CW
LR
26.1
12.0
37.1
0.41
0.12
0.22
25.12
10.13 28.35
61.11
24.79
60.05
Early
42.8
0.03
1.39
5.23
5.40
25.10
25.38
26.00
Cal.
LR.
42.4
42.8
43.2
0.65
0.17
0.37
27.48
7.86 25.40
68.81
23.07
54.03
IR
42.4
43.1
43.2
0.14
0.10
0.14
5.86
6.43 6.70
14.08
15.59 18.918111 15
N/A
N/A
42.4
43.1
43.2
36.45
5.01
11.23
29.30
17.48 15.79
45.42
40.00
32.33
CW
LR
12.4
13.1
23.22
1.36
2.69
359
11.19
9.03
12.01
20.200
29.5
33.76
Late
41.8
12.4
1.56
9.21
9.81 9.24
38.42 -3
40.34
Cal.
LR
42.4
43.0
43.2
5.83
4.46
1.63
13.86
9.46
11.88
37.79
29.58
29.91
TR.
42.3
43.1
43.2
3.15
4.51
1.62
8.93
9.51
9.61
35.72
37.35
35.50
35-
RS R-CNN (aarly calibrate)
(aarly calibrate)
30-
PAA (Early calibrate)
25
40
25
R CNN calibrate)
LSACE (In %67
ATSS (Late calibrate)
20
20
PAA (Late calib rate)
30
15.
25
10-
15
0-
(unc)
50
1.00
250
500
2500 (al)
((unc.)
50
100
255
500
2500 [al0
0 [unnc
50
100
250
500
2500 [all)
# Images used for calibration
Images used for calibration
Images used for calibration
(a) LaECE
(b) LaACE
(c) LaMCE
Figure A.6: The effect of number of images on calibration using CA IR. We find it sufficient to use only 500 images for calibration. The results are presented using COCO minitest.
Instance Segmentation We use three different Mask R-CNN variants for instance segmentation:
The vanilla Mask R-CNN (He et al. 2017 with ResNeXtt111 (Xie et al. 2016 backbone, softmax
classifier and using Repeat Factor Sampling (RFS) to address the long-tailed nature LVIS,
Mask R-CNN with ResNet-50,, sigmoid classifier, trained with RS Loss (Oksuz et al. 2021a] and
RFS,
Mask R-CNN with ResNet-50, softmax classifier, trained with Seesaw Loss Wang et al. 2020 but
RFS.
We obtain Vanilla Mask R-CNN and Seesaw Loss from mmdetection. As for Mask R-CNN trained with RS Loss, we use the official repository of RS Loss (Oksuz et al. 2021a) in which it is trained for 12 epochs using multi-scale training augmentation. The other Mask R-CNN variants also employ multi-scale training augmentation and the Vanilla Mask R-CNN trained for epochs as well. Differently, Mask R-CNN with Scesaw Loss is trained for 24 epochs and uses the mask normalization technique proposed in the same paper Wang al. 2020
26

---

Published in Transactions on Machine Learning Research (10/2024)
D.2 Further Ablation Experiments MoCaE
This section presents further ablation experiments by which we validate our design choices. These include the validation of calibration method and the design choices in Refining NMS.
D.2.1 Further Details and Ablation on Calibration
We use CA IR and LR to calibrate the models given the final scores in MOCAE, i.e., late calibration. More specifically, we use CA IR in all of the experiments unless otherwise explicitly specified. One notable exception where we use the CA LR instead of IR is the SOTA experiment in Tab. S in which we observe that CA LR performs 0.1 AP compared to CA IR. Please note that, we obtain those calibrators on a held-out validation set consisting of only 500 images. This section presents further experiments to validate these design choicos.
Validating the Calibrator While calibrating the predictive distribution of the classifiers, commonly the accuracy of the classifier is preserved. However, this might not be the case once the predictive confidence instead of the distribution is calibrated, which is the case in the common calibrators for object detection and in our case (Oksuz et al. 2023). This is mainly because the ranking among the detections can change unless the calibrator is monotonically increasing function of the confidence. Therefore, we would idcally expect calibrator to improve the calibration by at least preserving the accuracy of the detector. To ensure that, we investigate LR and IR both CA and Class-wise (CW) on late as well as carly calibration in Tab. A.15 Here, we obtain the calibrators on COCO minival and report the calibration error and accuracy on COCO minitest. We observe in the red cells in Tab. A. that all calibrators, except CA IR and CA IR, decrease AP especially for early calibration. Furthermore, these calibrators improve LAFCE in all cases. These observations on accuracy and calibration led us to choose CA IR and CA LR while calibrating the single models in MOCAE.
500 Images are Sufficient for calibration in MoCaE Equipped with the aforementioned insights, we then investigale the sulficient. number of images for calibration to enable MoEs using MOCAE. Our aim here is determine the cardinality of the held-out validation set to properly calibrate the models, which can help the practitioners to avoid reserving redundant data for this held-out validation set. Following the literature (Guo et al.) 2017 Oksuz et al.] 2023). we obtain calibrators on a held-out validation set (COCO minival) and then report the results on the test set (COCO minitest). Furthermore, we obtain several calibrators on both early and late calibration settings by using different number of images. Fig. [A.6 presents how LAECE, LAACE and LAMCE change when the cardinality of the hold-out validation set changes on three different detectors. We observe in general that calibration errors drop significantly even when only 50 images are used to learn the calibrators. Through introducing more images, while we do not observe notable gain in LAECE (Fig. (.6(()) a)), LAACE (Fig. A.)(b)) and LAMCE (Fig, AA,(c)) continue to improve especially for early calibration, implying the necessity for these calibration errors. Overall, as we have not observed a notable gain after 500 images, we keep 500 images on the held-out validation set while training the calibrators. A noteworthy point is that 500 images is a small number images compared to large training sets in object detection, demonstrating the ease of applicability of using MOCAE.
Comparison of Early and Late Calibration Here, we provide additional insights on early and late calibration.In Tab. A. we can see that while both approaches improve single models, late calibration performs slightly better than early calibration consistently. Furthermore, in practical terms, late calibration is significantly simpler the number of confidence scores obtained before post-processing is significantly larger than those obtained after postprocessing (i..,, final detections). To illustrate this more concretely, RS R-CNN outputs 1K proposals for cach image as raw detections, thus resulting with a single image containing 80K raw confidence scores for the COCO dataset and more than 1M scores for LVIS as each proposal has a score for each class. In addition, a very large amount of these raw detections do not even overlap with any objects, complicating the problem due to this imbalanced nature of the data. Furthermore, the number of raw confidence scores is significantly larger for one-stage detectors (ATSS and PAA in our case) as such detectors make predictions directly from very large number of anchors; making early calibration even more
27

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.16: Further experiments comparing early and late calibration. Standard NMS used for both the methods. While both approaches improve single models, late calibration performs slightly better. The results are presented on COCO minitest.
Calibration Type
RS R-CNN
ATSS
PAA
AP
AP50
APrs
AP's
APM
APL
N/A
42.4
62.1
46.2
26.8
46.3
56.9
(Single
43.1
61.5
47.1
27.8
47.5
54.2
Models)
43.2
47.1
27.0
47.0
57.6
Early
43.9
63.5
47.9
28.6
47.9
56.9
Late
44.1
63.0
48.4
28.5
48.4
56.8
Early
43.8
63.3
47.7
28.1
47.8
57.6
Late
44.0
62.7
47.9
28.2
48.2
58.1
Early
44.3
63.1
47.8
28.8
48,1
56.7
Late
44.4
62.5
48.5
29.2
48.5
57.3
Early
44.5
63.7
48.4
29.0
48.5
57.5
Late
44.7
63.1
48.9
29.2
49.0
58.2 2
Table A.17: Comparison of different calibration methods to obtain MoEs. CA IR performs better than other methods. The results are presented on COCO minitest.
Calibrated
RS R-CNN
ATSS
PAA
AP
AP50
AP75
APS
APM
APT.
N/A (Single Models)
42.4
62.1
46.2
26.8
46.3
56.9
43.1
61.5
47.1
27.8
47.5
54.2
43.2
60.8
17.1
27.0
47.0
57.6
X CW LR
42.4
62.1
16.3
26.8
46.3
56.9
43.0
61.4
28.1
47.2
54.4
CA LR
43.7
62.5
47.7
28.7
48.2 22
55.2
CA
44.1
63.0
48.4
28.5
48.4
56.8
x
1
43.4
62.5
47.1
27.3
47.3 .3
58.0
CW LRR
42.7
61.8
46.7
27.1
46.5
57.6
CA LR
43.5
63.0
47.7
27.9
47.5
57.8
CA
44.0
62.7
47.9
28.2
48.2
58.1
43.3
60.9
47.2
27.1
47.2
57.6
CW LR
42.6
60.7
46.5
26.6
47.2 2
53.9
CA LR
44.4
62.5
48.3
29.0
48.3
57.3
CA IR
44.4
62.5
48.5 18.5
29.2
48.5 55
57.3
43.4
62.5
47.1
27.3
47.3
58.0
CW LR
43.0
61.3
47.1
28.2
47.4
54.6
CA LR
44.0
62.7
48.1
29.0
48.5
55.8
CA IR
44.7
63.1
48.9
29.2
49.0
58.2
impractical for them On the other hand, we use only top-100 detections in COCO and top-300 detections in LVTS for each image following the evaluation specification of these datasets. Thereby resulting more practical scenarios with significantly smaller nunber of detections for late calibration compared to early. Consequently, considering its slight accuracy gain well as simplicity, we prefer late calibration over early to obtain MoEs in MOCAE.
"To keep this number manageable, use 1000 detections predicted from each pyramid level ATSS and PAA for early calibration.
28

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.18: Ablation analysis of MOCAE. We use the MoE combining RS R-CNN, ATSS and PAA on COCO mini-test
MOCAE
Refining NMS
Calibration
Pipeline
Soft NMS
Score Voting
Early
x
X
X
43.3
Early
/
X
X
44.5
Late
/
X
Late
/
X
44.8
Late
X
V
X
43,4
Late
X
V
V
44.4
Late
45.5
Table A.19: Using a subset of the training set for calibrating object detectors using MOCAE. The results are reported on COCO minitest. We compare the models in Tab.
Model Type
Detector
AP
AP
AP75
RS R-CNN
42.4
62.1
46.2
Single Models
ATSS
43.1
61.5
47.1
PAA
43.2
60.8
47.1
RS R- R-CNN X
43.4
63.0
47.7
Deep Ensembles
ATSS
44.1
62.3
48.4
PAA x
44.4
62.0
48 4
Vanilla MoE (RS R-CNN, ATSS, PAA)
43.4
62.5
47.1
MOCAE (ATSS and PAA) calibration on train MoCAE (ATSS and PAA) calibration on val
44.7
62.3
49.0
Mixtures of Experts
44.8
62.4
49.2
MOCAE (RS R-CNN, ATSS, PAA) calibration train MOCAE (RS R-CNN, ATSS, PAA) calibration on val
45.4
63.0
50.0
45.5
63.2
50.0
Effect of Different Calibration Methods on MoEs While we choose CA IR as our calibration method in MOCAE, here we present how different calibration methods perform obtaining MoEs. Specifically, we late calibration with CA LR and CW LR as these two methods also preserve the accuracy of single models as shown in Tab. A.17 Tab. A.17 presents the results where
that CA calibrators perform better than CW LR. Also, while CA LR obtains par performance with CA IR while combining ATSS and PAA, it performs worse once RS R-CNN is in the mixture. This might be because the calibration error of CA LR is higher than CA IR (for late calibration) in terms of all calibration measures as shown in Tab. A.17
presents more detailed version of the Tab. included in the paper. In this version, also include carly calibration, which performs similar with late calibration as shown in Tab. A.18
Ablation of MoCaE with Early Calibration Tab. A.18
Can we use training set to calibrate experts in MoCaE? Following the common convention in the classification literature (Mukhoti et al. 2020 Guo et al., 2017) and the
approaches object detection (Kuzucu et al.] 2024 Oksuz al.) 2023), we obtain the post-hoe calibrators on held-out validation set while using MoCAE. In experiments, this held-out validation set includes 500 images. While this is usually quite small dataset to keep for calibration, can be used for the purpose of combining detectors. To do so, we randomly sample 500 images from the training set of COCO and LVTS, and report the results in Tab. A.19 and Tab. A.20 The main observation in these tables that while colibration on a subset of the training set and on a volidation set perform almost on-par on COCO, there significant performance gap (1.0 AP) favor of using validation set for LVTS. Please note that, LVIS long-taile dataset with around 1K different classes and it has been shown in the literature that the models tend to overfit for the datasets with long-tailed examples (Feldman Zhang 2020
still investigate if subset of the training set
29

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.20: Using a subset of the training set for calibrating object detectors using MOCAE. The results are reported on LVIS val set (excluding 500 images used for calibration). We compare the models in Tab. I1
Method
AP
AP
AP75
AP,
APc c
AP;
Mask R-CNN
25.4
39.2
27.3
15.7
24.7
30.4
RS Loss
25.1
38.2
26.8
16.5
24.3
29.9
Seesaw Loss
25.4
39.5
26.9
15.8
24.7
30.4
Vanilla MoE
25.2
38.3
26.8
16.5
24.3
29.9
MoCAE calibration on train
26.7
40.7
28.6
17.6
25.9
31.4
MOCAE calibration on val
27.7
42.8
29.4
18.2
27.3
32.4
Table A.21: Sensitivity of Vanilla MoE and MOCAE to different configurations of Soft NMS. The results are presented on COCO mini-test and 500 validation images that used to train the calibrators for LVIS, We report box AP for COCO and mask AP for LVIS.
Method
Soft NMS
COCO
LVTS
X
43.4
37.5
Linear, NOUNMS 0.65
43.4
37.5
Gaussian, ONMS 0.20
41.6
37.9
Vanilla MoE
Gaussian, ONMS 0.40
42.1
37.8
Gaussian, ONMS 0.60
42.4
37.9
Gaussian, ONMS 0.80
42.7
37.8
Gaussian, NNMS 1.00
42.9
37.9
44.7
39.8
Linear, IoU NMS 0.65
44.8
39.9
Gaussian, ONMS 0.20
43.7
40.6
MoCxE
Gaussian, 0.10
14.4
40.8
Gaussian, ONMS 0.60
44.8
40.6
Gaussian, ONMS 0.80
44.7
40.2
Gaussian, ONMS 1.00
44.6
39.9
Feldman 2019), implying a larger generalisation gap (i.e., the difference between the test and the training errors) for such datasets. Subsequently, Mukhoti et al. (2020)and Bai et al. (2022) showed that the larger generalisation gap implies a larger calibration error. Accordingly, the results in Tab. A.20 shows the resulting drawback of using training sets for calibration compared to using a held-out validation set. Specifically, calibrating on validation set outperforms calibrating on a subset of the training set by 1.0 mask AP. We also note that, calibrating on the training set still outperforms Vanilla MoE by 1.5 mask AP, which demonstrates the importance of the calibration in obtaining an MoE.
D.2.2 Sensitivity of MoCaE to Design Choices in Refining NMS
Refining NMS combines Soft NMS and Score Voting. Specifically, Soft NMS can be linear or gaussian; furthermore both Soft NMS (either linear gaussian) and Score Voting have lyper-parameters. Here, we investigate the sensitivity of MOCAE to such design choices using Soft NMS ATSS and PAA for COCO; and the setting described in Sec. 4 for LVIS. We easily see in Tab. A.21 that Vanilla MoE does not benefit properly from Soft NMS without calibration. For example, there is no gain for Linear Soft NMS, the performance degrades for the Gaussian Soft NMS on COCO and the gain is only 0.4 for LVIS. This is expected as a single
example using RS R-CNN,
to reconciliate the scores all detectors might not be sufficient especially for the Gaussian Soft NMS. On the other hand, after calibration, we consistently see the gains for our MOCAE: MOCAE benefits slightly on COCO dataset both for linear and gaussian cases; and besides, the gain LVTS is 1.0 mask AP. This is because, the scores compatible for each detector
30

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.22: Object detection performance on COCO minitest with the detectors using multiscale testing. Test image scale refers the shorter side of the image, please refer to the text for the details. MOCAE) is complementary multiscale testing. Our gains in green are obtained compared to the best single model for each performance measure, represented underlined.
Model Type
Detector
Test Image Seale
AP
AP
AP
APs
APM
APt.
R CNN
42.4
62.1
46.2
26.8
46.3
56.9
ATSS
61.5
17.1
27.8
11.55
51.2
Single Models
PAA
43.2
60.8
47.1
27.0
47.0
57.6
RS R-CNN
100, 800, 1200
43.2
62.1
47.6
29.4
46.5
56.8
ATSS
400, 800, 1200
44.7
62.6
49.0
31.6
48.2
57.5
PAA
100, 800, 1200
11.7
62.0
19.0
31.6
18.2
58.5
Vanilla MoE (RS R-CNN, ATSS, PAA) Ours
800
43.4
62.5
47.1
27.3
47.33
58.0
MoCAE (RS R CNN, ATSS, PAA) Ours
800
45.5
63.2
50.0
29.7
49.7
59.3
Mixtures of Experts
Vanilla Mol (RS R-CNNN, ATSS, PAA)
400, 800, 1200
44.4
62.5
48.7
30.2
47. 8
59.6
MOCAE (RS R-CNN, ATSS, PAA) Ours
400, 800, 1200
46.4
63.3
51.6
32.9
50.0
61.2
+1.8
4.0.7
4-2.6
+-1.3
+1.8
-2.7
after calibration and a single hyperparameter allows Soft NMS to properly adjust the scores from different. detectors. We choose Linear Soft NMS on COCO resulting the best results for Vanilla MoE and MOCAF. For LVIS, we use Gaussian Soft NMS with ØNMS 0.40 for MOCAE. In similar way, we validate the pprar-prreeeter of Score Voting as 0.04.
D.2.3 MoCaE is Complementary Multiscale Testing
Multiscale test augmentation is another way of improving the detection performance of a detector, which has been commonly used in the literature (Oksuz et al. 2020 Zhang et al. 2020b Liu et al. 2018 Duan ltt al. 2019). In this section, we show that MoCAE is complementary to multiscale testing. To show this, we first apply multi-scale testing to RS R-CNN, ATSS and PAA as the individual detectors. Specifically, while we use only images with 800x1333 resolution for inference, here we use images with 400x667, 800x1333 and 1200x2000 resolutions as three different scales. Tab. A.22 shows that this way of testing improves the performance of the single models between 0.8 to 1.6 AP. Then, we combine the outputs of these models with multiscale augmentation using Vanilla MoE, and MOCAE and make two more important observations in the same Tab. A.22 (i) The performance of MOCAE with multi-scale detectors improves by 0.9 AP compared to the MOCAF with detectors without test. time a augmentation; and (ii) MOCAF improves the single best detector with multiscale testing by 1.8AP. These observations lead to the conclusion that MOCAE complementary to multiscale lesting.
D.3 Further Details and Analyses on Deep Ensembles
This section provides further details and analyses on DEs.
D.3.1 The Effect of Calibration on DEs
DEs combine the same models that are trained from different initialization of the parameters. Ideally, the expectation over the predictive distributions of the components in a DE yields the prediction of the DE. This can be casily obtained for classifiers which predict categorical distribution over the classes given an input image. On the other hand, it is not straightforward to use DEs for detectors there is no clear way to associate detections from differont detectors. As result, similar to MOCAE, we obtain DEs by using late calibration as shown in Fig. aaa,, which turns out to be an effective method. To see that, we first. present the single model performance of the five different components comprising DEs in Tab, [A.23] Then, from these single detectors, we obtain DEs for PAA with and without calibration using the standard NMS. Tab. shows shows that this way of obtaining DEs is effective as the performance increases when the number of components increases. We observe that increasing the number of components improve the performance
31

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.23:
Single model performance of the detectors that we used in DEs. While obtaining MoEs, we combine "Model 1 of different types of detectors.
Model
AP
AP
APra
APs
APx
APL
R- R-CNN (Model 1)
42.4
62.1
26.8
46.3
56.9
R$ R-CNN (Model 2)
42.6
62.7
46.7
27.3
46.5
55.8
RS R-CNN (Model 3)
42.6
62.6
46.5
27.2
46.8
56.
RS R-CNN (Model 4)
42.6
62.3
46.1
27.9
46.1
55.9
R-CNN (Model 5)
42.2
62.8
26.7
46.5
55.7
ATSS (Model 1)
43.1
61.5
47.1
27.8
47.5
54.2
ATSS (Model 2)
43.3
61.5
47.5
28.9
47.8
55.0
ATSS (Model 3)
43.3
61.5
28.9
47.9
55.8
ATSS (Model 4)
43.0
61.2
29.0
47.4
54.6
ATSS (Model 5)
43.3
61.5
47.5
28.2
47.7
54.8
PAA (Model
43.2
60.8
47.1
27.0
47.0
57.6
PAA (Model 2)
43.5
61.0
47.3
27.5
47.7
57.7
PAA (Model 3)
43.4
61.1
47.2
27.7
47.6
57.8
PAA (Model 4)
43.6
61.4
47.2
27.6
47.6
57-6
PAA (Model 5)
43.6
61.3
47.2
27.9
47.9
58.0
Table A.24: The effect of increasing the components and using calibration on DEs. Increasing the components improves the performance while calibration does not have notable effect performance for DEs unlike their importance for MoFs.
Model
Calibration
AP
AP
AP78
APs
APM
APL
43.8
61.4
28.2
48.3
58.3
PAA
43.9
61.4
17.7
28.0
48.3
58.6
X
44.1
61.5
48.0
28.8
48.7
58.8
PAA x 3
V
44.1
61.6
48.0
28.7
48.7
58.9
X
44.3
61.7
48.3
28.9
48.8
59.1
PAA x
44.3
61.7
48.3
28.9
48.8
59.1
44.4
61.7
48.6
29.1
49.0
59.3
PAA x
44.4
61.7
48.6
28.9
49.0
59.3
between 0.1 0.3 AP. On the other hand, there is incompatibiitty among different detectors in DE, the effect of calibration is not notable for DEs. Still, having observed that PAA with components has slightly better (0.1 AP) performance once calibrated, in our comparisons we use DEs with calibration.
D.3.2 DEs with Less Components and Refining NMS
In Tab. 6 we compared MOCAE with DEs with 5 components. Note that in this case, the DEs have more components, implying a higher number of parameters compared to MOCAE with or 3 components. For the sake of completeness and provide more fair comparison to MOCAE with maximum of components Tab. A.25 extends our comparison in Tab. 6 by including (i) the DEs with 3 components and (ii) the DEs with Refining NMS. Tab. A.25 presents that with equal number of components, our MOCAE outperforms the best DE with 3 components by 1.4 AP (44.1 PAA x3 vs 45.5 AP of MOCAE). Furthermore, in the case that Refining NMS used for MoCAEs, their performance consistenty improves; showing the effectiveness of our aggregator. Still, the performance of the best DE, i.c., PAA with 5 components and Refining NMS, is 0.5 AP lower compared to MOCAE; demonstrating the effectiveness of approach.
32

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.25: The effect of Refining NMS on Deep Ensembles and Vanilla MoE.
Model Type
Detector
AP
APao
AP35
APs
APx
APL
RS R.CNN
42.4
62.1
46.2
26.8
46.3
56.9
Single Models
ATSS PAA
43.1
61.5
47.1
27.8
47.5
54.2
47.1
27.0
57.6
RS R CNN X
43.3
63.1
47.4
27.7
47.5
57.1
47.9
29.5
PAA
58.0
RS R-CNN with Ref. NMS
18.1
58.5
ATSS x 3 with Ref. NMS
44.3
62.1
48.3
29.7
29.3
56.3
Deep Ensembles
PAA 3 with Ref. NMS
44.6
61.6
48.7
28.9
49. 1
59.5
RS R-ONN x
43.4
63.0
47.7
28.0
47.5
57.0
ATSS x
44.1
62.3
48
29.
49.0
56.3
PAA x
44.4
62.0
48.4
28.9
49.0
59.2
RS R-CCNN x with Ref. NMS
44.5
63.0
49.1
28.8
48.6
58.4
ATSS x with Ref. NMS PAA with Ref. NMS
44.6
48.9
29.8
49.5
45.0
19.3
Vanilla MoE (RS R. B.CNN, ATSS, PAA]
27.3
58.0
Vanilla MoE with Ref. NMS R-CNN, ATSS, PAA)
411
48.2
27.8
18.4
Mixtures of Experts
MOCAE (ATSS and PAA) Ours MOCAE (RS R-CNN, ATSS, PAA) Ours
44.8
49.2
29.4
57.6
45.5
63.2
50.0
29.7
49.7
59.3
+2.3
1.1
+12.9
++1.9
+-2.2
+1.77
25
45
CNNNN
ATSS
20
PAA
70-
McCal NMS
AR8100 Qin %0
35-
15.
McCaE Ref. NMS
30-
10-
25
500
20
40
15
Medium Objects
10
OLRP
TIDE Type
Type
(a) TtDE-style Analysis
(b) Average Recall
(c) LRP Error Components
Figure A.7: The contribution of MOCAE on performance aspects
D.4 Why does MoCaE Improve Performance of Single Detectors?
Here, we investigate how MOCAE improves the performance of single detectors and note two main takeaways: First, MOCAE combines the detectors in way that the resulting MoE benefits from the complementary nature of the detectors; and as expected, second MoCAE mainly improves the recall of the single detectors.
D.4.1 Complementary Nature of MoCaE
Ideally, an MoF. aims to combine the individual components in such way that they complement each other in different subsets of the input space to achieve better performance. Here, we show that this is, in fact, the case also for MOCAE. Similar to previous analyses, we combine ATSS, PAA and RS R-CNN and report the results COCO méni-test. Differently, in order to be able to discuss the results, we utilize 12 supercategorios such animal, food, which are already present in COCO datasct. Fig. A.8 shows the results in which we can easily observe that different experts prevail on different supercaleeorries. To illustrate, RS R- CNN (Oksuz et al. 2021a) is the best single model in the appliance supercategory, ATSS (Zhang et al.) 2020b the best single model in the kitchen supercategory and PAA (Kim & Loc 2020) the best in the indoor supercategory. noteworthy point is thatMoCAE performs better than any of its components in all of the supervalegories, achieving notable improvements the entirety of the input space. This effectively
33

---

Published in Transactions on Machine Learning Research (10/2024)
65
RS-RCNN
60
ATSS
55
PAA
50
MoCaE
AP
45
40
35
30
animal
person
electronic
vehicle
outdoor
appliance
sports
furniture
kitchen
food accessory indoor
Supercategories
Figure A.8: AP scores of single models (RS-RCNN, ATSS, PAA) and MOCAE (consisting of RS-RCNN, ATSS, PAA) on different supercategories of the COCO mini-lest dataset. MOCAF, performs consistently higher than any of the single models on every supercategory.
shows the capability of MOCAE to leverage the complementary expertise of the single models form a more accurate mixture.
D.4.2 The Contribution of MoCaE on Different Performance Aspects
We now analyse what performance aspects, among localisation, recall and precision, affected by MOCAE by expliting three different analyses tools from the dctection litcrature. First, use TIDE (Bolya et al. 2020) that defines oracle APs as the APs obtained when FP and false-negative (FN) errors are completely mitigated. Then, the difference between the oracle and actual AP correspond to the error of a detector for a specific performance aspect. More spocifically, while obtaining Oracle FP AP, the FP detections are simply removed from the final detection set. As for Oracle FN AP, the FN objects are removed from the dataset. Fig. A.7(a) shows that MoCAF variants perform similar to the single detectors terms of FP Error while they clearly outperform them on FN Error. This indicates that one of the contributions of MoE to find the objects that are not detected by at least one of the individual detector illustrated on Fig. 2 and Tab, 2 However, TIDE analysis does not provide insight on the localisation quality which is mainly targeted by Refining NMS. Therefore, in our second analysis we exploit AR defined as the average of the recall values over 10 IoUs from 0.50 to 0.95. Aligned with the observation in TIDE analysis, Fig, A.7(b) presents that MoCAE improves AR of the single detectors. Furthermore, the performance gain mainly originates from the improvement in small and medium objects as MoCAF does not improve the performance on large objects notably. This indicates that the resulting MoE is especially stronger than single detectors more challenging object categories. Using Refining NMS further boosts the AR performance it improves the localisation performance, which is critical for recalls with higher IoUs. To investigate the benefit of MoE in practical usc-casos, we finally conduct an LRP analysis (Oksuz et al. 2021c 2018) in Fig. A.7 In this figure, oLRP FP, FN and Loe correspond to the components Optimal LRP Error defined as I-Precision, 1-Recall and the average loU Error of TP detections. Aligned with our previous analysis, MoE mainly decreases the recall error and MOCAE with Refining NMS mainly contributes to the oLRP Loc component, outperforming all individual models in the end. These three different analyses confirm that MOCAE with NMS mainly decreases the recall error of the detector and using Refining NMS contributes to the localisation error.
34

---

Published in Transactions on Machine Learning Research (10/2024)
45
45
40
40
(in %
Ln %)
35
35
F R-CNN
AP
RS R-CNN
30
30
ATSS
PAA
25
AP=1.4
AP=21.0/
25
0.05
0.25
0.50
0.75
0.05
0.25
0.50
0.75
Background Removal Threshold
Background Removal Threshold
(a) Uncalibrated
(b) Early Calibrated
Figure A.9: The effect of background removal threshold on AP and NMS processing time for (a) uncalibrated and (b) early calibrated detectors on COCO. The area of the dots are proportional to the NMS processing time the detectors. This threshold is typically set to 0.05, in which case PAA and RS R-CNN have large NMS processing time in (a). Once uncalibrated, the detectors follow different trends and are sensitive the threshold in terms of AP and NMS processing time. In (b); early calibration (i) aligns the detector by reducing this sensitivity, and (ii) allows using 0.05 by both maximizing the AP and reducing NMS processing time for the over-confident PAA and RS R-CNN. F R-CNN refers to Faster R-CNN.
Table A.26:
The values used in Fig. A.9 Early calibration regularizes the behaviour of the detectors by reducing their sensitivity to background removal threshold with respect to AP and NMS time. NMS time is measured in terms of ms using single Nvidia 1080ti GPU.
Background Removal
Detector
Uncalibrated
Early Calibrated
Threshold
AP
NMS
AP
NMS time
Faster R-CNN
40.1
0.5
40.3
0.6
0.05
RS-RCNN
42.4
35.4
42.4
0.6
ATSS
43.1
0.6
43.1
0.7
PAA
43.3
29.2
43:2
0.8
Paster R-CNN
38.6
0.4
39.6
0.5
0.25
RS RS-N
42.4
2.0
41.9
0.5
ATSS
38.0
0.5
42.7
0.6
PAA
43.2
42.7
Faster R-CNN
36.2
35.7
0.50
RS-RCNN
42.1
ATSS
19.3
0.4
39.3
PAA
39.2
0.5
39.3
Paster R-CNN
32.1
0.4
26.5
0.4
0.75
RS-RCNN
25.1
0.4
30.4
0.4
ATSS
1.4
0.0
30.7
0.5
PAA
21.0
0.4
29.6
0.5
D.5 A Use Case for Early Calibration: Reducing the Sensitivity to Background Removal Threshold
Here, investigate an additional use-case of early calibration in which it reduces the sensitivity of the detectors to background removal threshold in terms of both AP and efficiency. As AP provably benefits from more detections (Oksuz al. 2023), detectors prefer small background removal threshold as the first step of post procosrccssing (Fig. b()).. To illustrate, 0.05 is the common choice for COCO [Zhnng et al.) 2020b
Kim &
Lee 2020: Ren el al. 2017) and il is as low as 10-4 for LVIS [Chen el al., 2019 Gupla el al, 2019 While
this convention preferred by AP, it can easily increase NMS processing time especially for over-confident detectors. This is because, for such dotectors, the background removal stop accepts redundant truo-negatives (TNs), which should have been rejected. Hence, due to this large number of redundant TNs propagated the NMS, NMS processing time significantly increases. To illustrate PAA, which uses threshold of 0.05
35

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.27: Experiments with Self-aware Object Detectors. We use the General Object Detection setting in Okssuz al. 2023). Please refer the text for the details of the performance measures.
Model Type
Self-aware Detector
OOD Detection Performance
Arc. and for In-distribution
Acc. and Cal. for Domain shiff
DAQ1 1
BA
IDQ
LaRCE
LRP
IDQ
LaRCE
IRP
BA-RS-RCNN
40.9
89.0
39.6
18.0
73.9
27.1
19.2
83.7
Single Models
SA-ATSS
40.0
87.9
19.4
17.9
73.9
27.3
20.6
83.5
SA-PAA
40.6
87.3
39.0
17.8
74.4
27.1
21.0
83.6
Vanila MoE
42.5
88.7
39.9
18.1
73.6
29.2
19.6
82.1
MoFs
MoCAE
87.6
for COCO, NMS takes 29.2 ms/ingn assimage on a Nvidia 1080Ti GPU, while only takes 0.6 ms/image for ATSS and Faster R-CNN (F R-CNN). This difference among the detectors casily be noticed by comparing Specifically,
the areas of the dots at 0.05 in Fig. A.9 a propagated to the NMS per image average. After early calibration, this number of detect from the same threshold reduces to ~ 2K per image, which now enables NMS to take only 0.8 ms/ ms/image as ideally expected. Fig. (b prcsents that NMS takes consistently between 0.6 to 0.8 tt/mmm//maaeeea /ssmmage for all detectors as the behaviour of the detectors are aligned.
observed for PAA that ~ 45K detections are
D.6 Further Discussion on SAOD Task
Details of the SAOD task Self-aware object detec- tion task (SAOD) (Oksuz al. 2023) provides a compre- hensive framework to evaluate the robustness of object detectors. Specifically, the performance aspects that are jointly considered in this task are out of distribution de- tection, calibration, domain shift as well as the accuracy. Specifically, SAOD evaluates an object detector in terms of the following criteria:
Table A.28: Object discovery the out-of- distribution set (SinObj110K-OOD) from SAOD (Oksuz et al. 2023). Our gains in green are highlighted with respect to the best single model (underlined).
Model Type
Detector
AR
Rejecting OOD images utilizing reliable image- level uncertainty estimates
R$ R- R-CNN
41.3
Single Models
ATSS
42.0
Yielding accurate and calibrated detections
PAA
39.7
Being robust to domain shift under varying sever- ities.
Vanilla MoE
42.7
MoEs
MOCAE Ours
45.6
The evaluation is conducted using more than 150K single images, which a large-scale test enabling thorough evaluation. Specifically, the SAOD task utilizes the fol- lowing split of the datasets for evaluating detector in terms of the aforementioned performance aspects:
+3.6
DID: The in-distribution dataset consisting of images containing the same set of foreground objects as the training set.
T(DIp) Domain-shift in-distribution dataset obtained through applying transformations from [Hendrycks & Dictterich 2019) with severitics 1, and on the in-distribution set.
DOOD The oui-of-listribution dataset that contains only the objects of different foreground classes than that of the DID.
On this datasct, the idcal behavior expected from a robust object detector for given input X is:
If E DID, "accept" the input and provide accurate and calibrated detections, any rejection is penalized.
If X E T(DIp), with severitics 1 and 3, "accept" the input and provide accurate and calibrated delections, any rejection is penalized.
Tab.
A.26 shows the exact values used obtain Fig. A.9
36

---

Published in Transactions on Machine Learning Research (10/2024)
If X E T(DID), with severity 5, provide the choice to "accept" the input though no penalty for rejections as transformed images might have severe deformities with respect to their original versions. X E DOOD, "reject" the input and refrain from providing any detections, any accept is be penalized.
In association with these datasets, the authors also propose the following evaluation measures:
Balanced Accuracy (BA) measures the OOD performance as the harmonic mean of TPR and TNR
in this binary classification problem.
LAECE measures the calibration performance, discussed in App A
In-Distribution Quality (IDQ) combines accuracy and calibration performance as the harmonic 2018) and I-LaECE on in-distribution data DID- Analogously, for domain-shifted data, T(DIn), IDQr is used.
mean of 1-LRP (Oksuz et al.
Finally, Distribution-ANe Awareness Quality (DAQ) as the main performance measure of the task, unifies these measures. Specifically, DAQ a higher the better measure, defined as the harmonic mean of BA, IDQ and IDQr-
Implementation Details Based the definition of this task, the authors also propose algorithm to convert any detector to a self-aware one We follow the proposed algorithm to make RS R-CNN, ATSS and PAA, as well as Vanilla MoE and MoCAE self-aware. We use the General Object Detection setting in (Oksuz et al. 2023) as our models are trained on COCO, aligned with this dataset. We utilise the official SAOD code throughout our experiments by keeping all the settings.
Discussion of the Results The results are presented in Tab. A.27 in which MOCAE improves DAQ measure, the main performance measures of this task, up to +2 points compared to the best single model while also showing notable improvements in terms of the rest of the measures. Furthermore, MOCAE also outperforms Vanilla MoE. This highlights that the MOCAE is more reliable than its counterparts in terms of the reliablity aspects considered within the SAOD (Oksuz et al. 2023) framework. When we examine the individual robustness aspects, we can easily see that MOCAF outperforms all of the single detectors and Vanilla MoE in terms of calibration and accuracy both on in-distribution and domain-shifted data. On the other hand, we observe that the OOD performance drops from ~ 1 BA compared to the best single model. In the following, we further discuss why this is the case and provide more insight.
Is Lower OOD Performance Pitfall of MoCaE or its Strength? MOCAE combines individual detectors to ideally benefit from the strength of each detector in the mixture. This commonly manifests itself to increase the recall performance of the individual components, i.e. Average Recall (AR), as presented in Tab. [ while motivating MOCAE as well as in Fig. A.7 AR consistently increases in such cases due to the fact that the detections of the individual experts are combined properly in in-distribution images.
Now, consider an alternative case, which the input image is out-of-distribbation (OOD), on which each detection is a FP an OOD image does not contain an in-distribution object (as defined by (Oksuz al., )) In this specific case, combining the detectors properly might result in more FPs casily. To illustrate loy example, assume there are two images X1 and X₂ and & represents the number of maximum detections in an image, which is typically h = 100. Also assume that, an overconfident detector yields high-confident FP on an out-of-distribution (OOD) image X1 with A: - 1 lower confident detections. As for X₂ this detector does not have a high-confident detection. This detector accepts the OOD image X1 and rejects X2. A second but an underconfident detector yields very low confidence detections on X1 and one relatively confident. detection on X2. In contrary to X2, this detector accepts the OOD image X₂ and rejects X1. When we combine these two detectors without calibration, the resulting Vanilla MoE will mimie the overconfident. detector, potentially rejecting X2 and accepting X1. Unfortunately, calibrating these two detectors in the form of MOCAE will have a confident detection in cach image, potentially resulting in the acceptance of both OOD images. This is why Vanilla MoE mimics the most confident detector in Tab. A.27] In this specific case, the most confident detector, RS R-CNN has the largest BA, and consequently Vanilla MoE outperforms MOCAE in terms OOD performance.
Wee refer the reader to the Algorithm A.I and Algorithm A.2 Oksuz
2023 for the details.
37

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.29: Oracle MoE and pitfalls of MOCAE. Object detection performance on COCO mini-test, When the single methods have significant performance gap, then MOCAE might not perform well as the calibrators are imperfect. Oracle MoE is an MoE in which each expert perfectly calibrated. As a result, all experts have 0 LAECE in Oracle MoE. Following Theorem
1 Oracle MoE outperforms all single detectors.
Model Type
Method
AP
AP 30
AP,s 75
APs
AP' M
APt.
Unc. LABCE
Cal. LAECE
RS R-CNN
42.4
62.1
46.2
26.8
46.3
56.9
36.45
3.19
ATSS
43.1
61.5
47.1
27.8
47.5
54.2
5.01
4.54
PAA
43.2
60.8
47.1
27.0
47.0
37.6
11.23
1.09
YOLOv7
55.6
73.1
60.6
41.2
60.4
69.5
9.23
4.60
Single Models
QueryInst.
55.9
75.4
61.3
38.5
60.8
73.2
4.54
3.19
DyHead
56.8
75.6
62.2
42.8
60.6
71.0
10.04
6.26
EVA
64.5
82.3
T1.0
50.6
68.9
78.1
13.39
7.13
Co-DETR
64.5
81.7
70.8
51.0
68.0 G
79.3
5.81
4.82
MoEs of EVA and Co-DETR
Vanilla MoE
64.6
82.3
71.3
50.7
79.0
N/A
N/A
MoCAE
65.0
82.6
71.5
51.0
69.0
79.6
N/A
N/A
Oracle MoE
81.9
96.6
91.1
72.7
85.2
92.1
N/A
N/A
MoEs of RS R-CNN, ATSS, PAA EVA, Co-DETR
Vanilla MoF
60.0
76.3
65.8
43.8
64.9
76.5
N/A
N/A
MoCAF
61.9
78.8
69.0
49.0
66.6
76.4
N/A
N/A
Oracle MoF
85.3
97.2
933.2
78.3
88..5
94.5
N/A
N/A
MoFs of YOLOV,, QueryInst, DyHead EVA, Co-DETR
Vanilla MoF.
64.1
81.4
70.6
50.2
68.5
78.4
N/A
N/A
MoCAF
63.1
80.1
69.5
49.9
68.0 0
77.5
N/A
N/A
Oracle MoE
86.1
97.4
93.8
77.4
89.1
95.0
N/A
N/A
MoEs of All Single Models
Vanilla MoE
60.1
76.3
66.0
44.1
65.1
76.6
N/A
N/A
MoCAE
61.7
78.3
68.1 1
48.7
66.9
76.3
N/A
N/A
Oracle MoE.
86.7
97.4
94.1
78.0
89.8
95.5
N/A
N/A
Following from the comparison of OOD performances of MoEs, the question arises why detectors generate high confidence detections on OOD images and whether there is a benefit of having them. More specifically, we conjecture that this mainly a result of object discovery as the DOoD merely contains the classes that are not exactly present in DID, in which objects of semantically similar classes still exist across the two. To support this claim, design an experiment on the same OOD split. In this experiment, the bounding box annotations already present in the OOD split and check whether which of the models find the most number of objects. We do not distinguish among the classes as all the classes in OOD split (SinObj110K-00
use
pertains to the unknown class for all of the models we use. We report AR in Tab. A.28 to evaluate the models with respect to their object discovery characteristics. The results show that MOCAE outperforms all individual models and Vanilla MoE with significant margin terms of AR; demonstrating that it finds more objects compared to the other models. This is because, unsurprisingly, the confident detections of individual models correspond to the objects and combining them after calibration in MOCAE results in finding more unknown objects. As result, while MoCAF does not perform the best in terms of image-level OOD detection, it is, in fact, a better alternative for object discovery.
D.7 The Effect of Theorem using an Oracle MoE
We proved Theorem I that the calibration target that we designed in Eq. provides the optimal AP under certain assumptions on post-procossing. Here, we show that the assumptions are, in fact, not very strict. and significantly higher AP values can easily be observed once the individual detectors in the mixture are perfectly calibrated as suggested in Theorem [ To do so, we design an Oracle MoE, in which we replace the confidence score of the detections by the corresponding calibration target in Eq. g such that the detectors are now perfectly calibrated. In Oracle MoE, we use the standard NMS to keep its design simple. Tab. A.29 presents the results of this Oracle MoE combining different object detectors on COCO mini-lest. When only two detectors, EVA and Co-DETR, are combined using Oracle MoE, we observe that the gain of the Oracle MoE compared to the best single model is more than 15AP, which is very significant. Furthermore, as the number of components increases, the performance of Oracle MoE consistently increases up to 86.7 AP and 97.4 AP when all of the eight detectors (with very different performances) are combined. The
38

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.30: Detailed results on LVIS val set. The detectors have different characteristics in terms of exploiting Repeat Factor Sampling (RFS), backbone, the number of training epochs and the loss function. While Vanilla MoE does not yield gain, MOCAE enables a stronger MoE than all single detectors.
Method
RFS Backbone
Epoch
Instance Sogmentation Performance AP,
Object Detection Performance
AP
APso
APm
AP.
AP;
AP
APsa
Ar
AP.
AP;
Seesaw Mask R-CNN (Wang et al.
2020
ResNet-50
24
25.4
39.5
26.9
15.8
21.7
30.4
25.6
11.6
26.6
14.0
21.0
32.3
RS Mask R&CNN Olosuz
Re-Net-50
25.1
38.2
26.8
16.5
29.9
25.8
39.7
27.8
24.5
32.0
Mask R.CNN
He
RosNeXt 101
12
25.4
39.2
27.3
15.7
24.7
30.4
26.6
42.1
28.5
15.4
25.2
33.1
Vanilla MoE
N/A
N/A
N/A
25.2
38.3
26.8
16.5
29.9
25.9
39.8
27.9
24.5
32.2
-0.2
-1.2
0.5
0.0
0.4
-0.5
2.3
-0.6
00.3
0.7
-0.9
MoCAE (Ours)
N/A
N/A
N/A
27.7
42.8
29.4
18.2
29.1
44.8
31.4
17.0
27.9
35.8
+2.3
+3.3
+221
+1.7
42.4
+2.0
+2.5
+1.6
+2.9
+1.6
+227
42.7
significance of these AP values validates Theorem from the practical perspective. On the other hand, once the performance gap among the combined detectors increases, observe that Vanilla MoE and MoCAE do not perform well, which we discuss in the following section.
D.8 Limitations of MoCaE
On the importancc of performance difference among the detectors As discussed in Sec. in the paper, the main limitation of MOCAE (as well as Vanilla MoE) is that it benefits from combining similarly performing detectors. This is presented in Tab. A.29 in which we see that combining low-performing detectors (e.g., RS R-CNN or YOLÓv7) with EVA and Co-DETR decreases the performance of MoE to an AP less than the best single detector. On the other hand, as just presented in App. D.7 this is not the case for Oracle MoF. This suggests that while the calibration decreases the LAECE of the single detectors significantly (Tab. A.29), they are still far from being well-calibrated to construct an accurate MoE. This is because, we keep the capacity of the calibrators low, that is Class-agnostic LR and IR are monotonically increasing function of the predicted confidence; thereby preserving the ranking among the detections and the AP of each detector. On the other hand, this capacity limitation might not result in better MoEs than the individual components once their performance gap is high. Furthermore, the Oracle MoE consistently improves upon the individual detectors even the performance gap is significantly high as shown in Tab. A.29 This demonstrates the importance of the calibration quality in obtaining an accurate MoE as well as the potential to improve the object detectors using MoEs, which is an open problem.
MoCaE requires more resource than single models Another limitation of MOCAE stems from using multiple models during inference, which is also the case for DEs. Still, as each detector can process the input in parallel, the overhead introduced by calibration would be negligible when separate GPU is allocated for each detector. As an example, on an Intel i7 3.60GHz CPU, obtaining the isotonic regression calibrator and inference take 12.0 ms and 0.30 mssimaae respectively. This inference cost is negligible it increases the inference time of ATSS only by 0.2% and EVA by 0.005%. Besides, comparing Fig. [bb) with Fig. aa), we showed that calibration balances the contribution of the detectors to MoE. However, PAA, as the most accurate detector. has still the lowest contribution with %25.61 in Fig, [b). This is because LAECE after calibration in Tab. [ is still non-zero for the detectors. Therefore, better calibration methods could give rise to more accurate MoFs.
D.9 Full Version of the Tables Pruned in the Paper
Tab. A.30 presents the detailed results on LVIS dataset for instance segmentation; Tab. A.31 includes the performance of all classes for DOTA dataset; Tab. A.32 shows the performance over different object scales; and Tab. A.33 includes COCO minitest results for common object detectors as well as their main differences in terms of pretraining data and the backbone. These tables are excluded from the main paper due to the space limitation.
39

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.31: The performance of all classes on DOTA v1.0. AP the main performance measure of DOTA.
Dataset Classes
Detector
All
Plame
Basebo. Bridge Ground-t. Small-veh. Large-veh.
Ship
Temmis Basket Storag. Socver Roundab. Harbor Swimn.. Iecico.
RTMDet
81.32
88.04
86.20
08.50
82.43
$1.21
84.87
88.70
90. 89
88.75
87.33
72.12
70.85
81.16
81.49
77.24
LSKN (prev. SOTA)
81.8
89.69
85.70
61.42
81.23
91.37
86.05
88.64
90.88
88.49
87.40
71.67
71.35
79.19
51.77
80.85
Vanilla Mol
50.60
87.76
83.27
61.77
78.25
81.26
85.33
58.334
89.93 0.36
85.54
86.35
70.98
63,92
84.28
82.45
77.57
1.25
1.93
2.93
+0.30
4.98
0.11
0.72
0.96
1.05
1.14
5.43
13.12
1..68
3.28
MoCAl
$2.62
89.09
86.47
61.38
83.28
81.43
85.03
68.72
90.86
88.31
87.11
75.50
74.12
84.49
81.63
81.93
(Ours)
10.77
0.60
10.27
0.00
-0.05
10.08
1.02
0.002
0.03
0.44
0.29
13.39
1.7.7
13.33
0.14
11.08
Table A.32: Effect of calibration MoE performance. All MoEs use Late calibration with standard NMS. While combining uncalibrated detectors do not provide notable gain over the single detectors, calibration is essential for strong MoE resulting in up to ~ 1.5 AP gain over single detectors.
Combined Detectors
Object Detection Performance
Model Type
Calibration
RS R-ONN
ATSS
PAA
AP
AP50
AP75
APs
APM
APr.
N/A
/
42.4
62.1
46.2
26.8
46.3
56.9
Single Models
N/A
43.1
61.5
47.1
27.8
47.5
54.2
N/A
/
43.2
60.8
47.1
27.0
47.0
57.6
/
42.4
62.1
46.3
26.8
46.3
56.9
44.1
63.0
48.4
28.5
48.4
56.8
43.4
62.5
47.1
27.3
47.3
58.0
/
/
/
44.0
62.7
47.9
28. 2
48.2
58.1
Mixtures of Experts
43.3
60.9
27.1
47.2
57.6
/
44.4
62.5
48.5
29.2
48.5
57.3
X
7
43.4
62.5
47.1
27.3
47.3
58.0
44.7
63.1
48.9
29.2
49.0
58.2
D.10 Reliability Diagrams of the Utilized Models
For further insights, include the reliability diagrams of the object detectors and instance segmentation models in Fig. A.10 Fig. A.14, The figures show that the calibration performance of the detectors improves after using post-hoc calibrators, enabling us to construct an effective MoE using MOCAE. Hore, we use 500 images to calibrate the detectors. This the reason of the small difference between LaECE values of Fig. A.10 and Tab. which uses all 2500 images in COCO minival for calibration.
40

---

Published in Transactions on Machine Learning Research (10/2024)
Table A.33: Object detection performance on COCO test-dev and måni-test using strong object detectors. The gains reported compared to the best single model as underlined. MOCAE maintains the significant. AP boost also for this challenging setting well.
Meshod
Pretzaining
Backhome
COCO test-dev
COCO minitest
Data
AP
AP
APts
APs
APM
APL
AP
AP
AF1:
APs
APи
APL
YOLOv7 (Wane al. QueryInst
2022a
None Nome IuageNel22K
L-size conv.
55.5
73.0
60.6
37.9
58.8
67.7
55.6
73.1
60.6
41.2
60.4
69.5
Ling
Swin-L.
55.7
75.7
61.4
36.2
58.4
70.9
55.9
75.4
61.3
38.5
80.8
73.2
DyHead
Dar
2021
Swin-L
56.6
75.5
61.8
39.1
59.8
68.7
56.8
75.6
62.2
42.8
60.6
71.0
Vanilla Mol
N/A
N/A
57.8
76.6
63.2
40.0
80.9
70.8
57.7
76.3
62.9
42.8
62.7
72.8
-0.4
MoCaE
A/A
N/A
(Ours)
+2.4
+1.5
+2.9
+1.7
+-288
+1.5
+2.1
+1.1
+2.1
+1.9
+2.8
+11
1.0
1.0
1.0
meen IoU
mean IoU
meen IoU
% Samples
0.8
Samples
LaECE
LaECE- 11.25%
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.0 0.2
0.4
0.6
0.8
1.0
0.0.0
0.2 0.4
0.6
0.8
1.0
0.0
0.0
0.2 0.4
0.6
0.8
1.0
Confidence
Confidence
Confidence
(a)
Uncalibrated
(b) Uncalibrated
(c) Uncalibrated
1.0
1.0
1.0
mean IoU
mean IoU
mean Iou
Samples
0.8
Samples
0.8
Samples
0.6
0.6
0.6
0.2
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2 0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8 1.0
Confidence
Confidence
Confidence
(d) Isotonic Regression
(c) Isotonic Regression
(f) Isotonic Regression
Figure A.10: Reliability diagrams the detectors presented on Tab. (a) Uncalibrated RS R-CNN, (b)
Uncalibratod ATSS, (c) Uncalibrated PAA, (d) RS R-CNN calibrated by Isotonic Regression, (c) ATSS
calibrated by Isotonic Regression, and (f) PAA calibrated by Isotonic Regression.
41

---

Published in Transactions on Machine Learning Research (10/2024)
1.0
1.0
1.0
mean IOU
mean Iou of Samples
% Samples
0.8
0.8
LaECE 7.03%
LaCCE= 5.00%
0.6
0.6
0.6
0.4
LacCE= 13.12%
0.2
0.2
mean of Samples
0.2
0.g
0.0
0.2 0.4
0.6
0.8
1.0
0.0
0.0
0.2 0.4
0.6 Confidence
0.8
1.0
0.0
0.0
1.0 0.2 0.4
0.6
0.8
Confidence
Confidence
(a) Uncalibrated
(b) Uncalibrated
(c) Uncalibrated
1.0
1.0
1.0
mean IoU
mean IoU
mean IoU
Samples
0.8
Samples
0.8
% of Samples
LaECE- 6.52%
6.38%
LaECE- 4.32%
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.0
0.2
0.6 Confidence
0.4
0.8
1.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Confidence
Confidence
(d) Tsotonic Regressioo
(e) Tsotonic Regression
(f) Tsotonic Regression
Figure A.11: Reliability diagrams of the detectors presented on Tab. 5 (a) Uncalibrated Co-DETR, (b)
Uncalibrated BRS-DETR, (c) Uncalibrated DINO, (d) Co-DETR calibrated by Isotonic Regression, (e)
BRS-DETR calibrated by Isotonic Regression, and (f) DINO calibrated by Isotonic Regression.
1.0
1.0
1.0
loU
mean
IoU
0.8
% of Samples
0,8
% of Samples
0.8
% of Samples
LaECE- 9 9.23%
LaECE- 4.54%
LaECE- 10.05%
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.0
0.0
0.6
0.8
0.0
0.0
0.2
0.4
0.8
1.0
0.0.
0.4 0.6 1.0
Confidence
Confidence
Confidence
Uncalibrated
(b) Uncalibrated
(c) Uncalibrated
1.0
1.0
1.0
mcan
mean IoU
mean IoU % of Semples
0.8
% of Samples
0.8
% of Semples
0.8
4.60%
LaFCE- 3.1 5.19%
LwFCE- 6.26%
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0 0.4 0.6 Confidence
0.0
0.0
0.2
0.4
1.0
0.0
0.2
0.6
0.8 1.0
Confidence
Confidence
(d) Isotonic Regression
(e) Isotonic Regression (f) Isotonic Regression
Figure 12 Reliability diagrams of the detectors presented on Tab. 7 (a) Uncalibrated YOLOv7, (b)
Uncalibrated QueryInst, (c) Uncalibrated DyHead, (d) YOLOv7 calibrated by Isotonic Regression, (e)
QueryInst calibrated by Isotonic Regression, and (f) DyHead calibrated by Isotonic Regression.
42

---

Published in Transactions on Machine Learning Research (10/2024)
meen loV
mean IoU
mean IeU
mean lou
0.3
Samples
0.8
Samples
0.8
Samples
0.8
LabiE- 5.1%
LaECE- 5.02%
LabCE 13.40%
LaECE- 10. 10.35%
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.2
0.4
0.6
1.0
0.0
0.6 0.8
1.0
0.0
0.2 Confidence
0.4
0.8
1.0
0.0
0.2
0.4
0.6
1.0
Confidence
Confidence
Confidence
(a) Uncalibrated
(b) Linear Regression
(c) Uncalibrated
(d) Linear Regression
Figure A.13: Reliability diagrams of the detectors from Tab.
(a) Uncalibrated Co-DETR, (b) Co-DETR
calibrated by Linear Regression, (c) Uncalibratod EVA, and
(d) EVA calibrated by Linear Regression.
1.0 Samples
1.0
1.0
1.0
moon IoU
mean loU
mean
mean lol)
0.8
0.8
Samples
0.8
Samples
0.8
LaECE= 15.59%
LaECE= 8.58%
LaBCE 14.18N
LaECE- 9.60%
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.8.0
0.2 Confidence
0.4
0.6
0.8
1.0
0.0.0
0.2
0.4
0.6
0.8
1.0
Confidence
Confidence
Confidence
(a) Uncalibrated
(b) Linear Regression
(c) Uncalibrated
(d) Linear Regression
Figure A.14: Reliability diagrams the detectors from Tab.
12 (a) Uncalibrated Mask2Former, (b)
FasmeFormer calibrated by Lincar Regression, (c) Uncalibrated ViTDet, and (d) ViTDet calibrated by Linear Regression.
43


Reviews:
[{"review_id": "VClISATxQ5", "reviewer": "Reviewer_1nWj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths include - \n1. The work proposes simple method of calibrating individual detection experts confidence score for improved results in mixture of experts.\n\n2. The work has extensive results with different datasets and models and usecases as well. For eg the work also shows result with open vocabulary detection and instance segmentation.\n\n3. The paper is nicely written with good illustrations on challenges faced by naive mixture of experts and how the work tackles it (fig 1 and 2).\n\nThe weaknesses include - \n1. Choice of detectors - The main result table (table 1) shows result with detectors which are old/ outdated (RS R-CNN- 2021, ATSS - 2020, PAA - 2020). While it shows promising results with these detector with the MoCAE, it would be good to have result on the main table with new detectors. The result with new detectors like co-DETR and EVA are presented quite late in table 6, also has less of improvement with proposed mixture of calibrated experts.\n\n2. Discussion not self contained: The work discusses about Localisation-aware Expected Calibration error (LAECE) in equation 1, which is not self contained. For example what does the bins do, what is IoU^c (j) etc. The work points out to supplement for details, however for the discussion is complete. The texts should be self contained. Similarly score voting , eq 3, does not have proper discussion as what is happening and how the final bounding box after NMS is obtained. The work says refer to App. D, but again the discussion is not complete and self contained. \n\n3. Instance segmentation results with quite old methods: Although the experiments work with mixture of experts of Mask R-CNN (2017) and its variant (2020, 2021), the results with SoTA works such as Mask2Former would boost the contribution claim.\n\n4. Learning calibration via linear regression during training:  The post processing for Mixture of calibrated experts is applied on part of the val splits, and does not require training. However one could have the same objective in the training, using training data, while learning individual experts. Sure learning on part of val split with less data is fast and efficient, however one can also better learn the calibration regression model on train data (maybe with bigger models). Any reason why this is not preferred ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work propose simple approach to combine multiple detection experts for improved detection performance. Naively combining different experts results in inferior performance due to difference in confidences of predictions of different experts, as some of the expert can dominate the results with high confident predictions. The work proposes calibration of the confidence followed by refining non maximal suppression for better selection of boundary boxes. Overall it improves over different baselines achieving state of the art results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Writing changes - The work should improve discussions of LAECE and score voting as presented in weakness 2. The current version is not self contained and is difficult to understand from the main paper what is happening. The reader will have to jump to supplement to understand it better. In general the main paper should be self contained. \n\nClarification on using old detectors as the main result as in limitation 1 and 3 : Overall I liked the work, however the limitation that it performs best with old detectors makes it less attractive. Further results using Mask2Former in instance segmentation result would make the claim stronger (limitation 3). Further clarification on why the learning of calibration cannot happen during training will be helpful.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AQFBkan9Jg", "reviewer": "Reviewer_iRbf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n**Robust Experimental Results**: The paper presents extensive testing across various tasks (e.g., COCO, DOTA), consistently showing improved performance and achieving state-of-the-art results.\n\n**Clear and Readable Presentation**: The paper is well-organized and easy to follow, with clear explanations, figures, and tables that effectively convey the proposed approach and its results.\n\n## Weakness\n\nHere are two weaknesses of the paper:\n\n**Limited Novelty**: The novelty of the proposed calibration technique is somewhat trivial, as similar calibration methods have been extensively studied in the context of classification. The approach might not offer significant new insights beyond existing work.\n\n**Limited Experimental Scope**: The experiments involve only three distinct models with different calibration characteristics. The paper would benefit from additional experiments with varying combinations of models, such as all low-confidence models or all high-confidence models, to better evaluate the effectiveness of MoCaE compared to Deep Ensembles (DE).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel approach to enhancing object detection performance by addressing the issue of miscalibration in the combination of multiple expert detectors. The authors propose the Mixture of Calibrated Experts (MoCaE) framework, which first calibrates the predictions of individual detectors and then refines them using a new strategy called Refining Non-Maximum Suppression (NMS). This method effectively balances the contributions of each expert, leading to significant improvements in detection accuracy. The paper's extensive experiments across multiple detection tasks demonstrate MoCaE's effectiveness, achieving state-of-the-art results on benchmarks such as COCO and DOTA. The work is supported by a theoretical analysis that explains why calibration is crucial for optimal performance in mixtures of experts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. \"predicted confidences do not match the accuracy.\" should be changed with in a more precise to illustrate \"calibration\". \n2. add experiments for different experts as in weakness.", "broader_impact_concerns": "There are no significant ethical concerns directly arising from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3CjhJ6kctO", "reviewer": "Reviewer_yiuu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The motivation is clear. The observation that confidence discrepancy leads to degraded performance of MoE is interesting. \n\n2. The qualitative and quantitative results demonstrate advantages over prior baselines.\n\n3. The paper is easy to read and understand.\n\n\nWeaknesses:\n\n1. Can you show more examples of various detectors (Faster-RCNN [1], Cascade-RCNN [2], GFL [3], Deformable-DETR [4], DINO [5], and Co-DETR [6]) just like Fig 3? Besides, can we just simply calibrate each expert by score normalization?\n\n2. The post-hoc calibrator described in Section 3.2.1 seems an additional branch that predicts the IoU score for each object. Please compare your method with the IoU branch design (only the IoU branch is trainable and it receives the object features as inputs).\n\n3. What's the performance upper bound of MoCaE (using GT IoU scores to calibrate experts)? \n\n4. The novelty of Refining NMS is limited as it seems a combination of Soft-NMS and Score Voting.\n\n5. This method employs a held-out validation set for recalibration and this can lead to data leakage. Therefore, the comparisons on the COCO minitest are not fair. The performance of models that utilize the COCO train for calibration should be presented.\n\n[1] Faster r-cnn. ICCV 2015.\n\n[2] Cascade r-cnn: Delving into high quality object detection. CVPR 2018.\n\n[3] Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. NeurIPS 2020.\n\n[4] Deformable detr: Deformable transformers for end-to-end object detection. ICLR 2020.\n\n[5] Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ICLR 2023.\n\n[6] Detrs with collaborative hybrid assignments training. ICCV 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This paper reveals that the simple model ensemble of different detectors may hurt the final performance due to the vastly different confidences of these detectors. The diversity in training regimes for these detectors leads to such prediction discrepancy.\n\n2. To address this issue, this paper proposes a MoE method called MoCaE. MoCaE first calibrates the results of each expert and fuse these calibrated predictions with an elaborate Refining NMS.\n\n3. Extensive experimental results are conducted to demonstrate the effectiveness of the proposed framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the Weaknesses listed above.", "broader_impact_concerns": "There are no ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, the reviewers appreciated the clear motivation (yiuu), the simplicity of the approach (1nWj), the experimental results (iRbf, yiuu, 1nWj) and the presentation (iRbf, yiuu, 1nWj). However, they also raised concerns about the limited novelty (iRbf, yiuu) especially w.r.t. other works considering calibration for object detection (Kuzucu et al. 2024), the variety of experts tested (iRbf, yiuu, 1nWj), additional baselines (yiuu, 1nWj), clarifying parts of the discussion (1nWj), and potential data leakage on one benchmark (yiuu). 

The authors' response clarified most of the concerns, especially those regarding including more experts/additional baselines, the discussion parts, and the potential leakage. 2/3 reviewers recommended acceptance, but 2/3 still raised concerns about the limited technical contribution as calibration has been already discussed in this field (Kuzucu et al. 2024) even if not in the context of the mixture of experts.

The AE went over reviews, answers, and updated manuscript and found the article to meet the criteria for acceptance to TMLR as 1) it clearly supports the statements and 2) despite the limited technical contribution, it is of interest to anyone who wants to develop mixture-of-experts models for object detection. Therefore, the AE recommends the acceptance of the work. 

As a (very) minor change in the final version, it would be helpful to point out within the introduction how this work contributes to the discussion and studies of the calibration of object detectors. This discussion is mostly included at the end of Appendix A, but readers may still find (even only part of) it interesting to better contextualize the contribution of the work.
Decision: Accept (poster)