Looking Beyond the Known: Towards a Data Discovery Guided Open-World Object Detection
Authors: Anay Majee
Amitesh Gangrade
Rishabh K Iyer
Venue: NeurIPS 2025 poster ()
Keywords: Open-World Object Detection
Data-Discovery
Representation Learning
Submodular Functions
Primary area: applications
TL;DR: Combinatorial Open-World Detection

Abstract: Open-World Object Detection (OWOD) enriches traditional object detectors by enabling continual discovery and integration of unknown objects via human guidance. However, existing OWOD approaches frequently suffer from semantic confusion between known and unknown classes, alongside catastrophic forgetting, leading to diminished unknown recall and degraded known-class accuracy. 
To overcome these challenges, we propose **C**ombinato**r**ial **O**pen-**W**orld **D**etection (**CROWD**), a unified framework reformulating unknown object discovery and adaptation as an interwoven combinatorial (set-based) data-discovery (CROWD-Discover) and representation learning (CROWD-Learn) task. CROWD-Discover strategically mines unknown instances by maximizing Submodular Conditional Gain (SCG) functions, selecting representative examples distinctly dissimilar from known objects. Subsequently, CROWD-Learn employs novel combinatorial objectives that jointly disentangle known and unknown representations while maintaining discriminative coherence among known classes, thus mitigating confusion and forgetting. Extensive evaluations on OWOD benchmarks illustrate that CROWD achieves improvements of 2.83% and 2.05% in known-class accuracy on M-OWODB and S-OWODB, respectively, and nearly 2.4$\times$ unknown recall compared to leading baselines.

Looking Beyond the Known: Towards a Data
Discovery Guided Open-World Object Detection
Anay Majee
Amitesh Gangrade"
Rishabh Iyer
The University of Texas at Dallas f firstname lastname@utdalla. edu
Abstract
Open- World Object Detection (OWOD) enriches traditional object detectors by abling continual discovery and integration of unknown objects via human guidance. However, existing OWOD approaches frequently suffer from semantic confusion between known and unknown classes, alongside catastrophic forgetting, leading diminished unknown recall and degraded known-class accuracy. To overcome these challenges, we propose Combinatorial Open-World World Detection (CROWD unified framework reformulating unknown object discovery and adaptation as an interwoven combinatorial (set-based) data-discovery (CROWD-Discover) and rep- resentation learning (CROWD-Learn) task. CROWD-Discover strategically mines unknown instances by maximizing Submodular Conditional Gain (SCG) functions, selecting representative examples distinctly dissimilar from known objects. Sub- sequently, CROWD-Lean employs novel combinatorial objectives that jointly dis- entangle known and unknown representations while maintaining discriminative co- herence among known classes, thus mitigating confusion and forgetting. Extensive evaluations on OWOD benchmarks illustrate that CROWD achieves improvements of 2.83% and 2.05% in known-class accuracy M-OWODB and S-OWODB, respectively, and nearly 2.4x unknown recall compared to leading baselines.
Qurennty objects
minedi objects
Collection
Known
K
AC Besocted,
Unkmowns
Targeted
Object
Solection
Desector
Both known and unknoww! Bojects deffectid
Detector
(a) CROWD- Discover (CROWD-D) mines unknown exemplars from confidence background objects.
(b) CROW CROW D--Lcarn (CROWD-LL) learss distinet embeddings known and unknown objects
Figure 1: Overall Architecture of CROWD showing our novel combinatorial data-discovery guided representation learning approach to (a) identify unknown objects² and (b) learn distinguishable representations of both known and unknown objects.
Introduction
Object Detection (OD) is central to numerous vision applications 35 47 30].
but shown in Figure T
(left) conventional OD models operate under closed-wond world assumption, where the object vocabulary remains fixed throughout deployment. This limitation hinders the model's ability generalize to
novel object categories. Open- World Objecl Delection (OWOD), introduced by Joseph et al. (21). addresses this by combining open-set recognition 63
74 65
with incremental learning [37 [2]
enabling models to detect unknown objects and subsequently recognize them with minimal supervision, thus supporting continual self-improvementt However, recent efforts (64 77,
reveal
Work done as graduate student at UTDallas.
Projject Page at htttp:://naamajjeee httpe//eeeeaaaaeeeeeoeteepeee//e
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

two enduring challenges: (1) confusion between known and unknown objects [2 44], and (2)
catastrophic forgetting of previously learned classes [6] 64 45]. Confusion arises due to visual similarity between unknown and known classes (e.g., truck vs. car headlamps), while forgetting stems from the lack of supervision for unknowns, causing them to be misclassified as background. Existing methods struggle to address both issues, evidenced by low unknown recall and high Wilderness Impact scores [21] (Table 3) These limitations motivate the need for a framework that can effectively discover unknown Region-of- Interests (RoIs) and learn representations that remain -nnteress--ss
distinct from known categories, thereby mitigating confusion and preserving prior knowledge.
We cast Open-Worid Object Detection (OWOD) as a set-based discovery and learning problem. For each task t, we view the known object classes as a collection of sets K", group all candidate unknowns into single pseudo-labeled set U [2I]. and treat everything else as background B This formulation (Section 3.T) facilitates the incorporation of submodular functions into OWOD and gives rise to our Combinatorial Open-World Detection (CROWD) framework. As illustrated in Figure | CROWD tackles OWOD as an interleaved process of data discovery (CROWD-D) and representation learning (CROWD-L), directly targeting the dual challenges of confusion and forgetting.
Starting from a OD trained only on known instances Kt (Figure CROWD-Discover (CROWD-D) identifies representative unknowns, formulated as combinatorial targeted selection problem. Specif- ically, CROWD-D maximizes the Submodular Conditional Gain (SCG, Section 3.2) between K* and candidate subsets, encouraging dissimilarity with known and background objects. Authors in [34] provide theoretical guarantees that greedy maximization
52 of SCG results in selection of samples in U which are dissimilar to K" as well as B¹, constituting informative pseudo-labeled unknowns.
CROWD-Lear (CROWD-L)) then fine-tunes the OD model both known and mined unknowns as shown in Figure via novel combinatorial joint objective (Equation (T). rooted in two families of submodular functions: SCG (34] and Total Submodular Information (II (SIM). Maximizing SCG increases diversity between known and unknown objects, reducing feature overlap and confusion, as corroborated by Table Conversely, minimizing SIM encourages intra-class cohesion within each known object, preserving discriminative features and mitigating forgetting. This formulation closely follows the observation in
48, 449 that submodular functions model cooperation (20) and diversity [38] when minimized and maximized respectively. Finally, we instantiate a family of submodular-basee based loss functions within CROWD-L. that jointly reduce confusion and forgetting, achieving notable gains in unknown recall and known-class accuracy (Table (5.. We validate our
approach on two standard OWOD benchmarks, M-OWOD 21 and S-OWOD [15), demonstrating its effectiveness across diverse open-world settings. Our main contributions are-
CROWD introduces a novel combinatorial viewpoint in OWOD by modeling the identification of unknown instances of given task as a data discovery problem (CROWD-D), selecting unknown Rols which maximize the SCG between and the known object instances.
CROWD also introduces a novel set-based learning paradigm CROWD-L, based on SCG functions which minimizes the cluster overlap between embeddings of known and unknown objects while retaining the discriminative feature information from the known ones.
Finally, CROWD demonstrates ~2.4 increase in unknown recall per task alongside up to 2.8% improvement on M-OWODB and 2. improvement on S-OWODB in known class performance (measured as mAP) over several existing OWOD baselines.
2 Related Work
Open- Open-World Object Detection (OWOD) first introduced in Joseph et al. 21 augmented a Faster R-CNN (58) model with contrastive clustering and an Energy-Based Unknown Classifier relying on objectness threshold based pseudo labeling strategy. Subsequent work such OW-DETR [15] adapted deformable DETR [76] and proposed an attention-
pscudo-labeling scheme that identifies high-activation regions as unknowns without requiring extra supervision. Further, CAT improves transformer- baased models by decoupling localization and classification, while introducing dual pseudo-labeling strategies, namely- model-driven and input-driven to robustly mine unknowns. PROB 777 advanced the state of the art by modeling objectness probabilistically in the embedding
$ Figure (a) shows subset of background and unknown Rols for clarity. The total number of Rols in the original experiment is set to 512 (as in (61) while the number of mined unknowns is set 10 (per image).

---

space using Gaussian likelihood, allowing better separation of unknowns from background without explicit negative examples. Other notable works include 2B-OCD 68), which integrates a localization- based objectness head 28 OCPI.
separation to reduce known-unknown confusion, and UC-OWOD 69 employs feature-space regularization to suppress background misclassification. Some recent methods leverage external supervision, e.g., MViT' [46), or multimodal cues such as text for class-agnostic detection, these often fall outside strict OWOD assumptions but highlight promising directions for future research. Complementary to these, recent approaches such as RandBox [64] sidesteps detection bias via random bounding box sampling and dynamic-k filtering, while OrthogonalDet [61] enforces angular decorrelation in object features to disentangle objectness and class semantics. Interestingly, Randbox and OrthogonalDet outperforms larger models like OW-DETR, UC-OWOD etc. while using simpler Fester-RCNN [58] based architecture. Despite substantial progress, OWOD methods continue to grapple with confusion between known and unknown objects and catastrophic forgetting during incremental adaptation recently evidenced in Xi et al. 770, motivating the development of our CROWD framework.1 In general our work is also related standard object detection while CROWD-Discover (Section 3.3.1)
73
enforces
is related to combinatorial subset selection, the related work for which is provided in Appendix A.2
3 Method
3.1 Problem Definition: OWOD
We largely adopt the problem formulation of OWOD from Joseph et al. [21] with modifications towards a combinatorial (set-based) formulation. Given an incoming task T, where / E [1, n]. an object detector h" (.;0) recognizes a set of known classes K {K1, _g...., Kzx J. |K* = C* while also accounting for unknown classes U that may appear during inference (classes in U are not labeled during training). Here, K? i E [1, C*1 indicates examples for each known class in Tr. The dataset D' = {( ) for each task T,, where each label yy contains K object instances (K can very for each image) defined by bounding box parameters y = [Cky 2k, yk Wk, h&]. with Ck E [1, C*] representing the class label. The object detection model h² (:;0) is trained learn newly introduced instances from labeled examples in K while identifying unknown objects U by assigning them placeholder label (0). Examples in U can be reviewed by human expert who identifies C" new classes, allowing the model to update incrementally and produce h (,;'(.;)) without retraining on the entire dataset. If Tix indicate the newly labeled set of unknown classes s.t. Kt U Ú+, enabling continual adaptation to new object categories over time.
| C", then K++1=
3.2 Preliminaries: Submodularity
Adopting a set-based formulation allows us to explore combinatorial functions for OWOD. In particular explore Submodular functions which are set functions exhibiting a unique diminishing returns property. Formally, a function f :22 + I defined on a ground set V is submodular
for any subset A,, A, C V, it holds that f(A)) + f(A,)
f(A, U A₁) + f(A, n A₂) [T]. These
functions have been widely studied for applications such as data subset selection (27 34 19 active learning [67 33
2 (25], and video summarization [24] 26). Typically, these tasks involve formulating subset selection or summarization as submodular maximization subject to a knapsack constraint,
- A classic result guarantees )1- approximation factor (53 using a greedy algorithm, which can be implemented more efficiently via improved greedy strategies (33 (2).. Within this framework, Submodular Information Functions (SIMs) 18 17.3, 3) such as Encility-Location or Graph-Cut. promote diversity when maximizing f(A). On the other hand, Submodular Conditional Gain (SCG) 177, y A,), captures elements in A, most dissimilar to Aj. Extrapolating this, Kothawade et al.
32
defines discovery of unseen, rare examples as targeted selection problem.
Further works 48
49
have demonstrated the utility of these combinatorial functions in continuous
optimization. Majee et al. 48) introduces losses inspired by SIMS to enforce autra-group compactness (when minimized) and inter-group separation (when maxinized), while 49 uses SMI-based losses to account for interactions between abundant and rare samples in few-shot learning. Motivated by these insights, CROWD proposes novel duta-discovery guided representation learning framework to dynamically identify and incrementally adapt to unknown objects.
3

---

From Previous Task (?t-1)
K
Seloungi
kc
K+111
k
K
g
Ret lesturna Ro features
Lenowa
Task Task (
Next
('()0)
th
(.;0)
(ax(0) (0))
GROWD detects known unknown objects detected
Targeted Selection
Rai
CROWD
(b) CROWD-Lear
Figure 2: Interleaved Data- Disco-Discvrryy and Representation Learning in CROWD on an incoming task Te- CROWD takes as input the model weights from Ti-1 and small replay buffer of previously known classes Kt-1, applies (a) CROWD-Leam to discover unknown Rols (b) CROWD-L lear discriminative features of both known and unknown instances return updated model h++1 and the current task replay buffer R
3.3 The CROWD Framework
The problem formulation in Section 3.1 surfaces two unique challenges in the domain of OWOD (1) How to identify instances of unlabeled unknown objects U given labeled examples of
only known ones in K (2) How to effectively learn representations of currently known ob- jects without forgetting the previously known (classes introduced in T,, where < l) ones ?
To this end, we introduce Combinatorial
Open-Worg Algorithm Discovering Unknown Rols in CROWD-D Detection (CROWD) framework. which models the OWOD task as interleaved set-based data discovery 132 and representation learning 48 problem. CROWD achieves this in two stages paggs-nameen namely CROWD-Discover (a.k.a. CROWD-
Require: A task t, set of Rol feature vectors R E R Rxxx Objectness scores 3(.) E R.W, Task specific Labels yK1, budget
f** Identify and Exclude outliers **y
2: R < {r R(S(r) 2 Te}
K HHUGGARLAM-AAACHHINGRR. yK:) Known class
Rol features
D) and CROWD-Learn (a.k.. CROWD-L) as shown in Figure 2 Given an incoming task T; we first train ht h.:00) on currently known classes in D At this point, CROWD- D utilizes the frozen model weights of ht' and uses small replay buffer (typically containing examples from both previously known and currently introduced objects) to discover highly representative proposals of unknown classes U+_ We clucidate this in Section 3.3.I Subsequently, CROWD-L introduces a novel combinatorial learning strategy rapidly finetune ht on this replay buffer (we adopt the predefined buffer in Joseph et al. [1] to distinguish between known classes K" and unknown U while preserving distinguishable features from the
V P Kt
5: *** Select Background Samples **/
B arg max ( Kt) B'CV |
Large feature
separation from K
/ Select Unknown samples from K #a/
8: U + arg max (U, K BBB Unknowns are U'CV,IU'SK
different from KPUR U
9: return U
previously known classes. We discuss this in detail in Section 3.3.2
3.3.1 CROWD-Discover
During training of (-; (), label information is available only for the currently known classes KT with labels of the previously known バT-1 and the unknown classes U CROWD-D tackles the challenge of identifying potentially unknown instances from Region-of- -n-teress- (Rol) proposals produced by the employ pseudo labeling
(RPN) in (.) Unlike existing OWOD methods (21), feature orthogonalization [61] etc. rely on the objectness score (probability an Rol proposal to contain foreground object), whereas CROWD-D achieves this by modeling this task combinatorial data discovery problem [32]. Given set of Rol proposals

---

Labeled K'
Background (DD) Rols are farthest from K' the feature space.
J R A
U
(U instances are dissimilar as well as
(a) Rol Feature Space
(b) Background 13*
(c)
Unknowns U
Figure 3: Illustration of the data-discovery pipeline in CROWD-D on a synthetic dataset with |R] = 500 and budget k 10 and the underlying submodular function as Graph-Cut. CROWD-D selects U which are both dissimilar background B' and known K* instances.
and a submodular function we define the data discovery task (Algorithm 7 as targeted selection problem which selects set of unknown instances U from V = R. K" that maximizes the SCG H given a query set comprising of known Kt and the background B' instances (line 8 Algorithm 7)
Here, k denotes a budget for the number of unknown samples mined per image. From the definition of SCG in Section 3.2 selected examples in U are largely dissimilar to examples in K U B indicating that they are neither background objects nor visually similar to known objects as shown in Figure cc.. Interestingly. the number of known Rols K* in are significantly fewer than the background Rols B (typically [R| = 500 (total number of Rols from the RPN) whereas | ~ 10 in MS-COCO 40 which leaves B3| 490) most OD models. To minimize the computation costs while selecting U from this large Rol pool we exclude all Rols with low objectness scores < Te (line Algorithm|| [ and likely background objects B' predicted with high confidence (line 6 of Algorithm Instances in B' are selected which maximize the SCG between themselves and known Rols K under a budget constraint of 76 7,% (samples that are significantly different from K¹). Known instances are identified by following the hungarian matching technique applied Wang et al. (64) as shown in line 3 of Algorithm T The exclusion thresholds Tn and тà are empirically determined to be 0.2 and 30% respectively and the underlying submodular function in our experiments is chosen to be Graph-Cut which has been evidenced Kothawade et al. (34) to model both representation and diversity among selected examples.
3.3.2 CROWD-Learn
Including unknown examples U can potentially inject noisy labels into the training data with detrimental effects. We show in Table 5 that CROWD-D alone does not handle the knowledge retention from previously known classes K f-1 despite significant improvements on unknown class recall, causing forgetting. CROWD-L overcomes the aforementioned challenges by introducing novel Combinatorial representation learning strategy inspired from recent developments [48, 49), that ensures orthogonality (separation) between embeddings in K and U while minimizing the effect of forgetting of kt-1, Here, k denotes replay buffer of previously known classes.
LCROWD (0) Σ O(K::))
(CROWD Σ H2 = fFU")
-
LCROWD (0) = self (0) 11 LROODD (0)
Given set of known Kt U R unknown U classes alongside a submodular function f we define a learning objective LCROWD (0) as shown in Equation (1 which jointly minimizes the Submodular
Total Information (L sel (Loo) over each known class K E [K* U KK-- and the SCG (Hl y as defined in Section 3.2 between known class K²₃ and the unknown set U* cross ). Note that CROWD-L is applied during training of task Te as finetuning step as shown in Figure ((b).
Note that / relies on the pairwise interaction between examples in a batch which we represent using A (xx..) .0) (2..))))
cosine similarity (0) [Ta" [1" and can be different from the one used in CROWD- decided through ablations Section 4.2 Our loss formulation in T-RRWD follows the observation in [88] which entails that submodular functions model cooperation [20] and diversity [38] when
5

---

Table I: Summary of various instantiations of CROWD-L by varying the submodular function in 'CROWD and J.
'CROWD- Here, 'T denotes batch with instances from K2 U U?.
Objective Name
Insfances of Lenin
Instances of L
CROWD-GC
2i Diex Zyer 8,(0) ALnnERy (1y(01)
CROWD-FL
IM
Σ max (aa(0)
max (9).
M TRT ZieTr Maxyery sy(()
NET
keN;
CROWD-LLoDer
2 TPT (ot)( (9) sa or (0)6 (010,
2is [
log (011)
Cree 2
Core 3
Norualzzdd Loss Valuo
CROWD-GO CROWWDSLD
6
0-
0
2
vs. Seporation:
Figure 4: Characterization of losses in CROWD-L (L on synthetie two-cluster imbalanced dataset by increasing known vs. unknown class separation (cases through 3) similar to the Rol embedding space of h² (:: The synthetic dataset generation is performed under the same seed. Here, CROWD is maximized as in Equation II..
minimized and maximized respectively. By varying the choice of between popular submodular
functions Facility-Location (FL), Sraph-Cut (GC) and Log-Detern Sog-Determinant (LogDet) we introduce
family of loss functions summarized in Table [ and derivations in Appendix [A.3] LCROWD is applied to the classification head of h': f) model during all training stages described in Sec. Our novel formulation entails some interesting properties-
(1) L CROWD retains informative known class features: Following the insights in Jegelka and Bilmes
20
self which minimizes the total information contained in K encourages intra-class com-
pactness retaining the most discrininative features from the known classes alleviating forgetting [49]
(2) L LCROWD models known vs. unknown separation As shown in Equation ( L LEROND minimizes the SCG (H between the embeddings of the known and unknown classes. As observed in Kothawade et al. (34) maximizing SCG models dissimilarity between two sets. In OWOD, LEROWD promotes large inter-class boundary between K and U*, minimizing their cluster overlap resulting in reduced confusion. Further, a hyper-parameter parameter controls the trade-off between intra-class compactness modeled by f and the inter-class separation in LCROWD-
(3) Sensitivity to unknown classes with varying f Table highlights the instances of LCROWD by varying f. Such variations injects domain specific properties into CROWD-L critical for the OWOD tasks. As depicted in Figure 4 for -CROWD- under varying known unknown class separation, CROWD-FL explicitly models representation [4]] by adopting the FL based submodular function while CROWD-Logg CROWD-GC models both representation and diversity but not resilient to imbalance between K and U as CROWD-FL 48 Thus, CROWD-FL emerges to be suitable choice for OWOD modeling both representation and resilience to imbalance.
critical
injects diversity by modeling cluster volume through Log-Determinant (18)
(4) Generalization to Incremental Object Detection (IOD) As observed in several related works 15] 61 (77, learning from unknown pseudo labels OWOD benefits Incremental Object Detection (IOD) tasks. However, its important to note that CROWD relies mined unknowns (from CROWD-D) while no unknown objects are provided in the IOD setting. This requires us to slightly modify our learning objective LCROWD model dissimilarity between currently known K" and the previously known objects πT- (a replay buffer as Joseph et al. (1] instead of unknowns, s.t.
Leroes (0) = - H

---

Table 2: Open-world object detection results across incremental tasks. U-Recall and mAP (%)
are reported for various baselimes on M-OWOD and S-OWOD benchmarks. Best results are in bold.
Method
U-Recall
mAP
U-Recall
Prew.
Curr:
IAP
Both
U-Recall
Prev.
MAP Curr.
Both
Prev.
mAP
Curr.
Both
Benchmark
ORE
81
4.9
56.0
2.9
52.7
26.0
19.4
3.9
38.2
12.7
29.7
29.6
12.4
25.1
39.9
OW-DETR 15
7.5
59.2
6.2
53.6
33.5
42.9
5.7
38.3
15.8
30.8
31.4
17.1
27.8
UC-OWOD
33.1
30.5
31.8
28.8
16.3
24.6
25.6
23.2
A.J.WW 45
13.6
39.3
10.0
53.2
34.0
45.6
14.3
42.6
26.7
38.0
33.5
21.8
30.6
PROB 77
19.4
59.5
17.4
55.7
32.2
44,0
19.6
43.0
22.2
36.0
35.7
18.9
31.5
23.7
19.1
55.5
44.1
24.4
18.7
29.9
RandBox 64
10.6
61.8
6.3
45.1
7.8
39.4
35.4
OrthogonalDet
30.6
CROWD (ours)
57,9
61.7
$3.6
56.7
38.9
47.8
69.6
48.0
31.4
42.5
42,9
25.4
38.5
S-OWOD Benchmark Results
ORE 21
1.5
61.4
3.9
56.5
26.1
40.5
3.6
38.7
23.7
33.7
33.6
26.3
31.8
OW-DETR
5.7
24.9
PROB 21
17.6
73.4
22.3
66.3
36.0
50.4
24.8
47.8
30.4
42.0
42.6
31.7
39.9
CAT ++
24.0
74.2
23.0
67.6
35.5
50,7
24.6
51.2
32.6
45.0
45
35.1
42.8
OrthogonalDet 61
24.6
71.6
27.9
64.0
39.9
51.3
31,9
52.1
42.2
48.8
48.7
38.8
46.2
CROWD (ours)
50.5
73.5
41.7
64.9
41.2
53.1
49.6
54.7
42.1
48.4
49.8
43.0
46.4
Table 3: Unknown Class Metries on M-OWODB. Comparison of U-Recall, WI, and A-OSE across lasks (excluding Task 4 where all classes are known U Φ). Best results are in bold.
Method
Tusk 1
Task
Task WI(.)
U-Recall (t)
WI
A-OSE(
U-Recall (?)
WI
A-OSE
U-Recall
A-OSE (i)
ORE 21
4,9
0.0621
10459
2.9
0.0282
10445
3.9
0.0211
7990
OST
0.0417
4889
0.0213
2546
0.0146
2120
OW-DETR 15
7,5
0.0571
10240
6.2
0.0278
8441
5.7
0.0156
6803
PROB 77
19.4
0.0569
5195
17.4
0.0344
6452
19.6
0.0151
2641
RandRox
4498
1880
OrthogonalDet
61
24.6
0.0299
4148
26.3
0.0099
1791
29.1
0.0077
1345
CROWD (Ours)
57.6
0/0380
3823
0.0101
1508
0.0066
1266
4 Experiments
Datasets: We evaluate our approach on two well established benchmarks M-OWOD (1 and (5) M-OWOD, (Superclass-Mixee OWOD Benchmark) consists of images from both MS-COCO 40 and PASCAL-VOC [IO] depicting 80 classes grouped into 4 tasks (20 classes per task). On the other hand, S-OWOD (Superclass-Spparute OWOD Benchmark) consists of images from only MS-COCO dataset. Both benchmarks split the underlying data points into four distinct (non-overlapping) tasks Tú, where / E 41. During training on a task T, the model provided labeled examples from T, alone while at inference the model expected to identify objects in tasks leading up to Tr, s.t & E t]. No prior knowledge of subsequent tasks E [t + 1, ni] (n refers to maximum number of tasks in experiment) contrast to M-OWODB, S-OWODB introduces a distinct separation between super-categoric (eg. animals, vehicles etc.) and distributes these super-categories between tasks (each task will have examples from one or more unique super-categories).
available during training and inference on T, In
Experimental Sctup: Following Sun et al. [61 we adopt Faster-RCNN 58 based model with a pretrained ResNet-50 [6) backbone. Our model is trained incrementally on tasks as described above with batch size of 12, an Adam optimizer, a base learning rate to 2.5 x 10-5 and weight decay 1 x 10 10-4, CROWD-D utilizes the Rol features ([R) 500) to mine k 10 unknown instances (determined through ablation study in Appendix A.4.I 1 per image. The CROWD-L loss is applied across tasks as an additional head and operates on Rol features projected to a 256-dimensional feature space. We train model 4 NVIDIA V100 GPUs, provide additional experimental details in Appendix A.4 and release our code at https:://// : //github httptaaabbccmm/maaaaaaaaa/////oa
git
Metrics We use mean average precision (mAP) to evaluate known classes, partitioned into previously seen and newly introduced categories. For unknown object class, we follow OWOD conventions [2 15 and report unknown object recall (U-Recall). as mAP is inapplicable due to incomplete annotations. To measure confusion between known and unknown classes, we report Wilderness Impact (WI) [7] and Absolute Open-Set Error (A-OSE) [51]
7

---

Table 5: Ablation Experiments the M-OWOD benchmark. We report the U-Recall and mAP (all known classes) by varying the choice of selection strategies in CROWD-D and learning objectives in CROWD-L. We show that a joint (data discovery combinatorial loss) strategy provides the best overall performance (denoted as CROWD (joint)).
Method
Baseline
CROWD
CROWD
Task
Task
Tusk
Task
U-Recall
mAP
U-Recall
mAP
U-Recall
mAp
mAP
Det 61
24.6
61.3gam
26.3gaza
47.0gamm
29.1
41.3
37.9
CROWD-D (w FLOG)
50.7
60.3
52:2
45.7
60.
40.6.00
38.3
CROWD-D (w GCCG)
57.0
61.2200
54.1
45.2
69,6
40.8.00
38.
CROWD-D LogDeCC)
56.4
61.2,440
54.1jars
44.
69.1
39.7,00
37.6
CROWD.J (wr
25.0n
61.7asm
86,8
47.7
28884030
42.4
CROWD-L. (w/ GCCG)
24. 2oois
61.3ga.1
27. 19.1
47.4
31.0a0as
40.2.a.
CROWD-L (w/ LogDerCG)
22.7
g9
27.0ga
44.6gmm
27.2
38.3
36.0
CROWD (joint)
61.7a0m
53.6an
47.820 on
31.4_0m
38.5ga0
4.1 Results on Benchmark OWOD and IOD tasks
OWOD: We compare the performance of CROWD against several existing baselines on M- OWOD and S-OWOD benchmarks as shown in Table 2
Note, that follow Sun et al. 61 and report our results on the same seed and compute settings for fair comparisons. CROWD surpasses the latest baseline OrthogonalDet (61 by up to 2.8% and 2.1% on M-OWOD and S-OWOD benchmarks while achieving up to 2.4x gains in U-recall. For approaches like PROB 777. CAT [44] which adopt selection strategies to mine unknowns our combinatorial approach achieves up to 8.4% (on M-OWOD) improve- ments. This can be attributed to the contributions of CROWD-D which mines representative unknown exam- ples effectively increasing the coverage on such objects. Also, we observe ~ 3% increase in mAP for previously forgetting. The competitive results on the currently known classes (Curr. in Table ( indicates that (,; 8) enforces a stronger decision boundary between Kt and U through CROWD while retaining performance on Kt through L LRROWD Additionally, in Table show that CROWD achieves lesser confusion over existing baselines while boosting U-Recall establishing the importance of modeling OWOD as a combinatorial data-discovery problem. This further highlighted qualitatively in Figure [
Table 4: Results of CROWD on PASCAL VOC for three IOD tasks shown in terms of Prev., Curr., and overall mAP.
10 + 10 setting
Prev.
Curr.
mAP
ILOD 60
63.2
63.2
63.2
Faster ILOD 55
69.8
54.5
62.1
PROB 77
66.0
67.2
66.5
OrthogonalDet 61
69.4
71.8
67.0
known classes indicating a reduction
CROWD (ours)
73.5
75.1
72.0
setting
ILOD
60
68.3
58.4
65,8
Faster ILOD 55
71.6
56.9
67.9
PROB 77
73.2
60.8
70.1
OrthogonalDet 61
74.5
66.9
72.6
CROWD (ours)
76.2
68.9
74.4
setting
IOD: Our novel loss formulation described in Sec- tion 3.3.2 (point 4) is applied to the finetuning stage of
IL.D C1
68.5
62.7
68.2
Faster ILOD 55
68.9
61.1
68.5
PROB ?7
73.9
48.5
72.6
IOD across three popular task splits from the PASCAL- VOC [10] dataset. Note, that for IOD we do not apply CROWD-D due to absence of unknown examples. Our results summarized in Table and detailed in Table (Appendix) shows up to 5.9% boost in overall mAP showing better generalization IOD tasks while minimizing the impact of forgetting via stronger retention of previously known classes, very common pitfall in IOD.
OrthogonalDet 661
73.5
74.5
73.6
CROWD (ours)
74.2
75.3
74.2
4.2 Ablations
We conduct ablations on the M-OWOD benchmark analyze the contributions of individual components of CROWD. On top of the baseline method OrthogonalDet 61] we first introduce instances of CROWD-D to assess the impact of data-discoveey under a fixed budget we decouple CROWD-D and introduce our novel learning objectives in CROWD-L to assess their impact on forgetting and confusion as discussed in Section 3.3.2 For each of the above steps we ablate among instances of ʃ Graph-Cut (GC), Log- Jog-Determinant (LogDet) and Facility-Location (FI.). Finally, we combine the best performing instances from CROWD-D and CROWD-L. into joint formulation (referred to CROWD (joint)) as shown in Table 5 which achieves the best
10. Next,
8

---

OrthogonalDer 6
CROWD (ours)
(b)
(c)
Figure 5: Qualitative results from CROWD contrasted against OrthogonalDet [61 showing that our approach mitigates (a) confusion (b) generalizes to unknowns and (c) reduces forgetting.
Table 6: Ablation Experiments on the exclusion criterion Te and background budget TT in CROWD-D. The submodular function is kept constant Graph-Cut in CROWD-D and CROWD- FL based learning objective is chosen in CROWD-L. under constant machine seed.
Method
Value
Task
Task 2
Task 3
Task 4 mAP
U-Recall
mAP
U-Recall
mAP
U-Recall
mAP
0.05
$7.5
61.7
52.7
47.1
41.5
38.0
CROWD (7.)
0,2
57.9
61.7
53.6
47.8
69.6
42.4
38.5
30%
0.5
53.3
59.5
49.4
45.9
65.0
40.1
37.8
10%
$7.9
61.7
53.1
46.8
69.2
41.3
37.4
CROWD (n.)
30%
57.9
61.7
53.6
47.8
69.6
42.4
38.5
0.2
50%
55.4
60.0
50.0
42.1
63.7
38,9
34.5
overall performance, balancing the tradeoff between boosting currently known class performance and retaining performance on previously learnt ones.
Impact of Dasa-Discovery in CROWD-D As shown in Table irrespective of the choice of f. CROWD-D boosts the U-Recall over the baseline by introducing additional information in the form of pseudo labeled unknowns. We observe that CROWD-D (w/GCCG) (f here is
Graph-Cur provides the best gains in U-Recall up to 2x over the latest baseline OrthogonalDet. This follows the observation in Kothawade et al. [4 which shows that greedy maximization of GCCG models relevance (examples which are dissimilar both K and U²) while others model FFacilitt-Locatione Thus, we adopt GCCG based selection strategy Algorithm | for our experiments in Table P
diversity (CROWD-D w/LogDet) and representation (CROWD-D
Impact of k As stated in Section 3.3.I k controls the number of potential unknown Rols identified by CROWD-D per image. We ablate among several plausible values of [0, 100) and summarize the results in Table I0 of the Appendix. Increasing the number of identified unknowns from (OrthogonalDet) to 10 shows an increase performance of the underlying model (U-Recall) while the performance does not increase beyond 20. The increase in U-Recall can be attributed to inclusion of informative Rols the training loop. In fact, the mAP on known classes slightly drops below existing baselines for k = 100 due to inclusion of spurious background Rols in the training pipeline.
Impact of Combinatorial Objectives in CROWD-L: Similar to CROWD-D ablate variations of f to contrast between formulations summarized in T As shown Table learning formu- lation, particularly CROWD-FL (based on Facility-Location) demonstrates better retention of previ- ously known class performance while achieving competitive results on latest baseline OrthogonalDet. This follows the observation Majec al. (88 which demonstrates that FL based objectives model
representation, retaining the most discriminative features through LEROWD while enforcing sufficient inter-cluster boundary between known and unknown Rol features (L-rrees))) This also re-establishes the properties described in Figure wherein CROWD-FL. shows larger sensitivity inter-cluster separation compared to CROWD-GC, CROWD-LogDel and Ldecorr introduced in OrthogonalDet.

---

Table 7: Ablation Experiments the variation in vj in CROWD-L. Given the Graph- based selection strategy in CROWD-D we vary 7 between [0.5, 1.0, 1.5] and adopt the best performing value for our pipeline in CROWD-L. The selection budget k in CROWD-D set to 10 for all experiments and a fixed seed value.
Task
Tusk
Method
U-Recall
Curr.
mAP
U-Recall
Prev.
mAP Curr.
Both
U-Recall
Prev.
mAP
Curr.
Both
Prev.
Curr.
mAP
Both
OthogomalDet [i]
24.6
61.3
26.3
55.5
38.5
47.0
29.1
46.7
30.6
41.3
42.4
24.3
37.9
57.1
32.8
50.2
38.3
CROWD (ours)
1.0
57.9
61.7
53.6
56.7
38.9
47.8
69.6
48.0
314
42.5
42.9
25.4
38.5
57.9
61.7
33.6
55.6
39.1
47.4
69.5
44.0
34.6
40.9
44.0
21.1
38.3
Ablation on Exclusion Criterion T. and 75 in CROWD-D- At first, Te is exclusion threshold which reduces the search space of CROWD-D by eliminating Rols which have a low confidence threshold. As shown in Table [ increasing тe from 0 to increases performance until Te 0.2 and then reduces. A lower value of Te allows for a large search space but includes lot of noisy background objects leading to reduced selection performance. On the other hand large value of re can potentially earmark unknown foregrounds as unknowns resulting in reduced performance.
Keeping Te fixed at 0.2 ablate T, which controls the selection budget for backgrounds (higher the value more are the number of background Rols identified). Increasing т (percentage here) increases the fraction of Rols treated as backgrounds. This widens the search space for the combinatorial function causing a small drop in performance due to confusions between true backgrounds and foreground unknowns. On the other hand, very large values of 96 shrink the search space oftentimes considering unknown foregrounds as background objects showing a steep drop in performance.
Ablation on Trade-off between LCROWD and LCROWD in CROWD-L The papeer-rrameter controls the trade-off between known-unknown class separation and known class cluster compactness discussed in Table 7 lower value of y) does not enforce separation between currently known and unknown exemplars but enforces intra-class compactness. This results in better retention of previously known objects but a drop in currently known objects due to increased confusion with unknown exemplars. On the other hand for large value of 7) the model enforces large separation between currently known and unknown objects boosting performance on the currently knowns but suffers from catastrophic forgetting of the previously known classes.
5 Conclusion, Limitations and Future Work
We introduced CROWD, a novel combinatorial framework in OWOD, which reformulates OWOD as interleaved set-based discovery (CROWD-D) and representation learning (CROWD-L) tasks. Leveraging Submodular Conditional Gain (SCG) functions, CROWD-D strategically selects representative unknown instances distinctly dissimilar from known objects while CROWD-L consumes mined unknowns to preserve discriminative coherence over known classes. Our evaluations confirm that CROWD effectively addresses known vs. unknown class confusion and forgetting, achieving significant improvements in unknown recall and known-class accuracy on standard OWOD and IOD benchmarks. Despite improvements in U-Recall, operating under fixed budget CROWD-D injects some spurious exemplars into the selected unknown pool (particularly in images with no known unknown objects). We aim to address this in future works by exploring alternative combinatorial formulations beyond SCG, and introducing stricter constraints in CROWD-D.
Acknowledgements
We gratefully thank anonymous reviewers for their valuable comments. We would also like to extend our gratitude to our fellow researchers from the CARAML lab at UT Dallas for their suggestions. This work supported by the National Science Foundation under Grant Numbers IIS-2106937, a gift from Google Research, an Amazon Research Award, and the Adobe Data Science Research award. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation, Google or Adobe.
10

---

References
[1] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh K. Iyer. Effective evaluation of deep active learning on image classification tasks, ArXiv, abs/2106. 15324, 2021.
[3] Jeff Bilmes. Submodularity in machine learning and artificial intelligencee, arXiv preprini 2201:2200..00132 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. Computer Vision ECCV 2020, 2020.
[5] Cody Coleman, Christopher Ych, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Perey Liang, Jure Leskovee, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations, 2020.
[6] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1601-1610, 2021.
[7] Akshay Dhamija. Manuel Gunther, Jonathan Ventura, and Terrance Boult. The overlooked elephant of object detection: Open set. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
[8] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6568 6577, 2019.
[9] Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09841 2018.
[10] Everingham, M. A. Eslami, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111())998-16, January 2015.
[11] Satoru Fujishige. Submodular Fanctions and Optimization, volume 58, Elsevier, 2005.
[12] Takuma Fukuda, Hiroshi Kera, and Kazuhiko Kawamoto. Adapter merging with centroid prototype mapping for scalable class-incremental learning. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pages 4884-4893, 2025.
[13] Ross Girshick. Fast R-CNN. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1440 1448, 2015.
[14] Ross Girshick. Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR '14, page 580-587, 2014.
[15] Akshita Gupta, Sanath Narayan, KI Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. OW-DETR: Open Open-world detection transformer. In CVPR, 2022.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[17] Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning. In Algorithmic Learning Theory, pages 722-754. PMI.R, 2021.
[18] Rishabh Iyer, Ninad Khargonkar, Jeff Bilmes, Himanshu Asnani. Generalized submodular information measures: Thcoretical propertics, examples, optimization algorithms, and applications. IEEE Transactions on Information Theory, 68(2):752-781, 2022.
[19] Eeshaan Jain, Tushar Nandy, Gaurav Aggarwal, Ashish V. Tendulkar, Rishabh Tyer, and Abir De. Efficient data subset selection to generalize training across models: Transductive and inductive networks. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023.
[20] Stefanie Jegelka and Jeff Bilmes. Submodularity beyond submodular energies: Coupling edges in graph In CVPR 2011, 2011.

---

[21] K Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). page 5826-5836, jun 2021.
[22] K. J. Joseph, Jathushan Rajasegaran, Salman Khan, Fahad Shahbaz Khan, and Vineeth N. Balasubramanian. Incremental object detection via meta- letanlear,ing, IEEE Transactions on Pattern Analysis and Machine Intelligence, 4+(12):920999216 2022.
[23] Athresh Karanam, Krishnateja Killamsetty, Harsha Kokel, and Rishabh Iyer. ORTENT: Submodular mutual information measures for data subset selection under distribution shift. Advances in Neural Information Processing Systems, volume 35, 2022.
[24] V. Kaushal, R. Iyer, K. Doctor, A. Sahoo, P. Dubal, S. Kothawade, R. Mahadev, K. Dargan, and G. Ra- makrishnan. Demystifying multi-faceted video summarization: Tradeoff between diversity, representation. coverage and importance. In 2019 JEEE Winter Conference on Applications of Computer Vision (WACV), pages 452 461, 2019.
[25] Vishal Kaushal. Rishabh Iyer, Suraj Kothawade, Rohan Mahadev. Khoshrav Doctor, and Ganesh Ra- makrishnan. I Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 2019.
[26] Vishal Kaushal, Sandeep Subramanian, Suraj Kothawade, Rishabh Iyer, and Ganesh Ramakrishnan. framework towards domain specific video summarization. In 2019 IEEE winter conference on applications of computer vision (WACV), pages 666-675. IEEE, 2019.
[27] Krishnateja Killamsetty. Guttu Sai Abhishek, Aakriti, Ganesh Ramakrishnan, Alexandre V. Evfimievski. Lucian Popa, and Rishabh Iyer. AUTOMATA: gradient based data subset selection for -efpuiienf aaperr-arameter tuning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2024.
[28] Dahun Kim, Tsung- Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open-wored object proposals without learning classify. IEEE Robotics and Automation Letters (RA-L), 2022.
[29] Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is NP-Hard. In Proceedings of the 37th International Conference on Machine Learning, ICML'20, 2020.
[30] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsuhramani, Weihua Hu, Michihiro Yasunaga. Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine. Chelsea Finn, and Percy Liang. Wilds: A benchmark of wi-l distribution shifts. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[31] Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. SIMILAR: Submodular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems, 34, 2021.
[32] Suraj Kothawade, Shivang Chopra, Saikat Ghash, and Rishabh Tyer. Active data discovery: Mining unknown data using submodular information measures, 2022.
[33] Suraj Kothawade, Saikat Ghosh, Sumit Shekhar, Yu Xiang, and Rishabh K. Iyer. Talisman: Targeted active learning for object detection with rare classes and slices using submodular mutual information. In Computer Vision ECCV 2022 17th European Conference, 2022.
[34] Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff A. Bilmes, and Rishabh K. Iyer. PRISM: A rich class of parameterized submodular information measures for guided data subset selection. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI, pages 10238-10246, 2022.
[35] Suraj Kothawade, Athary Savarkar, Venkat Iyer, Ganesh Ramakrishnan, and Rishabh Iyer. Clinical: Targeted active learning for imbalanced medical image classification. In Medical Image Learning with Limited and Noisy Data: First International Workshop. MILLanD 2022, Held in Conjunction with MICCAI, 2022.
[36] Changbin Li. Suraj Kothawade, Feng Chen, and Rishabh Iyer. PLATINUM: Semi-supervised model agnostic meta-learning using submodular mutual information. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12826- 12842. PMLR, 17-23 Jul 2022.
12

---

[37] Linhao Li, Yongzhang Tan, Siyuan Yang. Hao Cheng, Yongfeng Dong, and Liang Yang. Adaptive decision boundary for few-shot class-incremental learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 18359-18367, 2025.
[38] Hui Lin and Jeff Bilmes. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, 2011.
[39] Hui Lin and Jeff Bilmes. Learning mixtures of submodular shells with application document summa- rization. Proceedings of the Twenty-Eight Conference on Uncertainty in Artificial Intelligence, 12, 2012.
[40] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision - ECCV 2014, Cham, 2014. Springer International Publishing.
[41] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2999-3007, 2017.
[42] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV, 2016.
[43] Shuailei Ma, Yuefeng Wang, Jiaqi Fan, Ying yu Wei, Thomas H. Li, Hongli Liu, and Fanbing Lv. Cat Localization and identification cascade detection transformer for open-world object detection. In CVPR, 2023.
[44] Shuailei Ma. Yuefeng Wang, Ying Wei, Jiaqi Fan, Thomas H. Li. Hongli Liu, and Fanbing Lv. Cat Localization and identification cascade detection transformer for open-world object detection. In 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 19681-19690, 2023.
[45] Yuqing Ma, Hainan Li, Zhange Zhang. Jinyang Guo, Shanghang Zhang, Ruihao Gong, and Xianglong Liu. Annealing- label-transfer learning for open world object detection. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11444-11455 11463, 06 2023.
[46] Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang. Class-agnostic object detection with multi-modal transformer. In European Conference on Computer Vision (ECCV). Springer, 2022.
[47] Anay Majee, Kshitij Agrawal, and Anbumani Subramanian. Few-Shot Learning For Road Object Detection. In AAAI Workshop on Meta-ne and MetaDL Challenge, volume 140, pages 115-126. 2021.
[48] Anay Majee, Suraj Nandkishor Kothawade, Krishnateja Killamsetty, and Rishabh K Iyer. SCoRe: Sub- modular combinatorial representation learning. Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 34327-34349, 2024.
[49] Anay Majee, Ryan Sharp, and Rishabh Iyer. Smile: Leveraging submodular mutual information for robust few-shot object detection. In European Conference on Computer Vision (ECCV), 2024.
[50] Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. In Pruceedings of the 2021 Conference on Empirical Methods Natural Language Processing, 2021.
[51] Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhauf. Dropout sampling for robust object detection in open-set conditions. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018
[52] Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrak, and Andreas Krause. Lazier than lazy greedy. Proceedings of the AAAÍ Conference on Artificial Intelligence, 29(1), 2015.
[53] G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematicul Programming, 14(1):265-294, 1978.
[54] Patrik Okanovic, Roger Waleffe, Vasilis Mageirakos, Konstantinos Nikolakakis, Amin Karhasi, Dionysios Kalogerias, Nezihe Merve Gürel, and Theodoros Rekatsinas. Repeated random sampling for minimizing the -nncurryy of learning. In The Twelfth International Conference on Learning Representations. 2024.
[55] Can Peng, Kun Zhao, and Brian C Lovell. Faster ILOD: Incremental learning for object detectors based on faster renn. Pattern Recognition Letters, 2020.
13

---

[56] Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. 2017 IEEE Conference Computer Vision and Patiern Recognition (CVPR), pages 6517-6525, 2016.
[57] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection, 2015.
[58] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[59] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
[60] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In ICCV, 2017.
[61] Zhicheng Sun, Jinghan Li, and Yadong Mu. Exploring orthogonality in open world object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17302- 17312, 2024.
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, volume 30, 2017.
[63]
detection and open-set recognition: A critical analysis of methods and benchmarks. International Journal of Compuder Vision (IJCV), 2024.
Hongjun Wang. Sagar Vaze, and Kai Han. Dissecting
[64] Yanghao Wang, Zhongqi Yue, Xian-Sheer Shnn-Sheg Hua, and Hanwang Zhang. Random boxes are open-world object detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6233 6243, 2023.
[65] Yu Wang, Junxian Mu, Pengfei Zhu, and Qinghua Hu. Exploring diverse representations for open set recognition. AAAI, 2024.
[66] Kai Wei, Yuzong Liu. Katrin Kirchhoff, Chris Bartels, and Jeff Bilmes. Submodular subset selection for large-scale speech training data. In 2014 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2014.
[67] Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity data subset selection and active learning. In ICML, 2015.
[68] Yan Wu, Xiaowei Zhao, Yuqing Ma, Duorui Wang, and Xianglong Liu. Two-branch objectness-centie open world detection. In Proceedings of the 3rd International Workshop on Cemman-CentticcCoo Multimedia Analysis, 2022.
[69] Zhiheng Wu, Yue Lu, Xingyu Chen, Zhengxing Wu, Liwen Kang, and Junzhi Yu. Uc-owod: Unknown- classified open world object detection. In Computer Vision ECCV 2022. page 193 210, 2022.
[70] Xing Xi, Yangyang Huang, Zhijic Zhong, and Ronghua Luo. UMB: Understanding model behavior for open- world object detection. In The Tiirtt-eighth Annual Conference on Neural Information Processing Systems, 2024.
[71] Binbin Yang, Xi Deng, Han Shi, Changlin Li, Gengwei Zhang, Hang Xu, Shen Zhao, Liang Lin, and Xiaodan Liang. Continual object detection via prototypical task correlation guided gating mechanism. 2022 JEER/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9245-9254, 2022.
[72] Shuo Yang, Peize Sun, Yi Jiang, Xiaobo Xia, Ruiheng Zhang, Zehuan Yuan, Changhu Wang, Ping Luo, and Min Xu. Objects in semantic topology. In The Tenth International Conference on Learning Representations, ICLR, 2022.
[73] Jinan Yu, Liyan Ma, Zhenglin Li, Yan Peng, and Shaorong Xie. Open-world object detection via discriminative class prototype lcarning. 2022 IEEE International Conference on Image Processing (ICIP), pages 626-630. IEEE, 2022.
[74] Zhenyu Zhang, Guangyao Chen, Yixiong Zou, Yuhua Li, and Ruixuan Li. Learning unknowns from unknowns: Diversified negative prototypes generator for few-shot open set recognition. In Proceedings of the 32nd ACM International Conference Multimedia, pages 6053 6062, 2024.
14

---

[75] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable [detr]: Deformable transformers for end-to-end object detection. In International Conference Learning Representations, 2021.
[76] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In International Conference Learning Representations, 2021.
[77] Orr Zohar, Kuan-Chich Wang, and Serena Yeung. PROB: Probabilistic objectness for open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11444- 11044-1143,,, June 2023.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We call out our contributions in brief in the abstract and enlist them on page 2 the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations in Section of the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
15

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We define CROWD-L and CROWD-D in Section 3.3. and Section 3
respectively. Instances of CROWD-L (novel to CROWD) are derived based on Theorems A.3.1 through A.3.3 with proofs Sec. A.3 of the Appendix.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We release the anonymized version of our code at httpt::/ wtt.bbbbbtuutbtto// thub com/ mmajeellus//Coe git which houses the evaluation metries following the popular De- tectron2 framework (https:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
16

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We evaluate our work on MS-COCC 40 and PASCAL-VOC publicly available.
10 which are
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guide wtwas//oooeSumiisiiooooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/puut httppppwwppppppppppppppppppaaaep wtw.:iddde///eSSdmbmmeeeeee
ssioioolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail out the experimental settings in Section 4 with additional details in Appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
17

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide statistical significance experiments on key components of CROWD contrasted against State-of-the-Art approach OrthogonalDet in Table For every task 7, in the OWOD benchmark run experiments on three different randomly generated seed values (kept constant across methods) keeping other parameters constant and report results in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We run all our experiments on 4 NVIDIA V100 GPUs with an approximate training time of hours for all tasks. This is described Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't ittn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////waaaaaepccccppcc/ccaeaa
Answer: [Yes]
Justification: For our research and experiments, we use public datasets and models that have been used in prior published works. Our research does not involve direct interaction with human subjects.
18

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Object detection is extensively used in real world scenario. As broad impact, the proposed work would be meaningful in scenarios for such real-world scenario (where setting often open-world) and model has to continually adapt new tasks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [No]
Justification: Safeguards would fall outside the scope of our research and we mainly focus on subjects that are inanimate. No experiments explicitly use personally identifiable features of humans subject.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
19

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper relies several existing benchmarks which are cited and credited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The key asset result of this paper is the code base where provide execution instructions.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not crowdsource images by ourselves; we use images from publicly available datasets MS- MS-COCO and PASCAL-VOC which has been vetted through.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There is no crowdsourcing involved this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21

---

A
Appendix
Notation
22
Additional Related Work
22
A.3
Derivations of Instances of LCROND
23
A.3.I
Derivation of CROWD-FL
24
A.3.2
Derivation of CROWD-GC
25
A.3.3
Derivation of CROWD-LLogDet
26
A.4
Additional Experimental Details
27
A.4.1
Ablation on Selection Budget
28
A.4.2
Results on Synthetic Datasets CROWD-D
28
A.1 Notation
Following the problem definition in the main paper we introduce the notations used in Table throughout the paper.
Table 8: Collection of notations used in the paper.
Symbol
Description
Task identifier for each OWOD task.
Ti D T K K U Re
Each Task in OWOD.
Training dataset for each task.
The Ground set, here refers to the mini-batch at each iteration.
Complete set of currently known classes in T,
Complete set of previously known classes in Te.
Complete set of unknown classes T,.
Predefined Replay buffer currently known classes in Tç.
Ri-1
Predefined Replay buffer of previously known classes Tg.
(2, 0)
Task specific Object Detector used feature extractor.
CIS(..)
Multi-Layyr Perceptron as classifier. In our case a two layer network.
Parameters of the feature extractor.
(A)
Pross-Similarity between sets A, V. Self-Similarity between samples in set A E T. Submodular Information function over Submodular Conditional Gain function between sets A and Q. Loss value computed over all known and unknown objects.
SA(
f(A)
({{))))
LCROWD self L CRRWWD I CROWD
Combinatorial loss computed over all known classes K; T. Combinatorial loss computed between known classes K2 and unknown classes U
A.2 Additional Related Work
Data subset selection aims at identifying distinct set of examples from a large pool which accurately captures the properties of the data distribution. This has rendered subset selection to
be a natural choice for data-efficient machine learning tasks like Active Learning [31] (50] [59 [
Continual Learning 22 MD Data Summarization 34 etc. Traditionally subset selection has been defined as subsampling technique based on similarity 233 uncertainty [5] etc. random [54].
Orthogonally, a new line of work based on combinatorial functions. particularly submodular functions [I], I8 have emerged which effectively selects informative subsets by modeling the notions of cooperation, diversity and representation (34] These functions formulate subset selection as a greedy maximization task [52) based on several information theoretic measures like Total Information, Mutual Information, Conditional Gain etc. (discussed in Section 3.2 of the main
paper). Concurrent their success in vision 34 31. language [39], speech [66] etc. domains,
22

---

subset selection has been used in auxiliary learning mechanisms like meta-learnin [36 and data-discovery [32] targeting identification of rare or unseen examples from an unlabeled example pool. CROWD exploits this line of investigation adopting a combinatorial subset selection technique (detailed in Section 3.3.T to discover unknown objects in the open-world setting).
Object detection (OD) is a fundamental task in computer vision encapsulating both localization and recognition tasks under the same roof. OD methods are traditionally grouped into two principal paradigms: single-stage and two-stage detectors. Single-stage detectors, exemplified by SSD 22).
RetinaNet [11, and YOLO [57 [56) CenterNet []] unify the processes of object localization and classification into single feed-forward network, enabling real-time performance with relatively low computational overhead. In contrast, two-stage detectors, such as Faster R- R-CNN [14 [3 [58], adopt a cascaded architecture wherein a Region Proposal Network (RPN) first hypothesizes candidate object regions, followed by refinement stage that simultaneously predicts the class and precise bounding box of each proposal. CNN based architectures struggles with the long-range dependencies, which is important for understanding the complex spatial relationships between objects at varying scales (perspective views). Transformer based models [4]
improve upon this vulnerability by introducing a self-attention [62] mechanism based on an encoder-decoder architecture [4]. While these models achieve impressive performance in closed- world settings (all object categories present during testing are known and predefined in the training data) they under-perform in open-world scenarios when encountering unknown objects unseen during training.
75 6
Preliminaries of Submodularity (continued from Section 3.2) As discussed in Section 3.2 of
the main paper, submodular functions have been recognized to model notions of cooperation 200
diversity [88. representation (3 and coverage [24.. Following the combinatorial formulation in Section 3.1 of the main paper we define the ground set V = {A1, A2, ........ AN) }. s.t. [V] N and
explore lour different categories of submodular information functions in our work, namely-
(1) Submodular Total Information (Sy) which measures the total information contained in each
set [TT] expressed Sy(A,, Ag,.... ... AN) as in Equation (2). Maximizing Sy over a set A, models
diversity (38] while minimizing Sy models cooperation 20].
A2, AN) = Σ f(A.)
(2)
1=1
(2) Submodular Conditional Gain (H2) which models the gain in information when set A₁ is added
to Ag. H models the notion of dissimilarity between sets and can be expressed in Equation (3).
Hy(Ai A,) = -((A ((A)) Vi,j E (V]
Given a submodular function (can alterna- tively be H)) tasks like selection 19 27 and summarization 26 24 have been modeled as a discrete optimization problem to identify summarized set of examples A C V via submodular maximization under a cardinality constraint (IA) < k), i.c. max,svv|AAjs f (A). This can be fairly approximated with a (1-e-1) constant factor guarantee (53] using greedy optimization techniques (52) shown in Algo- rithm 2 Extending the definition of submodular functions to continuous optimization space Majec et al. [48] have proposed a set of novel family of learning objectives which minimize total information and total correlation among sets in Dtrain using continuous optimization techniques like SGD. These objectives have been shown to be significantly more robust to large imbalance demonstrated in real-world tasks like longtail recognition (48 and few-shot learning [49].
Algorithm 2 Greedy Submodular Maximiza- tion 5
Require: Submodular function f 2: + R, car- dinality constraint k
Ensure: Set A C V maximizing f ((A) under car- dinality constraint k
1: A x- 0
2: for j = 1 to k do
3:
E arg max [F(AU f(A))
eEVIA
4: A AUfe
5: return A
A.3 Derivations of Instances of LCROWD
As discussed in Section 3.3.2 of the main paper, varying the choice of Submodular function f in Equation [I] results in several instances of LCROWD- Based on three popular choices of f among
23

---

Racility-Locationn Graph-Cor Cut and Log- Dogg--eermmnat,, we derive the respective formulations of LCROWD- sell Note, that the derivations of CCROWD adapted from Majec et al. 48 and are thus not included below.
A.3.1 Derivation of CROWD-FL
Theorem A.I. Given set of known Rols KG E [1,C%, set of unknown RoIs U (T K U and the Facility-Location based submodular function defined over any set s.s. f(A)
EieT j Sig, we define CROWD-FL FI learning objective to learn the parameters @ of the model Lself h', containing two components LCROWD and LCRÖWD D as shown in Equation Here, sij resembles the similarity between samples and j respectively.
self CROWD
max (i)
K
JEK!
iET
(4)
LCROWD
crosss
Σ NET
max max 8 (0) - Sna (0), 0) REK
aEU
Proof. From the definition of Tgroes in Equation 1 we find,
Σ (H1(K
CROWD
(5)
cross CROWD
Substituting the definition of f(A) over any set from the theorem the above expression we get
Ca
I crross "CROWD
Σ
H2(Ki(
1=1
C
cross I. CROWD
ΣΣ
max
8kk
Σ
max
ke REK?U
NEU NET
i=1 NET
C¹
(6)
croRe CROWD
ΣΣ
max
max maxs kU
Σ
max 8YM
REK!
WEU
i=1 NET
EET
O³
CROWD
crozs
ΣΣ i=1 NET
max
max8nk REK!
max8
NEt
Term
Term
The Term 2 in the above equation controls the degree of separation between K and U Due to this
we introduce a hyper-paramett parameter y which we can control during model training. Sine is a constant does not affect the submodular properties of Laross The final loss formulation, normalized by the size of T thus becomes
cross CROWD
Σ
Σ NET
max
max KEK
1mmxss 0 WEtT
Additionally, we do not provide proofs for T.8 information formulation in Majec et al. [48).
T.RowD since this function largely resembles the total
D

---

A.3.2 Derivation of CROWD-GC
Theorem A.2. Given a set of known Rols K E [1,C", set of unknown Rols U (T Kº U and the Graph-Cuu Cut based submodular function f defined over any set A s.t. f(A)
EiET LjeASth Sij >EiJEA Sij, we define CROWD-GC learning objective to learn the parameters of the model he containing two components L ()) and LoRÖw (0) as shown in Equation Here, s.j resembles the similarity between samples and j respectively.
C²
CROWD
f
Σ jET
85)(0) -
Σ 81y(0)]
Ki
EK
iJEK!
(8)
C¹
DCROOD())
croos
Σ
iT
[((k 2λ
Σ
Sku 8ka( (0)
Ke KE
Proof. From the definition of CROWD in Equation
we find,
C² Σ
(H(
CROWD
(9)
O²
cross
Σ
-
=
Substituting the definition of S(A) over any set from the theorem in the above expression of LCROWD we get-
C² HHI(K:U
= Ct
L EToss
-
L CROWD
Cross
Σ Σ Σ
Syk
Σ
Snk Σ
Σ
Snu A
Σ
Spu
NET KER!UUE
n.KERSUEE
nET DELi
n.uEU
(10)
I CROWD
ΣΣ
Snk ΣΣ
Snu X
Σ
Snk
i=
NER!
NEU
M,KERŞUD
ΣΣ
8nu +1
Σ
8nu
NET DED
The second term and the fourth term cancels out (same value with opposite signs).
L CROWD
ΣΣΣ
8ak
Σ
8nk +
Σ 8nu
NET KEK!
In.EEUUU
n.uEU
(II)
I CROWD
ΣΣΣ
8nk
Σ MAKEK!
8uk +2
Σ
8na
d= NET KEK!
n,uEUe
Now, rearranging the terms of the equation get-
C¹
L CRRWD
8nk X
Σ
8nk
+ Σ
Snu
i=1
WET
KEK;
n.kEK
n.EEU
Ct f(K)
(12)
I cross CROWD
Σ
2X Σ
Snu
Term
n.EUUe
Term 2
25

---

Similar to CROWD-FL the Term 2 in the above equation controls the degree of separation between K² and U Due this we introduce a nyper-parameter which can control during model training. Sine is a constant it does not affect the submodular properties of CROWDD The final loss formulation, normalized by the size of 'T thus becomes-
CROWD
[(K) + 2A
Σ
8Ym
(13)
1=1
Additionally, we do not provide proofs for CROWD since this function largely resembles the total information formulation in Majee et al. 48
A.3.3 Derivation of CROWD-LogDet
Theorem A.3. Given a set of known Rols KG E [1,C9, a set of unknown Rols U (T K" UU' and the Log-Det Log-Determinant based submodular function f defined over any set A s.t. f(A) log det(ss) A), we define CROWD-LogDD learning objective which contains two components L and Lerose Cnown as shown in Equation 14). Here, 8.j resembles the similarity between samples and
j respectively.
C¹
LOROWD
log det((x: (6) + (1xx!))
1=1
(14)
CROWD
ICToss
(0)
log (st(sx: (0) v² s -x) (O)sk( (0)²)
Proof. From the definition of CROWD in Equation
find
LRROWD
Σ H1(Ki)
(15)
-
CROWD
Substituting the definition of f(A) over any set from the theorem in the above expression of .CROWD we get-
C²
at f(K u0' f(U))
L
cross CROWD
cross
log det det(sx;uur)) log det(s
(16)
dct(sx)u0r)
log det(s
From Schur's complement which states that given two sets and det(sAuB)
det(sa). det \ SA). Replacing the term dct(sk;uu=) with the above definition we get-
log det(sy=). det( det(ssuuuu \ sy*)
I CROWD
det(s
(17)
Σ
log det (s \ sp))
26

---

Table 9: Generalization performance on Incremental Object Detection (IOD) where we show that our CROWD approach (here only CROWD-L) when applied to the fintetuning stage of IOD tasks show better generalizability. Best results are in bold while new classes introduced in the task are shaded
gray
10 setting
sycle
bird
bottle
bus
chsir
cow
table
dog
bike
penson
plani
sheep
sofa
LOD [30]
69.9
7044
69.1
51.3
48
68.7
78.9
68.4
45.5
58.
59.7
72.7
73.5
73,2
66.3
29.5
63:1
61.6
69.3
62.2
63.2
Faster ILDD 663
72.8
75.7
71.2
60.5
61.7
704
K3.3
76.6
53.1
72.3
36.7
70.9
66.8
67.6
66.1
24.7
63.1
48
57
43.6
62.1
ORE 23
63.5
70.9
58.9
42.9
34.1
76.2
80.7
76.3
34.1
66.1
56.1
70.4
80.2
72.3
81.8
42.7
71.6
68.1
77.0
67.7
64.1
Mcta-1LOD
57.6
75.1
76.0
66.3
ROSETTA
74.2
76.2
64.9
574
76.1
84.4
68.8
52.4
67.0
62.9
63.3
79.8
72.8
78.1
40.1
62.3
61.2
72.4
66.8
66.8
OW-DETR 15
61.8
69.1
67.8
25.8
47.3
78.3
78.4
78.6
96.2
71.5
57.5
75.3
76.2
77.4
79.5
40.1
66.8
66.3
75.6
64.1
65.7
PROB Cid
70.4
75.4
67.3
48.1
55.9
73.5
78,5
75,4
42.8
72.2
64.2
73.8
76.0
74.8
75.3
40.2
66.2
73.3
64.4
64.0
66.5
CAT
76.5
75.7
67.0
62.4
73.2
82.3
83.7
42.7
64.4
56.8
74.1
75.8
79.2
78.1
39.9
65.1
67.4
67.7
OrthogooalDet [1
82.9
80.1
75.8
61.3
60.6
81.5
87.9
51.9
48
82.1
57.7
63.5
80.5
77.6
78.2
38.9
69.8
62.8
76.9
61.2
69:11
CROWD (ours)
84.1
84.5
73.9
60.0
65.1
80.1
89.3
82.7
53.3
77.4
633/
78.5
80.9
B3.4
83.9
46.5
72.6
60.9
77.9
71.5
73,5
setting
sero
cycle
bird
boot
bottle
bus
car
Cat
chair
COW
table
dog
hoose
bike
person
plant
sheep
sofa
train
TV
IAA
ILOD [0]
70.5
79.2
68.8
59.1
53.2
75.4
79.4
78.8
46.6
59.4
59.0
75.8
71.8
78.6
69.6
33.7
61.5
63.1
71.7
62.21
65.8
Faster ILOb
65
66.5
78.1
71.8
54.6
61.4
68.4
82.6
82.7
52.1
74.3
63.1
78.6
80.5
78.4
80.4
36.7
61.7
59.3
67.9
59.1
67.9
55.7
77.2
85.6
76.7
78.5
82.1
64.6
68.5
Mcta-1LOD 22
78.4
79.7
66.9
51.8
56.2
77.7
81.6
79.1
47.7
75.0
61.8
74.7
81.6
77.5
80.2
37.8
58.0
54.6
73.0
56.1
67.8
ROSETTA 11
76.5
77.5
65.1
56.0
60.0
78.3
85.5
78.7
49.5
68.2
67.4
71.2
83.9
75.7
82.0
43.0
60.6
64.1
72.8
67.4
69.2
OW-DETT
15
77.1
76.5
69.2
51.3
61.
79.8
84.2
81.0
49.7
79.6
58.1
79.0
83.1
67.8
85.4
33.2
65.1
62.0
73.9
65.0
69.4
PROB [7
63.9
75.0
75.8
78.3
77.2
38.4
57.1
61.9
70.1
CAT
75.3
81.0
84.4
64.5
56.6
74.4
84.1
86.6
53.0
70.1
72.4
83.4
85.5
81.6
81.0
32.0
58
6
60.7
81
63.5
72.2
OrthogonalDet BTP
81.8
79.3
71.0
71.0
58.8
62.1
82.6
89.7
79.8
47.0
80.5
61.1
79.9
80.2
81.6
44.2
65.5
71.5
75.6
74,2
72.6
CROWD
72.5
83.1
89.3
83.7
N0.3
W2.8
63.7
75.6
74.4
19 1 setting
pero
cycle
bird
boot
bottle
hus
car
CAR
chalir
COW
table
dog
horse
bike
person
plant
sheap
sofa
train
IAP
ILOD
15.4
78.4
76.3
77.2
77.5
70.1
42.3
68.2
Faster
ILDD
15
64.2
74.7
73.2
55.5
53.7
70.8
82.9
N2.6
51.6
74.7
58.7
78.8
81.8
75.3
77.4
43.1
73.8
61.7
69.8
61.1
68.5
ORE 21
67.3
76.8
60
28.4
58.8
81.1
66.5
75.8
41.5
79.6
54.6
72.8
85.9
81.7
82.4
44.8
75.8
68.2
75.7
661.
68.8
Meta-11.0
22
78.2
77.5
69.4
55.0
56.0
78.4
84.2
79.2
46.6
79.0
63.2
78.5
82.7
79.1
79.9
44.1
73.2
66.3
76.4
57.6
70.2
ROSETTA
75.3
65.3
58.3
79.6
84.6
72.9
49.2
73.7
68.3
71.0
78.9
77.7
80.7
44.0
68.5
76.1
68.3
69.6
OW-DETR
15
79.0
80.8
53.5
77.5
82.0
74.7
43.3
66.6
62.0
70.2
PROB 22
80.3
78.9
77.6
59.7
63.7
75.2
86.0
83.9
53.7
82.8
66.5
82.7
80.6
83.8
77.9
48.9
71.5
69.9
77.6
48.5
72,6
CAT 44
$6.0
85.8
78.8
65.3
613
714
84.8
84.8
52.9
78.4
71.
82.7
83.8
81.2
80.7
43.7
75.9
58.5
85.2
61.1
73.8
OrthogonalDet
61
66.0
744
844
78.8
68.6
74.5
CROWD (ours)
81.7
77.4
66.8
80.7
K7.1
67.9
49.4
87.3
65.6
84.3
85.4
79.9
81.6
48.6
690
74.2
Following Schur's complement yet again which states that SAUB \. SA 8B where SA refers to the cross-similarities between sets A and B while 8 and sy represent the corresponding self-similarities and substitute this definition into the aforementioned equation
3 A SA.B.
L
crozs
Σ
log det - -K..0 T & k(,)))
(18)
Normalizing this term with the size of the ground set [7] and introducing the dyper-parameter v which trades-off between inter-cluster separation and intra-cluster compactness, we derive the function for cross as
GrORS
log (st - u² s k.,)))
(19)
CROWD
Similar to previously derived objectives, we do not provide proofs for LRROWD since this function largely resembles the total information formulation in Majee et al. (48).
A.4 Additional Experimental Details
In this section we provide additional experimental details for training our CROWD approach on M-OWOD, S-OWOD and IOD benchmarks discussed in Section of the main paper.
M-OWOD and S-OWOD benchmarks M-OWOD and S-OWOD benchmarks are created from MS-COCO 40 and split into 4 tasks T,, where t E [1.4] detailed in the "Datasets" section in Section 4 For each task, the model in provided labeled examples from T, alone while at inference the model is expected to identify objects in tasks leading up to Ti, s.t t E [1, t]. We split the training
TTis reproduction of the results from OrthogonalDet from their public repo
27

---

Table 10: Ablation Experiments on the variation in k in CROWD-D. Given the Graph-Cur based selection strategy in Algorithm|] of CROWD-D and the CROWD-FL based learning objective in CROWD-L we vary k in [0, 100] and adopt the best performing budget for our pipeline.
Method
Budget k
Task
Task 2
Task 3
Task 4 mAP
U-Recall
mAP
U-Recall
mAP
U-Recall
mAP
OrthogonalDet 66
24.6
61.3
26.3
47.0
29.1
41.3
37.9
51.2
61.0
49.1
45.9
62.7
40.3
37.4
10
57.9
61.7
53.6
47.8
69.6
42.4
38.5
CROWD (ours)
30
58.4
61.7
53.5
48.0
70.1
42.4
38.5
100
57.5
59.3
53.7
44.3
70.9
38.8
32.0
into two splits. In the first stage the model is exposed only to the currently known classes K* and the learnt model h' biases on labeled examples in K' At the end of the first stage CROWD-D kicks in and selects representative unknowns as described in Section 3.3.1 Lets call in US. Next, we store a replay buffer of the currently known objects K', s.t. K' C K', Following this, we combine K', K and replay buffer from the previous task k into a single dataset to finetune h' using CROWD-L. As detailed in Section 3.3.2 this ensures known vs. unknown separation wile retaining discriminative features from known classes.
IOD benchmarks- In contrast to OWOD, IOD does not encounter unknowns during model training but experiences heavy catastrophic forgetting on previously known classes K Following recent benchmarks like Sun et al. [61] Zohar et al. [77]. Joseph et al. [2] we evaluate the IOD performance of CROWD on PASCAL-VOC benchmark on three settings produced by varying the number of newly added classes 10 10, 15 5, 19 + 1 as shown in Table 9 In the absence of unknowns we do not apply CROWD-D and only rely on CROWD-L applied to the finetuning stage of IOD. Following latest works we adopt replay based learning technique whichh stores small subset of the previously known objects Kt-1 in buffer. Kt-1 combined with the newly introduced classes K is used to finetune h². This also requires us to slightly modify the formulation of LEROWD as detailed in Section 3.3.2 For each setting h is trained on a batch size of 12 for 3000 iterations using an AdamW optimizer, base learning rate to 2.5 x 10-5 and weight decay of 1 x 10-4.
A.4.1 Ablation on Selection Budget
As detailed in Section 3.3.1 the parameter dictates how many candidate unknown Rols CROWD-D selects per image. We conduct an ablation over several plausible settings of k within the interval [0, 100], and present the outcome in Table 10 Fo this experiment we keep the choice of submodular function f in CROWD-D as Graph-Cut and Facility-Location (CROWD-FL.) for CROWD-L. following the results of the ablation experiments in Table in the main paper. Notably, raising from (i.e., OrthogonalDet) to 10 yields marked uplift in the model's uecaoln-recall (U-Recall), yet further increases beyond k- = 20 confer no additional gains. This initial boost in U-Recall stems from the integration of truly informative Rols into the training loop. However, when k reaches its upper bound of 100, the mean average precision (mAP) on known classes experiences a slight decline relative to existing baselines- a consequence of inadvertently incorporating spurious background proposals.
A.4.2 Results on Synthetic Datasets CROWD-D
In addition to the illustrations provided in Figure 3 we contrast the selection performance of CROWD- by varying the underlying submodular function f between Grap-Cut (GC), Facility (FL) and Log- Log-Determinant (LogDet) on synthetic datasets as shown in Figure 6 The use of synthetic datasets provide us with complete control over the embedding space allowing us pathologically inject imbalance, inter-cluster separation etc. in a compute efficient fashion. Particularly in our experiments we use two-cluster imbalanced setup mimicking the Rol embedding space in Faster- RCNN [58) model. Similar to Sun et al. [61] the number of known class and unknown class feature vectors are severely imbalanced with total number of Rols R = 500 and the number of knowns |K| = 10. R and K" are sampled from a normal distribution with fixed variance values. The LogDet based selection strategy enforces the notion of diversity in the selection mechanism which does not select representative unknowns negatively impacting OWOD performance shown in Table The
28

---

(a) Rol Feature Space R
(b) Background
(c) Unknowns
Bi K
GCCG
(a) Rol Feature Space R
(b) Background B
(c) Unknowns
Kt
R K"
E
FLCG
(a) Rol Feature Space
(b) Background B
Unknowns U
K*
R
LogDeiCG
Figure CROWD-D results on synthetic dataset contrasted against instances of popular sub- modular functions Graph-Cut, Facility-Locatii and Log-Determinan Graph-Cut based selection strategy models both representation and diversity resulting in the best possible choice of unknown instances in U
FL. based selection strategy models representation as shown in Figure during selection resulting in erroneous selection of background instances negatively affecting OWOD performance. Lastly, GC based selection strategy shown in Figure 6 models notions of both diversity and representation selecting diverse backgrounds B' farthest K well representative unknowns U². This results in GC based selection strategy to produce the best overall results as shown in Table 2
29


Reviews:
[{"review_id": "hErJhFcUIq", "reviewer": "Reviewer_9jJR", "summary": "This paper proposes a novel framework for Open-World Object Detection (OWOD) named Combinatorial Open-World Detection (CROWD), which consists of two tightly coupled components: CROWD-Discover for targeted unknown instance mining, and CROWD-Learn for submodular objective-based representation learning. Unlike previous approaches that rely on objectness heuristics or clustering, CROWD formalizes OWOD as a combinatorial problem using submodular conditional gain (SCG) and submodular information measures (SIM). This formulation aims to both identify informative unknown instances and ensure robust feature disentanglement, thereby mitigating known-unknown confusion and catastrophic forgetting. The method achieves state-of-the-art performance across multiple OWOD benchmarks.", "questions": "1. Include runtime/memory benchmarks or FLOPs to better quantify the computational trade-offs of CROWD compared to baselines.\n2. Provide a sensitivity analysis for thresholds τₑ and τ_b (e.g., a heatmap or performance curve).\n3. Add qualitative failure cases where CROWD-D misidentifies background as unknown, to better understand its limitations in complex scenes.\n2. In Figure 2, superscripts and subscripts are not very clear. Could u further explain them?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths:\n1. The paper articulates the inherent limitations of existing OWOD approaches, particularly in semantic confusion and forgetting, and motivates the need for a more principled and theoretically grounded framework. Modeling OWOD as a combinatorial data-discovery and learning task is a novel and theoretically sound perspective. The use of submodular functions introduces strong guarantees and interpretability.\n2. The method demonstrates significant gains in unknown recall and competitive improvements in known-class accuracy on both M-OWOD and S-OWOD benchmarks, validating its effectiveness.\n3. The paper is well-organized, with a clear separation of components (CROWD-D and CROWD-L), detailed methodology, and informative figures and tables that aid understanding.\n\n### Weaknesses:\n1. Lines 191–194 describe heuristics (e.g., filtering by objectness score and subset selection) to reduce computational cost. However, the actual computational complexity or runtime comparison to baselines is not discussed. Since CROWD introduces combinatorial subset selection, this may impact scalability. Could the authors provide complexity analysis or empirical runtime comparisons?\n2. The framework heavily relies on two thresholds, τₑ (objectness threshold) and τ_b (background selection budget). Although empirically set (0.2 and 30%), the sensitivity of performance to these hyperparameters is not systematically analyzed. How robust is the system to variations in τₑ and τ_b? Could performance degrade significantly if these are misconfigured?\n3. The paper positions CROWD-D as a more principled alternative to heuristic pseudo-labeling (e.g., ORE, PROB), yet concrete distinctions remain somewhat abstract. What is the core theoretical or empirical advantage of CROWD-D over attention-based or objectness-based mining (e.g., OW-DETR, CAT)? In real-world settings where unknowns and backgrounds may share features, how does CROWD-D avoid misidentifying complex background as unknown?\n4. As shown in Table 5, the standalone gain from CROWD-L is relatively modest (~0.4–1.2% mAP over baseline), especially considering the additional modeling complexity. Could the authors elaborate on why the representation learning stage has limited standalone impact? Is it because CROWD-L primarily complements CROWD-D, or is the performance bounded by the quality of mined unknowns?", "was_revised": true, "final_justification": "The response from authors addressed most of my concerns, so I'd like to increase my final rating from 4 to 5.", "rebuttal": "**[W1] “Lines 191-194 …”** - We thank the reviewer for the detailed question. The reduction in computation cost referred to in lines 191-194 results from the exclusion of very low confidence RoI proposals potentially containing (line 2 in Algorithm 1). To put things numerically we contrast against existing SoTA approach OrthogonalDet (Sun et al., 2024) which models interactions between every possible pair of RoI features to calculate a relevance score (to select unknowns). For the Faster-RCNN family of object detectors the number of RoIs are 500 (as set in our work as well as OrthogonalDet) resulting in a 500 x 500 sized similarity kernel for every image in the dataset. On the other hand, excluding very low confidence proposals (thresholded on the objectness scores) in CROWD naturally reduces the number of RoIs to ~ 200-300 proposals reducing the compute overhead. Also, in terms of runtime we observe that on average OrthogonalDet requires about 36 hours to train, reweight and finetune on a particular task while our method CROWD can perform this in ~32 hours (using the same compute infrastructure and seed setting). This can be attributed to the joint impact of the combinatorial selection and faster convergence of the combinatorial objectives (similar results observed in SMILe (Majee et al., 2024) ) due to modeling of the learning task as a set problem (30k steps in OrthogonalDet per task vs. 24k steps in CROWD per task). We will release the log files from the best performing runs from our framework (built upon detectron2) upon acceptance showing this statistic. \n\n**[W2] “The framework …”** - We appreciate the reviewer’s constructive criticism on the hyper-parameters in CROWD. We clarify that $\\tau_e$ is an exclusion threshold which reduces the search space of CROWD-D by eliminating RoIs which have a low confidence threshold. As shown in Tab. R1 below, increasing $\\tau_e$ from 0 to 1 increases performance until $\\tau_e = 0.2$ and then reduces. A lower value of $\\tau_e$ allows for a large search space but includes a lot of noisy background objects leading to reduced selection performance. On the other hand a large value of $\\tau_e$ can potentially earmark unknown foregrounds as background resulting in reduced performance.\n\nKeeping $\\tau_e$ fixed at 0.2 we ablate $\\tau_b$ which controls the selection budget for backgrounds (higher the value more are the number of background RoIs identified). Increasing $\\tau_b$ (percentage here) increases the fraction of RoIs treated as backgrounds. A small value of $\\tau_b$ widens the search space for the combinatorial function when mining unknowns causing a small drop in performance due to confusions between true background and foreground unknowns. On the other hand, very large values of $\\tau_b$ shrink the search space oftentimes considering unknown foregrounds as background objects showing a steep drop in performance.\n\n*Table R1: Ablation Experiments on the exclusion criterion $\\tau_e$ and background budget $\\tau_b$ in CROWD-D. The submodular function is kept constant as Graph-Cut in CROWD-D and a CROWD-FL based learning objective is chosen in CROWD-L under constant machine seed.*\n\n| **Method**             | Value    | **Task 1 U-Recall** | **Task 1 mAP** | **Task 2 U-Recall** | **Task 2 mAP** | **Task 3 U-Recall** | **Task 3 mAP** | **Task 4 mAP** |\n|------------------------|----------|---------------------|---------------|---------------------|---------------|---------------------|---------------|---------------|\n| **CROWD ($\\tau_e$)**     | 0.05     | 57.5                | 61.7          | 52.7                | 47.1          | 68.2                | 41.5          | 38.0          |\n|                                | **0.2**  | 57.9                | 61.7          | 53.6                | 47.8          | 69.6                | 42.4          | 38.5          |\n| $\\tau_b$ = 30%              | 0.5      | 53.3                | 59.5          | 49.4                | 45.9          | 65.0                | 40.1          | 37.8          |\n|------------------------|----------|---------------------|---------------|---------------------|---------------|---------------------|---------------|---------------|\n| **CROWD ($\\tau_b$)**  | 10%      | 57.9                | 61.7          | 53.1                | 46.8          | 69.2                | 41.3          | 37.4          |\n|                               | **30%**  | 57.9                | 61.7          | 53.6                | 47.8          | 69.6                | 42.4          | 38.5          |\n| $\\tau_e$ = 0.2                  | 50%      | 55.4                | 60.0          | 50.0                | 42.1          | 63.7                | 38.9          | 34.5          |\n\n\n**[W3] “The paper positions …”** - We thank the reviewer for the insightful question. As noted in lines 90–92 of the main paper, CAT employs a dual pseudo-labeling strategy: a model-driven approach leveraging attention scores and confidence to mine unknowns, and an input-driven approach using selective search to propose unknown candidate regions independent of model predictions. Similarly, OW-DETR uses unmatched object queries with high backbone activation scores as unknowns during training.\n\nIn contrast, CROWD-D takes a fundamentally different, principled approach by framing unknown mining as a greedy submodular maximization problem based on information-theoretic submodular conditional gain (SCG) functions (Sec. 3.3). Instead of relying solely on heuristic attention or objectness scores, CROWD-D first filters out clear background regions via low objectness scores (inspired by CAT) and then selects unknown candidates by maximizing the information gain relative to a fixed set of known RoIs. The optimization process also highlighted in Algorithm 2 (appendix) inherently encourages selection of examples that are maximally informative (high marginal gains) thereby dissimilar from known classes, reducing confusion with complex backgrounds. It is interesting to note that our selection strategy is further backed by strong theoretical guarantees (Khargonkar etal., 2021) and follows the laws of diminishing marginal return to achieve this behavior. Nonetheless, the effectiveness of CROWD-D does depend on the quality of underlying feature representations, motivating the joint use of CROWD-L to improve feature discriminability between known and unknown objects.\n\n**[W4] As shown in Table 5 …”** - We agree with the reviewer that the standalone gain of CROWD-L is 0.4 to 1.2% over SoTA approaches while CROWD-D shows up to 2$\\times$ gain in unknown class recall. The selection process in CROWD-D can potentially inject some noisy pseudo-labels in the learning process as shown by the drop in performance from 47.0 to 45.2 in Table 5 (row 2 vs. row 4). To maintain the performance on known classes CROWD-L complements CROWD-D and enforces sufficient separation between known and unknown RoI features without sacrificing on the boost in unknown recall resulting from CROWD-D (last row in Table 5).\n\n**[Q1] “Include runtime …”** - We would like to request the reviewer to kindly refer to our comment [W1] in the same rebuttal where we describe the runtime costs incurred in CROWD over SoTA approach OrthogonalDet (Sun et al., 2024).\n\n**[Q2] “Provide a sensitivity  …”** - We would like to point the reviewer to our comment in [W2] of the rebuttal where we provide ablation experiments to show the impact of these hyperparameters.\n\n**[Q3] “Add qualitative …”** - We understand the concern of the reviewer and we will definitely include some qualitative failure cases in the Appendix of our paper in the immediate next revision.\n\n**[Q4] “In Figure 2 …”** - We understand the confusion of the reviewer regarding the symbols used in Fig. 2 (main paper). At first, we would like to point the reviewer to Table 6 of the Appendix where a glossary of symbols and their respective meanings in the context of CROWD has been provided. Particularly for Figure 2 we provide below a step by step explanation below - \n\n\n(1) Fig. 2 is a detailed overview of a single task within OWOD. A task $T^t$ with task identifier $t$ accepts a model $h^t$ and a replay buffer $\\hat{K}^{t-1}$ containing examples from previously known objects.\n\n(2) At first we train $h^t$ on only currently known objects $K^t$ a replay buffer from which is stored as $\\hat{K}^t$. For every example in $\\hat{K}^t$ we also extract background RoIs $B^t$.\n\n(3) CROWD-D mines a set of unknowns $\\hat{U}^t$ from $\\hat{K}^t$ and $B^t$ which is used by CROWD-L.\n\n(4) CROWD-L learns an updated model $h^{t+1}$ by finetuning on $\\hat{K}^{t-1}$ (previously known), $\\hat{K}^t$ (currently known) and $\\hat{U}^t$ (mined unknowns) using our novel learning objectives.\n\n(5) A replay buffer of the currently learnt objects $\\hat{K}^t$ and the model weights $h^{t+1}$ are passed on to the next task."}, {"review_id": "eN688cgy3p", "reviewer": "Reviewer_MYTT", "summary": "The paper proposes a new method named CROWD (CombinatoRial Open-World Detection) for detecting known and unknown (semantic) object classes in 2D camera images. The method first trains an object detector in a supervision setting to detect known object classes. After that, the predictions of the trained detector are used to discover unknown objects in the “background”. The filtered background objects and known class instances are then used to train the detector again. The paper claims significantly improved recall for detecting unknown objects and also improves detection performance for known classes.", "questions": "1. Why is the selected orange bounding box for the chair in Figure 1 a different size than the red bounding box, which is part of the “background” bounding boxes. By just selecting a bounding box, the size should remain the same, right?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Paper claims to improve recall for detecting unknown object class instances by 2.4 and also improves detection performance for known object class instances.\n2. The considered task is relevant in the current AI field, i.e. a label-efficiënt method is proposed as a object detector is trained to detect object classes that where not part of the training dataset labels.\n\nWeaknesses\n1. The first part of the title is very general “Looking Beyond the Known”. It basically states the same as “Open-World Object Detection” which is in the second part of the title. It would better to name the method name in the title or to put the characteristics of the proposed method in the title.\n2. The method name CROWD is chosen a bit randomly, the “r” is just one of the letters of “CombinatoRial”.\n3. Line 27 is unclear. It is unclear whether “previously learned classes” are about “classes that were unknown” or about “known classes”. In line 29, it seems that “unknown” classes were meant.\n4. Lines 30-31 refer to the Wilderness Impact scores in Table 2 but these cannot be found in Table 2.\n5. The symbols introduced in lines 35-36 give the impression of an exponent. Subscript would be better.\n6. The second part of the introduction is mainly a methodology. Even some sections of the methodology are named. The last part of the introduction (58-75) is very vague / hard to understand. Last sentence of line 75 contains typo, it should be “Our main contributions are:”.\n7. Three contributions are stated but the third one is just empirical support for the first two contributions (both are designed components for the proposed method).", "was_revised": true, "final_justification": "The authors are thanked for their rebuttal. My questions are answered. I am satisfied with the response. I changed my rating to borderline accept. If the authors incorporate the rebuttal in their paper, it can become solid work.", "rebuttal": "**[W1] “The first part …”** - We understand the concern of the reviewer on the title. We would like to clarify that the reasoning behind the use of “Looking Beyond the Known” was to highlight the importance of discovering high-quality unknown examples (which is achieved in CROWD-D). We would be happy to update the title with the technical details of our method in the immediate next revision.\n\n**[W2] “The method name …”** - The choice of CROWD as an abbreviation highlights that our model is able to identify potential unknown foreground objects within a “crowd” of background RoIs. In our defence, the use of intermittent letters like “r” seems to be a common practise in literature, even within the domain of OWOD eg. PROB (Zohar et al., 2023) which is abbreviated form of “**Pr**obabilistic **Ob**jectness”.\n\n**[W3] “Line 27 …”** - We thank the reviewer for pointing this out. The phrase pointed to refers to “known classes” in current task $T^t$. We will definitely update this in the paper for better readability. In line 29 “unknowns” do refer to unknown classes.\n\n**[W4] “Lines 30-31 … “** - We apologize for the oversight. Wilderness Impact and Absolute Open-Set Error (ASE) are demonstrated in Table 3 of the main paper. We will correct this in the immediate next revision.\n\n**[W5] “The symbols …”** - We understand the concern of the reviewer and would be happy to replace the current symbols with the “t” in subscript. However, in our defence, these symbols of known and unknown classes are used as in OrthogonalDet (Sun et al., 2024) who use $t$ in superscript.\n\n**[W6] “The second part …”** - We appreciate the criticism of the reviewer on this. Our goal here was to give a complete technical overview of our CROWD approach in a summarized form under the introduction with suitable references to details from the methodology section. We will be happy to dial down the amount of details and provide a high-level overview in the immediate next revision.\n\n**[W7] “Three contributions …”** - We agree that the third claim in our paper is an empirical validation of the previous two claims. We would also like to emphasize that the empirical prowess of methods in OWOD has been enlisted as a core contribution in several previous SoTA methods like ORE (Joseph et al., 2021), PROB (Zohar et al., 2023), OW-DETR (Gupta et al., 2022) and OrthogonalDet (Sun et al., 2024). We follow a similar pattern in writing our contributions. These empirical gains (even though are results of the method used) show the applicability of CROWD to general Open-World scenarios which we deem to be a critical contribution. \n\nPlease find below our responses to the questions raised - \n\n**[Q1] “Why is the …”** - We understand the concern of the reviewer and would like to emphasize that Figure 1 is a broad overview of the entire learning mechanism in CROWD and the instances shown are for illustration purposes only. The number of RoIs returned by the RPN for each image in our model is set to 500 (lines 263-264) with a large number of overlapping ones (since NMS is not applied in this stage) which we could not show in the figure (on the left). The figure on the right (with unknown instances plotted) highlights only the selected RoIs (3 / 10 selected) due to which the sizes appear different. We do acknowledge the difference in sizes and will ensure they are consistent in the immediate next revision. \n\nPlease find below our responses to the formatting issues raised in the review - \n\n**[F1] “In Figure 1 …”** - We thank the reviewer for the suggestion and will update the figure as suggested in the next revision. We clarify here that Fig. 1a and 1b does not refer to two different sub-figures and points to the complete process of selection and learning in CROWD. The top two rows of Fig.1 resembles CROWD-D denoted as 1a and the bottom row points to CROWD-L, denoted as 1b.\n\n**[F2] “Put the …”** - We thank the reviewer and will update the position of the subfigure titles.\n\n**[F3] “Split the …”** - We understand the confusion of the reviewer. We will keep only the final formulation of $L_{\\text{CROWD}^{\\text{cross}}}$ here and split the equation to accommodate for space. The remaining would be referenced from the Appendix where we provide the detailed derivations.\n\n**[F4] “Put the …”** - We thank the reviewer and will update the position of the subplot titles."}, {"review_id": "SKot4QMq65", "reviewer": "Reviewer_3Pty", "summary": "This paper addresses two major bottleneck challenges in Open World Object Detection, namely confusion between known and unknown objects, and catastrophic forgetting of previous learned classes. This paper proposes a novel framework called CROWD (Combinatorial Open World Detection), innovatively reformulating the OWOD problem as an interleaved task of combinatorial optimization-based data discovery (CROWD-Discover) and representation learning (CROWD-Learn). The paper utilizes the submodular conditional gain function to effectively mine representative unknown instances that are significantly different from known classes from unlabeled data. The paper employs different novel combinatorial objective functions to decouple the features of known and unknown classes while maintaining the discriminative consistency within known classes, providing new ideas for the above two key issues. Experimental results show that this method achieves remarkable results in standard OWOD benchmark tests, significantly improving the recall rate of unknown objects and the detection accuracy of known classes.", "questions": "1. There seems to be some ambiguity between the selection of Bt and Ut in the framework proposed in the article described in Figure 1 and the subsequent description.\n2. Whether using the replay buffer to solve the problem of catastrophic forgetting can only alleviate it to a certain extent or completely solve it. If a large number of new categories are added, will the problem still occur due to the fixed size of the buffer? \n3. Have any experiments been conducted to verify the method of combining the designed loss function with the replay buffer? How should it be set to achieve the best effect? What’s the best settings of the replay buffer?\n4.  The paper selected a set of submodular function combinations with the best performance for the CROWD framework through experiments on the M-OWOD benchmark. However, it failed to verify whether this set of function combinations is also the optimal choice on other benchmarks with different data characteristics (such as S-OWOD), and no relevant experiments on the robustness of this combination were provided.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. This paper is dedicated to solving two important problems encountered in open-world object detection and proposes a novel and interesting approach. The article has clear logic and sufficient innovation.\n2. The comparative experiments in the article are very rich, which fully verify the feasibility of the framework proposed in the article.\n3. The ablation experiments are rigorously designed, fully analyzing the contributions of each innovative module. The paper not only demonstrates the superiority of the overall solution but also verifies the necessity and effectiveness of each component through a detailed ablation study, making the sources of the method's advantages clear and the argumentation process very solid.\n\nWeaknesses\n1. The paper's experiments on the sensitivity of hyperparameters involved in the network structure are not enough sufficient. For example, the article mentions a small replay buffer, but does not conduct relevant experimental analysis on specific settings such as the buffer size and sampling strategy.\n2. When comparing different SCG functions, although the paper selects a combination with relatively good results, it lacks a comparison of the performance of these SCG functions on different datasets,such as S-OWOD. The robustness of the model mentioned in the article still needs further relevant experiments to prove.", "was_revised": true, "final_justification": "The response makes some clarification regarding the \"replay buffer\", and some Figures will be optimized in the further version . Overall, I'm positive about the article and maintain my rating for the current version.", "rebuttal": "**[W1] “The paper’s experiments …”** - We understand the concern of the reviewer regarding the “replay buffer” used in our experiments. We would like to clarify that the use of replay buffer is not unique to CROWD and has been used in several benchmarks like ORE (Joseph et al., 2021) and OrthogonalDet (Sun et al., 2024). Since we built our work on top of OrthogonalDet’s framework we used the exact same replay buffer which was publicly released (a fixed image list which is also same in ORE) in their codebase, thus did not require ablation. The number of unknown instances mined in CROWD-D however is a contribution of CROWD and has been ablated upon in Table 8 of the Appendix.\n\n**[W2] “When comparing different …”** - Since the end to end model training and inference takes roughly 32 hours in CROWD requiring a relatively large compute infrastructure we only conduct our ablation experiments on M-OWODB to limit compute and budgetary needs. This also follows the convention in SoTA approaches. However, we would like to point the reviewer to Table 2 (in main paper) where we demonstrate results on S-OWOD benchmark (as requested) for the exact same settings determined from the ablation study in Table 5 showing improvements in performance over SoTA approaches in ¾ tasks.\n\n**[Q1] “There seems …”** - We apologize for the confusion caused and would like to clarify that Fig. 1 was intended to be a very high-level overview of CROWD so some minor technical details have been omitted. Here, we provide a step by step analogy between Fig. 1 and our method description and will update the figure to show all technical details mentioned below - \n\n(1) At first we show that a model trained only on known classes has no knowledge of unknowns (discarded as low confidence BG by the NMS) (top subfigure in Fig.1).\n\n(2) We then pull out the RoI proposals before application of NMS to include the BG proposals and discard proposals with the lowest confidence as true background. This is not currently shown in Fig. 1 but we highlight this in Fig. 3.\n\n(3) Given the known class embeddings and the true backgrounds we apply CROWD-D to select the unknown class examples (middle subfigure in Fig. 1) and store them as $U^t$ for each task (also shown in Fig. 1).\n\n(4) Finally, we apply CROWD-L to learn both known and unknown classes in incremental finetuning fashion (bottom subfigure in Fig. 1). \n\n**[Q2] “Whether using …”** - We appreciate the question raised by the reviewer in this context. Using replay buffers (a common practice in literature both in OWOD and incremental learning) is a compute and memory efficient technique but does not completely solve catastrophic forgetting. According to our claims in CROWD we see that mining representative unknowns (CROWD-D) alongside a fixed buffer (this is predetermined from earlier research) and a better learning objective (CROWD-L) reduces the impact of forgetting. This is definitely a topic of continuing research and has been mentioned in our limitations as well.\n\n**[Q3] “Have any …”** - Continuing from our response to a similar question in [W1] above we would like to emphasize that the replay buffer used in CROWD is not determined by us but is a fixed list list of images for both M-OWOD and S-OWOD benchmarks released publicly in ORE, OrthogonalDet etc. This fixed imagelist has been used in most existing SoTA approaches during finetuning to maintain a fair comparison.\n\n**[Q4] “The paper selects …”** - We agree with the reviewer that we conduct our ablations only on the M-OWOD benchmark but use the exact same settings across tasks in both M-OWOD and S-OWOD benchmarks. From our literature review, this seems to be a common practice in literature adopted in SoTA approaches like PROB, OrthogonalDet, OW-DETR etc. Similar to our response in [W2] we would like to point the reviewer to Table 2 where we provide results on both M-OWOD and S-OWOD benchmarks keeping the exact same hyper-parameter settings as the ablation study."}, {"review_id": "thvU6l45ZA", "reviewer": "Reviewer_bAGd", "summary": "The paper deals with the problem of classifying known objects and detecting unknown objects. The tasks are continually discovered, and the model has to adapt to new tasks, giving rise to two challenges: catastrophic forgetting and confusion among known/unknown objects. The proposed method tackles the problem by proposing two major components. First component discovers the known and unknown objects by leveraging Submodular Conditional Gain and second components learns the representation of known/unknown objects. Experiments on benchmarks reveal effectiveness of the proposed method for the task of Open-World Object Detection.", "questions": "See weakness section", "limitations": "YES for limitations\n\nFor societal impact, authors have given answer of [NA]. I suggest the following as the societal impact of the work.\n\nObject detection are extensively used in real world scenario. The proposed work would be meaningful in scenarios for such real-world scenario (where setting is often open-world) and model has to continually adapt to new tasks.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n- The paper tackles the challenging but important problem of continual object detection in open-world setting. \n- The design of the proposed components is well motivated: data discovery for each category, then learning representation. \n- Ablations on impact of each component illustrates the importance of the design.\n- Performance is compared against substantial number of baselines. \n\n**Weakness**\n\n- Missing study of the impact of each component of the proposed loss function (hyperparameter η). \n- The paper mentions how thresholds τe and τb are determined. Could authors show the details?", "was_revised": true, "final_justification": "My concerns about missing studies have been addressed in rebuttal. Overall, I am positive about the paper.", "rebuttal": "**[W1] “Missing study …”** - We thank the reviewer for pointing this out. As stated in lines 230 - 231 $\\eta$ controls the trade-off between known-unknown class separation and known class cluster compactness. A lower value of $\\eta$ does not enforce separation between currently known and unknown exemplars but enforces intra-class compactness. This results in better retention of previously known objects but a drop in currently known objects due to increased confusion with unknown exemplars. On the other hand for a large value of $\\eta$ the model enforces large separation between currently known and unknown objects boosting performance on the currently knowns but suffers from catastrophic forgetting of the previously known classes. This is shown in Tab. R2 below. We adopt the value of $\\eta$ which provides the best overall performance (a middle ground between forgetting previously knowns and improving upon currently known classes).\n\n*Table R2: Ablation Experiments on the variation in $\\eta$ in CROWD-L. Given the Graph-Cut based selection strategy in CROWD-D we vary $\\eta$ between [0.5, 1.0, 1.5] and adopt the best performing value for our pipeline in CROWD-L. The selection budget $\\mathtt{k}$ in CROWD-D was set to 10 for all experiments and a fixed seed value.*\n\n| **Method**                   | $\\eta$ | **Task 1** U-Recall | **Task 1** mAP Curr. | **Task 2** U-Recall | **Task 2** mAP Prev. | **Task 2** mAP Curr. | **Task 2** mAP Both | **Task 3** U-Recall | **Task 3** mAP Prev. | **Task 3** mAP Curr. | **Task 3** mAP Both | **Task 4** mAP Prev. | **Task 4** mAP Curr. | **Task 4** mAP Both |\n|------------------------------|-------|---------------------|----------------------|---------------------|----------------------|----------------------|---------------------|---------------------|----------------------|----------------------|---------------------|----------------------|----------------------|---------------------|\n| OrthogonalDet [Sun etal., 24]       |   -   | 24.6                | 61.3                 | 26.3                | 55.5                 | 38.5                 | 47.0                | 29.1                | 46.7                 | 30.6                 | 41.3                | 42.4                 | 24.3                 | 37.9                |\n| **CROWD (ours)**             | 0.5   | 57.8                | 58.8                 | 53.4                | 57.1                 | 32.8                 | 44.9                | 65.3                | 50.2                 | 25.9                 | 42.1                | 44.0                 | 21.1                 | 38.3                |\n| **CROWD (ours)**             | 1.0   | 57.9            | **61.7**             | 53.6            | 56.7             | 38.9             | **47.8**            | 69.6           | 48.0             | 31.4             | **42.5**            | 42.9             | 25.4             | **38.5**            |\n| **CROWD (ours)**             | 1.5   | 57.9                | 61.7                 | 53.6                | 55.6                 | 39.1                 | 47.4                | 69.5                | 44.0                 | 34.6                 | 40.9                | 40.2                 | 28.0                 | 37.2                |\n\n**[W2] “The paper mentions …”** - Both $\\tau_e$ and $\\tau_b$ are determined empirically. $\\tau_e$ is an exclusion threshold which reduces the search space of CROWD-D by eliminating RoIs which have a low confidence threshold. As shown in Tab R3 below, increasing $\\tau_e$ from 0 to 1 increases performance until $\\tau_e = 0.2$ and then reduces. A lower value of $\\tau_e$ allows for a large search space but includes a lot of noisy background objects leading to reduced selection performance. On the other hand a large value of $\\tau_e$ can potentially earmark unknown foregrounds as unknowns resulting in reduced performance.\n\nNext in Tab. R3 we keep $\\tau_e$ fixed at 0.2 and ablate $\\tau_b$ which controls the selection budget for backgrounds (higher the value more are the number of background RoIs identified). Increasing $\\tau_b$ (percentage here) increases the fraction of RoIs treated as backgrounds. A small value of $\\tau_b$ widens the search space for the combinatorial function when mining unknowns causing a small drop in performance due to confusions between true background and foreground unknowns. On the other hand, very large values of $\\tau_b$ shrink the search space oftentimes considering unknown foregrounds as background objects showing a steep drop in performance.\n\n*Table R3: Ablation Experiments on the exclusion criterion $\\tau_e$ and background budget $\\tau_b$ in CROWD-D. The submodular function is kept constant as Graph-Cut in CROWD-D and a CROWD-FL based learning objective is chosen in CROWD-L under constant machine seed.*\n\n| **Method**             | Value    | **Task 1 U-Recall** | **Task 1 mAP** | **Task 2 U-Recall** | **Task 2 mAP** | **Task 3 U-Recall** | **Task 3 mAP** | **Task 4 mAP** |\n|------------------------|----------|---------------------|---------------|---------------------|---------------|---------------------|---------------|---------------|\n| **CROWD ($\\tau_e$)**     | 0.05     | 57.5                | 61.7          | 52.7                | 47.1          | 68.2                | 41.5          | 38.0          |\n|                                | **0.2**  | 57.9                | 61.7          | 53.6                | 47.8          | 69.6                | 42.4          | 38.5          |\n| $\\tau_b$ = 30%              | 0.5      | 53.3                | 59.5          | 49.4                | 45.9          | 65.0                | 40.1          | 37.8          |\n|------------------------|----------|---------------------|---------------|---------------------|---------------|---------------------|---------------|---------------|\n| **CROWD ($\\tau_b$)**  | 10%      | 57.9                | 61.7          | 53.1                | 46.8          | 69.2                | 41.3          | 37.4          |\n|                               | **30%**  | 57.9                | 61.7          | 53.6                | 47.8          | 69.6                | 42.4          | 38.5          |\n| $\\tau_e$ = 0.2                  | 50%      | 55.4                | 60.0          | 50.0                | 42.1          | 63.7                | 38.9          | 34.5          |\n\n\nWe appreciate the reviewer’s comment on the societal impact of CROWD. We agree that object detection is a largely applied technique in real-world scenarios and OWOD helps object detectors continually adapt to new settings. We will include this in the questionnaire in the immediate next revision."}, {"review_id": "ZRttx9uhRU", "reviewer": "Reviewer_5m2P", "summary": "The paper proposes a novel Open World Object Detection (OWOD) approach named CROWD, which innovatively decomposes the OWOD task into two stages: CROWD-D and CROWD-L. By maximizing the SCG function, the method sequentially selects candidate regions that significantly differ from the known class feature representations as background and unknown objects, thereby effectively enhancing the detection capability for unknown objects. Furthermore, by minimizing the clustering overlap between the embeddings of known and unknown objects, the method mitigates confusion between known and unknown categories. Overall, CROWD demonstrates substantial innovation and advancement in the recognition of unknown classes compared to prior work.", "questions": "See weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.   This paper reformulates unknown object discovery and adaptation as an interwoven combinatorial data-discovery and representation learning task. This idea has novelty\n2.   CROWD has significantly improved the recall rate of unknown objects compared to previous methods\n\n\nWeaknesses ：\n \n1.\tThe CROWD-D module for unknown class discovery appears to exhibit a strong dependence on the annotations of known classes. In this framework, both background and unknown object pseudo-labels are derived based on their similarity to the feature representations of known classes. Under conditions where the annotations and feature diversity of known categories are limited, the reliability of unknown candidate selection，driven by the maximization of the SCG function may be compromised. Moreover, variations in the distribution of known classes could potentially impact the robustness of the model in discovering unknown categories.\n\n2.\tThe formulation and notations used in the paper could benefit from better consistency. Specifically, the connection between the SCG function introduced earlier and its usage in later sections could be clarified further. Additionally, the symbols used in the loss function formulation should be explained in more detail for better understanding.", "was_revised": true, "final_justification": "Thank you for the patient response. I maintain the positive score.", "rebuttal": "**[W1] “The CROWD-D module …”** - We agree with the reviewer that the selection of $U^t$ in CROWD-D is dependent on the known $K^t$. This dependence on $K^t$ has been a common trend in OWOD since its induction through ORE (even in latest SoTA paper OrthogonalDet) since examples $K^t$ are the only source of reliable labeled examples (often used as anchors). We would also like to point the reviewer to Gupta et al., 2022 (OW-DETR) who introduced a super-class separated benchmark (S-OWODB in our paper) ensuring varying distributions across tasks (a scenario in question) where CROWD demonstrates superior performance over SoTA methods. As shown by the authors of PRISM (Kothawade et al., 2022) for auxiliary tasks, submodular maximization of SCG shows exemplary performance in mining rare/unseen instances where the size of the query set is very small compared to the ground set. This is because SCG relies on gain in information in contrast to simple distance/similarity measures used in other approaches. Our work extends this property to OWOD tasks with modifications discussed in Sec. 3.3.\n\n**[W2] “The formulation …”** - We understand the concern of the reviewer. Due to lack of space in the main paper we provided the key property of SCG functions alone in Sec. 3.2 relevant to the formulation of CROWD in Sec. 3.3 and beyond. However, we would like to point the reviewer to Sec. A.2 of the Appendix alongside Sec. 3.2 (main paper) where we detail out the preliminaries of submodular functions and the mathematical modeling of SCG supporting the property of feature dissimilarity that SCG models upon submodular maximization (also detailed in Algorithm 2 of Appendix). This property is consistently used across CROWD-D and CROWD-L with differences only in the optimization strategy. In CROWD-D (a selection formulation) we maximize SCG using Greedy Optimization (Nemhauser etal., 1978) to select examples in $U^t$ dissimilar to $K^t$ while in CROWD-L (specifically in $L_{\\text{CROWD}}^{\\text{cross}}(\\theta)$) SCG is maximized using SGD to model separation (dissimilarity) between $K^t \\cup B^t$ and $U^t$ (mined in CROWD-D). \n\nAdditionally, for the notations used in the loss formulation of CROWD-L we would like to point the reviewer to Table 6 in the Appendix where we explicitly call out the definitions of each term used in our formulation. As suggested by the reviewer we would be happy to include this in the main paper to improve readability."}]

Author rebuttal:

Meta-review: This paper tackles the challenges of semantic confusion and catastrophic forgetting in the problem of open-world object detection. It introduces the CROWD framework, which combines combinatorial data discovery and representation learning. The former selects representative unknowns distinct from known classes, while the latter employs combinatorial objectives to disentangle representations and preserve discriminative power. Experiments show notable gains from the framework. 

This paper received comments from five reviewers. The main concerns before the rebuttal are reflected in (1) the threshold involved in the framework; (2) some empirical performance gains are marginal; (3) algorithm details and implementation are not clear enough; (4) the ablation studies are not sufficient to understand the key contributions of the work. The authors provided detailed feedback, and all reviewers reached a consensus. 

The AC has carefully reviewed the paper and the reviewers’ comments, and considers this to be a solid and meaningful work. The authors are encouraged to incorporate the reviewers’ constructive feedback in the final version to further strengthen the clarity and impact of this work.
Decision: Accept (poster)