Title: Looking Beyond the Known: Towards a Data Discovery Guided Open-World Object Detection
Venue: NeurIPS 2025
Accepted: yes
Keywords: Open-World Object Detection

Data-Discovery

Representation Learning

Submodular Functions
Abstract: Open-World Object Detection (OWOD) enriches traditional object detectors by enabling continual discovery and integration of unknown objects via human guidance. However, existing OWOD approaches frequently suffer from semantic confusion between known and unknown classes, alongside catastrophic forgetting, leading to diminished unknown recall and degraded known-class accuracy. 
To overcome these challenges, we propose **C**ombinato**r**ial **O**pen-**W**orld **D**etection (**CROWD**), a unified framework reformulating unknown object discovery and adaptation as an interwoven combinatorial (set-based) data-discovery (CROWD-Discover) and representation learning (CROWD-Learn) task. CROWD-Discover strategically mines unknown instances by maximizing Submodular Conditional Gain (SCG) functions, selecting representative examples distinctly dissimilar from known objects. Subsequently, CROWD-Learn employs novel combinatorial objectives that jointly disentangle known and unknown representations while maintaining discriminative coherence among known classes, thus mitigating confusion and forgetting. Extensive evaluations on OWOD benchmarks illustrate that CROWD achieves improvements of 2.83% and 2.05% in known-class accuracy on M-OWODB and S-OWODB, respectively, and nearly 2.4$\times$ unknown recall compared to leading baselines.

Looking Beyond the Known: Towards a Data Discovery Guided Open-World Object Detection

Abstract

Open-World Object Detection (OWOD) enriches traditional object detectors by enabling continual discovery and integration of unknown objects via human guidance. However, existing OWOD approaches frequently suffer from semantic confusion between known and unknown classes, alongside catastrophic forgetting, leading to diminished unknown recall and degraded known-class accuracy. To overcome these challenges, we propose C ombinato r ial O penW orld D etection ( CROWD 2 ), a unified framework reformulating unknown object discovery and adaptation as an interwoven combinatorial (set-based) data-discovery (CROWD-Discover) and representation learning (CROWD-Learn) task. CROWD-Discover strategically mines unknown instances by maximizing Submodular Conditional Gain (SCG) functions, selecting representative examples distinctly dissimilar from known objects. Subsequently, CROWD-Learn employs novel combinatorial objectives that jointly disentangle known and unknown representations while maintaining discriminative coherence among known classes, thus mitigating confusion and forgetting. Extensive evaluations on OWOD benchmarks illustrate that CROWD achieves improvements of 2.83% and 2.05% in known-class accuracy on M-OWODB and S-OWODB, respectively, and nearly 2.4 × unknown recall compared to leading baselines.

Currently

knownobjects

Unknownobjects

Known

Y

Kt

detected,

othersmissed

minedandstored

Bt

Targeted

Detector

Object

Selection

(a)CROWD-Discover(CROWD-D)mines unknown exemplarsfromlow confidencebackgroundobjects.

Person.

Ut

Unknowns

Unknown

DMOUNUO

Object

KnownKt

Detector

Bothknownand

objectsdetected

unknown

(b)CROWD-Learn(CROWD-L)learnsdistinct embeddingsofknownandunknownobjects

1 Introduction

Object Detection (OD) is central to numerous vision applications [35, 47, 30], but as shown in Figure 1 (left) conventional OD models operate under a closed-world assumption, where the object vocabulary remains fixed throughout deployment. This limitation hinders the model's ability to generalize to novel object categories. Open-World Object Detection (OWOD), introduced by Joseph et al. [21], addresses this by combining open-set recognition [63, 74, 65] with incremental learning [37, 12], enabling models to detect unknown objects and subsequently recognize them with minimal supervision, thus supporting continual self-improvement. However, recent efforts [64, 77, 61] reveal

two enduring challenges: (1) confusion between known and unknown objects [21, 15, 44], and (2) catastrophic forgetting of previously learned classes [61, 64, 45]. Confusion arises due to visual similarity between unknown and known classes (e.g., truck vs. car headlamps), while forgetting stems from the lack of supervision for unknowns, causing them to be misclassified as background. Existing methods struggle to address both issues, as evidenced by low unknown recall and high Wilderness Impact scores [21] (Table 3). These limitations motivate the need for a framework that can effectively discover unknown Region-of-Interests (RoIs) and learn representations that remain distinct from known categories, thereby mitigating confusion and preserving prior knowledge.

We cast Open-World Object Detection (OWOD) as a set-based discovery and learning problem. For each task t , we view the known object classes as a collection of sets K t , group all candidate unknowns into a single pseudo-labeled set U t [21], and treat everything else as background B t . This formulation (Section 3.1) facilitates the incorporation of submodular functions into OWOD and gives rise to our C ombinato r ial O penW orld D etection ( CROWD ) framework. As illustrated in Figure 1, CROWD tackles OWOD as an interleaved process of data discovery (CROWD-D) and representation learning (CROWD-L), directly targeting the dual challenges of confusion and forgetting.

Starting from a OD trained only on known instances K t (Figure 1 3 ), CROWD-Discover (CROWD-D) identifies representative unknowns, formulated as a combinatorial targeted selection problem. Specifically, CROWD-D maximizes the Submodular Conditional Gain (SCG, Section 3.2) between K t and candidate subsets, encouraging dissimilarity with known and background objects. Authors in [34] provide theoretical guarantees that greedy maximization [52] of SCG results in selection of samples in U t which are dissimilar to K t as well as B t , constituting informative pseudo-labeled unknowns.

CROWD-Learn (CROWD-L) then fine-tunes the OD model on both known and mined unknowns as shown in Figure 1(b) via a novel combinatorial joint objective (Equation (1)), rooted in two families of submodular functions: SCG [34] and Total Submodular Information [11] (SIM). Maximizing SCG increases diversity between known and unknown objects, reducing feature overlap and confusion, as corroborated by Table 5. Conversely, minimizing SIM encourages intra-class cohesion within each known object, preserving discriminative features and mitigating forgetting. This formulation closely follows the observation in [48, 49] that submodular functions model cooperation [20] and diversity [38] when minimized and maximized respectively. Finally, we instantiate a family of submodular-based loss functions within CROWD-L that jointly reduce confusion and forgetting, achieving notable gains in unknown recall and known-class accuracy (Table 5). We validate our approach on two standard OWOD benchmarks, M-OWOD [21] and S-OWOD [15], demonstrating its effectiveness across diverse open-world settings. Our main contributions are -

CROWD introduces a novel combinatorial viewpoint in OWOD by modeling the identification of unknown instances of a given task as a data discovery problem (CROWD-D), selecting unknown RoIs which maximize the SCG between and the known object instances.

CROWD also introduces a novel set-based learning paradigm CROWD-L, based on SCG functions which minimizes the cluster overlap between embeddings of known and unknown objects while retaining the discriminative feature information from the known ones.

Finally, CROWD demonstrates ∼ 2.4 × increase in unknown recall per task alongside up to 2.8% improvement on M-OWODB and 2.1% improvement on S-OWODB in known class performance (measured as mAP) over several existing OWOD baselines.

2 Related Work

Open-World Object Detection (OWOD) first introduced in Joseph et al. [21] augmented a Faster R-CNN [58] model with contrastive clustering and an Energy-Based Unknown Classifier relying on a objectness threshold based pseudo labeling strategy. Subsequent work such as OW-DETR [15] adapted deformable DETR [76] and proposed an attention-based pseudo-labeling scheme that identifies high-activation regions as unknowns without requiring extra supervision. Further, CAT [43] improves transformer-based models by decoupling localization and classification, while introducing dual pseudo-labeling strategies, namely - model-driven and input-driven-to robustly mine unknowns. PROB [77] advanced the state of the art by modeling objectness probabilistically in the embedding

space using a Gaussian likelihood, allowing better separation of unknowns from background without explicit negative examples. Other notable works include 2B-OCD [68], which integrates a localization-based objectness head [28], OCPL [73] enforces class-prototype separation to reduce known-unknown confusion, and UC-OWOD [69] employs feature-space regularization to suppress background misclassification. Some recent methods leverage external supervision, e.g., MViTs [46], or multimodal cues such as text for class-agnostic detection, these often fall outside strict OWOD assumptions but highlight promising directions for future research. Complementary to these, recent approaches such as RandBox [64] sidesteps detection bias via random bounding box sampling and dynamic-k filtering, while OrthogonalDet [61] enforces angular decorrelation in object features to disentangle objectness and class semantics. Interestingly, Randbox and OrthogonalDet outperforms larger models like OW-DETR, UC-OWOD etc. while using a simpler Faster-RCNN [58] based architecture. Despite substantial progress, OWOD methods continue to grapple with confusion between known and unknown objects and catastrophic forgetting during incremental adaptation recently evidenced in Xi et al. [70], motivating the development of our CROWD framework.In general our work is also related to standard object detection while CROWD-Discover (Section 3.3.1) is related to combinatorial subset selection, the related work for which is provided in Appendix A.2.

3 Method

3.1 Problem Definition: OWOD

We largely adopt the problem formulation of OWOD from Joseph et al. [21] with modifications towards a combinatorial (set-based) formulation. Given an incoming task T t where t ∈ [1 , n ] , an object detector h t ( . ; θ ) recognizes a set of known classes K t = { K t 1 , K t 2 , . . . , K t C k } , | K t | = C k while also accounting for unknown classes U t that may appear during inference (classes in U t are not labeled during training). Here, K t i , i ∈ [1 , C k ] indicates examples for each known class in T t . The dataset D t = { ( x t i , y t i ) } i M =1 for each task T t , where each label y t i contains K object instances ( K can very for each image) defined by bounding box parameters y t k = [ c k , x k , y k , w k , h k ] , with c k ∈ [1 , C k ] representing the class label. The object detection model h t ( . ; θ ) is trained to learn newly introduced instances from labeled examples in K t while identifying unknown objects U t by assigning them a placeholder label ( 0 ). Examples in U t can be reviewed by a human expert who identifies C u new classes, allowing the model to update incrementally and produce h t +1 ( . ; θ ) without retraining on the entire dataset. If ˆ U t indicate the newly labeled set of unknown classes s.t. | ˆ U t | = C u , then K t +1 = K t ∪ ˆ U t , enabling continual adaptation to new object categories over time.

3.2 Preliminaries: Submodularity

Adopting a set-based formulation allows us to explore combinatorial functions for OWOD. In particular we explore Submodular functions which are set functions exhibiting a unique diminishing returns property. Formally, a function f : 2 V → R defined on a ground set V is submodular if for any subset A i , A j ⊆ V , it holds that f ( A i ) + f ( A j ) ≥ f ( A i ∪ A j ) + f ( A i ∩ A j ) [11]. These functions have been widely studied for applications such as data subset selection [27, 34, 19], active learning [67, 33, 2, 25], and video summarization [24, 26]. Typically, these tasks involve formulating subset selection or summarization as submodular maximization subject to a knapsack constraint. A classic result guarantees a (1 -e -1 ) approximation factor [53] using a greedy algorithm, which can be implemented more efficiently via improved greedy strategies [53, 52]. Within this framework, Submodular Information Functions (SIMs) [18, 17, 3], such as Facility-Location or Graph-Cut, promote diversity when maximizing f ( A ) . On the other hand, Submodular Conditional Gain (SCG) [17], H f ( A i | A j ) , captures elements in A i most dissimilar to A j . Extrapolating this, Kothawade et al. [32] defines discovery of unseen, rare examples as a targeted selection problem. Further works [48, 49] have demonstrated the utility of these combinatorial functions in continuous optimization. Majee et al. [48] introduces losses inspired by SIMs to enforce intra-group compactness (when minimized) and inter-group separation (when maximized), while [49] uses SMI-based losses to account for interactions between abundant and rare samples in few-shot learning. Motivated by these insights, CROWD proposes a novel data-discovery guided representation learning framework to dynamically identify and incrementally adapt to unknown objects .

()

Kt

Kt

Subsample

t-1

Kt

Bt

To Next Task (Tt+1)

Rolfeatures

Rol features

LCROWD(0)

From Previous

ht(.;0)

-->

ht(.;0)

?

ht+1(.;0)

[-+2

Lcls(0)+Lbox(0)

CROWDdetects

Targeted

bothknownand

Selection

unknownobjects

detected

BG

Known

(a)CROWD-Discover

(b)CROWD-Learn

3.3 The CROWD Framework

The problem formulation in Section 3.1 surfaces two unique challenges in the domain of OWOD -(1) How to identify instances of unlabeled unknown objects U t given labeled examples of only known ones in K t ? (2) How to effectively learn representations of currently known objects without forgetting the previously known (classes introduced in T i , where i < t ) ones ?

To this end, we introduce C ombinato r ial O penW orld D etection ( CROWD ) framework, which models the OWOD task as a interleaved set-based data discovery [32] and representation learning [48] problem. CROWD achieves this in two stages - namely CROWD-Discover (a.k.a. CROWDD) and CROWD-Learn (a.k.a. CROWD-L) as shown in Figure 2. Given an incoming task T t we first train h t ( . ; θ ) on currently known classes in D t . At this point, CROWDD utilizes the frozen model weights of h t and uses a small replay buffer (typically containing examples from both previously known and currently introduced objects) to discover highly representative proposals of unknown classes U t . We elucidate this in

Algorithm 1 Discovering Unknown RoIs in CROWD-D

Require: A task t , set of RoI feature vectors R ∈ R N × d , Objectness scores S ( . ) ∈ R N , Task specific Labels y K t , budget k 1: /** Identify and Exclude outliers **/ 2: R ←{ r ∈ R | S ( r ) ≥ τ e } 3: K t ← HUNGARIAN-MATCHING ( R , y K t ) ▷ Known class RoI features 4: V ← R \ K t 5: /** Select Background Samples **/ 6: B t ← arg max B t ⊆V | B t |≤ τ b % |V| H f ( B t | K t ) ▷ Large feature separation from K t 7: /** Select Unknown samples from R \ K t **/ 8: U t ← arg max U t ⊆V , | U t |≤ k H f ( U t | K t ∪ B t ) ▷ Unknowns are different from K t ∪ B t 9: return U t

Section 3.3.1. Subsequently, CROWD-L introduces a novel combinatorial learning strategy to rapidly finetune h t on this replay buffer (we adopt the predefined buffer in Joseph et al. [21]) to distinguish between known classes K t and unknown U t while preserving distinguishable features from the previously known classes. We discuss this in detail in Section 3.3.2.

3.3.1 CROWD-Discover

During training of h t ( . ; θ ) , label information is available only for the currently known classes K t with no labels of the previously known K t -1 and the unknown classes U t . CROWD-D tackles the challenge of identifying potentially unknown instances from Region-of-Interest (RoI) proposals produced by the Region-Proposal-Network (RPN) in h t ( . ; θ ) . Unlike existing OWOD methods employ pseudo labeling [21], feature orthogonalization [61] etc. rely on the objectness score (probability of an RoI proposal to contain a foreground object), whereas CROWD-D achieves this by modeling this task as a combinatorial data discovery problem [32]. Given a set of RoI proposals

Labeled

Kt

Kt

R\Kt

Bt

Ut

Unknown(U)instances

are dissimilartoKtas

well as Bt.

Background(Bt)Rols are farthest

fromKtin thefeature space.

R and a submodular function f we define the data discovery task (Algorithm 1) as a targeted selection problem which selects a set of unknown instances U t from V = R \ K t that maximizes the SCG H f given a query set comprising of known K t and the background B t instances (line 8 in Algorithm 1) .

Here, k denotes a budget for the number of unknown samples mined per image. From the definition of SCG in Section 3.2 selected examples in U t are largely dissimilar to examples in K t ∪ B t indicating that they are neither background objects nor visually similar to known objects as shown in Figure 3(c). Interestingly, the number of known RoIs K t in R are significantly fewer than the background RoIs B t (typically | R | = 500 (total number of RoIs from the RPN) whereas | K t | ∼ 10 in MS-COCO [40] which leaves | B t | ∼ 490 ) in most OD models. To minimize the computation costs while selecting U t from this large RoI pool we exclude all RoIs with low objectness scores S < τ e (line 2 in Algorithm 1) and likely background objects B t predicted with high confidence (line 6 of Algorithm 1). Instances in B t are selected which maximize the SCG between themselves and known RoIs K t under a budget constraint of τ b % |V| (samples that are significantly different from K t ). Known instances are identified by following the hungarian matching technique applied in Wang et al. [64] as shown in line 3 of Algorithm 1. The exclusion thresholds τ e and τ b are empirically determined to be 0.2 and 30% respectively and the underlying submodular function in our experiments is chosen to be Graph-Cut which has been evidenced in Kothawade et al. [34] to model both representation and diversity among selected examples.

3.3.2 CROWD-Learn

Including unknown examples U t can potentially inject noisy labels into the training data with detrimental effects. We show in Table 5 that CROWD-D alone does not handle the knowledge retention from previously known classes K t -1 , despite significant improvements on unknown class recall, causing forgetting. CROWD-L overcomes the aforementioned challenges by introducing a novel Combinatorial representation learning strategy inspired from recent developments [48, 49], that ensures orthogonality (separation) between embeddings in K t and U t while minimizing the effect of forgetting of ˆ K t -1 . Here, ˆ K t -1 denotes a replay buffer of previously known classes.

Given a set of known K t ∪ ˆ K t -1 , unknown U t classes alongside a submodular function f we define a learning objective L CROWD ( θ ) as shown in Equation (1) which jointly minimizes the Submodular Total Information ( L self CROWD ) over each known class K t i ∈ { K t ∪ ˆ K t -1 } and the SCG ( H f as defined in Section 3.2) between known class K t i and the unknown set U t ( L cross CROWD ) . Note that CROWD-L is applied during training of task T t as a finetuning step as shown in Figure 2(b).

Note that f relies on the pairwise interaction between examples in a batch which we represent using cosine similarity s ku ( θ ) = h t ( x k ,θ ) T · h t ( x u ,θ ) || h t ( x k ,θ ) ||·|| h t ( x u ,θ ) || and can be different from the one used in CROWDD decided through ablations in Section 4.2. Our loss formulation in L CROWD follows the observation in [48] which entails that submodular functions model cooperation [20] and diversity [38] when

Case 1

Case 2

Case 3

Kt

Value

3.0

CROWD-GC

6

Ut

6

6

CROWD-FL

Loss

2.5

CROWD-LD

Normalized

2.0

2

2

2

1.5

1.0

0

0

0

0

2

4

0

2

4

0

2

4

Kt vs. Ut Separation

2

3

4

minimized and maximized respectively. By varying the choice of f between popular submodular functions - Facility-Location (FL), Graph-Cut (GC) and Log-Determinant (LogDet) we introduce a family of loss functions summarized in Table 1 and derivations in Appendix A.3. L CROWD is applied to the classification head of h t ( . ; θ ) model during all training stages described in Sec. 4. Our novel formulation entails some interesting properties -

L self CROWD retains informative known class features : Following the insights in Jegelka and Bilmes [20] L self CROWD which minimizes the total information contained in K t i encourages intra-class compactness retaining the most discriminative features from the known classes alleviating forgetting [49].

L cross CROWD models known vs. unknown separation : As shown in Equation (1) L cross CROWD minimizes the SCG ( H f ) between the embeddings of the known and unknown classes. As observed in Kothawade et al. [34] maximizing SCG models dissimilarity between two sets. In OWOD, L cross CROWD promotes a large inter-class boundary between K t i and U t , minimizing their cluster overlap resulting in reduced confusion . Further, a hyper-parameter η controls the trade-off between intra-class compactness modeled by L self CROWD and the inter-class separation in L CROWD.

Sensitivity to unknown classes with varying f : Table 1 highlights the instances of L CROWD by varying f . Such variations injects domain specific properties into CROWD-L critical for the OWOD tasks. As depicted in Figure 4 for L cross CROWD , under varying known vs. unknown class separation, CROWD-FL explicitly models representation [34] by adopting the FL based submodular function while CROWD-LogDet injects diversity by modeling cluster volume through Log-Determinant [18]. CROWD-GC models both representation and diversity but is not resilient to imbalance between K t and U t as CROWD-FL [48]. Thus, CROWD-FL emerges to be a suitable choice for OWOD modeling both representation and resilience to imbalance.

Generalization to Incremental Object Detection (IOD) : As observed in several related works [21, 15, 61, 77], learning from unknown pseudo labels in OWOD benefits Incremental Object Detection (IOD) tasks. However, its important to note that CROWD relies on mined unknowns (from CROWD-D) while no unknown objects are provided in the IOD setting. This requires us to slightly modify our learning objective L cross CROWD to model dissimilarity between currently known K t and the previously known objects ˆ K t -1 (a replay buffer as in Joseph et al. [21]) instead of unknowns, s.t. L cross CROWD ( θ ) = -∑ C t i =1 H f ( K t i | ˆ K t -1 ; θ ) .

4 Experiments

Datasets : We evaluate our approach on two well established benchmarks M-OWOD [21] and S-OWOD [15]. M-OWOD, ( Superclass-Mixed OWOD Benchmark ) consists of images from both MS-COCO [40] and PASCAL-VOC [10] depicting 80 classes grouped into 4 tasks (20 classes per task). On the other hand, S-OWOD ( Superclass-Separated OWOD Benchmark ) consists of images from only MS-COCO dataset. Both benchmarks split the underlying data points into four distinct (non-overlapping) tasks T t , where t ∈ [1 , 4] . During training on a task T t the model in provided labeled examples from T t alone while at inference the model is expected to identify objects in tasks leading up to T t , s.t t ∈ [1 , t ] . No prior knowledge of subsequent tasks t ∈ [ t +1 , n ] ( n refers to maximum number of tasks in an experiment) are available during training and inference on T t . In contrast to M-OWODB, S-OWODB introduces a distinct separation between super-categories (eg. animals, vehicles etc.) and distributes these super-categories between tasks (each task will have examples from one or more unique super-categories).

Experimental Setup : Following Sun et al. [61] we adopt a Faster-RCNN [58] based model with a pretrained ResNet-50 [16] backbone. Our model is trained incrementally on 4 tasks as described above with a batch size of 12, an AdamW optimizer, a base learning rate to 2 . 5 × 10 -5 and weight decay of 1 × 10 -4 . CROWD-D utilizes the RoI features ( | R | = 500 ) to mine k = 10 unknown instances (determined through ablation study in Appendix A.4.1) per image. The CROWD-L loss is applied across tasks as an additional head and operates on RoI features projected to a 256-dimensional feature space. We train our model on 4 NVIDIA V100 GPUs, provide additional experimental details in Appendix A.4 and release our code at https://github.com/amajee11us/CROWD.git .

Metrics : We use mean average precision (mAP) to evaluate known classes, partitioned into previously seen and newly introduced categories. For unknown object class, we follow OWOD conventions [21, 15] and report unknown object recall (U-Recall), as mAP is inapplicable due to incomplete annotations. To measure confusion between known and unknown classes, we report Wilderness Impact (WI) [7] and Absolute Open-Set Error (A-OSE) [51].

4.1 Results on Benchmark OWOD and IOD tasks

OWOD : We compare the performance of CROWD against several existing baselines on MOWOD and S-OWOD benchmarks as shown in Table 2. Note, that we follow Sun et al. [61] and report our results on the same seed and compute settings for fair comparisons.

CROWDsurpasses the latest baseline OrthogonalDet [61] by up to 2.8% and 2.1% on M-OWOD and S-OWOD benchmarks while achieving up to 2.4 × gains in U-recall. For approaches like PROB [77], CAT [44] which adopt selection strategies to mine unknowns our combinatorial approach achieves up to 8.4% (on M-OWOD) improvements. This can be attributed to the contributions of CROWD-D which mines representative unknown examples effectively increasing the coverage on such objects. Also, we observe ∼ 3 %increase in mAP for previously known classes indicating a reduction in forgetting. The competitive results on the currently known classes (Curr. in Table 2) indicates that h t ( . ; θ ) enforces a stronger decision boundary between K t and U t through L cross CROWD while retaining performance on K t through L self CROWD . Additionally, in Table 3 we show that CROWD achieves lesser confusion over existing baselines while boosting U-Recall establishing the importance of modeling OWOD as a combinatorial data-discovery problem. This is further highlighted qualitatively in Figure 5.

IOD : Our novel loss formulation described in Section 3.3.2 (point 4) is applied to the finetuning stage of IOD across three popular task splits from the PASCALVOC [10] dataset. Note, that for IOD we do not apply CROWD-D due to absence of unknown examples. Our

results summarized in Table 4 and detailed in Table 9 (Appendix) shows up to 5.9% boost in overall mAP showing better generalization to IOD tasks while minimizing the impact of forgetting via stronger retention of previously known classes , a very common pitfall in IOD.

4.2 Ablations

We conduct ablations on the M-OWOD benchmark to analyze the contributions of individual components of CROWD. On top of the baseline method OrthogonalDet [61] we first introduce instances of CROWD-D to assess the impact of data-discovery under a fixed budget k = 10 . Next, we decouple CROWD-D and introduce our novel learning objectives in CROWD-L to assess their impact on forgetting and confusion as discussed in Section 3.3.2. For each of the above steps we ablate among instances of f - Graph-Cut (GC), Log-Determinant (LogDet) and Facility-Location (FL). Finally, we combine the best performing instances from CROWD-D and CROWD-L into a joint formulation (referred to as CROWD (joint)) as shown in Table 5 which achieves the best

OrthogonalDet [61]

CROWD (ours)

(a)

(b)

(c)

MAM

L4851A5

DUS

HLC851A5

overall performance, balancing the tradeoff between boosting currently known class performance and retaining performance on previously learnt ones .

Impact of Data-Discovery in CROWD-D : As shown in Table 5, irrespective of the choice of f , CROWD-D boosts the U-Recall over the baseline by introducing additional information in the form of pseudo labeled unknowns. We observe that CROWD-D (w/GCCG) ( f here is Graph-Cut) provides the best gains in U-Recall up to 2 × over the latest baseline OrthogonalDet. This follows the observation in Kothawade et al. [34] which shows that greedy maximization of GCCG models relevance (examples which are dissimilar to both K t and U t ) while others model diversity (CROWD-D w/LogDet) and representation (CROWD-D w/Facility-Location). Thus, we adopt GCCG based selection strategy in Algorithm 1 for our experiments in Table 2.

Impact of k : As stated in Section 3.3.1, k controls the number of potential unknown RoIs identified by CROWD-D per image. We ablate among several plausible values of k ∈ [0 , 100] and summarize the results in Table 10 of the Appendix. Increasing the number of identified unknowns from 0 (OrthogonalDet) to 10 shows an increase in performance of the underlying model (U-Recall) while the performance does not increase beyond 20. The increase in U-Recall can be attributed to inclusion of informative RoIs in the training loop. In fact, the mAP on known classes slightly drops below existing baselines for k = 100 due to inclusion of spurious background RoIs in the training pipeline.

Impact of Combinatorial Objectives in CROWD-L : Similar to CROWD-D we ablate on variations of f to contrast between formulations summarized in Table 1. As shown in Table 5 our learning formulation, particularly CROWD-FL (based on Facility-Location) demonstrates better retention of previously known class performance while achieving competitive results on latest baseline OrthogonalDet. This follows the observation in Majee et al. [48] which demonstrates that FL based objectives model representation, retaining the most discriminative features through L self CROWD while enforcing sufficient inter-cluster boundary between known and unknown RoI features ( L cross CROWD ). This also re-establishes the properties described in Figure 4 wherein CROWD-FL shows larger sensitivity to inter-cluster separation as compared to CROWD-GC, CROWD-LogDet and L decorr introduced in OrthogonalDet.

Ablation on Exclusion Criterion τ e and τ b in CROWD-D - At first, τ e is an exclusion threshold which reduces the search space of CROWD-D by eliminating RoIs which have a low confidence threshold. As shown in Table 6, increasing τ e from 0 to 1 increases performance until τ e = 0 . 2 and then reduces. A lower value of τ e allows for a large search space but includes a lot of noisy background objects leading to reduced selection performance. On the other hand a large value of τ e can potentially earmark unknown foregrounds as unknowns resulting in reduced performance.

Keeping τ e fixed at 0.2 we ablate τ b which controls the selection budget for backgrounds (higher the value more are the number of background RoIs identified). Increasing τ b (percentage here) increases the fraction of RoIs treated as backgrounds. This widens the search space for the combinatorial function causing a small drop in performance due to confusions between true backgrounds and foreground unknowns. On the other hand, very large values of τ b shrink the search space oftentimes considering unknown foregrounds as background objects showing a steep drop in performance.

Ablation on Trade-off between L self CROWD and L cross CROWD in CROWD-L - The hyper-parameter η controls the trade-off between known-unknown class separation and known class cluster compactness discussed in Table 7. A lower value of η does not enforce separation between currently known and unknown exemplars but enforces intra-class compactness. This results in better retention of previously known objects but a drop in currently known objects due to increased confusion with unknown exemplars. On the other hand for a large value of η the model enforces large separation between currently known and unknown objects boosting performance on the currently knowns but suffers from catastrophic forgetting of the previously known classes.

5 Conclusion, Limitations and Future Work

We introduced CROWD, a novel combinatorial framework in OWOD, which reformulates OWOD as interleaved set-based discovery (CROWD-D) and representation learning (CROWD-L) tasks. Leveraging Submodular Conditional Gain (SCG) functions, CROWD-D strategically selects representative unknown instances distinctly dissimilar from known objects while CROWD-L consumes mined unknowns to preserve discriminative coherence over known classes. Our evaluations confirm that CROWD effectively addresses known vs. unknown class confusion and forgetting, achieving significant improvements in unknown recall and known-class accuracy on standard OWODand IOD benchmarks. Despite improvements in U-Recall, operating under a fixed budget CROWD-D injects some spurious exemplars into the selected unknown pool (particularly in images with no known unknown objects). We aim to address this in future works by exploring alternative combinatorial formulations beyond SCG, and introducing stricter constraints in CROWD-D.

References

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems , volume 32, 2019.

Nathan Beck, Durga Sivasubramanian, Apurva Dani, Ganesh Ramakrishnan, and Rishabh K. Iyer. Effective evaluation of deep active learning on image classification tasks. ArXiv , abs/2106.15324, 2021.

Jeff Bilmes. Submodularity in machine learning and artificial intelligence. arXiv preprint arXiv:2202.00132 , 2022.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision - ECCV 2020 , 2020.

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. In International Conference on Learning Representations , 2020.

Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1601-1610, 2021.

Akshay Dhamija, Manuel Gunther, Jonathan Ventura, and Terrance Boult. The overlooked elephant of object detection: Open set. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , March 2020.

Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6568-6577, 2019.

Melanie Ducoffe and Frederic Precioso. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09841 , 2018.

M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision , 111(1):98-136, January 2015.

Satoru Fujishige. Submodular Functions and Optimization , volume 58. Elsevier, 2005.

Takuma Fukuda, Hiroshi Kera, and Kazuhiko Kawamoto. Adapter merging with centroid prototype mapping for scalable class-incremental learning. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) , pages 4884-4893, 2025.

Ross Girshick. Fast R-CNN. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 1440-1448, 2015.

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR '14, page 580-587, 2014.

Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. OW-DETR: Open-world detection transformer. In CVPR , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016.

Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning. In Algorithmic Learning Theory , pages 722-754. PMLR, 2021.

Rishabh Iyer, Ninad Khargonkar, Jeff Bilmes, and Himanshu Asnani. Generalized submodular information measures: Theoretical properties, examples, optimization algorithms, and applications. IEEE Transactions on Information Theory , 68(2):752-781, 2022.

Eeshaan Jain, Tushar Nandy, Gaurav Aggarwal, Ashish V. Tendulkar, Rishabh K Iyer, and Abir De. Efficient data subset selection to generalize training across models: Transductive and inductive networks. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS) , 2023.

Stefanie Jegelka and Jeff Bilmes. Submodularity beyond submodular energies: Coupling edges in graph cuts. In CVPR 2011 , 2011.

K J Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , page 5826-5836, jun 2021.

K. J. Joseph, Jathushan Rajasegaran, Salman Khan, Fahad Shahbaz Khan, and Vineeth N. Balasubramanian. Incremental object detection via meta-learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):9209-9216, 2022.

Athresh Karanam, Krishnateja Killamsetty, Harsha Kokel, and Rishabh Iyer. ORIENT: Submodular mutual information measures for data subset selection under distribution shift. In Advances in Neural Information Processing Systems , volume 35, 2022.

V. Kaushal, R. Iyer, K. Doctor, A. Sahoo, P. Dubal, S. Kothawade, R. Mahadev, K. Dargan, and G. Ramakrishnan. Demystifying multi-faceted video summarization: Tradeoff between diversity, representation, coverage and importance. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 452-461, 2019.

Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) , 2019.

Vishal Kaushal, Sandeep Subramanian, Suraj Kothawade, Rishabh Iyer, and Ganesh Ramakrishnan. A framework towards domain specific video summarization. In 2019 IEEE winter conference on applications of computer vision (WACV) , pages 666-675. IEEE, 2019.

Krishnateja Killamsetty, Guttu Sai Abhishek, Aakriti, Ganesh Ramakrishnan, Alexandre V. Evfimievski, Lucian Popa, and Rishabh Iyer. AUTOMATA: gradient based data subset selection for compute-efficient hyper-parameter tuning. In Proceedings of the 36th International Conference on Neural Information Processing Systems , 2024.

Dahun Kim, Tsung-Yi Lin, Anelia Angelova, In So Kweon, and Weicheng Kuo. Learning open-world object proposals without learning to classify. IEEE Robotics and Automation Letters (RA-L) , 2022.

Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is NP-Hard. In Proceedings of the 37th International Conference on Machine Learning , ICML'20, 2020.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Proceedings of the 38th International Conference on Machine Learning , 2021.

Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. SIMILAR: Submodular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems , 34, 2021.

Suraj Kothawade, Shivang Chopra, Saikat Ghosh, and Rishabh Iyer. Active data discovery: Mining unknown data using submodular information measures, 2022.

Suraj Kothawade, Saikat Ghosh, Sumit Shekhar, Yu Xiang, and Rishabh K. Iyer. Talisman: Targeted active learning for object detection with rare classes and slices using submodular mutual information. In Computer Vision - ECCV 2022 - 17th European Conference , 2022.

Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff A. Bilmes, and Rishabh K. Iyer. PRISM: A rich class of parameterized submodular information measures for guided data subset selection. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI , pages 10238-10246, 2022.

Suraj Kothawade, Atharv Savarkar, Venkat Iyer, Ganesh Ramakrishnan, and Rishabh Iyer. Clinical: Targeted active learning for imbalanced medical image classification. In Medical Image Learning with Limited and Noisy Data: First International Workshop, MILLanD 2022, Held in Conjunction with MICCAI , 2022.

Changbin Li, Suraj Kothawade, Feng Chen, and Rishabh Iyer. PLATINUM: Semi-supervised model agnostic meta-learning using submodular mutual information. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 12826-12842. PMLR, 17-23 Jul 2022.

Linhao Li, Yongzhang Tan, Siyuan Yang, Hao Cheng, Yongfeng Dong, and Liang Yang. Adaptive decision boundary for few-shot class-incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 18359-18367, 2025.

Hui Lin and Jeff Bilmes. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , 2011.

Hui Lin and Jeff Bilmes. Learning mixtures of submodular shells with application to document summarization. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence , UAI'12, 2012.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision - ECCV 2014 , Cham, 2014. Springer International Publishing.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages 2999-3007, 2017.

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV , 2016.

Shuailei Ma, Yuefeng Wang, Jiaqi Fan, Ying yu Wei, Thomas H. Li, Hongli Liu, and Fanbing Lv. Cat: Localization and identification cascade detection transformer for open-world object detection. In CVPR , 2023.

Shuailei Ma, Yuefeng Wang, Ying Wei, Jiaqi Fan, Thomas H. Li, Hongli Liu, and Fanbing Lv. Cat: Localization and identification cascade detection transformer for open-world object detection. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19681-19690, 2023.

Yuqing Ma, Hainan Li, Zhange Zhang, Jinyang Guo, Shanghang Zhang, Ruihao Gong, and Xianglong Liu. Annealing-based label-transfer learning for open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11454-11463, 06 2023.

Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang. Class-agnostic object detection with multi-modal transformer. In European Conference on Computer Vision (ECCV) . Springer, 2022.

Anay Majee, Kshitij Agrawal, and Anbumani Subramanian. Few-Shot Learning For Road Object Detection. In AAAI Workshop on Meta-Learning and MetaDL Challenge , volume 140, pages 115-126, 2021.

Anay Majee, Suraj Nandkishor Kothawade, Krishnateja Killamsetty, and Rishabh K Iyer. SCoRe: Submodular combinatorial representation learning. In Proceedings of the 41st International Conference on Machine Learning , volume 235, pages 34327-34349, 2024.

Anay Majee, Ryan Sharp, and Rishabh Iyer. Smile: Leveraging submodular mutual information for robust few-shot object detection. In European Conference on Computer Vision (ECCV) , 2024.

Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , 2021.

Dimity Miller, Lachlan Nicholson, Feras Dayoub, and Niko Sünderhauf. Dropout sampling for robust object detection in open-set conditions. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , 2018.

Baharan Mirzasoleiman, Ashwinkumar Badanidiyuru, Amin Karbasi, Jan Vondrak, and Andreas Krause. Lazier than lazy greedy. Proceedings of the AAAI Conference on Artificial Intelligence , 29(1), 2015.

G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical Programming , 14(1):265-294, 1978.

Patrik Okanovic, Roger Waleffe, Vasilis Mageirakos, Konstantinos Nikolakakis, Amin Karbasi, Dionysios Kalogerias, Nezihe Merve Gürel, and Theodoros Rekatsinas. Repeated random sampling for minimizing the time-to-accuracy of learning. In The Twelfth International Conference on Learning Representations , 2024.

Can Peng, Kun Zhao, and Brian C Lovell. Faster ILOD: Incremental learning for object detectors based on faster rcnn. Pattern Recognition Letters , 2020.

Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6517-6525, 2016.

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection, 2015.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR , 2018.

Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In ICCV , 2017.

Zhicheng Sun, Jinghan Li, and Yadong Mu. Exploring orthogonality in open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1730217312, 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30, 2017.

Hongjun Wang, Sagar Vaze, and Kai Han. Dissecting out-of-distribution detection and open-set recognition: A critical analysis of methods and benchmarks. International Journal of Computer Vision (IJCV) , 2024.

Yanghao Wang, Zhongqi Yue, Xian-Sheng Hua, and Hanwang Zhang. Random boxes are open-world object detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6233-6243, 2023.

Yu Wang, Junxian Mu, Pengfei Zhu, and Qinghua Hu. Exploring diverse representations for open set recognition. AAAI, 2024.

Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, and Jeff Bilmes. Submodular subset selection for large-scale speech training data. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2014.

Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In ICML , 2015.

Yan Wu, Xiaowei Zhao, Yuqing Ma, Duorui Wang, and Xianglong Liu. Two-branch objectness-centric open world detection. In Proceedings of the 3rd International Workshop on Human-Centric Multimedia Analysis , 2022.

Zhiheng Wu, Yue Lu, Xingyu Chen, Zhengxing Wu, Liwen Kang, and Junzhi Yu. Uc-owod: Unknownclassified open world object detection. In Computer Vision - ECCV 2022 , page 193-210, 2022.

Xing Xi, Yangyang Huang, Zhijie Zhong, and Ronghua Luo. UMB: Understanding model behavior for open-world object detection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Binbin Yang, Xi Deng, Han Shi, Changlin Li, Gengwei Zhang, Hang Xu, Shen Zhao, Liang Lin, and Xiaodan Liang. Continual object detection via prototypical task correlation guided gating mechanism. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9245-9254, 2022.

Shuo Yang, Peize Sun, Yi Jiang, Xiaobo Xia, Ruiheng Zhang, Zehuan Yuan, Changhu Wang, Ping Luo, and Min Xu. Objects in semantic topology. In The Tenth International Conference on Learning Representations, ICLR , 2022.

Jinan Yu, Liyan Ma, Zhenglin Li, Yan Peng, and Shaorong Xie. Open-world object detection via discriminative class prototype learning. In 2022 IEEE International Conference on Image Processing (ICIP) , pages 626-630. IEEE, 2022.

Zhenyu Zhang, Guangyao Chen, Yixiong Zou, Yuhua Li, and Ruixuan Li. Learning unknowns from unknowns: Diversified negative prototypes generator for few-shot open-set recognition. In Proceedings of the 32nd ACM International Conference on Multimedia , pages 6053-6062, 2024.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable {detr}: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations , 2021.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations , 2021.

Orr Zohar, Kuan-Chieh Wang, and Serena Yeung. PROB: Probabilistic objectness for open world object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11444-11453, June 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We call out our contributions in brief in the abstract and enlist them on page 2 at the end of the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Section 5 of the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We define CROWD-L and CROWD-D in Section 3.3.1 and Section 3.3.2 respectively. Instances of CROWD-L (novel to CROWD) are derived based on Theorems A.3.1 through A.3.3 with proofs in Sec. A.3 of the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We release the anonymized version of our code at https://github.com/ amajee11us/CROWD.git which houses the evaluation metrics following the popular Detectron2 framework ( https://github.com/facebookresearch/detectron2 ).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We evaluate our work on MS-COCO [40] and PASCAL-VOC [10] which are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We detail out the experimental settings in Section 4 with additional details in Appendix A.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide statistical significance experiments on key components of CROWD contrasted against State-of-the-Art approach OrthogonalDet in Table 5. For every task T t in the OWOD benchmark we run our experiments on three different randomly generated seed values (kept constant across methods) keeping other parameters constant and report results in Table 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We run all our experiments on 4 NVIDIA V100 GPUs with an approximate training time of 36 hours for all tasks. This is described in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: For our research and experiments, we use public datasets and models that have been used in prior published works. Our research does not involve direct interaction with human subjects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Object detection is extensively used in real world scenario. As a broad impact, the proposed work would be meaningful in scenarios for such real-world scenario (where setting is often open-world) and model has to continually adapt to new tasks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: Safeguards would fall outside the scope of our research and we mainly focus on subjects that are inanimate. No experiments explicitly use personally identifiable features of humans as a subject.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper relies on several existing benchmarks which are cited and credited in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The key asset as a result of this paper is the code base where we provide execution instructions.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not crowdsource images by ourselves; we use images from publicly available datasets MS-COCO and PASCAL-VOC which has been vetted through.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: There is no crowdsourcing involved in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A Appendix

A.1 Notation

Following the problem definition in the main paper we introduce the notations used in Table 8 throughout the paper.

A.2 Additional Related Work

Data subset selection aims at identifying a distinct set of examples from a large pool which accurately captures the properties of the data distribution. This has rendered subset selection to be a natural choice for data-efficient machine learning tasks like Active Learning [31, 50, 59, 9], Continual Learning [29, 1], Data Summarization [31, 34] etc. Traditionally subset selection has been defined as a subsampling technique based on similarity [23], uncertainty [5] etc. or random [54]. Orthogonally, a new line of work based on combinatorial functions, particularly submodular functions [11, 18] have emerged which effectively selects informative subsets by modeling the notions of cooperation, diversity and representation [34]. These functions formulate subset selection as a greedy maximization task [52] based on several information theoretic measures like Total Information, Mutual Information, Conditional Gain etc. (discussed in Section 3.2 of the main paper). Concurrent to their success in vision [34, 31], language [39], speech [66] etc. domains,

subset selection has been used in auxiliary learning mechanisms like meta-learning [36] and data-discovery [32] targeting identification of rare or unseen examples from an unlabeled example pool. CROWD exploits this line of investigation adopting a combinatorial subset selection technique (detailed in Section 3.3.1 to discover unknown objects in the open-world setting).

Object detection (OD) is a fundamental task in computer vision encapsulating both localization and recognition tasks under the same roof. OD methods are traditionally grouped into two principal paradigms: single-stage and two-stage detectors. Single-stage detectors, exemplified by SSD [42], RetinaNet [41], and YOLO [57, 56], CenterNet [8] unify the processes of object localization and classification into a single feed-forward network, enabling real-time performance with relatively low computational overhead. In contrast, two-stage detectors, such as Faster R-CNN [14, 13, 58], adopt a cascaded architecture wherein a Region Proposal Network (RPN) first hypothesizes candidate object regions, followed by a refinement stage that simultaneously predicts the class and precise bounding box of each proposal. CNN based architectures struggles with the long-range dependencies, which is important for understanding the complex spatial relationships between objects at varying scales (perspective views). Transformer based models [4, 75, 6] improve upon this vulnerability by introducing a self-attention [62] mechanism based on an encoder-decoder architecture [4]. While these models achieve impressive performance in closed-world settings (all object categories present during testing are known and predefined in the training data) they under-perform in open-world scenarios when encountering unknown objects unseen during training.

Preliminaries of Submodularity (continued from Section 3.2) As discussed in Section 3.2 of the main paper, submodular functions have been recognized to model notions of cooperation [20], diversity [38], representation [34] and coverage [24]. Following the combinatorial formulation in Section 3.1 of the main paper we define the ground set V = { A 1 , A 2 , · · · A N } , s.t. |V| = N and explore four different categories of submodular information functions in our work, namely -

(1) Submodular Total Information ( S f ) which measures the total information contained in each set [11], expressed as S f ( A 1 , A 2 , . . . , A N ) as in Equation (2). Maximizing S f over a set A i models diversity [38] while minimizing S f models cooperation [20].

(2) Submodular Conditional Gain ( H f ) which models the gain in information when a set A j is added to A i . H f models the notion of dissimilarity between sets and can be expressed in Equation (3).

Given a submodular function f (can alternatively be H f ) tasks like selection [19, 27] and summarization [26, 24] have been modeled as a discrete optimization problem to identify a summarized set of examples A ⊆ V via submodular maximization under a cardinality constraint ( | A | ≤ k ), i.e. max A ⊆V , | A |≤ k f ( A ) . This can be fairly approximated with a (1 -e -1 ) constant factor guarantee [53] using greedy optimization techniques [52] as shown in Algorithm 2. Extending the definition of submodular functions to continuous optimization space Majee et al. [48] have proposed a set of novel family of learning objectives which minimize total information and total correlation among sets in D train using continuous optimization

Algorithm 2 Greedy Submodular Maximization [53]

Require: Submodular function f : 2 V → R , cardinality constraint k

Ensure: Set A ⊆ V maximizing f ( A ) under cardinality constraint k

techniques like SGD. These objectives have been shown to be significantly more robust to large imbalance demonstrated in real-world tasks like longtail recognition [48] and few-shot learning [49].

A.3 Derivations of Instances of L CROWD

As discussed in Section 3.3.2 of the main paper, varying the choice of Submodular function f in Equation (1) results in several instances of L CROWD. Based on three popular choices of f among

Facility-Location, Graph-Cut and Log-Determinant, we derive the respective formulations of L CROWD. Note, that the derivations of L self CROWD are adapted from Majee et al. [48] and are thus not included below.

A.3.1 Derivation of CROWD-FL

Theorem A.1. Given a set of known RoIs K t i , i ∈ [1 , C t ] , a set of unknown RoIs U t ( T = K t ∪ U t ) and the Facility-Location based submodular function f defined over any set A s.t. f ( A ) = ∑ i ∈T max j ∈ A s ij , we define CROWD-FL learning objective to learn the parameters θ of the model h t , containing two components L self CROWD and L cross CROWD as shown in Equation (4) . Here, s i j resembles the similarity between samples i and j respectively.

Proof. From the definition of L cross CROWD in Equation (1) we find,

Substituting the definition of f ( A ) over any set from the theorem in the above expression we get -

The Term 2 in the above equation controls the degree of separation between K t i and U t . Due to this we introduce a hyper-parameter ν which we can control during model training. Sine ν is a constant it does not affect the submodular properties of L cross CROWD . The final loss formulation, normalized by the size of T thus becomes -

Additionally, we do not provide proofs for L self CROWD since this function largely resembles the total information formulation in Majee et al. [48].

A.3.2 Derivation of CROWD-GC

Theorem A.2. Given a set of known RoIs K t i , i ∈ [1 , C t ] , a set of unknown RoIs U t ( T = K t ∪ U t ) and the Graph-Cut based submodular function f defined over any set A s.t. f ( A ) = ∑ i ∈T ∑ j ∈ A s ij -λ ∑ i,j ∈ A s ij , we define CROWD-GC learning objective to learn the parameters θ of the model h t containing two components L self CROWD ( θ ) and L cross CROWD ( θ ) as shown in Equation (8) . Here, s i j resembles the similarity between samples i and j respectively.

Proof. From the definition of L cross CROWD in Equation (1) we find,

Substituting the definition of f ( A ) over any set from the theorem in the above expression of L cross CROWD we get -

The second term and the fourth term cancels out (same value with opposite signs).

Now, rearranging the terms of the equation we get -

Similar to CROWD-FL the Term 2 in the above equation controls the degree of separation between K t i and U t . Due to this we introduce a hyper-parameter ν which we can control during model training. Sine ν is a constant it does not affect the submodular properties of L cross CROWD . The final loss formulation, normalized by the size of T thus becomes -

Additionally, we do not provide proofs for L self CROWD since this function largely resembles the total information formulation in Majee et al. [48].

A.3.3 Derivation of CROWD-LogDet

Theorem A.3. Given a set of known RoIs K t i , i ∈ [1 , C t ] , a set of unknown RoIs U t ( T = K t ∪ U t ) and the Log-Determinant based submodular function f defined over any set A s.t. f ( A ) = log det( s A ) , we define CROWD-LogDet learning objective which contains two components L self CROWD and L cross CROWD as shown in Equation (14) . Here, s i j resembles the similarity between samples i and j respectively.

Proof. From the definition of L cross CROWD in Equation (1) we find,

Substituting the definition of f ( A ) over any set from the theorem in the above expression of L cross CROWD we get -

From Schur's complement which states that given two sets A and B det( s A ∪ B ) = det( s A ) . det( s A ∪ B \ s A ) . Replacing the term det( s K t i ∪ U t ) with the above definition we get -

Following Schur's complement yet again which states that s A ∪ B \ s A = s B -s T A,B s -1 A s A,B , where s A,B refers to the cross-similarities between sets A and B while s A and s B represent the corresponding self-similarities and substitute this definition into the aforementioned equation as -

Normalizing this term with the size of the ground set |T | and introducing the hyper-parameter ν which trades-off between inter-cluster separation and intra-cluster compactness, we derive the function for L cross CROWD as -

Similar to previously derived objectives, we do not provide proofs for L self CROWD since this function largely resembles the total information formulation in Majee et al. [48].

A.4 Additional Experimental Details

In this section we provide additional experimental details for training our CROWD approach on M-OWOD, S-OWOD and IOD benchmarks discussed in Section 4 of the main paper.

M-OWODand S-OWOD benchmarks - M-OWOD and S-OWOD benchmarks are created from MS-COCO [40] and split into 4 tasks T t , where t ∈ [1 , 4] detailed in the "Datasets" section in Section 4. For each task, the model in provided labeled examples from T t alone while at inference the model is expected to identify objects in tasks leading up to T t , s.t t ∈ [1 , t ] . We split the training

into two splits. In the first stage the model is exposed only to the currently known classes K t and the learnt model h t biases on labeled examples in K t . At the end of the first stage CROWD-D kicks in and selects representative unknowns as described in Section 3.3.1. Lets call in U t . Next, we store a replay buffer of the currently known objects ˆ K t , s.t. ˆ K t ⊆ K t . Following this, we combine ˆ K t , K t and a replay buffer from the previous task ˆ K t -1 into a single dataset to finetune h t using CROWD-L. As detailed in Section 3.3.2 this ensures known vs. unknown separation wile retaining discriminative features from known classes.

IOD benchmarks - In contrast to OWOD, IOD does not encounter unknowns during model training but experiences heavy catastrophic forgetting on previously known classes K t -1 . Following recent benchmarks like Sun et al. [61], Zohar et al. [77], Joseph et al. [21] we evaluate the IOD performance of CROWD on PASCAL-VOC benchmark on three settings produced by varying the number of newly added classes - 10 + 10, 15 + 5, 19 + 1 as shown in Table 9. In the absence of unknowns we do not apply CROWD-D and only rely on CROWD-L applied to the finetuning stage of IOD. Following latest works we adopt a replay based learning technique whichh stores a small subset of the previously known objects ˆ K t -1 in a buffer. ˆ K t -1 combined with the newly introduced classes K t is used to finetune h t . This also requires us to slightly modify the formulation of L cross CROWD as detailed in Section 3.3.2. For each setting h t is trained on a batch size of 12 for 3000 iterations using an AdamW optimizer, a base learning rate to 2 . 5 × 10 -5 and weight decay of 1 × 10 -4 .

A.4.1 Ablation on Selection Budget k

As detailed in Section 3.3.1, the parameter k dictates how many candidate unknown RoIs CROWD-D selects per image. We conduct an ablation over several plausible settings of k within the interval [0 , 100] , and present the outcome in Table 10. Fo this experiment we keep the choice of submodular function f in CROWD-D as Graph-Cut and Facility-Location (CROWD-FL) for CROWD-L following the results of the ablation experiments in Table 5 in the main paper. Notably, raising k from 0 (i.e., OrthogonalDet) to 10 yields a marked uplift in the model's unknown-recall (U-Recall), yet further increases beyond k = 20 confer no additional gains. This initial boost in U-Recall stems from the integration of truly informative RoIs into the training loop. However, when k reaches its upper bound of 100, the mean average precision (mAP) on known classes experiences a slight decline relative to existing baselines-a consequence of inadvertently incorporating spurious background proposals.

A.4.2 Results on Synthetic Datasets - CROWD-D

In addition to the illustrations provided in Figure 3 we contrast the selection performance of CROWDD by varying the underlying submodular function f between Grap-Cut (GC), Facility-Location (FL) and Log-Determinant (LogDet) on synthetic datasets as shown in Figure 6. The use of synthetic datasets provide us with complete control over the embedding space allowing us to pathologically inject imbalance, inter-cluster separation etc. in a compute efficient fashion. Particularly in our experiments we use a two-cluster imbalanced setup mimicking the RoI embedding space in FasterRCNN [58] model. Similar to Sun et al. [61] the number of known class and unknown class feature vectors are severely imbalanced with total number of RoIs R = 500 and the number of knowns | K t | = 10 . R and K t are sampled from a normal distribution with fixed variance values. The LogDet based selection strategy enforces the notion of diversity in the selection mechanism which does not select representative unknowns negatively impacting OWOD performance as shown in Table 5. The

GCCG

FLCG

LogDetCG

(a) RoI Feature Space R

(c) Unknowns Ut

Kt

R\Kt

Bt

Ut

(a) RoI Feature Space R

(c) Unknowns Ut

Kt

R\Kt

Bt

(c) Unknowns Ut

Kt

R\Kt

Bt

Ut

FL based selection strategy models representation as shown in Figure 6 alone during selection resulting in erroneous selection of background instances negatively affecting OWOD performance. Lastly, GC based selection strategy shown in Figure 6 models notions of both diversity and representation selecting diverse backgrounds B t farthest to K t as well as representative unknowns U t . This results in GC based selection strategy to produce the best overall results as shown in Table 2.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel Open World Object Detection (OWOD) approach named CROWD, which innovatively decomposes the OWOD task into two stages: CROWD-D and CROWD-L. By maximizing the SCG function, the method sequentially selects candidate regions that significantly differ from the known class feature representations as background and unknown objects, thereby effectively enhancing the detection capability for unknown objects. Furthermore, by minimizing the clustering overlap between the embeddings of known and unknown objects, the method mitigates confusion between known and unknown categories. Overall, CROWD demonstrates substantial innovation and advancement in the recognition of unknown classes compared to prior work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.   This paper reformulates unknown object discovery and adaptation as an interwoven combinatorial data-discovery and representation learning task. This idea has novelty\n2.   CROWD has significantly improved the recall rate of unknown objects compared to previous methods\n\n\nWeaknesses ：\n \n1.\tThe CROWD-D module for unknown class discovery appears to exhibit a strong dependence on the annotations of known classes. In this framework, both background and unknown object pseudo-labels are derived based on their similarity to the feature representations of known classes. Under conditions where the annotations and feature diversity of known categories are limited, the reliability of unknown candidate selection，driven by the maximization of the SCG function may be compromised. Moreover, variations in the distribution of known classes could potentially impact the robustness of the model in discovering unknown categories.\n\n2.\tThe formulation and notations used in the paper could benefit from better consistency. Specifically, the connection between the SCG function introduced earlier and its usage in later sections could be clarified further. Additionally, the symbols used in the loss function formulation should be explained in more detail for better understanding.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper deals with the problem of classifying known objects and detecting unknown objects. The tasks are continually discovered, and the model has to adapt to new tasks, giving rise to two challenges: catastrophic forgetting and confusion among known/unknown objects. The proposed method tackles the problem by proposing two major components. First component discovers the known and unknown objects by leveraging Submodular Conditional Gain and second components learns the representation of known/unknown objects. Experiments on benchmarks reveal effectiveness of the proposed method for the task of Open-World Object Detection.

STRENGTHS AND WEAKNESSES:
**Strength**

- The paper tackles the challenging but important problem of continual object detection in open-world setting. 
- The design of the proposed components is well motivated: data discovery for each category, then learning representation. 
- Ablations on impact of each component illustrates the importance of the design.
- Performance is compared against substantial number of baselines. 

**Weakness**

- Missing study of the impact of each component of the proposed loss function (hyperparameter η). 
- The paper mentions how thresholds τe and τb are determined. Could authors show the details?

QUESTIONS:
See weakness section

LIMITATIONS:
YES for limitations

For societal impact, authors have given answer of [NA]. I suggest the following as the societal impact of the work.

Object detection are extensively used in real world scenario. The proposed work would be meaningful in scenarios for such real-world scenario (where setting is often open-world) and model has to continually adapt to new tasks.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses two major bottleneck challenges in Open World Object Detection, namely confusion between known and unknown objects, and catastrophic forgetting of previous learned classes. This paper proposes a novel framework called CROWD (Combinatorial Open World Detection), innovatively reformulating the OWOD problem as an interleaved task of combinatorial optimization-based data discovery (CROWD-Discover) and representation learning (CROWD-Learn). The paper utilizes the submodular conditional gain function to effectively mine representative unknown instances that are significantly different from known classes from unlabeled data. The paper employs different novel combinatorial objective functions to decouple the features of known and unknown classes while maintaining the discriminative consistency within known classes, providing new ideas for the above two key issues. Experimental results show that this method achieves remarkable results in standard OWOD benchmark tests, significantly improving the recall rate of unknown objects and the detection accuracy of known classes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper is dedicated to solving two important problems encountered in open-world object detection and proposes a novel and interesting approach. The article has clear logic and sufficient innovation.\n2. The comparative experiments in the article are very rich, which fully verify the feasibility of the framework proposed in the article.\n3. The ablation experiments are rigorously designed, fully analyzing the contributions of each innovative module. The paper not only demonstrates the superiority of the overall solution but also verifies the necessity and effectiveness of each component through a detailed ablation study, making the sources of the method's advantages clear and the argumentation process very solid.\n\nWeaknesses\n1. The paper's experiments on the sensitivity of hyperparameters involved in the network structure are not enough sufficient. For example, the article mentions a small replay buffer, but does not conduct relevant experimental analysis on specific settings such as the buffer size and sampling strategy.\n2. When comparing different SCG functions, although the paper selects a combination with relatively good results, it lacks a comparison of the performance of these SCG functions on different datasets,such as S-OWOD. The robustness of the model mentioned in the article still needs further relevant experiments to prove.\n\nQUESTIONS:\n1. There seems to be some ambiguity between the selection of Bt and Ut in the framework proposed in the article described in Figure 1 and the subsequent description.\n2. Whether using the replay buffer to solve the problem of catastrophic forgetting can only alleviate it to a certain extent or completely solve it. If a large number of new categories are added, will the problem still occur due to the fixed size of the buffer? \n3. Have any experiments been conducted to verify the method of combining the designed loss function with the replay buffer? How should it be set to achieve the best effect? What’s the best settings of the replay buffer?\n4.  The paper selected a set of submodular function combinations with the best performance for the CROWD framework through experiments on the M-OWOD benchmark. However, it failed to verify whether this set of function combinations is also the optimal choice on other benchmarks with different data characteristics (such as S-OWOD), and no relevant experiments on the robustness of this combination were provided.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method named CROWD (CombinatoRial Open-World Detection) for detecting known and unknown (semantic) object classes in 2D camera images. The method first trains an object detector in a supervision setting to detect known object classes. After that, the predictions of the trained detector are used to discover unknown objects in the “background”. The filtered background objects and known class instances are then used to train the detector again. The paper claims significantly improved recall for detecting unknown objects and also improves detection performance for known classes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Paper claims to improve recall for detecting unknown object class instances by 2.4 and also improves detection performance for known object class instances.\n2. The considered task is relevant in the current AI field, i.e. a label-efficiënt method is proposed as a object detector is trained to detect object classes that where not part of the training dataset labels.\n\nWeaknesses\n1. The first part of the title is very general “Looking Beyond the Known”. It basically states the same as “Open-World Object Detection” which is in the second part of the title. It would better to name the method name in the title or to put the characteristics of the proposed method in the title.\n2. The method name CROWD is chosen a bit randomly, the “r” is just one of the letters of “CombinatoRial”.\n3. Line 27 is unclear. It is unclear whether “previously learned classes” are about “classes that were unknown” or about “known classes”. In line 29, it seems that “unknown” classes were meant.\n4. Lines 30-31 refer to the Wilderness Impact scores in Table 2 but these cannot be found in Table 2.\n5. The symbols introduced in lines 35-36 give the impression of an exponent. Subscript would be better.\n6. The second part of the introduction is mainly a methodology. Even some sections of the methodology are named. The last part of the introduction (58-75) is very vague / hard to understand. Last sentence of line 75 contains typo, it should be “Our main contributions are:”.\n7. Three contributions are stated but the third one is just empirical support for the first two contributions (both are designed components for the proposed method).\n\nQUESTIONS:\n1. Why is the selected orange bounding box for the chair in Figure 1 a different size than the red bounding box, which is part of the “background” bounding boxes. By just selecting a bounding box, the size should remain the same, right?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for Open-World Object Detection (OWOD) named Combinatorial Open-World Detection (CROWD), which consists of two tightly coupled components: CROWD-Discover for targeted unknown instance mining, and CROWD-Learn for submodular objective-based representation learning. Unlike previous approaches that rely on objectness heuristics or clustering, CROWD formalizes OWOD as a combinatorial problem using submodular conditional gain (SCG) and submodular information measures (SIM). This formulation aims to both identify informative unknown instances and ensure robust feature disentanglement, thereby mitigating known-unknown confusion and catastrophic forgetting. The method achieves state-of-the-art performance across multiple OWOD benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n1. The paper articulates the inherent limitations of existing OWOD approaches, particularly in semantic confusion and forgetting, and motivates the need for a more principled and theoretically grounded framework. Modeling OWOD as a combinatorial data-discovery and learning task is a novel and theoretically sound perspective. The use of submodular functions introduces strong guarantees and interpretability.\n2. The method demonstrates significant gains in unknown recall and competitive improvements in known-class accuracy on both M-OWOD and S-OWOD benchmarks, validating its effectiveness.\n3. The paper is well-organized, with a clear separation of components (CROWD-D and CROWD-L), detailed methodology, and informative figures and tables that aid understanding.\n\n### Weaknesses:\n1. Lines 191–194 describe heuristics (e.g., filtering by objectness score and subset selection) to reduce computational cost. However, the actual computational complexity or runtime comparison to baselines is not discussed. Since CROWD introduces combinatorial subset selection, this may impact scalability. Could the authors provide complexity analysis or empirical runtime comparisons?\n2. The framework heavily relies on two thresholds, τₑ (objectness threshold) and τ_b (background selection budget). Although empirically set (0.2 and 30%), the sensitivity of performance to these hyperparameters is not systematically analyzed. How robust is the system to variations in τₑ and τ_b? Could performance degrade significantly if these are misconfigured?\n3. The paper positions CROWD-D as a more principled alternative to heuristic pseudo-labeling (e.g., ORE, PROB), yet concrete distinctions remain somewhat abstract. What is the core theoretical or empirical advantage of CROWD-D over attention-based or objectness-based mining (e.g., OW-DETR, CAT)? In real-world settings where unknowns and backgrounds may share features, how does CROWD-D avoid misidentifying complex background as unknown?\n4. As shown in Table 5, the standalone gain from CROWD-L is relatively modest (~0.4–1.2% mAP over baseline), especially considering the additional modeling complexity. Could the authors elaborate on why the representation learning stage has limited standalone impact? Is it because CROWD-L primarily complements CROWD-D, or is the performance bounded by the quality of mined unknowns?\n\nQUESTIONS:\n1. Include runtime/memory benchmarks or FLOPs to better quantify the computational trade-offs of CROWD compared to baselines.\n2. Provide a sensitivity analysis for thresholds τₑ and τ_b (e.g., a heatmap or performance curve).\n3. Add qualitative failure cases where CROWD-D misidentifies background as unknown, to better understand its limitations in complex scenes.\n2. In Figure 2, superscripts and subscripts are not very clear. Could u further explain them?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}