OpenCon: Open-world Contrastive Learning
Authors: Yiyou Sun
Yixuan Li
Venue: Accepted by TMLR ()

Abstract: Machine learning models deployed in the wild naturally encounter unlabeled samples from both known and novel classes. Challenges arise in learning from both the labeled and unlabeled data, in an open-world semi-supervised manner. In this paper, we introduce a new learning framework, open-world contrastive learning (OpenCon). OpenCon tackles the challenges of learning compact representations for both known and novel classes and facilitates novelty discovery along the way. We demonstrate the effectiveness of OpenCon on challenging benchmark datasets and establish competitive performance. On the ImageNet dataset, OpenCon significantly outperforms the current best method by 11.9% and 7.4% on novel and overall classification accuracy, respectively. Theoretically, OpenCon can be rigorously interpreted from an EM algorithm perspective—minimizing our contrastive loss partially maximizes the likelihood by clustering similar samples in the embedding space. The code is available at https://github.com/deeplearning-wisc/opencon.

Published in Transactions on Machine Learning Research (1/2023)
OpenCon: Open-world Contrastive Learning
Yiyou Sun
sungiyou Wws.wisc.. edu sharonli@cs. wisc. edu
Yixuan Li University of Wisconsin-Madiann
Reviewed on OpenReview: /ttps://openreveeee net/ forum? id= BuWJutpFer
Abstract
Machine learning models deployed in the wild naturally encounter unlabeled samples from both known and novel classes. Challenges arise in learning from both the labeled and unlabeled data, in an open-world semi-supervised manner. In this paper, we introduce a new learning framework, open-world contrastive learning (OpenCon). OpenCon tackles the challenges of learning compact representations for both known and novel classes, and facilitates novelty discovery along the way. We demonstrate the effectiveness of OpenCon on challenging benchmark datasets and establish compelitive performance. On the ImageNet dataset, OpenCon significantly outperforms the current best method by 11.9% and 7.4% on novel and overall classification accuracy, respectively. Theoretically, OpenCon can be rigorously interpreted from an EM algorithm perspective minimizing our contrastive loss partially maximizes the likelihood by clustering similar samples in the embedding space. The code is available https://github: wtwp:/hwwb.ccc//eelplaaannnnnnea wisciiipniiiccc/ccoooooo
1
Introduction
Modern machine learning methods have achieved remarkable success (Sun al.
2017 Van den Oord et al.
2018
Chen et al.)
2020a
Caron et al., 2020 He al. 2020 Zheng al., 2021 Wu al.) 2021 Cha al,
2021
Cui et al. 2021
Jiang al. 2021 Gao et al. 2021 Zhong al. 2021a Zhao & Han 2021 Fini
et al. 2021 Tsai et al. 2022 Zhang al.] 2022b Wang et al.) 2022). Noticeably, the vast majority of
learning algorithms have been driven by the wloled-world setting, where the classes are assumed stationary and unchanged. This assumption, however, rarely holds for models deployed in the wild. One important characteristic of open world is that the model will naturally encounter novel classes. Considering a realistic scenario, where a machine learning model for recognizing products in e-commercc may encounter brand-new products together with old products. Similarly, an autonomous driving model can run into novel objects on the road, in addition to known oncs. Under the setting, the model should ideally learn to distinguish not only the known classes, but also the novel categories. This problem is proposed as open-world semi-supervised learning CCao et al. 2022) generalized category discovery (Vaze et al. 2022). Research efforts have only started very recently to address this important and realistic problem.
Formally, we are given a labeled training dataset D; as well as an unlabeled dataset D.. The labeled dataset contains samples that belong a set of known classes, while the unlabeled dataset has a mixture of samples from both the known and novel classes. In practice, such unlabeled in-the-wild data can be collected almost for free upon deploying a model in the open world, and thus is available in abundance. Under the setting, our goal is to learn distinguishable representations for both known and novel classes simultaneously. While this setting naturally suits many real-world applications, it also poses unique challenges due to: (a) the lack of clear separation between known vs. novel data in Da, and (b) the lack of supervision for data in novel classes. Traditional representation learning methods are not designed for this new setting. For example, supervised contrastive learning (SupCon) (Khosla al., 2020) only assumes the labcled set D;, without considering the unlabeled data D.. Weakly supervised contrastive learning (Zheng et al. (021) assumes the same classes in labeled and unlabeled data, hence remaining closed-wol world and less generalizable to novel

---

Published in Transactions on Machine Learning Research (1/2023)
Labeled Data Di
Unlabeled Data Du
Prototype-based Learning
Novelty Recognition
Positive Set Selection
Prototype Update
Clusters
Open-world Contrastive Learning
Notation: Unlabeled Data
Data in Known Classes
Unlabeled Protolype
Prototype for Known Classes
Data in Novel Classes
Prototype for Novel Classes
Figure 1: Illustration of our learning framework Open-world Controstive Learning (OpenCon). The model is trained on a labeled dataset DD of known classes, and an unlabeled datasel D, (with samples from both known and novel classes). OpenCon aims to learn distinguishable representations for both known (blue and green) and novel (yellow and orange) classes simultaneously. See Section 3 for details.
samples. Self-supervised learning (Chen et al. 2020a) relies completely on the unlabeled set Dy and does not utilize the availability of the labeled dataset Dj.
Targeting these challenges, we formally introduce new learning framework, open-world contrastive learning (dubbed OpenCon). OpenCon is designed to produce a compact representation space for both known and novel classes, and facilitates novelty discovery along the way. Key to our framework, we propose a novel prototype-basd based learning strategy, which encapsulates two components. First, we leverage the prototype vectors to soparate known vs. novel classes unlabeled data D.. The prototypes can be viewed as set of representative embeddings, one for each class, and are updated by the evolving representations. Second, to mitigate the challenge of lack of supervision, we generate pseudo-positive pairs for contrastive comparison. We define the positive set to be those examples carrying the same approximated label, which is predicted based on the closest class prototype. In effect, the loss encourages closely aligned representations to all samples from the same predicted class, rendering a compact clustering of the representation.
Our framework offers several compelling advantages. (1) Empirically, OpenCon establishes strong performance on challenging benchmark datasets, outperforming existing basclines by a significant margin (Section 5
OpenCon is also competitive without knowing the number of novel classes in aivaance-acieveee similar even slightly better performance compared to the oracle (in which the number of classes is given). (2) Theoretically, we demonstrate that our prosolype-based learning can be rigorously interpreted from Expectation-Maxiine Maximization (EM) algorithm perspective. (3) Our framework is end-to-end trainable, and is compatible with both CNN-based and Transformer-basod architectures. Our main contributions are:
We propose a novel framework, open-world contrastive learning (OpenCon). tackling a largely unexplored problem in representation lcarning. As an integral part of our framework, we also introduce a prolotype-based learning algorithm, which facilitates novelty discovery and learning distinguishable representations.
Empirically, OpenCon establishes competitive performance on challenging tasks. For example, on the ImageNet dataset, OpenCon substantially outperforms the current best method ORCA (Cao
et al. 2022 by 11.9% and 7.4% in terms of novel and overall accuracy.
We provide insights through extensive ablations, showing the effectivencss of components in our frame- work. Theoretically, we show formal connection with EM algorithm- minimizing our contrastive loss partially maximizes the likelihood by clustering similar samples in the embedding space.
2

---

Published in Transactions on Machine Learning Research (1/2023)
Table 1: Comparison of different problem settings.
Unlabeled data
Problem Setting
Labeled data
Known classes
Novel classes
Semi-supervised learning
Yes
Yes
No
Robust suni-supervieed learning
Yes
Yes
Yes (Reject)
Supervised learning
Yes
No
No
Novel class discovery
Yes
No
Yes (Discover)
Open-world Semi-supervised learning
Yes
Yes
Yes (Cluster)
2
Problem Setup
The open-world setting emphasizes the fact that new classes can emerge jointly with existing classes in the wild. Formally, we describe the data setup and learning goal:
Data setup We consider the training dataset D Di U Du with two parts:
1. The labeled set D; {xi: Mi with yı 34. The label set 3/2 is known.
" The unlabeled set D, {x where each sample x; E X can come from either known or novel classed Note that we do not have access to the labels in Du. For mathematical convenience, we denote the underlying label set as Vall, where Yi с Vall implies category shift and expansion. Accordingly, the set of novel classes is Vn = YankOh where the subscript n stands for "novel". The model has no knowledge of the set Vn nor its size.
Goal Under the setting, the goal is to learn distinguishable representations for both known and nouel classes simultaneously.
Difference w.r.t. existing problem settings Our paper addresses a practical and underexplored problem, which differs from cxisting problem settings (scc Table [ for a summary). In particular, (a) we consider both labeled data and unlabeled data in training, and (b) we consider a mixture of both known and novel classes unlabeled data. Note that our setting generalizes traditional representation learning. For example, Supervised Contrastive Learning (SupCon) (Khosla et al. 2020) only assumes the labeled considering the unlabeled data D., Weakly supervised contrastive learning (Zheng et al. 2021) assumees the same classes in labeled and unlabeled data, i.e., })- = Van, and hence remains closed-won world. Self-supervise learning (Chen et al. 2020a) relies completely on the unlabeled set D, and does not assume the availability of the labeled dataset. The setup is also known as open-world semi-supervvsed learning, which is originally
Dr, without
introduced by (Cao et al. 2022 Despite the similar setup, our learning goal is fundamentally different: Cao
ct al.
2022 focus on classification accuracy, but not learning high-quality embeddings. Similar to GCD Vaze
et al. 20222. we aim learning compact embeddings for both known and novel classes.
3
Proposed Method: Open-world Contrastive Learning
We formally introduce new learning framework, open-world contrastive learning (dubbed OpenCon), which is designed to produce compact representation space for both known and novel classes. The open-world setting posits unique challenges for learning effective representations, namely due to (1) the lack of the separation between known vs. novel data in Da, (2) the lack of supervision for data in novel classes. Our learning framework targets these challenges.
3.1 Background: Generalized Contrastive Loss
We start by defining a generalized contrastive loss that can characterize the family of contrastive losses. We will later instantiate the formula to define our open-world contrastive loss (Section 3.2 and Section 3.3 Specifically, we consider a deep neural network encoder Φ X' + Rd that maps the input x to Lz-normalized
This is different from the problem Novel Class Discovery, which assumes unlaheled is purely from novel classes.
3

---

Published in Transactions on Machine Learning Research (1/2023)
feature embedding (x). Contrastive losses operate the normalized feature Ō(xx. In other words, the features have unit norm and lie on the unit hypersphere. For a given anchor point x, we define the per-sample contrastive loss:
exp(z
aat//)
L ((x): = |P(x) 2TEP(x)
Σ
log Σx- exp(z" z /T)
(1)
where T is the temperature parameter, z is the Le-normalized embedding vector of x, P(x) is the positive set of embeddings w.r.l. 2, and ((x is the negative set of embeddings.
In open-world contrastive learning, the crucial challenge is how to construct P(x) and MV() for different types of samples. Recall that we have two broad categorics of training data: (1) labeled data Dj with known class, and (2) unlabeled data Du with both known and novel classes. In conventional supervised CL frameworks with Dj only, the positive sample pairs can be easily drawn according to the boound-truth labels (Khosla ct al., 2020)- That is, Pxx) consists of embeddings of samples that carry the same label as the anchor point x, and (x) contains all the embeddings in the multi-viewed mini-batch excluding itself. However, this is not straightforward in the open-world setting with novel classes.
3.2 Learning from Wild Unlabeled Data
We now dive into the most challenging part of the data, Da, which contains both known and novel classes. We propose novel protolype-bssed learning strategy that tackles the challenges of: (1) the separation between known and novel classes in Du; and (2) pseudo label assignment that can be used for positive set. construction for novel classes. Both components facilitate the goal of learning compact representations, and enable end-to-end training.
Key to our framework, we keep a prototype embedding vector ee for each class c E Van- Here Van contains both known classes 3/2 and novel classes Vn- = PaulDr and 3i n,, = 0. The prototypes can be viewed as a set of representative embedding vectors. All the prototype vectors M [///--///c/.--eee are randomly initiated the beginning of training, and will be updated along with learned embeddings. We will also
discuss determining the cardinality [Manl (i.e., number of prototypes) in Section 6
Protedype-based OOD detection We leverage the prototype vectors to perform out-of-iistribution (OOD) detection, é.e., separate known vs. novel data in D.. For any given sample x; E DD,, we mcasure the cosine similarity between its embedding ((x,) and prototype vectors of known classes V/. If the sample embedding is far away from all the known class prototypes, it is more likely to be novel sample, and vice versa. Formally, propose the level set estimation:
D2 = (xxxx mxx/max
14] (xx)) < <}},
(2)
JEM
where a thresholding mechanism is exercised to distinguish between known and novel samples during training time. The threshold X can be chosen based on the labeled data Dj. Specifically, one can calculate the scores maxjed; ur] (xx,) for all the samples in Di, and use the score at the p-percentile example, when p 90, that means 90% of labeled data is above the threshold. We provide ablation on the effect of later in Scction t and thcoretical insights into why OOD detection helps open-world represcntation
the threshold. For
learning in Appendix D
Positive and negative set selection Now that we have identified novel samples from the unlabeled sample, we would like to facilitate learning compact representations for D,, where samples belonging to the same class are close to each other. As mentioned earlier, the crucial challenge is how to construct the positive set, denoted as (x). In particular, we do not have any supervision signal for unlabeled data in the novel classes. We propose utilizing the predicted label و = argmax +(x) for positive set selection.
For a mini-batch B, with samples drawn from D,, we apply two random augmentations for cach sample and generate a multi-viewed batch B,. We denote the embeddings of the multi-viewed batch as An, where the cardinality [An| = 2 B/ For any sample x in the mini-batch B,, we propose selecting the positive and
4

---

Published in Transactions on Machine Learning Research (1/2023)
negative set of embeddings as follows:
(x 1z'|2' E -
(3)
(x An |z,
(4)
where z is the Lg-normalized embedding of x, and j' is the predicted label for the corresponding training example of z'. In other words, we define the positive set x to be those examples carrying the same approzimated label prediction ÿ.
With the positive and negative sets defined, we are now ready introduce our new contrastive loss for open-world data. We desire embeddings where samples assigned with the same pseudo-label can form compact cluster. Following the general template in Equation [
we define a novel loss function:
f. Σ (x; Tay Pa(x). XEB,
(5)
For each anchor, the loss encourages the network to align embeddings of its positive pairs while repelling the negalives. All posilives in multi-viewed batch (i.e., the augnentatioon-based sample as well as any of the remaining samples with the same label) contribute to the numerator. The loss encourages the encoder to give closely aligned representations to all entries from the same predicted class, resulting in a compact representation space. We provide visualization in Figure 2 (right).
Prototype update The most canonical way to update the prototype embeddings is to compute it every iteration of training. However, this would extract heavy computational toll and in turn cause unbearable training latency. Instead, we update the coasicooni prototype vector in moving-average average style (Li
et al. 2020 Wang et al. 2022)
Me = Normalize (y/ + (1 7))z), for c= y (ground truth label), argmax z,
if x E Dr if z E Dn
(6)
Here, the prototype Me of class defined by the moving average of the normalized embeddings z, whose predicted class conforms to
embeddings of samples from D, U D,. 7 is tunable hyperparameter.
We exclude samples in D,AD because they may contain non-distinguishable data from known and unknown classes, which undesirably introduce noise to the prototype estimation. We verify this phenomenon by comparing the performance of mixing Du \D, with labeled data D, for training the known classes. The results verify our hypothesis that the non-distinguishabe data would be harmful to the overall accuracy. We provide more discussion on this in Appendix
Remark:
3.3 Open-world Contrastive Loss
Putting it all together, we define the open-world contrastive loss (dubbed OpenCon) as the following:
LopenCon = AnC., + Aehi + Auhhus
where Cn is the newly devised contrastive loss for the novel data, Li the supervised contrastive loss KKooll et al. 2020) employed on the labeled data Di, and Cg is the self-supervised contrastive loss (Chen al.) 2020a) employed on the unlabeled data D.. À are the coefficients of loss terms. Details of Lı and L. are in Appendix along with the complete pseudo-code in Algorithm I (Appendix).
Remark Our loss components work collaboratively to enhance the embedding quality in an open-world setting. The overall objective well suits the complex nature of our training data, which blends both labeled and unlabeled data. As we will show later in Section G a simple solution by combining supervised contrastive loss (on labeled data) and self-supervised loss (on unlabeled data) is suboptimal. Instcad, having C,, is critical to encourage closely aligned representations to all entries from the same predicted class, resulting in overall more compact representation for novel classes.
5

---

Published in Transactions on Machine Learning Research (1/2023)
4 Theoretical Understandings
Overview Our learning objective using wild data (c.f. Section 3.2 can be rigorously interpreted from an Exppecationn-axxiniaalll
(EM) algorithm perspective. We start by introducing the high-level ideas of how our method can be decomposed into E-step and M-step respectively. At the E-step, we assign each data example x E D,, to one specific cluster. In OpenCon, it estimated by using the prototypes: 0. = argmaayeee...
((xx). At the M-step, the EM algorithm aims to maximize the likelihood under the posterior class probability from the previous E-step. Theoretically, we show that minimizing our contrastive loss Cn (Bquation 5) partially maximizes the likelihood by clustering similar examples. In effect, our loss concentrates similar data to the corresponding prototypes, encouraging the compactness of features.
4.1 Analyzing the E-step
In E-step, the goal of the EM algorithm is maximize the likelihood with learnable feature encoder & and protolype matrix M = (pll...leel-.) which can be lower bounded:
[Pal
[Pal
clo,
> p 2 al() log
Σ
Σ CE
4((c)
where gi(c) (c) is denoted as the density function of a possible distribution over e for sample x,, By using the fact that log(-) function is concave, the inequality holds with equality when p(x,clo.M) is a constant value, thercfore we set:
(((e)
c/o, (c) ExEra p(x, co. M) px M)
p(x,, clo, M) p(c|x;, 6, M),
which is the posterior class probability. To estimate p(c[x;, o, M), model the data using the von Mises-
Fisher (yMF) (F [Fisher 1953) distribution since the normalized embedding locates in high-dimensional
hyperspherical space.
Assumption 4.1. The density function is given by I (x) ca(ks)eu'' d(x), where is the concentration parameter and ca(M) is coefficient.
With the vMF distribution assumption in 4.1 we have p(c]x;, o, M) ac ((x:)), where denotes
the softmax function and de is the c-th element. Empirically we take a one-hot prediction with d: argmax jeym ((x:) since each example inherently belongs to exactly one prototype, so we let (i(c) =
1/c = 1.).
4.2 Analyzing the M-step
In M-step, using the label distribution prediction qi(c) in the E-step, the optimization for the network & and the prototype matrix M given by:
P Σ
p (xx,clo M)
argmax d.M
Σ
qi(c) log
(8)
(((c)
1=1 cE
The joint optimization target in Equation g is then achieved by rewriting the Equation S
according to the
following Lemma 4.2 with proof in Appendix
Lemma 4.2. Zha et al. 2001) We define the set of samples with the same prediction S(c) = fx: E Drlo: = c).
The marimization slep is equévalent aligning the feature vector Φ(x) to the corvesponding protolype ptc:
Pal
(i(c) log p (xx, M)
Σ
Σ
Σ T(x) Mh
argmax
argmaa
6.M
9:(c)
e.M
1=1 Y Van

---

Published in Transactions on Machine Learning Research (1/2023)
In our algorithm, the maximization step is achieved by optimizing M and ø separately.
(a) Optimizing M:
For fixed Φ, the optimal prototype is given by 14r Normalizo(Exxste) [o(x)]). This optimal form empiri-
cally corresponds to our prototype estimation Equation [6) Empirically, it is expensive to collect all features in S(c). We use the estimation of pe by moving average: Pe = Normalize (7fe + (1 7))(x)) Vx e
S(c).
(b) Optimizing d:
We then show that the contrastive loss L, composed with the aligument loss part La encourages the closeness of features from positive pairs. By minimizing Ca, it is approximalely maximizing the target in Equation with the optimal prototypes u2 We can decompose the loss as follows:
1 Ln =
exp(z 5op(zT2t//7 .2//7)
log
P(x))
Ex-eN(x) exp(z z 2 /T)
=TEP(())
1
Σ
(z x /T)
Σ log Σ
exp(zT. .2 /T//T).
P(x)]
PP(x)
2tEP(x)
E=(((x)
(x(x)
fx(x)
In particular, the first term (x) is referred to as the alignment term Wang & Isola 2020), the compactness of features from positive pairs. To see this, we have the following lemma 4.3 with proof in Appendix F Lemma 4.3. Minimizing Lx(x) is equivalent to the marimization step w.r.l. parameter Φ.
which encourages
argmin Σ Lx(x) argmaa
ΣΣ
Σ d(x) pee
CEVOI
Summary These observations validate that our framework learns representation for novel classes in an EM fashion. Importantly, we extend EM from traditional learning setting to an open-world setting with the capability to handle real-world data arising in the wild. We proceed by introducing the empirical verification of our algorithm.
5 Experimental Results
Datasets We evaluate on standard benchmark image classification datasets CTFAR-100 (Krizhevsky et al. 2009) and ImageNet (Deng al. 2009). For the ImageNet, we sub-sample 100 classes, following the same setting ORCA (Cao al. 2022 for fair comparison. Results ImageNet-1k is also provided in Appendix Note that we focus on these tasks, as they are much more challenging than toy datasets with fewer classes. The additional comparison on CIFAR-10 is in Appendix G By default, classes are divided into 50% seen and 50% novel classes. We then select 50% of known classes as the labeled dataset, and the rest as the unlabeled set. The division is consistent with Cao et al. (2022), which allows us to compare the performance in a fair setting. Additionally, we explore different ratios of unlabeled data and novel classes (sce Section G).
Evaluation metrics We follow the evaluation strategy in Cao al. (2022) and report the following metrics: (1) classification accuracy known classes, (2) classification accuracy on the novel data, and (3) overall accuracy on all classes. The accuracy of the novel classes is measured by solving an optimal assignment problem using the Hungarian algorithm (Kuhn & Yaw 1955). When reporting accuracy on all classes, we solve optimal assignments using both known and novel classes.
Experimental details We use ResNet-18 as the backbone for CIFAR-100 and ResNet-50 as the backbone for ImageNet- 100. The pre-trained backbones (no final FC layer) are identical the ones Cao al. 2022 To ensure fair comparison, follow the same practice in Cao et al. (0222) and only update the
7

---

Published in Transactions on Machine Learning Research (1/2023)
Table 2: Main Results. Asterisk (*) denotes that the original method can not recognize seen classes. Dagger
()) denotes the original method can not detect novel classes (and we had to extend it). Results on ORCA, GCD and OpenCon (mean and standard deviation) are averaged over five different runs. The ORCA results are reported based on the official repo (Cao et a.
CIFAR-100
ImagNet-100
Method
All
Novel
Seen
All
Novel
Seen
1FixMatch (Kurakin al.
2020
20.3
23.5
39.6
34.9
36.7
65.8
!DS Guo al.
2020
24.0
23.7
55.1
30.8
32.5
71.2
tCGDL Sun et al.
2020
23.6
22.5
49.3
31.9
33.8
67.3
*DTC HHan et al. 2019
18.3
22.9
31.3
21.3
20.8
25.6
"RankStats (Zhao & Han
2021
23.1
28.4
36.4
40.3
28.7
47.3
SimCLR (Chen et al.
2020a.
22.3
21.2
28.6
36.9
35.7
39.5
ORCA (Cao et al.
2022
47.2
41.0+11.
66.7
76.4
68.9=0.8
89.1
GCD
Vaze et al. 2022
46.8+0.5
43.4
69.7-0.4
75.5
72.8=1.2
90.9 #0.2
OpenCon (Ours)
52.7+0.6
47.8=0.6
69.1+0.3
83.8+003
80.8+0.3
90.6#0.1
parameters of the last block of ResNet. In addition, we add a trainable two-layer MLP projection head that projects the feature from the penultimate layer to a lower-dimensional space IR (d= = 128), which is shown to be effective for contrastive loss (Chen et al. 2020a). We use the same data augmentation strategies as SimCI. (Chen al. 2020a). Same as Cao et al. (2022), we regularize the KT-divergence between the predicted label distribution p (0) and the class prior to prevent the network degenerating into trivial solution in which all instances assigned to few classes. We provide extensive details on the training configurations and all hyper-parameters in Appendix
OpenCon achieves SOTA performance As shown in Table 1, OpenCon outperforms the rivals by a significant margin on both CIFAR and ImageNet datasets. Our comparison covers an extensive collection of algorithms, including the best-performed methods to date. In particular, on ImageNet- 100, we improve upon the best bascline by 7.4% in terms of overall accuracy. It is also worth noting that OpenCon improves the accuracy of novel classes by 11.9%. Note that the open-world representation learning is a relatively setting. Closest to our setting is the open- woel-world semi-supervised learning (SSL) algorithms, namely ORCA (Cao ct al. 2 and GCD (Vaze et al. 2022 that directly optimize the classification performance. While our framework emphasizes representation learning, we demonstrate the quality of learned embeddings by also measuring the classification accuracy. This can be easily done by leveraging our learned prototypes on a converged model: 9 = argmax,Ex... 1u (xx). We discuss the significance w.r.t. existing works in detail:
OpenCon vs. ORCA Our framework bears significant differences w.r.t. ORCA in terms of learning goal and approach. (1) Our framework focuses on the representation learning problem, whereas ORCA optimizes for the classification performance using cross-entropy loss. Unlike ours, ORCA does not necessarily learn compact representations, as evidenced in Figure 2 (left). (2) We propose novel open-world contrastive learning framework, whereas ORCA does not employ contrastive learning. ORCA uses a pairwise loss to predict similarities between pairs of instances, and does not consider negative samples. In contrast, our approach constructs both positive and negative sample sets, which encourage aligning representations to all entries from the same ground-truth label or predicted pseudo label (for novel classes). explicitly considers OOD detection, which allows separating known vs. novel data in D,. ORCA does not. consider this and can suffer from noise in the pairwise loss (e.g., the loss may maximize the similarity between samples from known vs. novel classes).
(3) Our framework
OpenCon vs. GCD There are two key differences to highlight: (1) GCD (Vaze et al. 2022 requires a two-stage training procedure, whereas our learning framework proposes an end-lo-end training strategy. Specifically, GCD applies the SupCon loss (Khosla et al.) 2020) on the labeled data D, and SimCLR loss (Chen et al.) 2020a) on the unlabeled data D,,- The feature then clustered scparately by a semi-supervised K-means method. However, the two-stage method hinders the useful pseudo-labels to be incorporated into the training stage, which results in suboptimal performance. In contrast, our
8

---

Published in Transactions on Machine Learning Research (1/2023)
ORCA
GCD
Ours (More Compact)
Embeddings from Novel Classes
Embeddings from Seen Classes
Figure 2: UMAP MeInnes al. 2018 visualization of the feature embedding from 10 classes (5 for seen,
5 for novel) when the model is trained on ImageNet-1100 with ORCA (Cao et al. 2022 GCD Vaze et al. 2022 and OpenCon (ours).
prototype-based learning strategy alleviates the need for a separate clustering process (c.f. Section 3.2). which therefore easy to use in practice and provides meaningful supervision for the unlabeled data. (2) We propose a contrastive loss Ln better utilizing the psoudo-labels during training, which facilitates learning more compact representation space for the novel data From Table 2 outperforms GCD by 8.3% (overall accuracy) ImageNet-1000, showcasing the benefits of our framework.
we observe that OpenCon
Lastly, for completeness, we compare methods in related problem domains: (1) novel class detection: DTC Han
et al.
2019), RANKSTATS (Zhao Han 2021], ), (2)
semi-superoised learning: FIXMATCH (Kurakin et al. 2020 Guo et al. 2020) and CGDL (Sun et al. 2020). We also compare with the common representation
D
method SIMCLR (Chen et al., 22200a.. These methods are not designed for the Open-SSL task, therefore the performance is less competitive.
OpenCon is competitive VIT Going be- yond convolutional neural networks, show in Table 3 that the OpenCon is competitive for transformer-based VIT model Dosovitskiy ct al. 2020). We adopt the ViT-B/16 archi- fecture with DINO pre-trained weights (Caron 2021. following the pipeline used in Vaze
Table 3: Comparison of accuracy ViT-B/16 architec- ture (Dosovitskiy et al. 2020 Results are reported on ImageNet-100.
Methods
All
Novel
Seen
ORCA
Cao et al. 2022
73.5
64.6
89.3
al.
GCD
Vaze et al. 2022
74.1
66.3
89.8
ct al.
2022). In Table 3] compare Open Con's
k-Means MacQueen
1967
72.7
71.3
75.5
performance with ORCA (Cao et al.) 2022
RankStats+
Zhao & Han
2021
37.1
24.8
61.6
GCD
Vaze et al. 2022). k-Means (MacQueen
UNO+ Fini et al.
202
70.3
57.9
95.0
1967
RankStats+ + (Zhao & Han
2021] and
OpenCon (Ours)
84.0
81.2
93.8
UNO Fini et al. 2021) on ViT-B-16 16 architec-
ture. On ImageNNe-T000, we improve upon the
best bascline by 9.9 in terms of overall accuracy.
OpenCon learns more distinguishable representations We visualize feature embeddings using UMAP (Meinnes et al. 2018 in Figure 2 Different colors represent different ground-truth class labels. For clarity, we use the ImageNet-100 datasct and visualize subset of 10 classes. We can observe that OpenCon produces a better embedding space than GCD and ORCA. In particular, ORCA does not produce distinguishable representations for novel classes, especially when the number of classes increases. The features of GCD are improved, yet with some class overlapping (e.g., two orange classes). For reader's reference, we also include the version with a subset of 20 classes in Appendix where OpenCon displays more distinguishable representations.

---

Published in Transactions on Machine Learning Research (1/2023)
6 A Comprehensive Analysis of OpenCon
Prototype-based OOD detection is important In Figure 3
Accuracy-(Al) (All)
84.3
we ablate the contribution of a key component in OpenCon:
CIFAR-100 ImageNet-
84
83.1
83.7
83.7
83.5
83
prototype-based OOD detection (c.f- Section 3.2 1. To system-
82
81.9
atically analyze the effect, we report the performance under varying percentile p E {0, 10, 30, 50, 70, 90). Each p corresponds to different. threshold for separating known with p 0, Dn becomes equivalent
10
30
50
70
90
novel data
Accuracy-(AI))
55
in Da. In the extreme to Du, and hence the contrastive loss Ln is applied to the en-
54
53.1
53.3
53.1
53.7
53.2
53
51.7
tire unlabeled data. We highlight two findings: (1) Without OOD detection (p- 0), the unseen accuracy reduces by 2.4%,
51
0
10
30
50
70
90
compared to the best setting (p = 70%). This affirms the impor- fance of OOD detection for better representation learning. (2)
Figure 3: Effect of p on ImageNet-100 and CIFAR-100, measured by the overall accuracy. p= means no OOD detection.
A higher percentile p: in general, leads to better performance. We also provide thcoretical insights in Appendix D showing OOD detection helps contrastive learning of novel classes by having fewer candidate classes.
Table 4: Ablation study loss component.
CIFAR-100
ImageNet-100
Loss Components
All
Novel
Seen
All
Novel
Seen
w/o Li w/o La w/o fn
43.3
47.1
38.8
68.6
73.4
59.1
36.9
28.3
63.4
55.9
39.3
89.2
46.6
42.2
70.3
74.5
70.7
91.0
OpenCon (ours)
52.7#0.6
47.8+0.0
69.1
0..3
83.8+0.3
80.8=033
90.6+0.1
Ablation study on the loss components Recall that our overall objective function in Equation consists of three parts. We ablate the contributions of each component in Table g Specifically, we modify OpenCon by removing: (i) supervised objective (f.e., w/o Di), (ii) unsupervised objective on the entire unlabeled data (i.e., w/o Du), and (iii) prototype-baser contrastive learning novel data Dn,- We have the following key observations: (1) Both supervised objective L and unsupervised loss L, are indispensable parts of open-world representation learning. This suits the complex nature of our training data, which requires learning on both labeled and unlabeled data, across both known and novel classes.
(2) Purely combining SupCon (Khosla et al., 2020) and SimCLR (Chen et al.] 2020a) as used in GCD (Vaze et al., 2022) does not give competitive results. For example, the overall accuracy is 9.3% lower than our method on the ImageNet-100 dataset. In contrast, having Cn encourages closely aligned representations to all entries from the same predicted class, resulting in more compact representation space for novel data. Overall, the ablation suggests that all losses in our framework work together synergistically to enhance the representation quality.
Handling an unknown number of novel classes In practice, we often do not know the number classes Daul in advance. This is the dilemma faced by Open- Con and other basclines as well. In such cases, one can apply OpenCon by first estimating the number of classes. For fair comparison,
Table 5: Accuracy on CIFAR-100 dataset with an
unknown number of classes.
Methods
All
Novel
Secn
ORCA CCo et al.
2022
46.4
40.0
66.3
use the same
GCD Vaze et al. 2022
47.2
41.9
69.8
estimation technique? 2 as in Han et al. (2019): Cao
let al. 2022). On CIFAR-100, the estimated total mum-
OpenCon (Known Daul)
53.7
48.7
69.0
ber of classes is 124. At the beginning of training, we
OpenCon (Unknown (anl)
53.7
48.2
68.8
initialize the same number of prototypes accordingly.
Results in Table 5 show that OpenCon outperforms
2 A clustering algorithm performed the combination of laheled data and unlabeled data. The optimal number of classes chosen by validating clustering accuracy on the labeled data.
10

---

Published in Transactions on Machine Learning Research (1/2023)
the best baseline ORCA |Cao et al. 2022 by 7.3%. Interestingly, despite the initial number of classes, the training process will converge to solutions that closely match the ground truth number of classes. For example, at convergence, we observe a total number of 109 actual clusters. The remaining ones have no samples assigned, hence can be discarded. Overall, with the estimated number of classes, OpenCon can achieve similar performance compared to the seiting in which the number of classes is known.
OpenCon is robust under smaller number of labeled examples, and larger number of novel classes We show that OpenCon's strong performance holds under more challenging settings with: (1) reduced fractions of labeled examples, and (2) different ratios of known novel classes. The results summarized in Table 6 First, we reduce the labeling ratio from 50% (default) to 25% and 10%, while keeping the number of known classes to be the same (i.e., 50). With fewer labeled samples in the known classes, the unlabeled sample set size will expand accordingly. It posits more challenges for novelty discovery and representation learning. On InageNet-1000 OpenCon substantially improves the novel class accuracy by 10% compared ORCA and GCD, when only 10% samples are labeled. Secondly, we further increase the number of novel classes, from 50 (default) to 75 and 90 respectively. On ImageNet-100 with 75 novel classes ([Di] 25), OpenCon improves the novel class accuracy by 16.7% over ORCA (Cao et al. 2022. Overall our experiments confirm the robustness of OpenCon under various settings.
Table 6: Accuracy on CIFAR-100 under varying diffcrent labeling ratios (for labeled data) and different
numbers of known classes (())).. The number of novel classes 100
[D:].
CIFAR-100
Image.Net-100
Labeling Ratio
Dil
Method
All
Novel
Seen
All
Novel
Seen
ORCA
47.0
41.3
66.2
76.6
69.0
88.9
0.5
50
GCD
47.2
43.6
69.4
75.1
73.2
90.9
OpenCon
53.7
48.7
69.0
84.3
81.1
90.7
ORCA
47.6
42.5
61.8
72.2
64.2
87.3
0.25
50
GCD
41.4
39.3
66.0
76.7
69.1
89.1
OpenCon
51.3
44.6
65.5
82.0
77.1
90.6
ORCA
41.2
37.7
54.6
68.7
56.8
83.4
0.1
50
GCD
37.0
38.6
62.2
69.4
56.6
85.8
OpenCon
48.2
44.4
62.5
75.4
66.8
85.2
ORCA
40.4
38.8
66.0
57.8
54.0
89.4
0.5
25
GCD
41.6
39.2
70.0
65.3
63.2
90.8
OpenCon
43.9
41.9
70.2
74.5
72.7
91.2
ORCA
34.3
33.8
67.4
45.1
43.6
93.0
0.5
10
GCD
38.4
36.8
61.3
53.3
52.9
94.2
OpenCon
40.9
40.5
69.9
59.0
58.2
94.3
7
Related Work
Contrastive learning A great number of works have explored the effectiveness of contrastive loss in
unsupervised representation learning: InfoNCE (Van den Oord et al. 2018). SimCLR (Chen et al.
2020a
SWaV (Caron et al. 2020), MoCo (He al. 2020), SEER (Goyal al.
2021) and (Li et al. 2020
2021
Zhang et al. 2021)) It motivates follow-up works to on weakly supervised learning tasks (Zheng al.
202
Tsai et al. 2022; semi-supervised learning (Chen et al.
2020b
Li et al.
2021a
Zhang et al. 2022b Yang
et al. 2022a supervised learning with noise (Wu et al.
2021
Karim et al. 2022 Li et al. 2022a ). continual
learning (Cha et al. 2021), long-tailed recognition (Cui ct al.
2021 Tian et al. 2021 Jiang et al. 2021
et al. 2022b), few-shot learning (Gao et. al.]
2021 ) partial label learning (Wang el al. 2022) novel class
discovery (Zoong et al.) 2021a Zhao Han 2021 Fini et al.
2021), hierarchical multi-labed learning (Zang
ct al., (022a)). Under different circumstances, all works adopt different choices of the positive set, which is
not limited to the self-angmented view in SimCLR (Chen et al.) 2020a). Specifically, with label information
available, SupCon (Khosla et al. 2020
improved representation quality by aligning features within the same
11

---

Published in Transactions on Machine Learning Research (1/2023)
class. Without supervision for the unlabeled data, Dwibedi et al. (2021) used the nearest neighbor positive pair to learn compact embedding space. Different from prior works, we focus on the open-world representation learning problem, which is largely unexplored.
Out-of-distribution detection The problem of classification with rejection can date back to early works which considered simple model families such as SVMs (Fumera & Roli 2002]. In deep learning, the problem of out-of-distribution (OOD) detection has received significant research attention in recent years. Yang et al.
(2022b) survey line of works, including confidence-basee methods (Bendale & Boult 2016 Hendrycks &
Gimpel
2017 Liang et al. 2018
Huang & Li
2021),
emergy-based score (Lin et al.
2021 Liu et al.) 2020
Wang ct al.
2021| Sun et al., 2021; Sun & Li
2022
Morteza & Li 2022p, distancc- based approaches (Loo
et al. 2018
Tack et al. 2020 Sehwag et al.
2021:
Sun et al., 2022 Ming et al. 2022a b), gradient-based
score
Huang et al. 2021b). and Bayesian approaches (Gal & Ghahramani 2016
Lakshminarayanan et al.,
2017:
Maddox et al. 2019 Malinin & Gales 2018
2019). In our framework, OOD
detection is based on the
distance to the prototype of the closest known class. We show in Appendix J that several popular OOD
detection methods give similar performance.
Novel category discovery At an earlier stage, the problem of novel category discovery (NCD) is targeted as transfer learning problem in DTC (Han et al., 2019), KCL (Hsu et al. 2018), MCL (Hsu et al. 2019 The learning is generally in a two-stage manner: the model is firstly trained with the labeled data and then transfers knowledge to learn the unlabeled data. OpenMix (Zhong et al. 2021b) further proposes an end-to-end framework by mixing the seen and novel classes in a joint space. In recent studies, many researchers incorporate representation learning for NCD like RankStats (Zhao & Han 2021), NCL (Zhong 2021a) and UNO (Fini et al. 2021). In our setting, the unlabeled test set consists of novel classes but also classes previously seen in the labeled data that need to be separated.
et al.
Semi-supervised learning A great number of early works (Chapelle et al. 2006 Lee et al. 2013 Sajjadi
et al.] 2016; Laine & Aila 2017; Zhai et al.) 2019) Rebuffi et al. 2020 Kurakin et al. 2020) have been proposed to tackle the problem of semi-supervised learning (SSL). T Typically, a standard cross-entropy loss is
applied to the labeled data, and a consistency loss (Taine & Aila 2017 Kurakin et al. 2020 or self-supervised
loss (Sajjadi et al.) 2016 Zhai et al. 2019 Rebulti et al.) 2020) applied the unlabeled data. Under the
closed-world assumption, SSL methods achieve competitive performance which is close to the supervised
methods. Later works (Oliver et al. 2018 Chen et al.
2020c point out that including novel classes in the
unlabeled set can downgrade the performance. In
Guo et al. (2020): Chen et al. (2020c); Yu et al. (2000);
Park et al.] [0021); ; Saito et al. (2021 Huang et al.
(2021a); Yang et al. (2022a). OOD detection techniques
are wielded to separate the OOD samples in the unlabeled data. Recent works (Vaze et al. 2022 Cao et al,
2022: Rizve et al., 2022) further require the model to group samples from novel classes into semantically
meaningful clusters. In our framework, we unify the novelty class detection and the representation learning and achieve competitive performance.
8 Conclusion
This paper provides a new learning framework, open-world contrastive learning (OpenCon) that learns highly distinguishable representations for both known and novel classes in an open-world setting. Our open-world setting can generalize traditional representation learning and offers stronger flexibility. We provide important insights that the separation between known novel data in the unlabeled data and the pseudo supervision for data in novel classes is critical. Extensive experiments show that OpenCon can notably improve the accuracy on both known and novel classes compared to the current best method ORCA. As a shared challenge by all methods, one limitation is that the prototype number in our end-to-end training framework needs to be pre-specified. An interesting future work may include the mechanism to dynamically estimate and adjust the class number during the training stage. The broader impact statement is included Appendix A
Acknowledgement
The authors would also like to thank Haobo Wang and TMLR reviewers for the helpful suggestions and feedback.
12

---

Published in Transactions on Machine Learning Research (1/2023)
References
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. URL https://
com/saaap-ttafforrdooraaa
Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In Proceedings of the International Conference on Learning Representations, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Proceedings of Advonces in Neural Information Processing Systems, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9516 9525, 2021.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien (eds.). Semi-Superoised Learning. The MIT Press, 2006.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In Proceedings of the international conference machine learning, pp- 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Proceedings of Advances in Neurol Information Processing Systems, 01-22243-222533 2020b.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3569-3576, 2020c.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric rastivee learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 715-724. 2021.
Jia Deng, Wei Dong, Richard Socher, Ti-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2020.
Debidatta Dwibodi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanct, and Andrew Zisserman. With a little help from my friends: Nearest-neighbor rastivetive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9588 9597, 2021.
Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9284 9292, 2021.
Ronald Aylmer Fisher. Dispersion on sphere. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 217((130):295 3055, 1953.
13

---

Published in Transactions on Machine Learning Research (1/2023)
Giorgio Fumera and Fabio Roli. Support vector machines with embedded reject option. In International Workshop on Support Vector Machines, pp. 68 02-88. Springer, 2002.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning, pp- 1050- 1050,1059, 2016.
Yizhao Gao, Nanyi Fei, Guangzhen Liu, Zhiwu Lu, and Tao Xiang. Contrastive prototype learning with aug- mented embeddings for few-shot learning. In Proceedings of the Sheenh-Seeenth Conference on Uncertoinly in Artificial Intelligence, pp. 440-150,400 2021.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, al. superviee pretraining of visual features in the wild. arXiv preprint or orXiw:203..00 01801988, 2021.
Lan-Zhe Guo, Zhen-X Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi-supervised
learning for unscen-class unlabeled data. In Proceedings of the international conference on machine learning, volume 119 of Proceedings of Machine Learning Research, pp. 3897 3906. PMLR, 13 18 Jul 2020.
Kai Han, Andrea Vodaldi, and Andrew Zisserman. Learning to discover novel visual categorics via deep transfer clustering. In Proceedings of the IEEE/CVE International Conference on Computer Vision, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning- In Proceedings of the IEFE Conference on Compuler Vision and Pollern Recognition, pp. 9729 9738, 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and ont-of-distribution examples in neural networks. Proceedings of International Conference on Learning Representations, 2017.
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. Proceedings of the International Conference on Learning Representations, 2018.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. Proceedings of the International Conference on Learning Representations, 2019.
Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning. In Proceedings of the HEEE/CVE International Conference on Computer Vision, pp. 8310 00-9,,9, 2021a.
Rui Huang and Yixuan Li. Towards scaling out-of-listribution detection for large semantic space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting dist ributional shifts in the wild. In Proceedings of the Advances in Neural Information Processing Systems, 2021b.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Improving contrastive learning on imbalanced data via open- world sampling. Proceedings of Advonces in Neurol Information Processing Systems, 34, 2021.
Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Proceedings of Advances in Neural Information Processing Systems, 33:18661-18673 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fino-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
14

---

Published in Transactions on Machine Learning Research (1/2023)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
H. W. Kuhn and Bryn Yaw. The hungarian method for the assignment problem. Naval Res. Logist. Quart, pp. 83 97, 1955.
Alex Kurakin, Chun-Liang Li, Colin Raffel, David Berthelot, Ekin Dogus Cubuk, Han Zhang, Kihyuk Sohn, Nicholas Carlini, and Zizhao Zhang. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Proceedings of Advances in Neural Information Processing Systems, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. Proceedings of the
International Conference on Learning Representations, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of Advances in Newral Information Processing Systems, pp. 6402 6413, 2017.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of- distribution samples and adversarial atlacks. In Proceedings of Advances in Neural Information Processing Systems, pp. 7167-7177, 2018.
Junnan Li, Caiming Xiong, and Steven Hoi. Mopro: Webly supervised learning with momentum protolypes. In International Conference on Learning Representations, 2020.
Junnan Li, Caiming Xiong, and Steven C.H. Hoi. Semi-supervised learning with contrastive graph regulariza- tion. In Proceedings of the International Conference on Learning Representations, 2021a.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. Procecdings of the International Conference on Learning Representations, 2021b.
Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervieed contrastive learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022a.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogério Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, 2022b.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In Proceedings of the International Conference on Learning Representations, 2018.
Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi-level out-of-distribntion detection. Tn Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15313 15323, June 2021.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Snergy-based Proceedings of Advances in Neural Information Processing Systems, 33:21464-2144 2020.
detection.
J MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, pp. 281-297, 1967.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning, Advances in Neural Information Processing Systems, 32: 13153-13164, 2019.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Andrey Malinón and Mark Gales. Reverse ki-divergence training of prior networks: Improved uncertainty and adversarial robustness. In Advonces in Neural Information Processing Systems, 2019.
15

---

Published in Transactions on Machine Learning Research (1/2023)
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source. Software, 3(29):861, 2018.
Yifei Ming, Ziyang Cai, Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. Advances in neural information processing systems, 2022a.
Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. Cider: Exploiting hyperspherical embeddings for out-of-distribution detection. arXiv preprint 0rXiv22003.04550 2022b.
Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection. Proceedings of the AAAI Conference on Artificial Intellignnee, 2022.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and lan Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. Proceedings of Advances in Neural Information Processing Systems, 31, 2018.
Jongjin Park, Sukmin Yun, Jongheon Jeong, and Jinwoo Shin. Opencos: Contrastive semi-superrised learning for handling open-set unlabeled data. arXiv preprint arXiv:2107.00 08943, 2021.
Sylvestre-Alvee Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Semi- supervised learning with scarce annotations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognition Workshops, pp. 762 763, 2020.
Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Openldn: Learning to discover novel classes for open- open-world semi-supervised learning. Proceedings of the Europeon Conference on Computer Vision, 2022.
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set consistency regularization for semi-supervised learning with outliers. Proceedings of Advances in Neural Information Processing Systems, 2021.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Proceedings of Advances in Neural Information Processing Systems, 29, 2016.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2021.
Xin Sun, Zhemning Yang, Chi Zhang, Keck Voon Ling, and Guohao Peng. Conditional gaussian distribution lcarning for open set recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 13480 13489, 2020.
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribation dctection. In Europeon Conference on Computer Vision, 2022.
Yiyou Sun, Tonghua Su, and Zhiying Tu. Faster r-enn based autonomous navigation for vehicles in warehouse. In 2017 IEEE International Conference on Advanced Intelligent Mechatronics (AIM), pp. 2017.
IEEE,
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distributioo detection with rectified activations. In Advances in Neural Information Processing Systems, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribatioo detection with deep nearest neighbors. In Proceedings of the International Conference on Machine Learning, 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
Kiat Chuan Tan, Yulong Liu, Barbara Ambrose, Mclissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. Xix preprint ar 1eo::990005574 2019.
16

---

Published in Transactions on Machine Learning Research (1/2023)
Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. Divide and contrast: from uncurated data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10063 10074, 2021.
Self-supervised learning
Yao-Hung Hubert Tsai, Tianqin Li, Weixin Liu, Peiyuan Liao, Ruslan Salakhutdinov, and Louis-Philippe Morency. Learning weakly-supervisee contrastive represenlations. In International Conference on Learning Representations, 2022.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. a807, 2018.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. Proceedings of the International Conference on Learning Representations, 2021.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao, Pico: Contrastive label disambiguation for partial label learning. Proceedings of the International Conference on Learning Represenlations, 2022.
Haoran Wang, Weitang Liu, Alex Bocchieri, and Yixuan Li. Can multi-label classification networks know what they don't know? Proceedings of the Advances in Newral Information Processing Systems, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In Proceedings of the International Conference on Machine Learning, pp. PMLR, 2020.
Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Minggian Tang, and Yu-Feng Li. Nge: A unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 62 71, 2021.
Fan Yang, Kai Wu, Shuyi Zhang, Guannan Jiang, Yong Liu, Feng Zheng, Wei Zhang, Chengjie Wang, and Long Zeng. Class-aware contrastive semi-supervieed learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2022a.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-of-distribution detection. Xiviv preprint arxi:222000079 07242, 2022b.
Qing Yu, Daiki Ikami, Go Trie, and Kiyoharu Aizawa. Multi-task eurriculum framework for open-set semi- supervised learning. In Proceedings of the European Conference on Computer Vision, pp. 438-454. Springer, 2020.
Hongyuan Zha, Xiaofeng He, Chris Ding, Ming Gu, and Horst Simon. Spectral relaxation for k-means clustering. Advances in neural information processing systems, 14, 2001.
Xiaohua Zhai, Avital Oliver, Alexander Kolesníkov, and Lucas Beyer. S41: Self-supervised semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1476 1485, 2019.
Dejiao Zhang, Feng Nan, Xiaokai Wei, Shangwen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati, Andrew Arnold, and Bing Xiang. Supporting clustering with contrastive learning. Proceedings of the North American Chapter of the Association for Computational Linguistics, 2021.
Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hicrarchical multi-label contrastive learning framework. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022a.
17

---

Published in Transactions on Machine Learning Research (1/2023)
Yuhang Zhang, Xiaopeng Zhang, Jie Li, Robert Qiu, Haohang Xu, and Qi Tian. Semi-supervised contrastive learning with similarity co-calibration. IEEE Trunsactions on Mullimedia, 2022b.
Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Proceedings of Advances in Neural Information Processing Systems, 34, 2021.
Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. Proceedings of the IEEE/CVE International Conference on Computer Vision, 2021.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In Proceedings of the IEEE Conference on Computer Vision and Pullern Recognition, pp. 10877,18875, 2021a.
Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Opemmix: Reviving known knowledge for discovering novel visual categories in an open world. In Proceedings of the IEEE Conference on Computer Vision and Poltern Recognition, pp. 9462 9470, 2021b.
18

---

Published in Transactions on Machine Learning Research (1/2023)
Appendix OpenCon: Open-world Contrastive Learning
A
Broader Impacts
Our project aims to improve representation learning in the open-workd setting. Our study leads to direct benefits and societal impacts for real-world applications such e-commerce which may encounter brand-new products together with known products. Our study does not involve any human subjects or violation of legal compliance. In some rare cases, the algorithm may identify unlabeled minority sub-groups applied to images of humans or reflect biases in the data collection. Through our study and releasing our code, we hope that our work will inspire more future works to tackle this important problem.
B Preliminaries of Supervised Contrastive Loss and Self- Self-supervised Contrastive Loss
Recall in the main paper, provide general form of the per-sample contrastive loss:
exp(z' T = .z²/T) /T)
P(xx,
log
[P(x) 2+EP(x)
Σ Ex-eN(x) exp(z z /T)²
where 7 is the temperature parameter, z is the L2 normalized embedding of P(x) is the positive set of embeddings w.r.t. and (x)) is the negative set of embeddings.
In this section, we provide a detailed definition of Supervised Contrastive Loss (SupCon) Khosla et al. 2020 and Self-supervised Contrastive Loss (SimCIR) (Chen et al. 2020a
Supervised Contrastive Loss For mini-batch B₁ with samples drawn from Di, apply two random augmentations for each sample and generate multi-viewed batch B, We denote the embeddings of the multi-viewed batch as Ar, where the cardinality |Ail = 2/ For any sample x in the mini-batch Bi, the positive and negalive set of embeddings are as follows:
(((x) = (z' z' (A1z), y M((x = Ai)z,
where y the ground-truth label of x, and y/ the predicted label for the corresponding sample of z'. Formally, the supervised contrastive loss is defined as:
where 77 is the temperature.
Self-Supervised Contrastive Loss we apply two random augmentations for each sample and generate a multi-viewed batch B,, We denote the embeddings of the multi-viewed batch as Ay, where the cardinality [Aul = 2|B,]. For any sample x in the mini-batch B,, the positive and negative set of embeddings as follows:
For a mini-batch B, with samples drawn from unlabeled dataset D.,
P.(x)= (z' 2'' x(x'), is augmented from the same sample x)
N.(x): An
19

---

Published in Transactions on Machine Learning Research (1/2023)
The self-supervised contrastive loss is then defined as:
Lu
where is the temperature.
C
Algorithm
Below we summarize the full algorithm of open-world contrastive learning. The notation of Bu, Bi, Aus Ai is defined in Appendix
Algorithm Open-world Contrastive Learning
Input: Labeled set Dj = (xi, (x,,wi== and unlabeled set D, {x:};=) neural network encoder Φ, randomly initialized prototypes M.
Training Stage:
repeat
Data Preparation:
Sample mini-batch of labeled data Bi {x, y4 3 and unlabeled data B. {x Generate augmented batch and extract normalized embedding set Ar,
OOD detection:
Calculate OOD detection threshold x by A
Separate An from Ar
Dositive/Negative Set Selection:
Assign pseudo-labels by prototypes for each sample in An
Obtain P, from An pack-prooaggtion::
Calculate loss LOpenCon
Update network & using the gradients.
Prototype Update:
Update prototype vectors with Equation until Convergence
D Theoretical Justification of OOD Detection for OpenCon
In this section, we theoretically show that OOD detection helps open-world representation learning by reducing the lower bound of loss E' (x). We start with the definition of the supervised loss of the Mean xED.
Classifier, which provides the lower bound Lemma D.2
Definition D.1. (Mean Classifier) the mean classifter is linear layer with weight matrix M whose e-th is the mean n. of representations of inputs with class e: A.
XES(e) [Ø(x),, where S(c) defined in Appendix
R
is the set of samples with predicted label c. The average supervised loss of its mean classifier is:
Czup ==
E
((x) R.-) 1ct 4c
(9)
Ey22
xES(e+)
Lemma D.2. Let 7 = plct = =c))cttee x Via it holds that
1 Cruu
E
Cx >
xED,
20

---

Published in Transactions on Machine Learning Research (1/2023)
Proof.
exp(2 10o(zT.2tp) .2+/7)
E
(x)
E
Σ log
xED
xED,
|P(x) x+EP(x)
Σx EN(()) exp(z 2 /T)
E
(zJ.. zzz [T) + [(xx
log Σ exp(z) exp(zT.z .2- EN(x
xED,
[(x)
xt((((
x+EP(x)
E
E
+0(x))()))))))) 6(x) +xxx"" 1/T log
E c (EAaEES(C))
exp(o(x) 66x 1//))
EVa Ex+
E
E
[x()T ((xx)) ((xx)/== -=Eaxes(e-) d(x) ((x))(((*))
ct x ES2(ct)
E
(fx)(1Ac) - - 1/T
at EDvi xES(ct)
= ppcc # c))
E
- -E
E ((x) - ( 1 c 46 + +plct (c7)
ct,e-Eda
xxS(et)
1 Suupp
where in (a) approximate the summation over the positive/negative set by taking the expectation over the positive/negative sample in set S(c) (defined in Appendix 4 and in (b) we apply the Jensen Incquality since the log is a concave function.
In the first step, we show in Lemma D.2 that EE Cx(x) is lower-bounder bounded by a constant 1-1 times supervised xED, loss Siup defined in Definition D.I Note that Ciup is non non-positive and close to -1 in practice. Then the lower bound of E (x) has a positive correlation with 7 = p(c =c Note that y can be reduced by xED,,
OOD detection. To explain this:
When we separate novelty samples and form Dn, it has fewer hidden classes than D.. With fewer hidden classes, the probability of ct being equal to cm in random sampling is decreased, and thus reduces the lower bound of the (x).
xED
In summary, OOD detection facilitates open-world contrastive learning by having fewer candidate classes.
E Discussion on Using Samples in D.,DD
We discussed in Scction 3.2 that samples from D, Dr,D, contain indistinguishable data from known and novel classes. In this section, we show that using these samples for protolype-based learning may be undesirable.
We first show that the overlapping between the novel and known classes in D, (2n can be non-trivial. In Figure 4 we show the distribution plot of the scores maXjey Hj o(x;). It is notable that there exists a large overlapping area when maxje> M. x(x,) > A. For visualization clarity, we color the known classes in blue and the novel classes in gray.
We next show that using this part of data will be harmful to representation learning. Specifically, we replace Ci to be the following loss:
Lknown = (x: (xx))
x
where we define B3 to be a minibatch with samples drawn from D, U (D, (..)) data with known classes, along with the unlabeled data predicted as known classes. And we apply two random augmentations for each sample and generate a multi-viewed batch Br. We denote the embeddings of the multi-viewed batch
21

---

Published in Transactions on Machine Learning Research (1/2023)
D
Du Dn
Density
1.0
maxx μjj T jEM
Φ (x;)
Score
Figure 4: Distribution of OOD score maxjeM m. o(x,) for unlabeled data Du CIFAR-100. For visual clarity, gray indicates samples from novel classes and blue indicates samples from known classes. The shaded area highlights samples from novel classes but misidentified as known classes.
as Ag. The positive set of embeddings is as follows:
Px(x) = {z) E 8' Nx(()) = Asiz,
where
Dt (pseudo-label)
if x; E D.. if x; E Dt ys (ground-truth label),
0:
Intuitively, Lynown is an extension of L1, where we utilize both labeled and unlabeled data from known classes for representation learning. The final loss now becomes:
LModified >n AgCnnown + AuCas
(10)
We show results in Table 7 suggests that using Da DD, for prototype-based lcarning is suboptimal.
Compared to the original loss, the seen accuracy drops by 6.6%. This finding
Table 7: Comparison with loss Mooiiiodd on CIFAR-100.
CIFAR-100
Method
All
Novel
Seen
LModified
47.7
46.4
62.4
COpenCon
53.7
48.7
69.0
F Proof Details
Proof of Lemma 4.2
22

---

Published in Transactions on Machine Learning Research (1/2023)
Proof.
P pxx, M)
argina 6.M
Σ
Σ
a(() log
argmax 6.M
Σ
(())oogp((((
(i(c)
CE
12
argmax 6.M
Σ
Σ
1(0 = (x
eEVin
argmax 6.M
Σ
Σ
l,
CEJAn XES(c)
argmax 6.M
Σ
Σ
x(x)²
Pe:
eE XES(e)
where equation (a) is given by removing the constant term q(c)) log p(c) in argmax, (b) is by plugging (c)),, (d) is by reorganizing the index, and (e) is by plugging the vMF density function and removing the constant.
qí(c
Proof of Lemma 4.3
Proof.
argmni E
Lx(x) = argmin
Σ
o(x)
-z
[P()]
x
x
Ex(
argmin
Σ
E
o(x) +(xx)
S
c xE)(e)
x+ES(c))x
Σ o(x) d(xxt) ) -11
argmin
Seel -
oE)aan XES(e)
a+tE(())
argmin
Σ
Σ
ns, 1
eEVan XES(c)
argmax
Σ
Σ
o(x)
xES(e)
where in (a) ns. = Sa.l
Js. - (x [1x)]]l2 a constant value close to 1, and in (b) we show the approximation
to the optimization target in Equation with the fixed prototypes. The proof is done by using the equation in Lemma
G Results on CIFAR-10
We show results for CIFAR-10 in Table g where OpenCon consistently outperforms strong baselines, particularly ORCA and GCD. Classes are divided into 50% known and 50% novel classes. We then select 50% of known classes as the labeled dataset and the rest the unlabeled set. The division is consistent with Cao et al. 2022 which allows us to compare the performance in fair setting.
H More Qualitative Comparisons of Embeddings
we visualize the feature embeddings for a subset 20 classes using UMAP MMcInnes et al. 2018). This covers more classes than what has been shown in the main paper (Figure D The model trained on ImageNet-100. OpenCon produces more compact and distinguishable embedding space than GCD and ORCA.
In Figure 5
23

---

Published in Transactions on Machine Learning Research (1/2023)
Table Results on CIFAR-10. Asterisk (*) denotes that the original method can not recognize seen classes. Dagger (1) denotes the original method can not detect novel classes (and we had extend it). Results on GCD, ORCA, and OpenCon (mean and standard deviation) are averaged over five different. runs. The ORCA
results are reported by running the official repo (Cao et al.).
CIFAR-10 Novel
Method
All
Secn
Fix (Kurkin et al.
2020
49.5
50.4
71.5
D Guo et al.
2020
40.2
45.3
77.6
tCGDL Sun
2020)
39.7
44.6
72.3
DTC (Han et al.
2019
38.3
39.5
53.9
RankStats [Zhao & Han
2021
82.9
81.0
86.6
SimCLR (Chen et al. 2020a
51.7
63.4
58.3
ORCA Cao et al. 20222
88.3
87.5
89.920.4
GCD
Vaze et al. 2022
87.5
86,7
90.1 #:033
Open (Ours)
90.4 20.6
91.1 #0.1
89.3 3.0.2
ORCA
GCD
Ours (More Compact)
Embeddings from Novel Classes
Embeddings from Seen Classes
Figure 5: UMAP MeInnes et al. 2018 visualization of the feature embedding from 20 classes (10 for seen, 10 for novel). The model is trained on ImageNet-100 with ORCA (Cao et al. 2022), GCD (Vaze al. 2022). and OpenCon (ours).
Hyperparameters and Sensitivity Analysis
In this section, we introduce the hyper-parameter settings for OpenCon. We also show a validation strategy to determine important typer-parameters (weight and temperature) in loss LOpenCon and conduct a sensitivity analysis to show that the validation strategy can select near-optimal the basic training setting.
We start by introducing
For CIFAR-100 CITARR-100/ImageNee 1000 the model is trained for 200/120 epochs with batch-size 512 using stochastic
gradient descent with momentum 0.9, and weight decay 10-4, The learning rate starts 0.02 and decays by factor of 10 at the 50% and the 75% training stage. The momentum for prototype updating is fixed at 0.9. The percentile for OOD detection is 70%. We fix the weight for the KL-divergence regularizer to be 0.05.
Since the label for D, is not available, we propose a validation strategy by using labeled data D,. Specifically, we split the classes in Yi equally into two parts: known classes and "novel" classes (for which we know the labels). Moreover, 50% samples of the selected known classes are labeled. We further use the new validation dataset to select the best hyper-parameters by grid searching. The selected hyper-parameter groups are
24

---

Published in Transactions on Machine Learning Research (1/2023)
55
55
56
54.2
54
53.7
53.7
53.7
53.7
Accuracy (ccuracy-(Al) (All)
Accuracy-(Al)
53.3
53
Accuracy-(AN)
53.1
53
52.5
52.8
52.6
0.01
0.05
0.1
0.2
0.3
0.1
0.15
0.2
0.25
0.3
0.5
0.75
1,0
1.5
2.0
Xn
X
X
55
55
55
54.5
54
53,7
53.8
54
53.7
53.7
54
53.7
Accuracy (Al))
Accuracy-(Al))
Accuracy-(AN) (All)
53
53
52.9
53
52.6
52.6
53
53
52.7
52.8
52
52
52
51
50
0.5
0.6
0.7
0.8
0.9
0.05
0.075
0.1
0.125
0.15
0.2
0.3
0.4
0.5
0.6
Tn
T
T 2
Figure 6: Sensitivity analysis of hyper-parameters CIFAR-100. The overall accuracy reported on three (weight, temperature) pairs: (Ans Tn) for loss Lns (As, π) for loss LJ, and the (Aus Tu) for loss L., The middle bar in each plot corresponds to the hyperparameter value used in our main experiments.
summarized in Table 9 Note that the only difference between CIFAR-100 and ImageNet-100 scttings is the temperature of the self-supervised loss Cu.
We show the sensitivity of reported by fixing other hyper-parameters. We see that our validation strategy successfully selects A Au and 7, with the optimal one, and the other three Ans Ta and Tn are close to the optimal (with <1% gap in overall accuracy).
in Figure 6 The performance comparison the bar plot for
each typer-parameter
Table 9: Hyperparameters in OpenCon.
An
X
X
ImageNett100
0.1
0.7
0.2
0.1
0.6
CIFAR-100
0.1
0.7
0.2
0.1
0.4
J OOD Detection Comparison
We compare different OOD detection methods in Table 10 methods produce similar OOD detection performance. Note that Mahalanobis (Lee al.) 2018 require heavier computation which causes an unbearable burden in the training stage. Our method incurs minimal computational overhead.
Results show that several popular 0OD detection
25

---

Published in Transactions on Machine Learning Research (1/2023)
Table 10: Comparison of OOD detection performance with popular methods. Results are reported on CIFAR-100. Samples from 50 known classes treated in-distribution (ID) data and samples from the remaining 50 classes are used as oul-of-distribution (OOD) data. The OOD detection threshold is estimated on the labeled known classes Dr.
Method
FPR95
AUROC
MSP (Hendrycks & Gimpel 2017)
58.9
86.0
Energy
(Liu ct al. 2020
57.1
87.4
Mahalanobis Lee et al. 2018
54.6
88.7
Ours
57.1
87.4
K Sensitivity Analysis on Estimated Class Number
In this section, we provide the sensitivity analysis of the estimated class mumber on CIFAR-100. We noticed that when the estimated class number is much larger than the actual class number (100), some prototypes will never be predicted by any of the training samples. So the training process will converge to solutions that closely match the ground truth number of classes. We show Table 11 comparing the initial class number and the class number after converging. It shows that an overestimation of the class number has a negligible effect on the final converged class number.
Table 11: Converged class numbers under different initial class numbers.
Initial Class Number
100
110
120
130
150
200
Converged Class Number
100
106
108
109
109
109
L Performance on ImageNet-1k
To verify the effectiveness of OpenCon in large-scale setting, we provide the results on ImageNet- Table 12 The experiments are based on 500 known classes and 500 novel classes, with a labeling ratio 0.5 for known classes. We notice that ORCA's performance degrades significantly in a large-scale setting.
OpenCon significantly outperforms the best baseline GCD by 8.8%, in terms of overall accuracy.
Table 12: Comparison of classification accuracy on ImageNet-
ImageNet-1k Novel
Method
All
Seen
ORCA
17.6
20.5
35.3
GCD
35.8
36.9
69.0
OpenCon (ours)
44.6
42.5
73.9
M Differences w.r.t. Prototypical Contrastive Learning
Our work is inspired by yet differs substantially from unsupervised prototypical contrastive learning (PCL) ct al. 2021b), in terms of problem setup (Section D learning objective (Section 3), and theoretical insights (Section] 4 Importantly, PCL relies entirely on the unlabeled data D,, and does not leverage the availability of the labeled dataset. Instead, we consider the open-world cemi-supervised learning setting (Cao et al.) (022),, and leverage both labeled data (from known classes) and unlabeled data (from both known and novel classes). This setting leads to unique challenges that were not considered in PCL, such as how to learn strong representations for both the known and novel classes in unified framework; and how to perform OOD
26

---

Published in Transactions on Machine Learning Research (1/2023)
detection to separate known and novel samples in the unlabeled data. OpenCon makes contributions by addressing these challenges in an end-to-end trainable framework, supported by theoretical analysis.
In Table 13 we show evidence that the PCL does not trivially apply in this open-world setting, and under-performs OpenCon.
Table 13: Comparison with PCL.
CIFAR-100
Method
All
Novel
Seen
PCL (Li et al. 2021b)
42.1
44.5
39.0
OpenCon (Ours)
53.7
48.7
69.0
N Experiments on Fine-grained Datasets
The fino-grained task is more challenging since the known and novel classes can be similar. We provide the results of the fine-grained task on Stanford Cars Krauss et al., 2013), CUB-200 (Vize et al. 2021 and Herbarium19 (Tan et al. 2019) in Table 14 We adopt the ViT-B 16 architecture with DINO pre- pre-trained weights (Caron et al. 2021). Results show that OpenCon outperforms the best baseline GCD by 10.1% on the Stanford Cars dataset, terms of overall accuracy.
Table 14: Comparison on fine-grained datasets.
CUB-200
Stanford Cars
Herbarium19 Novel
Method
All
Novel
Seen
All
Novel
Seen
All
Seen
k-Means
MacQueen,
1967
34.3
32.1
38.9
13.8
10.6
12.8
12.9
12.8
12.9
RankStats+ ats+
Zhao
Han 2021
33.3
24.2
51.6
28.3
12.1
61.8
27.9
12.8
55.8
UNO+
F
Fini et al.
202
35.1
28.1
49.0
35.5
18.6
70.5
28.3
14.7
53.7
GCD
Vaze
2022
51.3
48.7
56.6
39.0
29.9
57.6
35.4
27.0
51.0
OpenCon (Ours)
54.7
52.1
63.8
49.1
32.7
78.6
39.3
28.6
58.9
O Hardware and Software
We run all experiments with Python 3.7 and Py Porch 1.7.1, using NVIDIA GeForce RTX 2080Ti GPUs.
27


Reviews:
[{"review_id": "AhGId3Hyqr", "reviewer": "Reviewer_MY9C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The proposed framework seems to be performing well.\n\nS2: Other than the generalized category discovery problem, the proposed framework also brings good performance for OOD detection, which is very interesting.\n\n\nW1: The paper itself has many overclaims, which may mislead the readers and does not give enough credit for existing works.\n\nW2: Some important baselines are missing, see Requested changes.\n\nW3: Also some important datasets for evaluating generalized category discovery are missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the generalized category discovery problem where the unlabelled data contains classes from both seen classes in the labelled data and novel classes, the goal is to learn to classify all classes.\nThis paper proposed a contrastive learning frameworks which selects the positive and negative pair via a moving average prototype, good performances are obtained on two datasets.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. In Table 1, the bottom row should be 'Generalized Category Discovery' [R1] or 'Open-World Semi-Supervised Learning' [R2], marking it as 'Ours' does not give enough credit to previous works and may mislead the reader (This problem setting is not new).\n2. As shown in [R1], performing generalized category discovery on fine-grained datasets is a more challenging and practical scenario, because the class-wise relation between seen and novel classes are clear (They are all same entry-level classes, e.g. all bird classes for CUB dataset), and the fine-grained differences between classes make it a harder problem to tackle for representation learning, thus I think the paper should include experiments on these datasets to support the claim of state-of-the-art performance.\n3. In [R1], the authors also provide strong baselines modified from strong novel category discovery methods, namely RankStat+[R3] and UNO+[R4], it is also shown that UNO+ can outperform GCD in some cases, I think this paper should also include these two baselines for a comparison to give the readers more context. It is also shown in [R1] that a simple k-means on DINO[R8] pretrained features could performs very well, I would suggest the author to include this baseline into the paper.\n4. As the overall framework for tackling the GCD problem is very similar to [R5], some more ablations should be conducted in my opinion, such as 1) using the prototypes from k-means instead of moving average, maybe also using the prototypes from a semi-supervised k-means [R1], 2) using the prototypical contrastive learning from [R5] (i.e. pull closer positive prototypes, push away negative prototypes) instead of the instance contrastive learning in the paper, 3) how well does the prototypes from k-means or semi-supervised k-means performs in the prototype-based novelty detection? 4) How does overclustering like in [R5] influences the performance?\n5. I think this paper is confusing the use of 'representation learning', in my opinion, 'representation learning' refers to 'learning representations of the data that make it easier to extract useful information when building classifiers or other predictors' [R6], yet this paper only evaluates on classification tasks, it is hard to say if the representation learned is good for transfer learning (which is widely adopted for evaluating representation in computer vision) [R7]. I would suggest the author to tune down the claim about representation learning, and change to a more appropriate title, or try to evaluate the representation learned by the proposed methods using [R7].\n6. As [R1] adopts DINO [R8] pretrained checkpoints on ImageNet, and the fact that in practice, one can always have the ImageNet pretrained weights available, I would suggest the paper to have more comparison on the DINO pretrained weights, e.g. apply DINO weights for all methods in table 2.\n7. The second sentence in the abstract is not correct in my opinion, the vast majority of representation learning methods indeed could work in the setting this paper is trying to tackle, in fact, the state-of-the-art representation learning methods are self-supervised, thus they do not face the 'open world' problem mentioned in the paper [R9, R10].\n8. In my opinion, Lemma 2 is trying to prove that pull positive pairs determined by the moving average prototypes closer is equivalent to aligning the examples to the corresponding prototypes, this is not a new discovery, although novelty is not a criteria for TMLR submission, I would suggest the author to cite previous works already showing this [R11] and [R12].\n9. One thing that is not very clear after I read the paper, how is the label assigned to each testing sample? In [R1], the label assign is done by using semi-supervised k-means, in [R2], it is done by a parametric classifier, how does the proposed method do this? Is it k-means or a parametric classifier?\n\n[R1] Generalized Category Discovery, CVPR 2022.\n\n[R2] Open-World Semi-Supervised Learning, ICLR 2022.\n\n[R3] Automatically discovering and learning new visual categories with ranking statistics, TPAMI 2021.\n\n[R4] A Unified Objective for Novel Class Discovery, ICCV 2021.\n\n[R5] Prototypical Contrastive Learning of Unsupervised Representations, ICLR 2021.\n\n[R6] Representation Learning: A Review and New Perspectives, https://arxiv.org/abs/1206.5538\n\n[R7] A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark, https://arxiv.org/abs/1910.04867\n\n[R8] Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021.\n\n[R9] Momentum Contrast for Unsupervised Visual Representation Learning, CVPR 2020.\n\n[R10] Masked Autoencoders Are Scalable Vision Learners, CVPR 2022.\n\n[R11] Relax, no need to round: integrality of clustering formulations, ITCS 2015.\n\n[R12] Spectral relaxation for k-means clustering, NeuIPS 2001.\n\nSubjective comments\n\n1. Although this is highly subjective, I would suggest the author to change the last sentence in the abstract, as I think the paper does not provide enough context for the readers, a sentence like this may mislead the readers.\n2. I would suggest the author to tune down the claims made in the abstract, as some of them may not be correct as already mentioned previously.\n\n\n\nFormatting issues\n\n1. The first equation in sec 4.1 should remove the argmax, this equation is comparing the value of the log likelihood, not the argmax of the log likelihood. The same goes for the equation in Lemma 4.2\n2. Last sentence of sec 8, 'in included' -> 'is included'\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m9lHq96wER", "reviewer": "Reviewer_wCPe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed setting/problem/direction is important and more realistic than previous methods. Studying in the OpenWorld with data from unknown classes for which we have no prior knowledge is important. The proposed prototype-based framework is straightforward, and all the components are designed reasonably. Some theoretical analyses are included to provide insights and intuition about the proposed framework. Overall the paper is easy to follow. \n\nExtensive experiments and comparisons are conducted in two datasets, CIFAR-100 and IN-100. An extensible to ViT is included. Proper ablation studies are also conducted to prove the effectiveness of the proposed components. \n\nAll current experiments are limited to 100 classes, a huge gap with practical applications. It is not convinced that the proposed method works in this scenery can achieve great performance in scale data.\n\nSince the prototype-based novelty discovery mechanism is one of the key points proposed by this paper, it is probably better to also include ablation studies in IN-100 instead of only testing on CIFAR-100. \n\nFor Table 6, it would be better to also include the Labeling Ratio = 0.5, $y_l$ = 50 result for better comparisons. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a new setting, open-world contrastive learning, and a framework accordingly. This setting emphasizes we should utilize both labeled data and unlabeled data from unknown classes for representation learning. A prototype-based framework is proposed with minor modifications to the existing prototype framework and contrastive learning formulation. Some theoretical analyses are included to help understand the proposed framework. They tested in two common benchmarks, CIFAR-100 and IN-100, and shown the proposed method consistently outperforms the baselines. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please address all the above concerns, especially the performance in scale-up setting concerns.\n- The author claims, \"Cao et al. (2022) focus on classification accuracy, but not learning high-quality embeddings.\", while they only report classification accuracy. Is there any other evidence to support the proposed framework can learn \"stronger\" learning representations?\n- Table 6, The proposed method under Labeling Ratio = 0.25, $y_l$ = 50 has the same IN-100 Seen results as Labeling Ratio = 0.5, $y_l$ = 50. This is weird, and the reviewer is unsure if there are any inputting errors.\n- OCRA achieved higher accuracy in Novel classes under Labeling Ratio = 0.25, $y_l$ = 50 than Labeling Ratio = 0.5, $y_l$ = 50. Could the author please explain the reason or provide analyses here?", "broader_impact_concerns": "There are no concerns about the ethical aspects in the reviewer's mind currently.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rmB8rcmYnu", "reviewer": "Reviewer_6dAv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. The problem considered in this work (i.e. semi-supervised learning with unlabeled data for unknown classes) is of clear practical relevance due to the usual unconstrained collection of unlabeled data) and interesting since few previous works considered it.\n2. The OpenCon framework is tailored to the task (e.g. considering separate loss components for each part of the dataset) and intuitive. It achieves remarkable performance and, due to its simplicity, can serve as a good baseline method for this task.\n4. Section 4 provides also a theoretical analysis of OpenCon.\n\n**Weaknesses**:\n1. My main concern is that the manuscript currently does not give enough credit to OCRA and GCD, and the fact that the tackled setting is not novel in terms of problem setup. To clarify: the article does not claim that the problem set is new in terms of data but stresses that it is new in terms of learning objective (i.e. representation vs classification accuracy of OCRA). My concern raises from the fact that multiple parts of the manuscript hint at the first statement. Examples are Table 1, where the problem is denoted as \"ours\", without referring e.g. to the open-world SSL problem pushed by OCRA. The same applies to the second paragraph of Section 1. Similarly for Section 2 both the first sentence and the second paragraph (i.e. difference w.r.t. existing problem setting), something that is clarified only at the end of the section with the explicit reference to OCRA. These statements should be revised (e.g. Sections 1-2, Table 1) and the differences clarified to avoid misunderstandings regarding the contributions.  \n2. Following on the previous, also GCD has the same objective as OpenCon, i.e. learning good representations for unknown classes (as clarified in Section 5). GCD should also be discussed in Section 2. Eventually, the title could also be made more specific to highlight the main contribution of the work (i.e. OpenCon, the contrastive objective/framework tailored for this problem). \n3. In Section 8, it would be interesting to discuss some weaknesses of the model and potential avenues for future work.\n\n**Minors**:\n1. I find the analyses in Appendix E (i.e. why not use all unlabeled samples) very insightful, given that how to deal with unlabeled samples is a crucial part of the approach (as shown in Table 4).  Due to its importance, it would be helpful to include part of this analysis in the main manuscript. \n2. Typo \"in included\", the last sentence of Section 8.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article proposes an approach, Open-world Contrastive learning (OpenCon) for the task of open-world representation learning. Similarly to semi-supervised learning, in this task, there are both labeled data and unlabeled data but the latter may belong to both known (labeled) and unknown (unlabeled) classes. OpenCon builds on five components: \n1. Supervised contrastive loss on the labeled samples.\n2. Prototype-based novelty recognition, where all samples very dissimilar (according to a threshold) w.r.t. the existing prototypes are considered as belonging to unknown classes. Note that prototypes for unknown classes are pre-initialized.\n3. Contrastive loss on novel classes, where the identified unknown class samples belong to the same positive set if they share the same closest prototype reserved for novel classes. \n4. Prototypes update through the moving average of their respective samples.\n5. Unsupervised loss (i.e. SimCLR) on unlabeled samples.\n\nExperiments on CIFAR-100 and ImageNet-100 show that OpenCon outperforms the state-of-the-art methods for this setup, ORCA (Cao et al., 2022) and GCD (Vaze et al., 2022). ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am leaning toward recommending the acceptance of the manuscript. Following on the points above, my request is on revising Table 1, and Sections 1-2 (and eventually the title) to better clarify the contribution of the manuscript w.r.t. OCRA and GCD, and avoid that a reader may consider the problem setup as an actual contribution (as correctly not indicated in the last part of Section 1).\n\nAs an additional final point: including a discussion of the weaknesses of OpenCon and potential future works in Section 8 would make the article more complete.", "broader_impact_concerns": "The article discusses the broader impact in Appendix A. There are no clear concerns but the ones related to the actual available data: the algorithm may identify unlabeled minority sub-groups if applied to images of humans and/or reflect biases in the data collection (e.g. specific attributes associated only with specific sub-groups). These of course are almost unrealistic setups/cases that would go way beyond the scope for which the algorithm has been developed: they may be nevertheless added to Appendix A for completeness.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "W5YLmyj5Xo", "reviewer": "Reviewer_6PFD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Paper writing is clear and easy to understand.\n2. Ablation study and analysis are comprehensive.\n\nWeakness:\n1. I think the proposed method is very similar to the cited Li et al. 2021b, and also shares a similar spirit with previous self-training works [1]. Basically, the idea is to assign labels to high-confident unlabeled data, plus unsupervised learning loss to encourage diverse sample-wise features. I think the concept of prototype learning, even combined with supervised and unsupervised learning, is not novel at this moment.\n\n2. The benchmark is very limited. CIFAR-100 and ImageNet-100 are far from being large enough. For example, most contrastive learning works tested on the standard ImageNet dataset. As this work also involves unsupervised learning, experiments on the standard ImageNet are required for a fair comparison.\n\n3. Lemmas that claims the equivalence between the proposed method and the EM algorithm are not significant. Prototypes are natively centroids in EM. This is also explained in [1].\n\n[1] Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes to solve open-world representation learning. In addition to the standard supervised learning loss and the contrastive loss for unsupervised learning, the authors further estimate and update the prototypes of unknown classes to promote the learning of distinguishable features on the unlabeled set.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The most change I hope the authors could include is the experiments on the standard ImageNet. Other tasks like detection (as did in Li et al. 2021b) would be good but not required.\n\n2. The authors should also elaborate more on the core contribution and benefit of the proposed method compared with previous EM-like unsupervised learning methods.", "broader_impact_concerns": "In general, I think the contribution and impact of this work are not enough. Although the authors provide clear writing and extensive ablation studies, the core idea is not novel.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an algorithm for performing representation learning in an open world semi-supervised setting (i.e. with both known and unknown classes). The paper provides good empirical evidence for the efficacy of the approach on a number of tasks. The paper also links OpenCon to the EM algorithm to provide further justification for the approach.

Initially, reviewers had concerns with the positioning of the paper with respect to prior work on this problem, but in the updated version the claims have been revised to address these concerns. The reviewers also raised concerns with narrow experiments. With the additional of a larger dataset (full ImageNet), fine-grained recognition problems, additional baselines, and additional ablations, these have been adequately addressed.

Open world SSL / Generalized Category Discovery is a topic of interest to the community, and this paper makes a good contribution to the area.

Congratulations on the acceptance!
Decision: Accept (poster)