Title: A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: open-world learning

clustering

spectral analysis
Abstract: Open-world semi-supervised learning aims at inferring both known and novel classes in unlabeled data, by harnessing prior knowledge from a labeled set with known classes. Despite its importance, there is a lack of theoretical foundations for this problem. This paper bridges the gap by formalizing a graph-theoretic framework tailored for the open-world setting, where the clustering can be theoretically characterized by graph factorization. Our graph-theoretic framework illuminates practical algorithms and provides guarantees. In particular, based on our graph formulation, we apply the algorithm called Spectral Open-world Representation Learning (SORL), and show that minimizing our loss is equivalent to performing spectral decomposition on the graph. Such equivalence allows us to derive a provable error bound on the clustering performance for both known and novel classes, and analyze rigorously when labeled data helps. Empirically, SORL can match or outperform several strong baselines on common benchmark datasets, which is appealing for practical usage while enjoying theoretical guarantees.

A Graph-Theoretic Framework for Understanding Open-World Semi-Supervised Learning

Abstract

Open-world semi-supervised learning aims at inferring both known and novel classes in unlabeled data, by harnessing prior knowledge from a labeled set with known classes. Despite its importance, there is a lack of theoretical foundations for this problem. This paper bridges the gap by formalizing a graph-theoretic framework tailored for the open-world setting, where the clustering can be theoretically characterized by graph factorization. Our graph-theoretic framework illuminates practical algorithms and provides guarantees. In particular, based on our graph formulation, we apply the algorithm called Spectral Open-world Representation Learning (SORL), and show that minimizing our loss is equivalent to performing spectral decomposition on the graph. Such equivalence allows us to derive a provable error bound on the clustering performance for both known and novel classes, and analyze rigorously when labeled data helps. Empirically, SORL can match or outperform several strong baselines on common benchmark datasets, which is appealing for practical usage while enjoying theoretical guarantees. Our code is available at https://github.com/deeplearning-wisc/sorl .

1 Introduction

Machine learning models in the open world inevitably encounter data from both known and novel classes [2, 15, 16, 65, 79]. Traditional supervised machine learning models are trained on a closed set of labels, and thus can struggle to effectively cluster new semantic concepts. On the other hand, open-world semi-supervised learning approaches, such as those discussed in studies [7, 63, 69], enable models to distinguish both known and novel classes , making them highly desirable for real-world scenarios. As shown in Figure 1, the learner has access to a labeled training dataset D l (from known classes) as well as a large unlabeled dataset D u (from both known and novel classes). By optimizing feature representations jointly from both labeled and unlabeled data, the learner aims to create meaningful cluster structures that correspond to either known or novel classes. With the explosive growth of data generated in various

<latexi sh

1_b

64="J

f

B

Z0q

SITO

/

yr

g

>A

c

VDL

M

F

2p du

k

w

W

o

5

j

m

U

+

XC

Y

Q

3P

G8

R

N

K

v

E

z

n

7

D

Labeled Data

l

Unlabeled Data

u

?

?

?

D

<latexi sh

1_b

64="np

2

TCY

83v

KGwJ

FrARQ

>

B9H

c

VDLS

N

f

Zdu

o

gq

y

0

U

mk

+

/

P

O

7

W

z

I

5

M

X

E

j

domains, open-world semi-supervised learning has emerged as a crucial problem in the field of machine learning.

Motivation. Different from self-supervised learning [5, 8, 11, 12, 23, 26, 68, 77], open-world semi-supervised learning allows harnessing the power of the labeled data for possible knowledge sharing and transfer to unlabeled data, and from known classes to novel classes. In this joint learning process, we argue that interesting intricacies can arise-the labeled data provided may be beneficial or unhelpful to the resulting clusters. We exemplify the nuances in Figure 1. In one scenario, when the model learns the labeled known classes (e.g., traffic light) by pushing red and green lights closer, such a relationship might transfer to help cluster green and red apples into a coherent cluster. Alternatively, when the connection between the labeled data and the novel class (e.g., flower) is weak, the benefits might be negligible. We argue-perhaps obviously-that a formalized understanding of the intricate phenomenon is needed.

Theoretical significance. To date, theoretical understanding of open-world semi-supervised learning is still in its infancy. In this paper, we aim to fill the critical blank by analyzing this important learning problem from a rigorous theoretical standpoint. Our exposition gravitates around the open question: what is the role of labeled data in shaping representations for both known and novel classes? To answer this question, we formalize a graph-theoretic framework tailored for the open-world setting, where the vertices are all the data points and connected sub-graphs form classes (either known or novel). The edges are defined by a combination of supervised and self-supervised signals, which reflects the availability of both labeled and unlabeled data. Importantly, this graph facilitates the understanding of open-world semi-supervised learning from a spectral analysis perspective, where the clustering can be theoretically characterized by graph factorization. Based on the graph-theoretic formulation, we derive a formal error bound by contrasting the clustering performance for all classes, before and after adding the labeling information. Our Theorem 4.2 reveals the sufficient condition for the improved clustering performance for a class. Under the K-means measurement, the unlabeled samples in one class can be better clustered, if their overall connection to the labeled data is stronger than their self-clusterability.

Practical significance. Our graph-theoretic framework also illuminates practical algorithms with provided guarantees. In particular, based on our graph formulation, we present the algorithm called Spectral Open-world Representation Learning (SORL) adapted from Sun et al. [64]. Minimizing this loss is equivalent to performing spectral decomposition on the graph (Section 3.2), which brings two key benefits: (1) it allows us to analyze the representation space and resulting clustering performance in closed-form; (2) practically, it enables end-to-end training in the context of deep networks. We show that our learning algorithm leads to strong empirical performance while enjoying theoretical guarantees. The learning objective can be effectively optimized using stochastic gradient descent on modern neural network architecture, making it desirable for real-world applications.

2 Problem Setup

We formally describe the data setup and learning goal of open-world semi-supervised learning [7].

Data setup. We consider the empirical training set D l ∪ D u as a union of labeled and unlabeled data.

The unlabeled set D u = { ¯ x i } m i =1 , where each sample ¯ x i can come from either known or novel classes 1 . Note that we do not have access to the labels in D u . For mathematical convenience, we denote the underlying label set as Y all, where Y l ⊂ Y all . We denote C = |Y all | the total number of classes.

The labeled set D l = { ¯ x i , y i } n i =1 , with y i ∈ Y l . The label set Y l is known.

The data setup has practical value for real-world applications. For example, the labeled set is common in supervised learning; and the unlabeled set can be gathered for free from the model's operating environment or the internet. We use P l and P to denote the marginal distributions of labeled data and all data in the input space, respectively. Further, we let P l i denote the distribution of labeled samples with class label i ∈ Y l .

Learning target. Under the setting, our goal is to learn distinguishable representations for both known and novel classes simultaneously. The representation quality will be measured using classic metrics, such as K-means clustering accuracy, which we will define mathematically in Section 4.2.2. Unlike classic semi-supervised learning [86], we place no assumption on the unlabeled data and allow its semantic space to cover both known and novel classes. The problem is also referred to as open-world representation learning [63], which emphasizes the role of good representation in distinguishing both known and novel classes.

Theoretical analysis goal. We aim to comprehend the role of label information in shaping representations for both known and novel classes. It's important to note that our theoretical approach aims to understand the perturbation in the clustering performance by labeling existing, previously unlabeled data points within the dataset. By contrasting the clustering performance before and after labeling these instances, we uncover the underlying structure and relations that the labels may reveal. This analysis provides invaluable insights into how labeling information can be effectively leveraged to enhance the representations of both known and novel classes.

3 A Spectral Approach for Open-world Semi-Supervised Learning

In this section, we formalize and tackle the open-world semi-supervised learning problem from a graph-theoretic view. Our fundamental idea is to formulate it as a clustering problem-where similar data points are grouped into the same cluster, by way of possibly utilizing helpful information from the labeled data D l . This clustering process can be modeled by a graph, where the vertices are all the data points and classes form connected sub-graphs. Specifically, utilizing our graph formulation, we present the algorithm - Spectral Open-world Representation Learning (SORL) in Section 3.2. The process of minimizing the corresponding loss is fundamentally analogous to executing a spectral decomposition on the graph.

3.1 A Graph-Theoretic Formulation

We start by formally defining the augmentation graph and adjacency matrix. For clarity, we use ¯ x to indicate the natural sample (raw inputs without augmentation). Given an ¯ x , we use T ( x | ¯ x ) to denote the probability of x being augmented from ¯ x . For instance, when ¯ x represents an image, T ( ·| ¯ x ) can be the distribution of common augmentations [11] such as Gaussian blur, color distortion, and random cropping. The augmentation allows us to define a general population space X , which contains all the original images along with their augmentations. In our case, X is composed of augmented samples from both labeled and unlabeled data, with cardinality |X| = N . We further denote X l as the set of samples (along with augmentations) from the labeled data part.

We define the graph G ( X , w ) with vertex set X and edge weights w . To define edge weights w , we decompose the graph connectivity into two components: (1) self-supervised connectivity w ( u ) by treating all points in X as entirely unlabeled, and (2) supervised connectivity w ( l ) by adding labeled information from P l to the graph. We proceed to define these two cases separately.

First, by assuming all points as unlabeled, two samples ( x , x + ) are considered a positive pair if:

Unlabeled Case (u): x and x + are augmented from the same image ¯ x ∼ P .

For any two augmented data x, x ′ ∈ X , w ( u ) xx ′ denotes the marginal probability of generating the pair:

which can be viewed as self-supervised connectivity [11, 23]. However, different from self-supervised learning, we have access to the labeled information for a subset of nodes, which allows adding additional connectivity to the graph . Accordingly, the positive pair can be defined as:

Labeled Case (l): x and x + are augmented from two labeled samples ¯ x l and ¯ x ′ l with the same known class i . In other words, both ¯ x l and ¯ x ′ l are drawn independently from P l i .

Considering both case (u) and case (l), the overall edge weight for any pair of data ( x, x ′ ) is given by:

and η u , η l modulates the importance between the two cases. The magnitude of w xx ′ indicates the 'positiveness' or similarity between x and x ′ . We then use w x = ∑ x ′ ∈X w xx ′ to denote the total edge weights connected to a vertex x .

Remark: A graph perturbation view. With the graph connectivity defined above, we can now define the adjacency matrix A ∈ R N × N with entries A xx ′ = w xx ′ . Importantly, the adjacency matrix can be decomposed into two parts:

which can be regarded as the self-supervised adjacency matrix A ( u ) perturbed by additional labeling information encoded in A ( l ) . This graph perturbation view serves as a critical foundation for our theoretical analysis of the clustering performance in Section 4. As a standard technique in graph theory [14], we use the normalized adjacency matrix of G ( X , w ) :

where D ∈ R N × N is a diagonal matrix with D xx = w x . The normalization balances the degree of each node, reducing the influence of vertices with very large degrees. The normalized adjacency matrix defines the probability of x and x ′ being considered as the positive pair from the perspective of augmentation, which helps derive the learning loss as we show next.

3.2 SORL: Spectral Open-World Representation Learning

We present an algorithm called Spectral Open-world Representation Learning (SORL), which can be derived from a spectral decomposition of ˜ A . The algorithm has both practical and theoretical values. First, it enables efficient end-to-end training in the context of modern neural networks. More importantly, it allows drawing a theoretical equivalence between learned representations and the topk singular vectors of ˜ A . Such equivalence facilitates theoretical understanding of the clustering structure encoded in ˜ A . Specifically, we consider low-rank matrix approximation:

According to the Eckart-Young-Mirsky theorem [17], the minimizer of this loss function is F k ∈ R N × k such that F k F ⊤ k contains the topk components of ˜ A 's SVD decomposition.

Now, if we view each row f ⊤ x of F as a scaled version of learned feature embedding f : X ↦→ R k , the L mf ( F, A ) can be written as a form of the contrastive learning objective. We formalize this connection in Theorem 3.1 below 2 .

Theorem 3.1. We define f x = √ w x f ( x ) for some function f . Recall η u , η l are coefficients defined in Eq. (2) . Then minimizing the loss function L mf ( F, A ) is equivalent to minimizing the following loss function for f , which we term Spectral Open-world Representation Learning (SORL) :

where

Proof. ( sketch ) We can expand L mf ( F, A ) and obtain

The form of L SORL ( f ) is derived from plugging w xx ′ (defined in Eq. (1)) and w x . Full proof is in Appendix A.

Interpretation of L SORL ( f ) . At a high level, L 1 and L 2 push the embeddings of positive pairs to be closer while L 3 , L 4 and L 5 pull away the embeddings of negative pairs . In particular, L 1 samples two random augmentation views of two images from labeled data with the same class label, and L 2 samples two views from the same image in X . For negative pairs, L 3 uses two augmentation views from two samples in X l with any class label. L 4 uses two views of one sample in X l and another one in X . L 5 uses two views from two random samples in X . This training objective, though bearing similarities to NSCL [64], operates within a distinct problem domain. Accordingly, we derive novel theoretical analysis uniquely tailored to our problem setting, which we present next.

4 Theoretical Analysis

So far we have presented a spectral approach for open-world semi-supervised learning based on graph factorization. Under this framework, we now formally analyze: how does the labeling information shape the representations for known and novel classes?

4.1 An Illustrative Example

We consider a toy example that helps illustrate the core idea of our theoretical findings. Specifically, the example aims to distinguish 3D objects with different shapes, as shown in Figure 2. These images are generated by a 3D rendering software [31] with user-defined properties including colors, shape, size, position, etc. We are interested in contrasting the representations (in the form of singular vectors), when the label information is either incorporated in training or not.

Data design. Suppose the training samples come from three types, X , X , X . Let X be the sample space with known class, and X , X be the sample space with novel classes. Further, the two novel classes are constructed to have different relationships with the known class. Specifically, X shares some similarity with X in color (red and blue); whereas another novel class X has no obvious similarity with the known class. Without any labeling information, it can be difficult to distinguish X from X since samples share common colors. We aim to verify the hypothesis that: adding labeling information to X (i.e., connecting and ) has a larger (beneficial) impact to cluster X than X .

(a) Definition of Augmentation Probability

(b) Adjacency Matrix

τ

s

τ

c

τ

0

τ

s

τ

c

τ

0

τ

1

τ

1

τ

1

τ

1

τ

0

τ

1

Known

Class

Novel

Class

Novel

Class

(1,2,3)

A

=

η

u

A

(

u

)

+

η

u

A

(

u

)

=

τ

0

τ

1

τ

1

T²+ T²+ ²

2T1Ts

2T1Tc

2TcTs

口

0

0

1

2T1TS

2

2TcTs

2T1Tc

1

0

0

2T1Tc

2TcTs

Ti

2

十

T

T

2

2T1Ts

0

0

2TcTs--

2T1Tc-

1

2T1Ts

T

2

1

0

0

0

0

1

0

0

2T2

2T2

口

0

0

0

0

2T2

2T2

?

(T1 +Ts)²

(T1 + Ts)²

Tc(T1+Ts）

Tc(T1+Ts)

0

0]

(1+Ta)²

(+Ts)²

Te(T1+Ts)_

Tc（T1+Ts)

0

0

Tc(T1 + Ts)

Tc(T1 + Ts)

T2

0

0

T

Tc(T1 +Ts)

Tc(T1 + Ts)

0

0

0

0

0

0

0

0

0

0

1

1

0

0

0

0

x

C

x

x

O

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="T

uMFzGO+jW

Cg9/5BnL

Rd2

oA

>

7X

c

V

NS8

EJ3Ur

q

0mk

y

Z

I

Q

P

f

Y

H

D

w

p

v

K

<latexi sh

1_b

64="d02/zNZv

YW

pFmM

q

O3

g

>A

B7X

c

V

S8

EJ

Ur

9L

k

o

y

I

Q

PCj

f

H

n

+

G

T

D

R

w

5

K

u

<latexi sh

1_b

64="

T

F

PMSL9

f

/

Kngy

N8

>A

B7X

c

V

EJ3Ur

q

0mk oW

2

Z

I

Q

Cj

z

Y

H

+

G

O

Dd

R

w

p

u

5

v

<latexi sh

1_b

64="d02/zNZv

YW

pFmM

q

O3

g

>A

B7X

c

V

S8

EJ

Ur

9L

k

o

y

I

Q

PCj

f

H

n

+

G

T

D

R

w

5

K

u

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

T

F

PMSL9

f

/

Kngy

N8

>A

B7X

c

V

EJ3Ur

q

0mk oW

2

Z

I

Q

Cj

z

Y

H

+

G

O

Dd

R

w

p

u

5

v

<latexi sh

1_b

64="d02/zNZv

YW

pFmM

q

O3

g

>A

B7X

c

V

S8

EJ

Ur

9L

k

o

y

I

Q

PCj

f

H

n

+

G

T

D

R

w

5

K

u

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="T

uMFzGO+jW

Cg9/5BnL

Rd2

oA

>

7X

c

V

NS8

EJ3Ur

q

0mk

y

Z

I

Q

P

f

Y

H

D

w

p

v

K

<latexi sh

1_b

64="d02/zNZv

YW

pFmM

q

O3

g

>A

B7X

c

V

S8

EJ

Ur

9L

k

o

y

I

Q

PCj

f

H

n

+

G

T

D

R

w

5

K

u

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

u0

Pov

7NBU

T

W

Gr

>A

Cc3

dVF

M

E

Xy

8

SJ

w9YDZ

I

n

2

/

j

g

R

K

p

f

H

+

5

z

L

Q

q

m

<latexi sh

1_b

64="QTB

DN3vpmK

u/

rwP

Vn

>A

CcX

d

F

M5

0

yIq

ELo

GW

Z

28J

z

R

Y

7

+

j

H

U

k

O

9

S

f

g

Augmentation graph. Based on the data design, we formally define the augmentation graph, which encodes the probability of augmenting a source image ¯ x to the augmented view x :

̸

̸

̸

With Eq. (7) and the definition of the adjacency matrix in Section 3.1, we can derive the analytic form of A ( u ) and A , as shown in Figure 2(b). We refer readers to Appendix B for the detailed derivation. The two matrices allow us to contrast the connectivity changes in the graph, before and after the labeling information is added. Insights. We are primarily interested in analyzing the difference of the representation space derived from A ( u ) and A . We visualize the top-3 eigenvectors 3 of the normalized adjacency matrix ˜ A ( u ) and ˜ A in Figure 3(a), where the results are based on the magnitude order τ 1 ≫ τ c > τ s > 0 . Our key takeaway is: adding labeling information to known class X helps better distinguish the known class itself and the novel class X , which has a stronger connection/similarity with X .

Qualitative analysis. Our theoretical insight can also be verified empirically, by learning representations on over 10,000 samples using the loss defined in Section 3.2. Due to the space limitation, we include experimental details in Appendix E.1. In Figure 3(b), we visualize the learned features through UMAP [43]. Indeed, we observe that samples become more concentrated around different shape classes after adding labeling information to the cube class.

(a2) Feature space with Top-3

eigenvectors of

(b2) UMAP visualization with

empirical feature space of

(a1) Feature space with Top-3

eigenvectors of

Add labels to 'cube'

Annotation:

(b1) UMAP visualization with

empirical feature space of

(More separable in color)

(More separable in shape)

˜

A

˜

A

(

u

)

˜

A

(

u

)

˜

A

1.2

1.2

1

1

0.5

0.5

0

0.2

0.4

0.4

0.6

0.6

-0.6-0.4-0.2

0

0.2

0.4

0.6

-0.6-0.4-0.2

0

0.20.40.6

4.2 Main Theory

The toy example offers an important insight that the added labeled information is more helpful for the class with a stronger connection to the known class. In this section, we formalize this insight by extending the toy example to a more general setting. As a roadmap, we derive the result through three steps: (1) derive the closed-form solution of the learned representations; (2) define the clustering performance by the K-means measure; (3) contrast the resulting clustering performance before and after adding labels. We start by deriving the representations.

4.2.1 Learned Representations in Analytic Form

Representation without labels. To obtain the representations, one can train the neural network f : X ↦→ R k using the spectral loss defined in Equation 6. We assume that the optimizer is capable to obtain the representation Z ( u ) ∈ R N × k that minimizes the loss, where each row vector z i = f ( x i ) ⊤ . Recall that Theorem 3.1 allows us to derive a closed-form solution for the learned feature space by the spectral decomposition of the adjacency matrix, which is ˜ A ( u ) in the case without labeling information. Specifically, we have F ( u ) k = √ D ( u ) Z ( u ) , where F ( u ) k F ( u ) ⊤ k contains the

̸

<latexi sh

1_b

64="Fn dR

W+k vqmYfuD

r

I

N

M

>A

B

H

c

V

LS

J3U

2P

G

C

Z

E

o

9

5

Qy

T

g

X

/

z

8

7

Kw

0

j

O

p

<latexi sh

1_b

64="

5p

z

J7RG

E

3

ku

K

qw

>A

B8H

c

V

NS

M

/

9

Ivg

UY

L

2

o

FC

T

f

Q

D

j

n

+

m

O

P

0

Z

d

X

y

<latexi sh

1_b

64="Fn dR

W+k vqmYfuD

r

I

N

M

>A

B

H

c

V

LS

J3U

2P

G

C

Z

E

o

9

5

Qy

T

g

X

/

z

8

7

Kw

0

j

O

p

<latexi sh

1_b

64="

5p

z

J7RG

E

3

ku

K

qw

>A

B8H

c

V

NS

M

/

9

Ivg

UY

L

2

o

FC

T

f

Q

D

j

n

+

m

O

P

0

Z

d

X

y

topk components of ˜ A ( u ) 's SVD decomposition and D ( u ) is the diagonal matrix defined based on the row sum of A ( u ) . We further define the topk singular vectors of ˜ A ( u ) as V ( u ) k ∈ R N × k , so we have F ( u ) k = V ( u ) k √ Σ ( u ) k , where Σ ( u ) k is a diagonal matrix of the topk singular values of ˜ A ( u ) . By equalizing the two forms of F ( u ) k , the closed-formed solution of the learned feature space is given by Z ( u ) = [ D ( u ) ] -1 2 V ( u ) k √ Σ ( u ) k .

Representation perturbation by adding labels. We now analyze how the representation is 'perturbed' as a result of adding label information. We consider |Y l | = 1 4 to facilitate a better understanding of our key insight. We can rewrite A in Eq. 3 as:

where we replace η l to δ to be more apparent in representing the perturbation and define l ∈ R N , ( l ) x = E ¯ x l ∼P l 1 T ( x | ¯ x l ) . Note that l can be interpreted as the vector of ' the semantic connection for sample x to the labeled data '. One can easily extend to r classes by letting l ∈ R N × r .

Here we treat the adjacency matrix as a function of the perturbation. In a similar manner as above, we can derive the normalized adjacency matrix ˜ A ( δ ) and the feature representation Z ( δ ) in closed form. The details are included in Appendix C.4.

4.2.2 Evaluation Target

With the learned representations, we can evaluate their quality by the clustering performance. Our theoretical analysis of the clustering performance can well connect to empirical evaluation strategy in the literature [75] using K -means clustering accuracy/error. Formally, we define the ground-truth partition of clusters by Π = { π 1 , π 2 , ..., π C } , where π i is the set of samples' indices with underlying label y i and C is the total number of classes (including both known and novel). We further let µ π = E i ∈ π z i be the center of features in π , and the average of all feature vectors be µ Π = E j ∈ [ N ] z j .

The clustering performance of K-means depends on two measurements: Intra-class measure and Inter-class measure. Specifically, we let the intra-class measure be the average Euclidean distance from the samples' feature to the corresponding cluster center and we measure the inter-class separation as the distances between cluster centers:

Strong clustering results translate into low M intra-class and high M inter-class . Thus we define the K-means measure as:

We also formally show in Theorem 4.1 that the K-means clustering error 5 is asymptotically equivalent to the K-means measure we defined above.

Theorem 4.1. ( Relationship between the K-means measure and K-means error .) We define the ξ π → π ′ as the index set of samples that is from class division π however is closer to µ π ′ than µ π . In other word, ξ π → π ′ = { i : i ∈ π, ∥ z i -µ π ∥ 2 ≥ ∥ z i -µ π ′ ∥ 2 } . Assuming | ξ π → π ′ | > 0 , we define below the clustering error ratio from π to π ′ as E π → π ′ and the overall cluster error ratio E Π ,Z as the Harmonic Mean of E π → π ′ among all class pairs:

̸

The K-means measure M kms (Π , Z ) has the same order of the Harmonic Mean of the cluster error ratio between all cluster pairs with proof in Appendix C.3.

The K-means measure M kms (Π , Z ) have a nice matrix form as shown in Appendix C.2 which facilitates theoretical analysis. Our analysis revolves around contrasting the resulting clustering performance before and after adding labels as we will shown next.

4.2.3 Perturbation in Clustering Performance

With the evaluation target defined above, our main analysis will revolve around analyzing 'how the extra label information help reduces M kms (Π , Z ) '. Formally, we investigate the following error difference, as a result of added label information:

where the closed-form solution is given by the following theorem. Positive ∆ kms ( δ ) means improved clustering, as a result of adding labeling information.

Theorem 4.2. (Main result.) Denote V ( u ) ∅ ∈ R N × ( N -k ) as the null space of V ( u ) k and ˜ A ( u ) k = V ( u ) k Σ ( u ) k V ( u ) ⊤ k as the rankk approximation for ˜ A ( u ) . Given δ, η 1 > 0 and let G k as the spectral gap between k -th and k +1 -th singular values of ˜ A ( u ) , we have:

where diag ( · ) converts the vector to the corresponding diagonal matrix and Υ ∈ R N × N is a matrix encoding the ground-truth clustering structure in the way that Υ xx ′ > 0 if x and x ′ has the same label and Υ xx ′ < 0 otherwise. The concrete form and the proof are in Appendix C.4.

Theorem 4.2 is more general but less intuitive to understand. To gain a better insight, we introduce Theorem 4.3 which provides more direct implications. We provide the justification of the assumptions and the formal proof in Appendix C.5.

Theorem 4.3. (Intuitive result.) Assuming the spectral gap G k is sufficiently large and l lies in the linear span of V ( u ) k . We also assume ∀ π c ∈ Π , ∀ i ∈ π c , l ( i ) =: l π c which represents the connection between class c to the labeled data. Given δ, η 1 , η 2 > 0 , we have:

Implications. In Theorem 4.3, we define the class-wise perturbation of the K-means measure as ∆ π c ( δ ) . This way, we can interpret the effect of adding labels for a specific class c . If we desire ∆ π c ( δ ) to be large, the sufficient condition is that connection of class c to the labeled data > intra-class similarity - inter-class similarity .

We use examples in Figure 1 to epitomize the core idea. Specifically, our unlabeled samples consist of three underlying classes: traffic lights (known), apples (novel), and flowers (novel). (a) For unlabeled traffic lights from known classes which are strongly connected to the labeled data, adding labels to traffic lights can largely improve the clustering performance; (b) For novel classes like apples, it may also help when they have a strong connection to the traffic light, and their intra-class similarity is not as strong (due to different colors); (c) However, labeled data may offer little improvement in clustering the flower class, due to the minimal connection to the labeled data and that flowers' self-clusterability is already strong.

5 Empirical Validation of Theory

Beyond theoretical insights, we show empirically that SORL is effective on standard benchmark image classification datasets CIFAR-10/100 [35]. Following the seminal work ORCA [7], classes are divided into 50% known and 50% novel classes. We then use 50% of samples from the known classes as the labeled dataset, and the rest as the unlabeled set. We follow the evaluation strategy in [7] and report the following metrics: (1) classification accuracy on known classes, (2) clustering accuracy on the novel data, and (3) overall accuracy on all classes. More experiment details are in Appendix E.2.

SORL achieves competitive performance. Our proposed loss SORL is amenable to the theoretical understanding, which is our primary goal of this work. Beyond theory, we show that SORL is equally desirable in empirical performance. In particular, SORL displays competitive performance compared to existing methods, as evidenced in Table 1. Our comparison covers an extensive collection of very recent algorithms developed for this problem, including ORCA [7], GCD [69], and OpenCon [63]. We also compare methods in related problem domains: (1) Semi-Supervised Learning [21, 37, 62], (2) Novel Class Discovery [22, 82], (3) common representation learning method SimCLR [11]. In particular, on CIFAR-100, we improve upon the best baseline OpenCon by 3.4 %in terms of overall accuracy. Our result further validates that putting analysis on SORL is appealing for both theoretical and empirical reasons.

6 Broader Impact

From a theoretical perspective, our graph-theoretic framework can facilitate and deepen the understanding of other representation learning methods that commonly involve the notion of positive/negative pairs. In Appendix D, we exemplify how our framework can be potentially generalized to other common contrastive loss functions [11, 34, 68], and baseline methods that are tailored for the open-world semi-supervised learning problem (e.g., GCD [69], OpenCon [63]). Hence, we believe our theoretical framework has a broader utility and significance.

From a practical perspective, our work can directly impact and benefit many real-world applications, where unlabeled data are produced at an incredible rate today. Major companies exhibit a strong need for making their machine learning systems and services amendable for the open-world setting but lack fundamental and systematic knowledge. Hence, our research advances the understanding of open-world machine learning and helps the industry improve ML systems by discovering insights and structures from unlabeled data.

7 Related Work

Semi-supervised learning. Semi-supervised learning (SSL) is a classic problem in machine learning. SSL typically assumes the same class space between labeled and unlabeled data, and hence remains closed-world. A rich line of empirical works [9, 13, 21, 29, 37, 38, 39, 42, 48, 50, 53, 54, 74, 76, 78] and theoretical efforts [3, 46, 47, 51, 60, 61, 73] have been made to address this problem. An important class of SSL methods is to represent data as graphs and predict labels by aggregating proximal nodes' labels [1, 18, 30, 71, 80, 84, 85]. Different from classic SSL, we allow its semantic space to cover both known and novel classes. Accordingly, we contribute a graph-theoretic framework tailored to the open-world setting, and reveal new insights on how the labeled data can benefit the clustering performance on both known and novel classes.

Open-world semi-supervised learning . The learning setting that considers both labeled and unlabeled data with a mixture of known and novel classes is first proposed in [7] and inspires a proliferation of follow-up works [49, 52, 63, 69, 81] advancing empirical success. Most works put emphasis on learning high-quality embeddings [49, 63, 69, 81]. In particular, Sun and Li [63] employ contrastive learning with both supervised and self-supervised signals, which aligns with our theoretical setup in Sec. 3.1. Different from prior works, our paper focuses on advancing theoretical understanding . To the best of our knowledge, we are the first to theoretically investigate the problem from a graph-theoretic perspective and provide a rigorous error bound.

Spectral graph theory. Spectral graph theory is a classic research problem [10, 14, 33, 40, 44, 70], which aims to partition the graph by studying the eigenspace of the adjacency matrix. The spectral graph theory is also widely applied in machine learning [1, 6, 45, 56, 58, 64, 86]. Recently, HaoChen et al. [23] derive a spectral contrastive loss from the factorization of the graph's adjacency matrix which facilitates theoretical study in unsupervised domain adaptation [24, 57]. In these works, the graph's formulation is exclusively based on unlabeled data. Sun et al. [64] later expanded this spectral contrastive loss approach to cater to learning environments that encompass both labeled data from known classes and unlabeled data from novel ones. In this paper, our adaptation of the loss function from [64] is tailored to address the open-world semi-supervised learning challenge, considering known class samples within unlabeled data.

Theory for self-supervised learning. A proliferation of works in self-supervised representation learning demonstrates the empirical success [5, 8, 11, 12, 23, 26, 68, 77] with the theoretical foundation by providing provable guarantees on the representations learned by contrastive learning for linear probing [4, 41, 55, 59, 66, 67]. From the graphic view, HaoChen et al. [23, 24], Shen et al. [57] model the pairwise relation by the augmentation probability and provided error analysis of the downstream tasks. The existing body of work has mostly focused on unsupervised learning . In this paper, we systematically investigate how the label information can change the representation manifold and affect the downstream clustering performance on both known and novel classes.

8 Conclusion

In this paper, we present a graph-theoretic framework for open-world semi-supervised learning. The framework facilitates the understanding of how representations change as a result of adding labeling information to the graph. Specifically, we learn representation through Spectral Open-world Representation Learning (SORL). Minimizing this objective is equivalent to factorizing the graph's adjacency matrix, which allows us to analyze the clustering error difference between having vs. excluding labeled data. Our main results suggest that the clustering error can be significantly reduced if the connectivity to the labeled data is stronger than their self-clusterability. Our framework is also empirically appealing to use since it achieves competitive performance on par with existing baselines. Nevertheless, we acknowledge two limitations to practical application within our theoretical construct:

The augmentation graph serves as a potent theoretical tool for elucidating the success of modern representation learning methods. However, it is challenging to ensure that current augmentation strategies, such as cropping, color jittering, can transform two dissimilar images into identical ones.

The utilization of Theorems 4.1 and 4.2 necessitates an explicit knowledge of the adjacency matrix of the augmentation graph, a requirement that can be intractable in practice.

In light of these limitations, we encourage further research to enhance the practicality of these theoretical findings. We also hope our framework and insights can inspire the broader representation learning community to understand the role of labeling prior.

References

Andreas Argyriou, Mark Herbster, and Massimiliano Pontil. Combining graph laplacians for semi-supervised learning. Advances in Neural Information Processing Systems , 18, 2005.

Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D Nowak, and Yixuan Li. Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. In International Conference on Machine Learning , pages 1454-1471. PMLR, 2023.

Maria-Florina Balcan and Avrim Blum. A pac-style model for learning from labeled and unlabeled data. In Colt , pages 111-126. Springer, 2005.

Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. Advances in Neural Information Processing Systems , 2022.

Adrien Bardes, Jean Ponce, and Yann Lecun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. In ICLR 2022-10th International Conference on Learning Representations , 2022.

Avrim Blum. Learning form labeled and unlabeled data using graph mincuts. In Proc. 18th International Conference on Machine Learning, 2001 , 2001.

Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In Proceedings of the International Conference on Learning Representations , 2022.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Proceedings of Advances in Neural Information Processing Systems , 2020.

Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien, editors. Semi-Supervised Learning . The MIT Press, 2006.

Jeff Cheeger. A lower bound for the smallest eigenvalue of the laplacian. In Problems in analysis , pages 195-200. Princeton University Press, 2015.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the international conference on machine learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15750-15758, 2021.

Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3569-3576, 2020.

Fan RK Chung. Spectral graph theory , volume 92. American Mathematical Soc., 1997.

Xuefeng Du, Gabriel Gozum, Yifei Ming, and Yixuan Li. Siren: Shaping representations for detecting out-of-distribution objects. Advances in Neural Information Processing Systems , 35: 20434-20449, 2022.

Xuefeng Du, Yiyou Sun, Xiaojin Zhu, and Yixuan Li. Dream the impossible: Outlier imagination with diffusion models. Advances in Neural Information Processing Systems , 2023.

Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika , 1(3):211-218, 1936.

Rob Fergus, Yair Weiss, and Antonio Torralba. Semi-supervised learning in gigantic image collections. Advances in neural information processing systems , 22, 2009.

Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9284-9292, 2021.

Anne Greenbaum, Ren-cang Li, and Michael L Overton. First-order perturbation theory for eigenvalues and eigenvectors. SIAM review , 62(2):463-482, 2020.

Lan-Zhe Guo, Zhen-Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semisupervised learning for unseen-class unlabeled data. In Proceedings of the international conference on machine learning , volume 119 of Proceedings of Machine Learning Research , pages 3897-3906. PMLR, 13-18 Jul 2020.

Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019.

Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems , 34:5000-5011, 2021.

Jeff Z HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: Analyzing the linear transferability of contrastive representations to related subpopulations. Advances in Neural Information Processing Systems , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020.

Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. Proceedings of the International Conference on Learning Representations , 2018.

Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. Proceedings of the International Conference on Learning Representations , 2019.

Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8310-8319, 2021.

Tony Jebara, Jun Wang, and Shih-Fu Chang. Graph construction and b-matching for semisupervised learning. In Proceedings of the 26th annual international conference on machine learning , pages 441-448, 2009.

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR , 2017.

Antony Joseph and Bin Yu. Impact of regularization on spectral clustering. The Annals of Statistics , 44(4):1765-1791, 2016.

Ravi Kannan, Santosh Vempala, and Adrian Vetta. On clusterings: Good, bad and spectral. Journal of the ACM (JACM) , 51(3):497-515, 2004.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems , 33:18661-18673, 2020.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 2(1-2):83-97, 1955.

Alex Kurakin, Chun-Liang Li, Colin Raffel, David Berthelot, Ekin Dogus Cubuk, Han Zhang, Kihyuk Sohn, Nicholas Carlini, and Zizhao Zhang. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Proceedings of Advances in Neural Information Processing Systems , 2020.

Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. Proceedings of the International Conference on Learning Representations , 2017.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , page 896, 2013.

James R Lee, Shayan Oveis Gharan, and Luca Trevisan. Multiway spectral partitioning and higher-order cheeger inequalities. Journal of the ACM (JACM) , 61(6):1-30, 2014.

Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. Predicting what you already know helps: Provable self-supervised learning. Advances in Neural Information Processing Systems , 34:309-323, 2021.

Wei Liu, Junfeng He, and Shih-Fu Chang. Large graph construction for scalable semi-supervised learning. In Proceedings of the 27th international conference on machine learning (ICML-10) , pages 679-686. Citeseer, 2010.

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software , 3(29):861, 2018.

Frank McSherry. Spectral partitioning of random graphs. In Proceedings 42nd IEEE Symposium on Foundations of Computer Science , pages 529-537. IEEE, 2001.

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems , 14, 2001.

Partha Niyogi. Manifold regularization and semi-supervised learning: Some theoretical analyses. Journal of Machine Learning Research , 14(5), 2013.

Samet Oymak and Talha Cihad Gulcu. A theoretical characterization of semi-supervised learning with self-training for gaussian mixture models. In International Conference on Artificial Intelligence and Statistics , pages 3601-3609. PMLR, 2021.

Jongjin Park, Sukmin Yun, Jongheon Jeong, and Jinwoo Shin. Opencos: Contrastive semisupervised learning for handling open-set unlabeled data. In Computer Vision-ECCV 2022 Workshops: Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II , pages 134-149. Springer, 2023.

Nan Pu, Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. 2023.

Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Semi-supervised learning with scarce annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 762-763, 2020.

Philippe Rigollet. Generalization error bounds in semi-supervised classification under the cluster assumption. Journal of Machine Learning Research , 8(7), 2007.

Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Openldn: Learning to discover novel classes for open-world semi-supervised learning. Proceedings of the European Conference on Computer Vision , 2022.

Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set consistency regularization for semi-supervised learning with outliers. Proceedings of Advances in Neural Information Processing Systems , 2021.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Proceedings of Advances in Neural Information Processing Systems , 29, 2016.

Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning , pages 5628-5637. PMLR, 2019.

Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. In 6th International Conference on Learning Representations, ICLR 2018 , 2018.

Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference on Machine Learning , pages 19847-19878. PMLR, 2022.

Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence , 22(8):888-905, 2000.

Zhenmei Shi, Jiefeng Chen, Kunyang Li, Jayaram Raghuram, Xi Wu, Yingyu Liang, and Somesh Jha. The trade-off between universality and label efficiency of representations from contrastive learning. In The Eleventh International Conference on Learning Representations , 2023.

Aarti Singh, Robert Nowak, and Jerry Zhu. Unlabeled data: Now it helps, now it doesn't. Advances in neural information processing systems , 21, 2008.

Nataliya Sokolovska, Olivier Cappé, and François Yvon. The asymptotics of semi-supervised learning in discriminative probabilistic models. In Proceedings of the 25th international conference on Machine learning , pages 984-991, 2008.

Xin Sun, Zhenning Yang, Chi Zhang, Keck-Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 13480-13489, 2020.

Yiyou Sun and Yixuan Li. Opencon: Open-world contrastive learning. Transaction on Machine Learning Research , 2023.

Yiyou Sun, Zhenmei Shi, Yingyu Liang, and Yixuan Li. When and how does known class help discover unknown ones? Provable understanding through spectral analysis. In Proceedings of the 40th International Conference on Machine Learning , Proceedings of Machine Learning Research, pages 33014-33043. PMLR, 2023.

Leitian Tao, Xuefeng Du, Xiaojin Zhu, and Yixuan Li. Non-parametric outlier synthesis. International Conference on Learning Representations , 2023.

Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. J. Mach. Learn. Res. , 22:281-1, 2021.

Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory , pages 1179-1206. PMLR, 2021.

Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints , pages arXiv-1807, 2018.

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing , 17(4):395-416, 2007.

Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. In Proceedings of the 23rd international conference on Machine learning , pages 985-992, 2006.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning , pages 9929-9939. PMLR, 2020.

Larry Wasserman and John Lafferty. Statistical analysis of semi-supervised regression. Advances in Neural Information Processing Systems , 20, 2007.

Fan Yang, Kai Wu, Shuyi Zhang, Guannan Jiang, Yong Liu, Feng Zheng, Wei Zhang, Chengjie Wang, and Long Zeng. Class-aware contrastive semi-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Muli Yang, Yuehua Zhu, Jiaping Yu, Aming Wu, and Cheng Deng. Divide and conquer: Compositional experts for generalized novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14268-14277, 2022.

Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In Proceedings of the European Conference on Computer Vision , pages 438-454. Springer, 2020.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S4l: Self-supervised semisupervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1476-1485, 2019.

Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, et al. Openood v1. 5: Enhanced benchmark for out-of-distribution detection. arXiv preprint arXiv:2306.09301 , 2023.

Kai Zhang, James T Kwok, and Bahram Parvin. Prototype vector machine for large scale semi-supervised learning. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 1233-1240, 2009.

Sheng Zhang, Salman Khan, Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fahad Khan. Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. 2022.

Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Proceedings of Advances in Neural Information Processing Systems , 34, 2021.

Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 10867-10875, 2021.

Dengyong Zhou, Thomas Hofmann, and Bernhard Schölkopf. Semi-supervised learning on directed graphs. Advances in neural information processing systems , 17, 2004.

Xiaojin Zhu. Learning from labeled and unlabeled data with label propagation. Tech. Report , 2002.

Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03) , pages 912-919, 2003.

where

Proof. We can expand L mf ( F, A ) and obtain

where f x = √ w x f ( x ) is a re-scaled version of f ( x ) . At a high level, we follow the proof in [23], while the specific form of loss varies with the different definitions of positive/negative pairs. The form of L SORL ( f ) is derived from plugging w xx ′ and w x .

Recall that w xx ′ is defined by

and w x is given by

Plugging in w xx ′ we have,

Appendix

A Technical Details of Spectral Open-world Representation Learning

Theorem A.1. (Recap of Theorem 3.1) We define f x = √ w x f ( x ) for some function f . Recall η u , η l are two hyper-parameters defined in Eq. (1) . Then minimizing the loss function L mf ( F, A ) is equivalent to minimizing the following loss function for f , which we term Spectral Open-world Representation Learning (SORL) :

Plugging w x and w x ′ we have,

B Technical Details for Toy Example

B.1 Calculation Details for Figure 2.

We first recap the toy example, which illustrates the core idea of our theoretical findings. Specifically, the example aims to distinguish 3D objects with different shapes, as shown in Figure 2. These images are generated by a 3D rendering software [31] with user-defined properties including colors, shape, size, position, etc.

Data design. Suppose the training samples come from three types, X , X , X . Let X be the sample space with known class, and X , X be the sample space with novel classes. Further, the two novel classes are constructed to have different relationships with the known class. Specifically, we construct the toy dataset with 6 elements as shown in Figure 4(a).

Augmentation graph. Based on the data design, we formally define the augmentation graph, which encodes the probability of augmenting a source image ¯ x to the augmented view x :

̸

̸

̸

According to the definition above, the corresponding augmentation matrix T with each element formed by T ( · | · ) is given in Figure 4(b). We proceed by showing the details to derive A ( u ) and A using T .

τ

s

τ

c

τ

0

τ

s

τ

c

τ

0

τ

1

τ

1

τ

1

τ

1

τ

0

τ

1

Known

Class

Novel

Class

Novel

Class

τ

0

τ

1

τ

1

T1

Ts

Tc

0

0

01

Ts

T1

0

Tc

0

0

T

=

Tc

0

T1

Ts

0

0

0

Tc

Ts

T1

0

0

0

0

0

0

T1

T1

0

0

0

0

T1

T1

O

x

Figure 4: An illustrative example for theoretical analysis. We consider a 6-node graph with one known class (cube) and two novel classes (sphere, cylinder). (a) The augmentation probabilities between nodes are defined by their color and shape in Eq. (11). (b) The augmentation matrices T derived by Eq. (11) where we let τ 0 = 0 .

Derivation details for A ( u ) and A . Recall that each element of A ( u ) is formed by w ( u ) xx ′ = E ¯ x ∼P T ( x | ¯ x ) T ( x ′ | ¯ x ) . In this toy example, one can then see that A ( u ) = 1 6 TT ⊤ since augmentation matrix T is defined that each element T x ¯ x = T ( x | ¯ x ) . Note that T is explicitly given in Figure 4(b) and then if we let η u = 6 , we have the close-from:

̸

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="T

uMFzGO+jW

Cg9/5BnL

Rd2

oA

>

7X

c

V

NS8

EJ3Ur

q

0mk

y

Z

I

Q

P

f

Y

H

D

w

p

v

K

<latexi sh

1_b

64="d02/zNZv

YW

pFmM

q

O3

g

>A

B7X

c

V

S8

EJ

Ur

9L

k

o

y

I

Q

PCj

f

H

n

+

G

T

D

R

w

5

K

u

<latexi sh

1_b

64="

T

F

PMSL9

f

/

Kngy

N8

>A

B7X

c

V

EJ3Ur

q

0mk oW

2

Z

I

Q

Cj

z

Y

H

+

G

O

Dd

R

w

p

u

5

v

<latexi sh

1_b

64="d02/zNZv

YW

pFmM

q

O3

g

>A

B7X

c

V

S8

EJ

Ur

9L

k

o

y

I

Q

PCj

f

H

n

+

G

T

D

R

w

5

K

u

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

T

F

PMSL9

f

/

Kngy

N8

>A

B7X

c

V

EJ3Ur

q

0mk oW

2

Z

I

Q

Cj

z

Y

H

+

G

O

Dd

R

w

p

u

5

v

<latexi sh

1_b

64="d02/zNZv

YW

pFmM

q

O3

g

>A

B7X

c

V

S8

EJ

Ur

9L

k

o

y

I

Q

PCj

f

H

n

+

G

T

D

R

w

5

K

u

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="T

uMFzGO+jW

Cg9/5BnL

Rd2

oA

>

7X

c

V

NS8

EJ3Ur

q

0mk

y

Z

I

Q

P

f

Y

H

D

w

p

v

K

<latexi sh

1_b

64="d02/zNZv

YW

pFmM

q

O3

g

>A

B7X

c

V

S8

EJ

Ur

9L

k

o

y

I

Q

PCj

f

H

n

+

G

T

D

R

w

5

K

u

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

<latexi sh

1_b

64="

5+FEX83NH

mSz2j fU

w

>A

CZn cdV

Q

GM

q

r

L

D

T

up

Y

W

y

v

B

k

9

g

PJ

/

R

O

K

I

7

0

o

We then derive the second part A ( l ) whose element is given by:

Such a form can be simplified in Section 4 by defining l ∈ R N , ( l ) x = E ¯ x l ∼P l 1 T ( x | ¯ x l ) and by letting |Y l | = 1 . In this toy example, the known class only has two elements, so l = 1 2 ( T : , 1 + T : , 2 ) (average of T 's 1st & 2nd column), we then have:

Finally, if we let η l = 4 and A = η u A ( u ) + η l A ( l ) , we have the full results in Figure 2.

B.2 Calculation Details for Figure 3.

In this section, we present the analysis of eigenvectors and their orders for toy examples shown in Figure 2. In Theorem B.1 we present the spectral analysis for the adjacency matrix with additional label information while in Theorem B.2, we show the spectral analysis for the unlabeled case.

Theorem B.1. Let

and we assume that 1 ≫ τ c τ 1 > τ s τ 1 > 0 , 4 9 τ c ≤ τ s ≤ τ c and τ 1 + τ c + τ s = 1 .

Let λ 1 , λ 2 , λ 3 and v 1 , v 2 , v 3 be the largest three eigenvalues and their corresponding eigenvectors of D -1 2 AD -1 2 , which is the normalized adjacency matrix of A . Then the concrete form of λ 1 , λ 2 , λ 3 and v 1 , v 2 , v 3 can be approximately given by:

Note that the approximation gap can be tightly bounded. Specifically, for i ∈ { 1 , 2 , 3 } , we have | λ i -ˆ λ i | ≤ O (( τ c τ 1 ) 2 ) and ∥ sin( U, ˆ U ) 6 ∥ F ≤ O ( τ c τ 1 ) , where U = [ v 1 , v 2 , v 3 ] , ˆ U = [ˆ v 1 , ˆ v 2 , ˆ v 3 ] .

Proof. By τ 1 + τ c + τ s = 1 and 1 ≫ τ c τ 1 > τ s τ 1 > 0 , we define the following equation which approximates the corresponding terms up to error O (( τ c τ 1 ) 2 ) :

And we have

Let ˆ λ a , . . . , ˆ λ f be six eigenvalues of ̂ D -1 2 ̂ A ̂ D -1 2 , and ˆ v a , . . . , ˆ v f be corresponding eigenvectors. By direct calculation we have

and corresponding eigenvectors as

For the remaining two eigenvectors, by the symmetric property, they have the formula

where α, β are some real functions. Then, by solving

we get

Now, we show that ˆ λ c > ˆ λ e . By τ c τ 1 ≪ 1 and 4 9 τ c ≤ τ s ≤ τ c

Thus, we have 1 = ˆ λ a = ˆ λ b > ˆ λ c > ˆ λ e > ˆ λ f > ˆ λ d = 0 . Moreover, we also have

Let ˆ λ 1 = ˆ λ a , ˆ λ 2 = ˆ λ b , ˆ λ 3 = ˆ λ c . Then, by Weyl's Theorem, for i ∈ { 1 , 2 , 3 } , we have

We finish the proof.

Theorem B.2. Recall η u A ( u ) is defined in Theorem B.1. Assume 1 ≫ τ c τ 1 > τ s τ 1 > 0 and τ 1 + τ c + τ s = 1 . Let λ ( u ) 1 , λ ( u ) 2 , λ ( u ) 3 and v ( u ) 1 , v ( u ) 2 , v ( u ) 3 be the largest three eigenvalues and their corresponding eigenvectors of D ( u ) -1 2 ( η u A ( u ) ) D ( u ) -1 2 , which is the normalized adjacency matrix of η u A ( u ) . Let

Let U ( u ) = [ v ( u ) 1 , v ( u ) 2 , v ( u ) 3 ] , ˆ U ( u ) = [ˆ v ( u ) 1 , ˆ v ( u ) 2 , ˆ v ( u ) 3 ] . Then, for i ∈ { 1 , 2 , 3 } , we have | λ ( u ) i -ˆ λ ( u ) i | ≤ O (( τ c τ 1 ) 2 ) and ∥ sin( U ( u ) , ˆ U ( u ) ) ∥ F ≤ O ( τ 2 c τ 1 ( τ c -τ s ) ) .

Proof. Similar to the proof of Theorem B.1, up to error O (( τ c τ 1 ) 2 ) , we have the following equation,

Let ˆ λ ( u ) 1 , . . . , ˆ λ ( u ) 6 be six eigenvalue of ̂ D ( u ) -1 2 ̂ η u A ( u ) ̂ D ( u ) -1 2 , and ˆ v ( u ) 1 , . . . , ˆ v ( u ) 6 be corresponding eigenvectors. By direct calculation we have

and corresponding eigenvector as

Then, by Weyl's Theorem, for i ∈ { 1 , 2 , 3 } , we have

We finish the proof.

C Technical Details for Main Theory

C.1 Notation

We let 1 n , 0 n be the n -dimensional vector with all 1 or 0 values respectively. 1 m × n , 0 m × n are similarly defined for m -byn matrix. I n is the identity matrix with shape n × n . For any matrix V , V ( i,j ) indictes the value at i -th row and j -th column of V . If the matrix is subscripted like V k , we use a comma in-between like V k, ( i,j ) . Similarly, v ( i ) and v k, ( i ) are the i -th value for vector v and v k respectively. [ n ] is used to abbreviate the set { 1 , 2 , ..., n } .

C.2 Matrix Form of K-means and the Derivative

Recall that we defined the K-means clustering measure of features in Sec. 4:

where the numerator measures the intra-class distance:

and the denominator measures the inter-class distance:

We will show next how to convert the intra-class and the inter-class measure into a matrix form, which is desirable for analysis.

Intra-class measure. Note that the K -means intra-class measure can be rewritten in a matrix form:

where H Π is a matrix to convert Z to mean vectors w.r.t clusters defined by Π . Without losing the generality, we assume Z is ordered according to the partition in Π -first | π 1 | vectors are in π 1 , next | π 2 | vectors are in π 2 , etc. Then H Π is given by:

Going further, we have:

Inter-class measure. The inter-class measure can be equivalently given by:

where H Π is defined as above. And we can also derive:

C.3 K-means Measure Has the Same Order as K-means Error

Theorem C.1. (Recap of Theorem 4.1) We define the ξ π → π ′ as the index of samples that is from class division π however is closer to µ π ′ than µ π . In other word, ξ π → π ′ = { i : i ∈ π, ∥ z i -µ π ∥ 2 ≥ ∥ z i -µ π ′ ∥ 2 } . Assuming | ξ π → π ′ | > 0 , we define below the clustering error ratio from π to π ′ as E π → π ′ and the overall cluster error ratio E Π ,Z as the Harmonic Mean of E π → π ′ among all class pairs:

̸

The K-means measure M kms (Π , Z ) has the same order as the Harmonic Mean of the cluster error ratio between all cluster pairs:

Proof. We have the following inequality for i ∈ ξ π → π ′ :

Then we have:

Note that the inter-class measure can be decomposed into the summation of cluster center distances:

̸

̸

where ∑ π = π ′ is enumerating over any two different class partitions in Π . Combining together, we have:

̸

̸

̸

C.4 Proof of Theorem 4.2

We start by providing more details to supplement Sec. 4.2.1.

Matrix perturbation by adding labels. Recall that we define in Eq. 3 that the adjacency matrix is the unlabeled one A ( u ) plus the perturbation of the label information A ( l ) :

We study the perturbation from two aspects: (1) The direction of the perturbation which is given by A ( l ) , (2) The perturbation magnitude η l . We first consider the perturbation direction A ( l ) and recall that we defined the concrete form in Eq. 2:

For simplicity, we consider |Y l | = 1 in this theoretical analysis. Then we observe that A ( l ) xx ′ is a rank-1 matrix can be written as

where l ∈ R N × 1 with ( l ) x = E ¯ x l ∼P l 1 T ( x | ¯ x l ) . And we define D l ≜ diag ( l ) .

The perturbation function of representation. We then consider a more generalized form for the adjacency matrix:

where we treat the adjacency matrix as a function of the 'labeling perturbation' degree δ . It is clear that A (0) = η u A ( u ) which is the scaled adjacency matrix for the unlabeled case and that A ( η l ) = A . When we let the adjacency matrix be a function of δ , the normalized form and the derived feature representation should also be the function of δ . We proceed by defining these terms.

Without losing the generality, we let diag ( 1 ⊤ N A (0)) = I N which means the node in the unlabeled graph has equal degree. We then have:

The normalized adjacency matrix is given by:

For feature representation Z ( δ ) , it is derived from the topk SVD components of ˜ A ( δ ) . Specifically, we have:

where we define ˜ A k ( δ ) as the topk SVD components of ˜ A ( δ ) and can be further written as ˜ A k ( δ ) = ∑ k j =1 λ j ( δ )Φ j ( δ ) . Here the λ j ( δ ) is the j -th singular value and Φ j ( δ ) is the j -th singular projector ( Φ j ( δ ) = v j ( δ ) v j ( δ ) ⊤ ) defined by the j -th singular vector v j ( δ ) . For brevity, when δ = 0 , we remove the suffix (0) since it is equivalent to the unperturbed version of notations. For example, we let

Theorem C.2. (Recap of Th. 4.2) Denote V ( u ) ∅ ∈ R N × ( N -k ) as the null space of V ( u ) k and ˜ A ( u ) k = V ( u ) k Σ ( u ) k V ( u ) ⊤ k as the rankk approximation for ˜ A ( u ) . Given δ, η 1 > 0 and let G k as the spectral gap between k -th and k +1 -th singular values of ˜ A ( u ) , we have:

where diag ( · ) converts the vector to the corresponding diagonal matrix and Υ ∈ R N × N is a matrix encoding the ground-truth clustering structure in the way that Υ xx ′ > 0 if x and x ′ has the same label and Υ xx ′ < 0 otherwise.

Proof. As we shown in Sec C.2, we can now also write the K-means measure as the function of perturbation:

The proof is directly given by the following Lemma C.3.

Lemma C.3. Let η 1 , η 2 be two real values and Υ = (1 + η 2 ) H Π -I -η 2 N 1 N 1 ⊤ N . Let the spectrum gap G k = λ ( u ) k λ ( u ) k +1 , we have the derivative of the K-means measure evaluated at δ = 0 :

The proof for Lemma C.3 is lengthy. We postpone it to Sec. C.6.

C.5 Proof of Theorem 4.3

We start by showing the justification of the assumptions made in Theorem 4.3.

Assumption C.4. We assume the spectral gap G k is large. Such an assumption is commonly used in theory works using spectral analysis [32, 57].

Assumption C.5. We assume l lies in the linear span of V ( u ) k . i.e. , V ( u ) k V ( u ) ⊤ k l = l , V ( u ) ⊤ ∅ l = 0 . The goal of this assumption is to simplify ( V ( u ) k V ( u ) ⊤ k ll ⊤ + V ( u ) k V ( u ) ⊤ k ll ⊤ V ( u ) ∅ V ( u ) ⊤ ∅ ) to ll ⊤ .

Assumption C.6. For any π c ∈ Π , ∀ i, j ∈ π c , l ( i ) = l ( j ) =: l π c . Recall that the l ( i ) means the connection between the i -th sample to the labeled data. Here we can view l π c as the connection between class c to the labeled data .

Theorem C.7. (Recap of Theorem 4.3.) With Assumption C.4, C.5 and C.6. Given δ, η 1 , η 2 > 0 , we have:

where

Proof. The proof is directly given by Lemma C.8 and plugging the definition of ∆ kms ( δ ) .

Lemma C.8. With Assumption C.4 C.5 and C.6, we have the derivative of K-means measure with the upper bound:

Proof. By Assumption C.4 C.5 and C.6 and Theorem 4.2, we have

where and

We observe that

where the last inequality is by Jensen's Inequality. We then have

C.6 Proof of Lemma C.3

Notation Recap: We define ˜ A k ( δ ) as the topk SVD components of ˜ A ( δ ) and can be further written as ˜ A k ( δ ) = ∑ k j =1 λ j ( δ )Φ j ( δ ) . Here the λ j ( δ ) is the j -th singular value and Φ j ( δ ) is the j -th singular projector ( Φ j ( δ ) = v j ( δ ) v j ( δ ) ⊤ ) defined by the j -th singular vector v j ( δ ) . For brevity, when δ = 0 , we remove the suffix (0) since it is equivalent to the unperturbed version of notations. For example, we let

Proof. By the derivative rule, we have,

where we let η 1 = 1 M inter (Π ,Z ) , η 2 = M intra (Π ,Z ) M inter (Π ,Z ) and Υ = (1 + η 2 ) H Π -I Π -η 2 N 1 N 1 ⊤ N . We proceed by showing the calculation of [ D ( δ ) -1 2 ] ′ , [ λ j ( δ )] ′ and [Φ j ( δ )] ′ .

Since D ( δ ) = I + δD l , then [ D ( δ ) -1 2 ] ′ ∣ ∣ ∣ δ =0 = -1 2 D l . To calculate [ λ j ( δ )] ′ and [Φ j ( δ )] ′ , we first need:

Then, according to Equation (3) in [20], we have:

According to Equation (10) in [20], we have:

̸

̸

Now we calculate the derivative of the K -means loss:

̸

where

̸

̸

̸

̸

Thus, we have:

Then [ M kms -all ( δ )] ′ ∣ ∣ ∣ δ =0 /η 1 is given by:

We can represent λ ( u ) j λ ( u ) j -λ ( u ) i = 1 + ∑ ∞ p =1 ( λ ( u ) i λ ( u ) j ) p . Denote the residual term as :

We then have:

D Analysis on Other Contrastive Losses

In this section, we discuss the extension of our graphic-theoretic analysis to one of the most common contrastive loss functions - SimCLR [11]. SimCLR loss is an extended version of InfoNCE loss [68] that achieves great empirical success and inspires a proliferation of follow-up works [5, 8, 12, 26, 34, 69, 77]. Specifically, SupCon [34] extends SimCLR to the supervised setting. GCD [69] and OpenCon [63] further leverage the SupCon and SimCLR losses, and are tailored to the open-world representation learning setting considering both labeled and unlabeled data.

At a high level, we consider a general form of the SimCLR and its extensions (including SupCon, GCD, OpenCon) as:

̸

where we let the P + as the distribution of positive pairs defined in Section 3.1. In SimCLR [11], the positive pairs are purely sampled in the unlabeled case (u) while SupCon [34] considers the labeled case (l) . With both labeled and unlabeled data, GCD [69] and OpenCon [63] sample positive pairs in both cases.

In this section, we investigate an alternative form that eases the theoretical analysis (also applied in [72]):

̸

which serves an upper bound of L gnl ( f ) according to Jensen's Inequality.

A graph-theoretic view. Recall in Section 3.1, we define the graph G ( X , w ) with vertex set X and edge weights w . Each entry of adjacency matrix A is given by w xx ′ , which denotes the marginal probability of generating the pair for any two augmented data x, x ′ ∈ X :

and w x measures the degree of node x :

One can view the difference between SimCLR and its variants in the following way: (1) SimCLR [11] corresponds to η l = 0 when there is no labeled case; (2) SupCon [34] corresponds to η u = 0 when only labeled case is considered. (3) GCD [69] and OpenCon [63] correspond to the cases when η u , η l are both non-zero due to the availability of both labeled and unlabeled data.

With the define marginal probability of sampling positive pairs w xx ′ and the marginal probability of sampling a single sample w x , we have:

̸

When τ is large:

If we further consider the constraint that the Z ⊤ Z = I , minimizing ̂ L simclr ( Z ; G ( X , w )) boils down to the eigenvalue problem such that Z is formed by the topk eigenvectors of matrix ( A -D 1 N 1 ⊤ N D -D 2 Tr( D ) 2 -Tr( D 2 ) ) . Recall that our main analysis for Theorem 4.2 and Theorem 4.3 is based on the insight that the feature space is formed by the topk eigenvectors of the normalized adjacency matrix D -1 2 AD -1 2 . Viewed in this light, the same analysis could be applied to the SimCLR loss as well, which only differs in the concrete matrix form. We do not include the details in this paper but leave it as a future work.

E Additional Experiments Details

E.1 Experimental Details of Toy Example

Recap of set up . In Section 4.1 we consider a toy example that helps illustrate the core idea of our theoretical findings. Specifically, the example aims to cluster 3D objects of different colors and shapes, generated by a 3D rendering software [31] with user-defined properties including colors, shape, size, position, etc. Suppose the training samples come from three shapes, X , X , X . Let X be the sample space with known class, and X , X be the sample space with novel classes. Further, the two novel classes are constructed to have different relationships with the known class. Specifically, the toy dataset contains elements with 5 unique types:

where

Experimental details for Figure 3(b) . We rendered 2500 samples for each type of data. In total, we have 12500 samples. For known class X , we randomly select 50% as labeled data and treat the rest as unlabeled. For training, we use the same data augmentation strategy as in SimSiam [12]. We use ResNet18 and train the model for 40 epochs (sufficient for convergence) with a fixed learning rate of 0.005, using SORL defined in Eq. (6). We set η l = 0 . 04 and η u = 1 , respectively. Our visualization is by PyTorch implementation of UMAP [43], with parameters ( n_neighbors=30, min_dist=1.5, spread=2, metric=euclidean ) .

E.2 Experimental Details for Benchmarks

Hardware and software. We run all experiments with Python 3.7 and PyTorch 1.7.1, using NVIDIA GeForce RTX 2080Ti and A6000 GPUs.

Training settings. For a fair comparison, we use ResNet-18 [25] as the backbone for all methods. Similar to [7], we pre-train the backbone using the unsupervised Spectral Contrastive Learning [23] for 1200 epochs. The configuration for the pre-training stage is consistent with [23]. Note that the pre-training stage does not incorporate any label information. At the training stage, we follow the same practice in [7, 63], and train our model f ( · ) by only updating the parameters of the last block of ResNet. In addition, we add a trainable two-layer MLP projection head that projects the feature from the penultimate layer to an embedding space R k ( k = 1000 ). We use the same data augmentation strategies as SimSiam [12, 23]. For CIFAR-10, we set η l = 0 . 25 , η u = 1 with training epoch 100, and we evaluate using features extracted from the layer preceding the projection. For CIFAR-100, we set η l = 0 . 0225 , η u = 3 with 400 training epochs and assess based on the projection layer's features. We use SGD with momentum 0.9 as an optimizer with cosine annealing (lr=0.05), weight decay 5e-4, and batch size 512.

Evaluation settings. At the inference stage, we evaluate the performance in a transductive manner (evaluate on D u ). We run a semi-supervised K-means algorithm as proposed in [69]. We follow the evaluation strategy in [7] and report the following metrics: (1) classification accuracy on known classes, (2) clustering accuracy on the novel data, and (3) overall accuracy on all classes. The accuracy of the novel classes is measured by solving an optimal assignment problem using the Hungarian algorithm [36]. When reporting accuracy on all classes, we solve optimal assignments using both known and novel classes.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors describe a theoretical framework for contrastive learning of representations in an open-world setting, where both labelled data and unlabelled data of potentially new classes is available. They explicitly describe the graph encoding positive sample connections and formulate a contrastive loss that amounts to a spectral decomposition of the adjacency matrix. Using this framework, they prove two theorems about the change in clustering quality of their representations via k-means when incorporating some labels. The main take-away is that labelled data helps to cluster another class if this other class has strong connections to the labelled class via data augmentation and was previously poorly clustered. Their method also outperforms several competitors for open-world and self-supervised representation learning with subsequent clustering.\n\nSTRENGTHS:\nI appreciate the theoretical nature of the paper. Understanding the behavior of machine learning models when a clear learning signal via labels is unavailable or just partially available is important for their development and trustworthiness. It is also encouraging that their framework does not only allow a formal investigation but also holds up practically. I find the paper is well written with an appropriate deferral of more technical parts to the appendix. The authors also discuss connections of their work to the more common contrastive SimCLR loss. Their theory seems to be novel for the setting of open-world representation learning.\n\nWEAKNESSES:\n**Strength of different types of augmentations:**  \nIn the illustrative example in section 4.1. it is assumed that the probability $\\tau_c$ of augmenting across classes if there is a shared attribute (e.g. augmenting a blue cube to a blue ball) is larger than the probability $\\tau_s$ of just changing attributes within a class (blue ball to red ball). I find this quite counterintuitive.  \nFirst, I am surprised that $\\tau_c$ should be non-zero. This might be achievable in the toy setting, but for any real world data, say CIFAR10, it is very unlikely, if not impossible, to augment two images of different classes to the same image. Maybe a plausible change of class can be achieved via cropping (cropping an image of a dog in front of flowers to just the flower) but it still seems unlikely that there should be another image in the dataset that augments to the same view. The same argument applies, to a lesser extend, also to the question if two images of the same class can be augmented to the same view (positivity of $\\tau_s$). In the toy setting, I also wonder how positions and sizes were treated. Can a large blue ball in the top left corner only augment to a large blue cube in the top left corner or to any blue cube?  \nSecond, I am surprised by the order of augmentation probabilities. Is it not more likely for two data points of the same class to augment to the same view than for two data points of different classes? A similar requirement seems to relevant for Theorem 4.2 (l255 ff). How crucial are these assumptions (or corresponding ones) for the outcome of the illustrative example, the main theory, and the empirical validation? How relevant does this make the theoretical analysis in real settings?    \n\n**Empirical illustration of the main theorems:**  \nIt would strengthen the paper if the authors had included experiments that showed how the predicted quantities in Thm 4.1 and 4.2 behave empirically in real / toy settings. In particular, a graph that shows the $\\Delta(\\delta)$ terms in both theorems for different $\\delta$ values both according to their theory and in terms of actual experiments. This would also make it more evident how restrictive the assumptions of the theorems are and how tight the inequality is. The size and evolution of the three summands in the last equation of Thm 4.2 would also be interesting.  \n\n\n**Minor:**  \n- I think that highlighting which feature is more separable in Fig 3b) is a bit problematic. First, UMAP plots are known to artificially tear apart structures. Second, the shape separability in 3b1) is also very good (but admittedly less obvious when looking at the plot as the color of the marker is much more prominent that its shape in the 2D plot). Therefore, I would recommend to rather stress the fact that when incorporating labels, the islands of the same shape but of different color seem to be much closer together (i.e. much less separable by color) than without labels. \n- It would have helped my understanding of the paper if it was stated more explicitly that label information is made available by providing the label of existing but previously unlabelled points as opposed to adding new labelled points.   \n- I would suggest to also assign the bottom matrix in Fig 2b) as $\\eta_l A^{(l)}$ as in the line above and then add another line $A=...$ below.\n\nQUESTIONS:\n**Size of the data augmentation graph:**  \nThe node set of the graph described in 3.1 is the set of all augmented views of all available data. Why is this even finite? Consider, for instance, image data. Are there not infinitely many ways to color jitter even a single image? Or is the idea to only consider a fixed and finite set of augmentations of each data point that is selected prior to training? This would be different than what is typically done in contrastive learning, where augmentations are applied on the fly. Even when considering a fixed set of augmentations, the size of the graph would have to be many times the size of the original dataset, right?  \n\n**Empirical Validation:**  \n- Were the same positive samples (including label information) used when comparing to SimCLR? From the discussion in appendix D, I think they were not. If that is the case and the SimCLR experiment is purely self-supervised, why is there such a large difference between the known, novel and all classes? Shouldn't all the completely un-/self-supervised methods be agnostic to which class has labels (which they cannot use anyway)?   \n- Using a ResNet-18 typically leads to about 90% linear probing and kNN accuracy on CIFAR10 for SimCLR. I appreciate that the evaluation is different and k-means is used to infer the clusters here. Nevertheless, I am surprised by the drastic drop in performance. Do the authors have an idea why this happens? Do the same SimCLR representations still achieve about 90% accuracy with linear probing / kNN classifier?    \n- I appreciate that the paper is about the open-world setting, where there might be novel classes without any labels, so that one can only do clustering and not classification for evaluation. Still the label information is available for CIFAR10 and CIFAR100 and I would have found it interesting to see how the proposed method fares under the SimCLR evaluation protocol with a linear probe / kNN classifier. This setting is standard in the self-supervised learning world and thus makes the quality of the proposed method more interpretable to an audience more familiar with self-supervised rather than open-world representation learning. I assume SORL should outperform SimCLR since it has some label information available at train time, right?   \n- Why is the performance for the \"known\" classes lower than that of the \"novel\" classes and why is the performance for \"all\" classes (typically) worse still than for the \"novel\" classes for the methods above the horizontal line in Table 1? For the open-world methods below the horizontal line, the performance is (typically) best on the \"known\" class, worse on the \"novel\" class and somewhere in between for \"all\" classes. This it what I would have intuitively expected.  \n- The method LSD-C [1] achieves over 80% clustering accuracy on CIFAR-10 without using any labels (and no supervised training of a shallow classifier). It might therefore be a relevant competitor.    \n\n**Minor:**  \n- The batch size of 512 is rather small for contrastive learning. Why is that?  \n- Are the different islands of points that share both color in shape in Fig 3b meaningful? Do they encode perhaps the size or position?\n\n**References:**  \n[1] Rebuffi, Sylvestre-Alvise, et al. \"Lsd-c: Linearly separable deep clusters.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\nLIMITATIONS:\nLimitations were not explicitly discussed. I suggest a more detailed discussion of how their theory and assumptions hold up on real world data, see weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper presents spectral open-world representation learning that aims to learn low-rank approximation of a constructed adjacency matrix. From this perspective, the authors study how the label information help (or hurt) the classification performance by analyzing the error bound. Experiments on image classification benchmarks show the effectiveness of the proposed approach.

STRENGTHS:
1. The formulation of open-world representation learning problem as a low-rank approximation of a weighted graph adjacency matrix is novel and reasonable.
2. The representation is clear. The provided illustrative example is helpful for understanding the theory.
3. The new ORL approach itself is simple yet effective.

WEAKNESSES:
1. The theoretical analysis presented in the main text is only applicable to the proposed SORL approach. Generalizing to other approaches needs independent considerations as stated by the authors in section 6.
2. The empirical validation seems somewhat disconnected with the main theoretical result (theorem 4.1 and 4.2) as it only focuses on the performance of SORL. It would be better to conduct experiments to directly validate the main theorems e.g. by showing the connection between performance improvement by adding different label information and factors affecting $\Delta_{\pi_s}(\delta)$.

QUESTIONS:
1. Is it possible to generalize the analysis to the SSL and supervised learning settings?
2. Is the spectral gap in theorem 4.1 crucial for the result?

LIMITATIONS:
Please see the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formulate the open-world representation learning using the graph. \nThis paper provides theoretical analyses for the framework (Thm 3.1) as well as the framework's performance (Thm 4.1 & 4.2).\nThe experimental results show that the framework outperforms the exiting method in the open-world learning setting.\n\nSTRENGTHS:\nOverall, I enjoy reading this paper and have a positive feeling.\n\n- Formulation. Of Eq. (6). By Thm. 3.1 this formulation corresponds to the SVD of normalized adjacency matrix Eq. (4). See more on the question (*).\n\n- The theoretical insights (Thm 4.1 & 4.2). The analyses using $k$-means measure strengthen the formulation. Particularly, the insight in L251-L258 is interesting; knowing more about green light increases the knowledge about the green apple, but not the flowers. This corresponds to our intuition.\n\nWEAKNESSES:\nThere's little I can say, but if I need to raise one, I would like to point out on the experiment.\n\nThe experiment is on the only one split, but I am curious the other splits. Ideally it'd be nice to have results like Fig 4 in [79], y-axis accuracy vs x-axis # of the known labels. \nBy seeing this, we can know \"how many pictures of the green light do we need to learn in order to know the green apple?\"\n\nI also would like to know the computational time, see more on the question section (**)\n\nFrom this viewpoint, I would give 6.\n\nQUESTIONS:\n- How much do the things need to be different in order to learn?\n\nBy learning something, how much the SORL can distinguish some similar things?\nIf we do not know anything about the green apple, can we increase the chance to distinguish the different variety of the green apple by learning green signals? Maybe the difference between the variety of the green apple is embedded in the spectral gap, but do you have any insights on this?\n\n- I'm just curious how did you come up with Eq. (6)?\n\n- This is not the question but a comment connecting to (*) in the strength section.\n\nThe Eq.(5) gives the SVD, but in this case Eq.(5) gives the eigendecomposition. Since the normalized Laplacian $L$ is defined as L := I -A, the Eq. (5) gives the k smallest eigenvalues. Thus, since the $k$ smallest eigendecompoition of the normalized Laplacian corresponds to the relaxed solution of the normalized graph cut, the SORL is actually corresponds to the relaxed solution of the normalized graph cut (see Sec.5.3 in [i]). \nFrom this viewpoint, as far as I understand, Thm 3.1 can be more powerful for the spectral clustering community -- the Eq. (6) corresponds the result of the spectral clustering (i.e., the relaxed solution of the normalized graph cut).\n\nWith this, the proposed method is not surprising (in a good sense); the proposed SORL is an open-world learning hinted normalized graph cut.\n\nFrom this view, I'd like to connect the point (**), if this method is faster than SVD, this indicates that there can be some faster way of the spectral clustering?\n \n- This is also minor but the authors might want to cite [i], which is the one of the most cited paper in spectral clustering.\n\n[i] Von Luxburg, Ulrike. \"A tutorial on spectral clustering.\" Statistics and computing 17 (2007): 395-416.\n\n\n\n----\n\nPOST REBUTTAL\n\nI'm satisfied with the authors' rebuttal, therefore I increase my score from 6 to 7.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the domain of open-world representation learning, which aims to learn representations that can correctly cluster samples in the novel class and classify samples in the known classes by utilizing knowledge from the labeled data. Notably, the motivation of this paper is to provide a theoretical foundation in this area by introducing a novel graph-theoretic framework specifically designed for open-world settings. Theoretically, the crux of the framework involves characterizing clustering through graph factorization. A new algorithm called Spectral Open-world Representation Learning (SORL) is introduced, which essentially involves spectral decomposition on the graph. The authors provide theoretical guarantees by deriving a provable error bound on the clustering performance for both known and novel classes. Practically, the paper also offers empirical evidence, demonstrating that SORL is competitive with, or even superior to, existing methods on benchmark datasets.\n\nSTRENGTHS:\nThe paper is well-written with clear motivation and structure. \nOverall interesting problem; good mathematical exposition; solid theoretical results and analysis; interesting and good experimental results\n\nWEAKNESSES:\n1. The experiments are not sufficient. Other large commonly used benchmark dataset is not used, like ImageNet-100. \n2. The paper has a very good theoretical analysis. But it is still unclear what contributes to the performance improvement.\n\nQUESTIONS:\n1. It would be interesting to see the results in a larger benchmark dataset, ImageNet 100. \n\n2. What is the effect of using different known and novel classes ratio besides 50%? Like, 0.25, 0.1.\n\n3. How do you choose the hyper-parameters, $\\eta_{u}$, $\\eta_{l}$?\n\n4. In the supplementary material, it is mentioned that pre-training is used. I wonder why this pre-training is necessary for this method and what it will be without pertaining.\n\nLIMITATIONS:\nThe broader impact has been discussed. It has a positive potential impact on the community.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}