SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yiyou Sun, Zhenmei Shi, Yixuan Li
Department of Computer Sciences
University of Wisconsin, Madison
{sunyiyou,zhmeishi,sharonli}@cs.wisc.edu
Open-world semi-supervised learning aims at inferring both known and novel classes in unlabeled data, by harnessing prior knowledge from a labeled set with known classes. Despite its importance, there is a lack of theoretical foundations for this problem. This paper bridges the gap by formalizing a graph-theoretic framework tailored for the open-world setting, where the clustering can be theoretically characterized by graph factorization. Our graph-theoretic framework illuminates practical algorithms and provides guarantees. In particular, based on our graph formulation, we apply the algorithm called Spectral Open-world Representation Learning (SORL), and show that minimizing our loss is equivalent to performing spectral decomposition on the graph. Such equivalence allows us to derive a provable error bound on the clustering performance for both known and novel classes, and analyze rigorously when labeled data helps. Empirically, SORL can match or outperform several strong baselines on common benchmark datasets, which is appealing for practical usage while enjoying theoretical guarantees. Our code is available at https://github.com/deeplearning-wisc/sorl.
Machine learning models in the open world inevitably encounter data from both known and novel classes [2, 15, 16, 65, 79]. Traditional supervised machine learning models are trained on a closed set of labels, and thus can struggle to effectively cluster new semantic concepts. On the other hand, open-world semi-supervised learning approaches, such as those discussed in studies [7, 63, 69], enable models to distinguish both known and novel classes, making them highly desirable for real-world scenarios. As shown in Figure 1, the learner has access to a labeled training dataset  $\mathcal{D}_l$  (from known classes) as well as a large unlabeled dataset  $\mathcal{D}_u$  (from both known and novel classes). By optimizing feature representations jointly from both labeled and unlabeled data, the learner aims to create meaningful cluster structures that correspond to either known or novel classes. With the explosive growth of data generated in various
Figure 1: Open-world Semi-supervised Learning aims to correctly cluster samples in the novel class and classify samples in the known classes by utilizing knowledge from the labeled data. An open question is "what is the role of the label information in shaping representations for both known and novel classes?" This paper aims to provide a formal understanding.
domains, open-world semi-supervised learning has emerged as a crucial problem in the field of machine learning.
Motivation. Different from self-supervised learning [5, 8, 11, 12, 23, 26, 68, 77], open-world semi-supervised learning allows harnessing the power of the labeled data for possible knowledge sharing and transfer to unlabeled data, and from known classes to novel classes. In this joint learning process, we argue that interesting intricacies can arise—the labeled data provided may be beneficial or unhelpful to the resulting clusters. We exemplify the nuances in Figure 1. In one scenario, when the model learns the labeled known classes (e.g., traffic light) by pushing red and green lights closer, such a relationship might transfer to help cluster green and red apples into a coherent cluster. Alternatively, when the connection between the labeled data and the novel class (e.g., flower) is weak, the benefits might be negligible. We argue—perhaps obviously—that a formalized understanding of the intricate phenomenon is needed.
Theoretical significance. To date, theoretical understanding of open-world semi-supervised learning is still in its infancy. In this paper, we aim to fill the critical blank by analyzing this important learning problem from a rigorous theoretical standpoint. Our exposition gravitates around the open question: what is the role of labeled data in shaping representations for both known and novel classes? To answer this question, we formalize a graph-theoretic framework tailored for the open-world setting, where the vertices are all the data points and connected sub-graphs form classes (either known or novel). The edges are defined by a combination of supervised and self-supervised signals, which reflects the availability of both labeled and unlabeled data. Importantly, this graph facilitates the understanding of open-world semi-supervised learning from a spectral analysis perspective, where the clustering can be theoretically characterized by graph factorization. Based on the graph-theoretic formulation, we derive a formal error bound by contrasting the clustering performance for all classes, before and after adding the labeling information. Our Theorem 4.2 reveals the sufficient condition for the improved clustering performance for a class. Under the K-means measurement, the unlabeled samples in one class can be better clustered, if their overall connection to the labeled data is stronger than their self-clusterability.
Practical significance. Our graph-theoretic framework also illuminates practical algorithms with provided guarantees. In particular, based on our graph formulation, we present the algorithm called Spectral Open-world Representation Learning (SORL) adapted from Sun et al. [64]. Minimizing this loss is equivalent to performing spectral decomposition on the graph (Section 3.2), which brings two key benefits: (1) it allows us to analyze the representation space and resulting clustering performance in closed-form; (2) practically, it enables end-to-end training in the context of deep networks. We show that our learning algorithm leads to strong empirical performance while enjoying theoretical guarantees. The learning objective can be effectively optimized using stochastic gradient descent on modern neural network architecture, making it desirable for real-world applications.
2 Problem Setup
We formally describe the data setup and learning goal of open-world semi-supervised learning [7].
Data setup. We consider the empirical training set  $\mathcal{D}_l\cup \mathcal{D}_u$  as a union of labeled and unlabeled data.
1. The labeled set  $\mathcal{D}_l = \{\bar{x}_i, y_i\}_{i=1}^n$ , with  $y_i \in \mathcal{Y}_l$ . The label set  $\mathcal{Y}_l$  is known.
2. The unlabeled set  $\mathcal{D}_u = \{\bar{x}_i\}_{i=1}^m$ , where each sample  $\bar{x}_i$  can come from either known or novel classes<sup>1</sup>. Note that we do not have access to the labels in  $\mathcal{D}_u$ . For mathematical convenience, we denote the underlying label set as  $\mathcal{V}_{\mathrm{all}}$ , where  $\mathcal{V}_l \subset \mathcal{V}_{\mathrm{all}}$ . We denote  $C = |\mathcal{V}_{\mathrm{all}}|$  the total number of classes.
The data setup has practical value for real-world applications. For example, the labeled set is common in supervised learning; and the unlabeled set can be gathered for free from the model's operating environment or the internet. We use  $\mathcal{P}_l$  and  $\mathcal{P}$  to denote the marginal distributions of labeled data and all data in the input space, respectively. Further, we let  $\mathcal{P}_{l_i}$  denote the distribution of labeled samples with class label  $i\in \mathcal{V}_l$ .
This generalizes the problem of Novel Class Discovery [19, 22, 27, 28, 82, 83], which assumes the unlabeled set is purely from novel classes.
Learning target. Under the setting, our goal is to learn distinguishable representations for both known and novel classes simultaneously. The representation quality will be measured using classic metrics, such as K-means clustering accuracy, which we will define mathematically in Section 4.2.2. Unlike classic semi-supervised learning [86], we place no assumption on the unlabeled data and allow its semantic space to cover both known and novel classes. The problem is also referred to as open-world representation learning [63], which emphasizes the role of good representation in distinguishing both known and novel classes.
Theoretical analysis goal. We aim to comprehend the role of label information in shaping representations for both known and novel classes. It's important to note that our theoretical approach aims to understand the perturbation in the clustering performance by labeling existing, previously unlabeled data points within the dataset. By contrasting the clustering performance before and after labeling these instances, we uncover the underlying structure and relations that the labels may reveal. This analysis provides invaluable insights into how labeling information can be effectively leveraged to enhance the representations of both known and novel classes.
3 A Spectral Approach for Open-world Semi-Supervised Learning
In this section, we formalize and tackle the open-world semi-supervised learning problem from a graph-theoretic view. Our fundamental idea is to formulate it as a clustering problem—where similar data points are grouped into the same cluster, by way of possibly utilizing helpful information from the labeled data  $\mathcal{D}_l$ . This clustering process can be modeled by a graph, where the vertices are all the data points and classes form connected sub-graphs. Specifically, utilizing our graph formulation, we present the algorithm — Spectral Open-world Representation Learning (SORL) in Section 3.2. The process of minimizing the corresponding loss is fundamentally analogous to executing a spectral decomposition on the graph.
3.1 A Graph-Theoretic Formulation
We start by formally defining the augmentation graph and adjacency matrix. For clarity, we use  $\bar{x}$  to indicate the natural sample (raw inputs without augmentation). Given an  $\bar{x}$ , we use  $\mathcal{T}(x|\bar{x})$  to denote the probability of  $x$  being augmented from  $\bar{x}$ . For instance, when  $\bar{x}$  represents an image,  $\mathcal{T}(\cdot |\bar{x})$  can be the distribution of common augmentations [11] such as Gaussian blur, color distortion, and random cropping. The augmentation allows us to define a general population space  $\mathcal{X}$ , which contains all the original images along with their augmentations. In our case,  $\mathcal{X}$  is composed of augmented samples from both labeled and unlabeled data, with cardinality  $|\mathcal{X}| = N$ . We further denote  $\mathcal{X}_l$  as the set of samples (along with augmentations) from the labeled data part.
We define the graph  $G(\mathcal{X}, w)$  with vertex set  $\mathcal{X}$  and edge weights  $w$ . To define edge weights  $w$ , we decompose the graph connectivity into two components: (1) self-supervised connectivity  $w^{(u)}$  by treating all points in  $\mathcal{X}$  as entirely unlabeled, and (2) supervised connectivity  $w^{(l)}$  by adding labeled information from  $\mathcal{P}_l$  to the graph. We proceed to define these two cases separately.
First, by assuming all points as unlabeled, two samples  $(x,x^{+})$  are considered a positive pair if:
Unlabeled Case (u):  $x$  and  $x^{+}$  are augmented from the same image  $\bar{x} \sim \mathcal{P}$ .
For any two augmented data  $x, x' \in \mathcal{X}$ ,  $w_{xx'}^{(u)}$  denotes the marginal probability of generating the pair:
$$
w _ {x x ^ {\prime}} ^ {(u)} \triangleq \mathbb {E} _ {\bar {x} \sim \mathcal {P}} \mathcal {T} (x | \bar {x}) \mathcal {T} \left(x ^ {\prime} | \bar {x}\right), \tag {1}
$$
which can be viewed as self-supervised connectivity [11, 23]. However, different from self-supervised learning, we have access to the labeled information for a subset of nodes, which allows adding additional connectivity to the graph. Accordingly, the positive pair can be defined as:
Labeled Case (l):  $x$  and  $x^{+}$  are augmented from two labeled samples  $\bar{x}_l$  and  $\bar{x}_l'$  with the same known class  $i$ . In other words, both  $\bar{x}_l$  and  $\bar{x}_l'$  are drawn independently from  $\mathcal{P}_{l_i}$ .
Considering both case (u) and case (l), the overall edge weight for any pair of data  $(x,x^{\prime})$  is given by:
$$
w _ {x x ^ {\prime}} = \eta_ {u} w _ {x x ^ {\prime}} ^ {(u)} + \eta_ {l} w _ {x x ^ {\prime}} ^ {(l)}, \text {w h e r e} w _ {x x ^ {\prime}} ^ {(l)} \triangleq \sum_ {i \in \mathcal {Y} _ {l}} \mathbb {E} _ {\bar {x} _ {l} \sim \mathcal {P} _ {l _ {i}}} \mathbb {E} _ {\bar {x} _ {l} ^ {\prime} \sim \mathcal {P} _ {l _ {i}}} \mathcal {T} (x | \bar {x} _ {l}) \mathcal {T} \left(x ^ {\prime} | \bar {x} _ {l} ^ {\prime}\right), \tag {2}
$$
and  $\eta_u, \eta_l$  modulates the importance between the two cases. The magnitude of  $w_{xx'}$  indicates the "positiveness" or similarity between  $x$  and  $x'$ . We then use  $w_x = \sum_{x' \in \mathcal{X}} w_{xx'}$  to denote the total edge weights connected to a vertex  $x$ .
Remark: A graph perturbation view. With the graph connectivity defined above, we can now define the adjacency matrix  $A \in \mathbb{R}^{N \times N}$  with entries  $A_{xx'} = w_{xx'}$ . Importantly, the adjacency matrix can be decomposed into two parts:
$\boxed{\mathrm{Perturbation by adding labels}}$
$$
A = \eta_ {u} A ^ {(u)} + \left| \eta_ {l} A ^ {(l)} \right., \tag {3}
$$
which can be regarded as the self-supervised adjacency matrix  $A^{(u)}$  perturbed by additional labeling information encoded in  $A^{(l)}$ . This graph perturbation view serves as a critical foundation for our theoretical analysis of the clustering performance in Section 4. As a standard technique in graph theory [14], we use the normalized adjacency matrix of  $G(\mathcal{X}, w)$ :
$$
\tilde {A} \triangleq D ^ {- \frac {1}{2}} A D ^ {- \frac {1}{2}}, \tag {4}
$$
where  $D \in \mathbb{R}^{N \times N}$  is a diagonal matrix with  $D_{xx} = w_x$ . The normalization balances the degree of each node, reducing the influence of vertices with very large degrees. The normalized adjacency matrix defines the probability of  $x$  and  $x'$  being considered as the positive pair from the perspective of augmentation, which helps derive the learning loss as we show next.
3.2 SORL: Spectral Open-World Representation Learning
We present an algorithm called Spectral Open-world Representation Learning (SORL), which can be derived from a spectral decomposition of  $\tilde{A}$ . The algorithm has both practical and theoretical values. First, it enables efficient end-to-end training in the context of modern neural networks. More importantly, it allows drawing a theoretical equivalence between learned representations and the top- $k$  singular vectors of  $\tilde{A}$ . Such equivalence facilitates theoretical understanding of the clustering structure encoded in  $\tilde{A}$ . Specifically, we consider low-rank matrix approximation:
$$
\min  _ {F \in \mathbb {R} ^ {N \times k}} \mathcal {L} _ {\mathrm {m f}} (F, A) \triangleq \left\| \tilde {A} - F F ^ {\top} \right\| _ {F} ^ {2} \tag {5}
$$
According to the Eckart-Young-Mirsky theorem [17], the minimizer of this loss function is  $F_{k} \in \mathbb{R}^{N \times k}$  such that  $F_{k}F_{k}^{\top}$  contains the top- $k$  components of  $\tilde{A}$ 's SVD decomposition.
Now, if we view each row  $\mathbf{f}_x^\top$  of  $F$  as a scaled version of learned feature embedding  $f: \mathcal{X} \mapsto \mathbb{R}^k$ , the  $\mathcal{L}_{\mathrm{mf}}(F, A)$  can be written as a form of the contrastive learning objective. We formalize this connection in Theorem 3.1 below<sup>2</sup>.
Theorem 3.1. We define  $\mathbf{f}_x = \sqrt{w_x} f(x)$  for some function  $f$ . Recall  $\eta_u, \eta_l$  are coefficients defined in Eq. (2). Then minimizing the loss function  $\mathcal{L}_{\mathrm{mf}}(F, A)$  is equivalent to minimizing the following loss function for  $f$ , which we term Spectral Open-world Representation Learning (SORL):
$$
\mathcal {L} _ {S O R L} (f) \triangleq - 2 \eta_ {l} \mathcal {L} _ {1} (f) - 2 \eta_ {u} \mathcal {L} _ {2} (f) + \eta_ {l} ^ {2} \mathcal {L} _ {3} (f) + 2 \eta_ {l} \eta_ {u} \mathcal {L} _ {4} (f) + \eta_ {u} ^ {2} \mathcal {L} _ {5} (f), \tag {6}
$$
$$
\mathcal{L}_{1}(f) = \sum_{i\in \mathcal{Y}_{l}}\underset { \begin{array}{c}\bar{x}_{l}\sim \mathcal{P}_{l_{i}},\bar{x}_{l}^{\prime}\sim \mathcal{P}_{l_{i}},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{l}),x^{+}\sim \mathcal{T}(\cdot |\bar{x}_{l}^{\prime}) \end{array} }{\mathbb{E}}\left[f(x)^{\top}f\left(x^{+}\right)\right],\\ \mathcal{L}_{2}(f) = \underset { \begin{array}{c} \bar{x}_{u}\sim \mathcal{P},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{u}),x^{+}\sim \mathcal{T}(\cdot |\bar{x}_{u}) \end{array} }{\mathbb{E}}\left[f(x)^{\top}f\left(x^{+}\right)\right],
$$
$$
\mathcal{L}_{3}(f) = \sum_{i,j\in \mathcal{Y}_{l}}\underset { \begin{array}{c}\bar{x}_{l}\sim \mathcal{P}_{l_{i}},\bar{x}_{l}^{\prime}\sim \mathcal{P}_{l_{j}},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{l}),x^{-}\sim \mathcal{T}(\cdot |\bar{x}_{l}^{\prime}) \end{array} }{\mathbb{E}}\left[  \big(f(x)^{\top}f\left(x^{-}\right)\big)^{2}\right],
$$
$$
\mathcal{L}_{4}(f) = \sum_{i\in \mathcal{Y}_{l}}\underset { \begin{array}{c}\bar{x}_{l}\sim \mathcal{P}_{l_{i}},\bar{x}_{u}\sim \mathcal{P},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{l}),x^{-}\sim \mathcal{T}(\cdot |\bar{x}_{u}) \end{array} }{\mathbb{E}}\left[ \big(f(x)^{\top}f\left(x^{-}\right)\big)^{2}\right],\\ \mathcal{L}_{5}(f) = \underset { \begin{array}{c}\bar{x}_{u}\sim \mathcal{P},\bar{x}_{u}^{\prime}\sim \mathcal{P},\\ x\sim \mathcal{T}(\cdot |\bar{x}_{u}),x^{-}\sim \mathcal{T}(\cdot |\bar{x}_{u}^{\prime}) \end{array} }{\mathbb{E}}\left[ \big(f(x)^{\top}f\left(x^{-}\right)\big)^{2}\right].
$$
Theorem 3.1 is primarily adapted from Theorem 4.1 in [64]. However, there is a distinction in the data setting, as Sun et al. [64] do not consider known class samples within the unlabeled dataset.
Proof. (sketch) We can expand  $\mathcal{L}_{\mathrm{mf}}(F,A)$  and obtain
$$
\mathcal {L} _ {\mathrm {m f}} (F, A) = \sum_ {x, x ^ {\prime} \in \mathcal {X}} \left(\frac {w _ {x x ^ {\prime}}}{\sqrt {w _ {x} w _ {x ^ {\prime}}}} - \mathbf {f} _ {x} ^ {\top} \mathbf {f} _ {x ^ {\prime}}\right) ^ {2} = c o n s t + \sum_ {x, x ^ {\prime} \in \mathcal {X}} \left(- 2 w _ {x x ^ {\prime}} f (x) ^ {\top} f (x ^ {\prime}) + w _ {x} w _ {x ^ {\prime}} (f (x) ^ {\top} f (x ^ {\prime})) ^ {2}\right)
$$
The form of  $\mathcal{L}_{\mathrm{SORL}}(f)$  is derived from plugging  $w_{xx'}$  (defined in Eq. (1)) and  $w_x$ . Full proof is in Appendix A.
Interpretation of  $\mathcal{L}_{\mathrm{SORL}}(f)$ . At a high level,  $\mathcal{L}_1$  and  $\mathcal{L}_2$  push the embeddings of positive pairs to be closer while  $\mathcal{L}_3$ ,  $\mathcal{L}_4$  and  $\mathcal{L}_5$  pull away the embeddings of negative pairs. In particular,  $\mathcal{L}_1$  samples two random augmentation views of two images from labeled data with the same class label, and  $\mathcal{L}_2$  samples two views from the same image in  $\mathcal{X}$ . For negative pairs,  $\mathcal{L}_3$  uses two augmentation views from two samples in  $\mathcal{X}_l$  with any class label.  $\mathcal{L}_4$  uses two views of one sample in  $\mathcal{X}_l$  and another one in  $\mathcal{X}$ .  $\mathcal{L}_5$  uses two views from two random samples in  $\mathcal{X}$ . This training objective, though bearing similarities to NSCL [64], operates within a distinct problem domain. Accordingly, we derive novel theoretical analysis uniquely tailored to our problem setting, which we present next.
So far we have presented a spectral approach for open-world semi-supervised learning based on graph factorization. Under this framework, we now formally analyze: how does the labeling information shape the representations for known and novel classes?
4.1 An Illustrative Example
We consider a toy example that helps illustrate the core idea of our theoretical findings. Specifically, the example aims to distinguish 3D objects with different shapes, as shown in Figure 2. These images are generated by a 3D rendering software [31] with user-defined properties including colors, shape, size, position, etc. We are interested in contrasting the representations (in the form of singular vectors), when the label information is either incorporated in training or not.
Data design. Suppose the training samples come from three types,  $\mathcal{X}_{\square}, \mathcal{X}_{\bigcirc}, \mathcal{X}_{\bigtriangledown}$ . Let  $\mathcal{X}_{\square}$  be the sample space with known class, and  $\mathcal{X}_{\bigcirc}, \mathcal{X}_{\bigtriangledown}$  be the sample space with novel classes. Further, the two novel classes are constructed to have different relationships with the known class. Specifically,  $\mathcal{X}_{\bigcirc}$  shares some similarity with  $\mathcal{X}_{\square}$  in color (red and blue); whereas another novel class  $\mathcal{X}_{\bigtriangledown}$  has no obvious similarity with the known class. Without any labeling information, it can be difficult to distinguish  $\mathcal{X}_{\bigcirc}$  from  $\mathcal{X}_{\square}$  since samples share common colors. We aim to verify the hypothesis that: adding labeling information to  $\mathcal{X}_{\square}$  (i.e., connecting  $\square$  and  $\triangleright$ ) has a larger (beneficial) impact to cluster  $\mathcal{X}_{\bigcirc}$  than  $\mathcal{X}_{\bigtriangledown}$ .
Figure 2: An illustrative example for theoretical analysis. We consider a 6-node graph with one known class (cube) and two novel classes (sphere, cylinder). (a) The augmentation probabilities between nodes are defined by their color and shape in Eq. (7). (b) The adjacency matrix can then be calculated by Equations in Sec. 3.1 where we let  $\tau_0 = 0$ ,  $\eta_u = 6$ ,  $\eta_l = 4$ . The calculation details are in Appendix B. The magnitude order follows  $\tau_1 \gg \tau_c > \tau_s > 0$ .
Augmentation graph. Based on the data design, we formally define the augmentation graph, which encodes the probability of augmenting a source image  $\bar{x}$  to the augmented view  $x$ :
$$
\mathcal {T} (x \mid \bar {x}) = \left\{ \begin{array}{l l} \tau_ {1} & \text {i f c o l o r} (x) = \operatorname {c o l o r} (\bar {x}), \operatorname {s h a p e} (x) = \operatorname {s h a p e} (\bar {x}); \\ \tau_ {c} & \text {i f c o l o r} (x) = \operatorname {c o l o r} (\bar {x}), \operatorname {s h a p e} (x) \neq \operatorname {s h a p e} (\bar {x}); \\ \tau_ {s} & \text {i f c o l o r} (x) \neq \operatorname {c o l o r} (\bar {x}), \operatorname {s h a p e} (x) = \operatorname {s h a p e} (\bar {x}); \\ \tau_ {0} & \text {i f c o l o r} (x) \neq \operatorname {c o l o r} (\bar {x}), \operatorname {s h a p e} (x) \neq \operatorname {s h a p e} (\bar {x}). \end{array} \right. \tag {7}
$$
With Eq. (7) and the definition of the adjacency matrix in Section 3.1, we can derive the analytic form of  $A^{(u)}$  and  $A$ , as shown in Figure 2(b). We refer readers to Appendix B for the detailed derivation. The two matrices allow us to contrast the connectivity changes in the graph, before and after the labeling information is added. Insights. We are primarily interested in analyzing the difference of the representation space derived from  $A^{(u)}$  and  $A$ . We visualize the top-3 eigenvectors<sup>3</sup> of the normalized adjacency matrix  $\tilde{A}^{(u)}$  and  $\tilde{A}$  in Figure 3(a), where the results are based on the magnitude order  $\tau_{1} \gg \tau_{c} > \tau_{s} > 0$ . Our key takeaway is: adding labeling information to known class  $\mathcal{X} \square$  helps better distinguish the known class itself and the novel class  $\mathcal{X}_{\bigcirc}$ , which has a stronger connection/similarity with  $\mathcal{X} \square$ .
Qualitative analysis. Our theoretical insight can also be verified empirically, by learning representations on over 10,000 samples using the loss defined in Section 3.2. Due to the space limitation, we include experimental details in Appendix E.1. In Figure 3(b), we visualize the learned features through UMAP [43]. Indeed, we observe that samples become more concentrated around different shape classes after adding labeling information to the cube class.