SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuheng Jia $^{1,3*}$ , Fuchao Yang $^{2}$ , Yongqiang Dong $^{1*}$
<sup>2</sup> College of Software Engineering, Southeast University Nanjing 210096, China
<sup>3</sup> Key Laboratory of New Generation Artificial Intelligence Technology and Its
yhjia@seu.edu.cn, yangfc@seu.edu.cn, dongyq@seu.edu.cn
In partial label learning (PLL), each sample is associated with a group of candidate labels, among which only one label is correct. The key of PLL is to disambiguate the candidate label set to find the ground-truth label. To this end, we first construct a constrained regression model to capture the confidence of the candidate labels, and multiply the label confidence matrix by its transpose to build a second-order similarity matrix, whose elements indicate the pairwise similarity relationships of samples globally. Then we develop a semantic dissimilarity matrix by considering the complement of the intersection of the candidate label set, and further propagate the initial dissimilarity relationships to the whole data set by leveraging the local geometric structure of samples. The similarity and dissimilarity matrices form an adversarial relationship, which is further utilized to shrink the solution space of the label confidence matrix and promote the dissimilarity matrix. We finally extend the proposed model to a kernel version to exploit the non-linear structure of samples and solve the proposed model by the inexact augmented Lagrange multiplier method. By exploiting the adversarial prior, the proposed method can significantly outperform state-of-the-art PLL algorithms when evaluated on 10 artificial and 7 real-world partial label data sets. We also prove the effectiveness of our method with some theoretical guarantees. The code is publicly available at https://github.com/Yangfc-ML/DPCLS.
Partial label learning (PLL) [14, 15, 17] is an emerging weakly supervised learning framework. In PLL, each sample is associated with a set of candidate labels, among which only one is the ground-truth label. Different from conventional supervised learning, PLL avoids precisely annotating label on each sample, which greatly reduces the labeling cost. Accordingly, PLL has been applied to many real-world scenarios such as automatic image annotation [1, 20], web mining [9], and ecoinformatics [8].
Formally speaking, let  $\mathcal{X} = \mathbb{R}^d$  be the  $d$ -dimensional feature space and  $\mathcal{V} = \{1,2,\dots,q\}$  be the label space with  $q$  labels. Given the partial label training set  $\mathcal{D} = \{(x_i,S_i)\mid 1\leq i\leq m\}$ , where  $x_{i}\in \mathcal{X}$  is a  $d$ -dimensional feature vector to represent the  $i$ -th sample and  $S_{i}\subseteq \mathcal{V}$  is the associated candidate label set, among which only one label is correct. PLL aims to induce a multi-class classifier  $f:\mathcal{X}\to \mathcal{V}$  from  $\mathcal{D}$ , which is very challenging as the ground-truth label of a sample is concealed in the candidate label set.
The key to solving PLL is label disambiguation, i.e., identifying the ground-truth label of a sample from its candidate label set. For example, [6, 12] considered the ground-truth label as a latent variable and identified it through an iterative refining procedure. [3] narrowed the candidate label set through a sparsity-based self-training procedure. Some works [19, 22, 23] leveraged the similarity relationship of samples in the feature space to disambiguation, i.e., samples are similar to each other in the feature space are likely to share the same ground-truth label. Recently, some researches exploited the label space to achieve label disambiguation [2, 7, 13]. For example, SDIM [2] first built a pairwise dissimilarity matrix through the candidate label sets, and then maximized the difference of the label confidence between two samples if their pairwise dissimilarity between them is large according to the constructed dissimilarity matrix. However, the dissimilarity matrix constructed by SDIM is predefined and relatively sparse, which depresses its effectiveness.
Realizing the effectiveness of the dissimilarity relationship in PLL, we propose a novel PLL method named DPCLS (partial label learning with Dissimilarity Propagation guided Candidate Label Shrinkage). Specifically, we first construct a semantic dissimilarity matrix by considering the complement of the intersection of the candidate label set, i.e., if two samples do not share any common candidate labels, they must belong to different classes, and their semantic dissimilarity is large. The above constructed dissimilarity relationships are still sparse and fixed, we therefore propose to propagate the initial dissimilarity relationships to the whole data set by leveraging the local geometric structure of samples, i.e., if two samples are similar to each other in feature space, they are expected to share the similar dissimilarity codings. Second, to include the enhanced semantic dissimilarity matrix in label disambiguation, we design a second-order similarity matrix by multiplying the label confidence matrix with its transpose. Under the ideal condition, the enhanced semantic dissimilarity matrix and the similarity matrix naturally form an adversarial relationship, i.e., a larger (resp. smaller) dissimilarity between two samples means a smaller (resp. larger) similarity between them. By exploiting this adversarial prior, the enhanced dissimilarity matrix can shrink the solution space of the label confidence matrix, and meanwhile, the similarity matrix induced from the label confidence matrix also contributes to build a better dissimilarity matrix. We theoretically confirm the above statement under some general conditions. Furthermore, we extend our method to a kernel version to model the non-linear structure of samples. The proposed model is finally formulated as a constrained regression problem with adversarial learning and graph regularization, which is optimized by inexact augmented Lagrange multiplier (IALM). Extensive experiments on artificial and real-world partial label data sets demonstrate the effectiveness of the proposed PLL method.
Let  $\mathbf{X} = [x_1, x_2, \dots, x_m]^{\mathsf{T}} \in \mathbb{R}^{m \times d}$  denote the feature matrix, where  $m$  and  $d$  represent the number of samples and the dimension of features.  $\mathbf{Y} = [y_1, y_2, \dots, y_m]^{\mathsf{T}} \in \{0, 1\}^{m \times q}$  represents the partial label matrix, where  $q$  is the number of classes. Moreover,  $y_{ij} = 1$  indicates that the  $j$ -th label is one of the candidate labels of the sample  $x_i$ . Note that in the candidate label set, only one label is correct.
To fulfill PLL, we first build the following constrained regression model
$$
\min  _ {\mathbf {W}, \mathbf {F}} \| \mathbf {X} \mathbf {W} - \mathbf {F} \| _ {F} ^ {2} + \lambda \| \mathbf {W} \| _ {F} ^ {2} \tag {1}
$$
$$
\mathrm {s . t .} \mathbf {F 1} _ {q} = \mathbf {1} _ {m}, \mathbf {0} _ {m \times q} \leq \mathbf {F} \leq \mathbf {Y},
$$
where  $\mathbf{F} \in \mathbb{R}^{m \times q}$  is the label confidence matrix with  $\mathbf{F}_{ij}$  representing the probability of the  $j$ -th label being the ground-truth label for the  $i$ -th sample.  $\mathbf{W} \in \mathbb{R}^{d \times q}$  is the coefficient matrix that maps the feature matrix to the label confidence matrix  $\mathbf{F}$ . To avoid over-fitting, we impose the widely-used squared Frobenius norm on  $\mathbf{W}$  as the regularization term, which is introduced by  $\lambda > 0$ .  $\mathbf{1}_q \in \mathbb{R}^q$  and  $\mathbf{1}_m \in \mathbb{R}^m$  are two all ones vectors,  $\mathbf{0}_{m \times q} \in \mathbb{R}^{m \times q}$  is an all zeroes matrix. The first constraint  $\mathbf{F1}_q = \mathbf{1}_m$  normalizes the label confidence vector for all samples. The second constraint  $\mathbf{0}_{m \times q} \leq \mathbf{F} \leq \mathbf{Y}$  means each element of  $\mathbf{F}$  is no less than 0, and no more than the corresponding element of  $\mathbf{Y}$ , which implies the ground-truth label of each sample must reside in the candidate label set and the label confidence of the non-candidate labels must be 0. We initialize the label confidence matrix as  $\mathbf{F}_{ij} = 1 / \sum_{j} y_{ij}$  if  $y_{ij} = 1$ , otherwise  $\mathbf{F}_{ij} = 0$ .
By minimizing Eq. (1), we construct a linear regression model that maps the feature space to the label confidence  $\mathbf{F}$ . We assume that the mapping from the features to the ground-truth label may
be easier, while that to the false-positive label residing in the candidate label set is relatively harder. Accordingly, optimizing Eq. (1) will help disambiguate the candidate labels and produce a preliminary label confidence matrix by exploring the useful information in the feature space.
Dissimilarity Propagation guided Candidate Label Shrinkage
To further exploit the valuable information in the label space, we first use candidate labels to construct a dissimilarity matrix  $\mathbf{D}_0\in \mathbb{R}^{m\times m}$ , i.e.,
$$
\mathbf {D} _ {0 i j} = \left\{ \begin{array}{l} 1, \text {i f} y _ {i} y _ {j} ^ {\top} = 0 \\ 0, \text {o t h e r w i s e .} \end{array} \right. \tag {2}
$$
If  $y_{i}y_{j}^{\top} = 0$ , the  $i$ -th sample  $x_{i}$  and the  $j$ -th sample  $x_{j}$  do not share any common candidate labels, which means they must belong to the different classes. Otherwise,  $x_{i}$  and  $x_{j}$  have a chance to belong to the same class. Therefore,  $\mathbf{D}_0$  indicates the semantic dissimilarity of samples. We then multiply the label confidence matrix with its transpose to create a similarity matrix termed as  $\mathbf{F}\mathbf{F}^{\top}$ , whose  $(i,j)$ -th element indicates the similarity between  $x_{i}$  and  $x_{j}$ . As the semantic dissimilarity matrix  $\mathbf{D}_0$  and similarity matrix  $\mathbf{F}\mathbf{F}^{\top}$  form an adversarial relationship, i.e., a larger (resp. smaller) element in  $\mathbf{D}_0$  implies a smaller (resp. larger) element in  $\mathbf{F}\mathbf{F}^{\top}$ , we use this adversarial prior to shrink the solution space of  $\mathbf{F}$  by:
$$
\left. \min  _ {\mathbf {F}} \left\| \mathbf {D} _ {0} \odot \mathbf {F} \mathbf {F} ^ {\top} \right\| _ {1}, \right. \tag {3}
$$
where  $\odot$  denotes the element-wise product of two matrices and  $||\cdot ||_1$  represents the  $l_{1}$  norm (i.e., the sum of absolute values of all elements in a matrix). Minimizing Eq. (3) ensues that the positive elements of  $\mathbf{D}_0$  and  $\mathbf{F}\mathbf{F}^{\top}$  will lie in the different locations. Unfortunately, directly minimizing Eq. (3) cannot help produce a better label confidence matrix  $\mathbf{F}$ , because  $\mathbf{D}_0$  is inferred from the candidate label set, and the  $(i,j)$ -th element of  $\mathbf{D}_0$  is positive only when  $y_i y_j^{\top} = 0$ , while  $\mathbf{F}$  is upper bounded by  $\mathbf{Y}$ , the  $(i,j)$ -th element of  $\mathbf{F}\mathbf{F}^{\top}$  is positive only when  $y_i y_j^{\top} \neq 0$ . That is the locations of the positive elements of  $\mathbf{D}_0$  and  $\mathbf{F}\mathbf{F}^{\top}$  are complementary.
To solve this problem, we propose to promote the initial dissimilarity matrix  $\mathbf{D}_0$  to produce a denser dissimilarity matrix  $\mathbf{D} \in \mathbb{R}^{m \times m}$  by a novel dissimilarity propagation method. Specifically, we leverage the local geometric structure of samples to enhance  $\mathbf{D}_0$ . Note that each column of  $\mathbf{D}$  (e.g.,  $\mathbf{D}_{.i}$ ) can represent the dissimilarity relationships between a sample (e.g.,  $x_i$ ) and the other samples. If two samples  $x_i$  and  $x_j$  are close to each other in the feature space, their dissimilarity relationships  $(\mathbf{D}_{.i}$  and  $\mathbf{D}_{.j}$ ) should also be similar. To capture the feature similarity, we build a local geometric matrix  $\mathbf{S} \in \mathbb{R}^{m \times m}$  using a radial basis function (RBF) kernel:
$$
\mathbf {S} _ {i j} = \left\{\begin{array}{l}\exp \left( \right.- \left| | x _ {i} - x _ {j} \right|\left. \right| _ {2} ^ {2} / \sigma^ {2}\left. \right), \text {i f} j \in \mathcal {N} _ {i}\\0, \text {o t h e r w i s e},\end{array}\right. \tag {4}
$$
where  $j \in \mathcal{N}_i$  indicates that  $x_j$  is a  $k$ -nearest neighbor of  $x_i$ , and  $\sigma$  is a hyper-parameter controlling the bandwidth of the RBF kernel. Based on S, the dissimilarity propagation guided candidate label shrinkage module becomes
$$
\min  _ {\mathbf {F}, \mathbf {D}} \left\| \mathbf {D} \odot \mathbf {F F} ^ {\top} \right\| _ {1} + \sum_ {i, j = 1} ^ {m} \mathbf {S} _ {i j} \| \mathbf {D}. i - \mathbf {D}. j \| _ {2} ^ {2} \tag {5}
$$
$$
\mathrm {s . t .} \mathbf {0} _ {m \times m} \leq \mathbf {D} \leq \mathbf {1} _ {m \times m}, \mathbf {D} _ {i j} = \mathbf {D} _ {0 i j}, \text {i f} \mathbf {D} _ {0 i j} = 1.
$$
The second term makes the highly similar samples in the feature samples share the similar dissimilarity codings. Moreover,  $\mathbf{D}_{ij} = \bar{\mathbf{D}}_{0ij}$  if  $\mathbf{D}_{0ij} = 1$  means the reliable semantic relationship should be retained in  $\mathbf{D}$ . In order to make  $\mathbf{D}$  a well-defined dissimilarity matrix, each element in  $\mathbf{D}$  should lie in the range of [0,1]. Note that in Eq. (5), both  $\mathbf{F}$  and  $\mathbf{D}$  are optimization variables. By minimizing Eq. (5), the enhanced semantic dissimilarity matrix will shrink the solution space of  $\mathbf{F}$  and at the same time  $\mathbf{F}$  will also help promote the quality of  $\mathbf{D}$  by the adversarial term.
Model Formulation
Taking all the above considerations into account, the proposed model finally becomes:
$$
\min  _ {\mathbf {W}, \mathbf {F}, \mathbf {D}} \| \mathbf {X} \mathbf {W} - \mathbf {F} \| _ {F} ^ {2} + \lambda \| \mathbf {W} \| _ {F} ^ {2} + \alpha \left\| \mathbf {D} \odot \mathbf {F} \mathbf {F} ^ {\top} \right\| _ {1} + \beta \operatorname {T r} (\mathbf {D L D} ^ {\top}) \tag {6}
$$
$$
\text {s . t .} \mathbf {F 1} _ {q} = \mathbf {1} _ {m}, \mathbf {0} _ {m \times q} \leq \mathbf {F} \leq \mathbf {Y}, \mathbf {0} _ {m \times m} \leq \mathbf {D} \leq \mathbf {1} _ {m \times m}, \mathbf {D} _ {i j} = \mathbf {D} _ {0 i j}, \text {i f} \mathbf {D} _ {0 i j} = 1,
$$
where  $\mathbf{L} \in \mathbb{R}^{m \times m} = \mathbf{D}_{\mathbf{S}} - \mathbf{S}$  is a graph Laplacian matrix, and  $\mathbf{D}_{\mathbf{S}} \in \mathbb{R}^{m \times m}$  is a diagonal matrix with the  $i$ -th diagonal element  $\mathbf{D}_{\mathbf{S}ii} = \sum_{j=1}^{m} \mathbf{S}_{ij}$ .  $\mathrm{Tr}(\cdot)$  returns the trace of a matrix.  $\alpha, \beta \geq 0$  are two hyper-parameters to balance different terms. By solving Eq. (6), the semantic dissimilarity is enhanced by a dissimilarity propagation process, and further utilized to shrink the solution space of the label confidence matrix by a novel adversarial prior.
3 Optimization and Setting
We adopt IALM to solve the problem in Eq. (6). To simplify Eq. (6), we introduce an auxiliary matrix  $\mathbf{A} = \mathbf{D}\in \mathbb{R}^{m\times m}$  and equivalently rewrite it as
$$
\min  _ {\mathbf {W}, \mathbf {F}, \mathbf {D}, \mathbf {A}} \| \mathbf {X} \mathbf {W} - \mathbf {F} \| _ {F} ^ {2} + \lambda \| \mathbf {W} \| _ {F} ^ {2} + \alpha \| \mathbf {A} \odot \mathbf {F} \mathbf {F} ^ {\top} \| _ {1} + \beta \operatorname {T r} (\mathbf {D L D} ^ {\top}) \tag {7}
$$
$$
\text {s . t .} \mathbf {F 1} _ {q} = \mathbf {1} _ {m}, \mathbf {0} _ {m \times q} \leq \mathbf {F} \leq \mathbf {Y}, \mathbf {D} = \mathbf {A}, \mathbf {0} _ {m \times m} \leq \mathbf {A} \leq \mathbf {1} _ {m \times m}, \mathbf {A} _ {i j} = \mathbf {D} _ {0 i j}, \text {i f} \mathbf {D} _ {0 i j} = 1.
$$
The solution of Eq. (7) can be obtained by solving the following augmented Lagrange equation:
$$
\min  _ {\mathbf {W}, \mathbf {F}, \mathbf {D}, \mathbf {A}} \| \mathbf {X} \mathbf {W} - \mathbf {F} \| _ {F} ^ {2} + \lambda \| \mathbf {W} \| _ {F} ^ {2} + \alpha \left\| \mathbf {A} \odot \mathbf {F} \mathbf {F} ^ {\top} \right\| _ {1} + \beta \operatorname {T r} (\mathbf {D L D} ^ {\top}) + \langle \boldsymbol {\Phi}, \mathbf {D} - \mathbf {A} \rangle + \frac {\mu}{2} \| \mathbf {D} - \mathbf {A} \| _ {F} ^ {2}
$$
$$
\text {s . t .} \mathbf {F 1} _ {q} = \mathbf {1} _ {m}, \mathbf {0} _ {m \times q} \leq \mathbf {F} \leq \mathbf {Y}, \mathbf {0} _ {m \times m} \leq \mathbf {A} \leq \mathbf {1} _ {m \times m}, \mathbf {A} _ {i j} = \mathbf {D} _ {0 i j}, \text {i f} \mathbf {D} _ {0 i j} = 1, \tag {8}
$$
where  $\Phi \in \mathbb{R}^{m\times m}$  is the Lagrange multiplier matrix,  $\mu \geq 0$  is a penalty parameter, and  $\langle \cdot ,\cdot \rangle$  returns the inner product of two matrices. We can optimize Eq. (8) by solving the following subproblems alternatively and iteratively.
1) W subproblem is formulated as
$$
\min  _ {\mathbf {W}} \| \mathbf {X} \mathbf {W} - \mathbf {F} \| _ {F} ^ {2} + \lambda \| \mathbf {W} \| _ {F} ^ {2}, \tag {9}
$$
Kernel Extension
The linear mapping in Eq. (9) may fail to model the nonlinear relationship. Therefore, we extend the above model to a kernel-based non-linear version. Let  $\phi(\cdot): \mathbb{R}^d \to \mathbb{R}^h$  denote the feature transformation that maps the origin feature space  $\mathbf{X}$  to a higher dimensional Hilbert space  $\phi(\mathbf{X})$ . According to the Representer Theorem,  $\mathbf{W}$  can be expressed as a linear combination of the input features, i.e.,  $\mathbf{W} = \phi(\mathbf{X})^\top \mathbf{H}$ , where  $\mathbf{H} \in \mathbb{R}^{m \times q}$  stores the combination weights. Then, we have  $\phi(\mathbf{X})\mathbf{W} = \mathbf{K}\mathbf{H}$ , where  $\mathbf{K} = \phi(\mathbf{X})\phi(\mathbf{X})^\top \in \mathbb{R}^{m \times m}$  is the kernel matrix and each element  $\mathbf{K}_{ij} = \mathcal{K}(x_i, x_j)$ . Finally, the nonlinear version is represented as:
$$
\min  _ {\mathbf {H}, \mathbf {b}} \left\| \mathbf {K} \mathbf {H} + \mathbf {1} _ {m} \mathbf {b} ^ {\top} - \mathbf {F} \right\| _ {F} ^ {2} + \lambda \operatorname {T r} \left(\mathbf {H} ^ {\top} \mathbf {K} \mathbf {H}\right), \tag {10}
$$
where  $\mathbf{b} \in \mathbb{R}^q$  is the bias term. When the first derivatives of  $\mathbf{H}$  and  $\mathbf{b}$  reach 0, Eq. (10) is solved, i.e.,
$$
\mathbf {H} = \left(\mathbf {K} + \lambda \mathbf {I} _ {m \times m} - \frac {\mathbf {1} _ {m} \mathbf {1} _ {m} ^ {\top} \mathbf {K}}{m}\right) ^ {- 1} \left(\mathbf {F} - \frac {\mathbf {1} _ {m} \mathbf {1} _ {m} ^ {\top} \mathbf {F}}{m}\right), \mathbf {b} = \frac {1}{m} \left(\mathbf {F} ^ {\top} \mathbf {1} _ {m} - \mathbf {H} ^ {\top} \mathbf {K} ^ {\top} \mathbf {1} _ {m}\right), \tag {11}
$$
where  $\mathbf{I}_{m\times m}\in \mathbb{R}^{m\times m}$  denotes an identity matrix. In the experiments, we use the RBF kernel as the kernel function, i.e.,  $\mathcal{K}(x_i,x_j) = \exp (-| |x_i - x_j||_2^2 /\sigma^2)$ , for our method and the compared ones.
2)  $\mathbf{F}$  subproblem is written as
$$
\left. \min  _ {\mathbf {F}} \| \mathbf {F} - \mathbf {P} \| _ {F} ^ {2} + \alpha \left\| \mathbf {A} \odot \mathbf {F} \mathbf {F} ^ {\top} \right\| _ {1} \right. \tag {12}
$$
$$
\begin{array}{l} \text {s . t .} \mathbf {F 1} _ {q} = \mathbf {1} _ {m}, \mathbf {0} _ {m \times q} \leq \mathbf {F} \leq \mathbf {Y}, \end{array}
$$
where  $\mathbf{P} = \mathbf{K}\mathbf{H} + \mathbf{1}_m\mathbf{b}^\top \in \mathbb{R}^{m\times q}$  is the output matrix of the model. Eq. (12) can be formulated as a standard quadratic programming (QP) problem, and solved by any QP tools. The detailed derivation process can be found in Section A of the supplementary file.
3) D subproblem is represented as
$$
\min  _ {\mathbf {D}} \beta \operatorname {T r} \left(\mathbf {D} \mathbf {L} \mathbf {D} ^ {\top}\right) + \frac {\mu}{2} \left\| \mathbf {D} - \mathbf {A} + \frac {\boldsymbol {\Phi}}{\mu} \right\| _ {F} ^ {2}. \tag {13}
$$
Eq. (13) reaches the minimum when its first-order derivative with respect to  $\mathbf{D}$  vanishes, leading to
$$
\mathbf {D} = (\mu \mathbf {A} - \boldsymbol {\Phi}) (2 \beta \mathbf {L} + \mu \mathbf {I} _ {m \times m}) ^ {- 1}. \tag {14}
$$
4) A subproblem is expressed as
$$
\min  _ {\mathbf {A}} \alpha \left\| \mathbf {A} \odot \mathbf {F F} ^ {\top} \right\| _ {1} + \frac {\mu}{2} \left\| \mathbf {D} - \mathbf {A} + \frac {\Phi}{\mu} \right\| _ {F} ^ {2} \tag {15}
$$
$$
\mathrm {s . t .} \mathbf {0} _ {m \times m} \leq \mathbf {A} \leq \mathbf {1} _ {m \times m}, \mathbf {A} _ {i j} = \mathbf {D} _ {0 i j}, \text {i f} \mathbf {D} _ {0 i j} = 1.
$$
Eq. (15) can solved element-wisely, i.e.,
$$
\mathbf {A} = \mathcal {T} \left(\mathcal {T} _ {1} \left(\mathcal {T} _ {0} \left(\frac {\mu \mathbf {D} + \boldsymbol {\Phi} - \alpha \mathbf {F} \mathbf {F} ^ {\top}}{\mu}\right)\right)\right), \tag {16}
$$
Algorithm 1 The Pseudo Code of the Proposed Method
Input:  $\mathcal{D}$ : the partial label training set;  $\lambda, \alpha, \beta, k$ : the parameters of model;  $\widehat{x}$ : an unseen test sample
Output:  $\widehat{y}$ : the predicted label for sample  $\widehat{x}$
1: Construct the dissimilarity matrix  $\mathbf{D}_0$  according to Eq. (2) and the kernel matrix  $\mathbf{K} = [\mathcal{K}(x_i,x_j)]_{m\times m}$
2: Initialize  $\mathbf{D} = \mathbf{A} = \boldsymbol{\Phi} = \mathbf{0}_{m\times m},\mu = 10^{-4}$
3: while not converged do
4: Update  $\mathbf{H}$  and  $\mathbf{b}$  by Eq. (11)
5: Update  $\mathbf{F}$  by solving Eq. (12)
6: Update  $\mathbf{D}$  by Eq. (14)
7: Update A by Eq. (16)
8: Update  $\Phi, \mu$  by Eq. (17)
9: Check the convergence condition  $\| \mathbf{D} - \mathbf{A}\|_{\infty} < 10^{-8}$
11: Return the predicted label  $\widehat{y}$  according to Eq. (18).
where  $\mathcal{T},\mathcal{T}_1,\mathcal{T}_0$  are three thresholding operators in elementwise, i.e.,  $\mathcal{T}(\mathbf{C}_{ij}) = 1$ , if  $\mathbf{D}_{0ij} = 1$ ,  $\mathcal{T}_1(\mathbf{C}_{ij})\coloneqq \min (1,\mathbf{C}_{ij}),\mathcal{T}_0(\mathbf{C}_{ij})\coloneqq \max (0,\mathbf{C}_{ij})$ .
Finally, the Lagrangian multiplier matrix and  $\mu$  are updated by
$$
\left\{ \begin{array}{l} \boldsymbol {\Phi} \leftarrow \boldsymbol {\Phi} + \mu (\mathbf {D} - \mathbf {A}) \\ \mu \leftarrow \min  (1. 1 \mu , \mu_ {\max }, \end{array} \right. \tag {17}
$$
where  $\mu_{\mathrm{max}} = 10^{10}$  is a predefined upper bound for  $\mu$
Model Prediction
Given an unseen test example  $\widehat{x}$ , our method predicts its label by
$$
\widehat {y} = \arg \max  _ {k} \sum_ {i = 1} ^ {m} \mathbf {H} _ {i k} \mathcal {K} (\widehat {x}, x _ {i}) + \mathbf {b} _ {k}. \tag {18}
$$
The overall pseudo code of our method is summarized in Algorithm 1, where it stops when  $\| \mathbf{D} - \mathbf{A}\|_{\infty} < 10^{-8}$ , where  $\| \cdot \|_{\infty}$  denotes the infinity norm of a matrix.
Hyper-parameter Settings of Our Method
Parameter  $\lambda$  is used to control the model complexity. A too large (resp. small) value of  $\lambda$  will lead to under-fitting (resp. over-fitting). Therefore, we set  $\lambda = 0.05$  for our method. Parameters  $\alpha$  and  $\beta$  are used to control the importance of the adversarial term and dissimilarity propagation term respectively.
According to a number of experiments, we fix  $\beta = 0.001$  and select  $\alpha$  from  $\{0.001, 0.01\}$ . Parameter  $k$  controls the number of  $k$ -nearest neighbors. Following the previous works [16, 22], we set  $k = 10$ .
Computational Complexity Analysis
The computational complexity of Algorithm 1 is mainly determined by steps 4-7. Specifically, steps 4 and 6 involve the inversion of  $m \times m$  matrices with the complexity of  $\mathcal{O}(m^3)$ . Step 5 solves a QP problem, leading to the complexity of  $\mathcal{O}(m^3 q^3)$ . Step 7 can be efficiently solved by linear thresholding operations with the complexity of  $\mathcal{O}(m^2)$ . Therefore, the overall computational complexity of Algorithm 1 in each iteration is  $\mathcal{O}(2m^3 + m^2 + m^3 q^3)$ . More analysis about computational complexity can be found in Section A of the supplementary file.