Title: Optimal Block-wise Asymmetric Graph Construction for Graph-based Semi-supervised Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph-based Semi-supervised Learning

Affinity Graph Construction
Abstract: Graph-based semi-supervised learning (GSSL) serves as a powerful tool to model the underlying manifold structures of samples in high-dimensional spaces. It involves two phases: constructing an affinity graph from available data and inferring labels for unlabeled nodes on this graph. While numerous algorithms have been developed for label inference, the crucial graph construction phase has received comparatively less attention, despite its significant influence on the subsequent phase. In this paper, we present an optimal asymmetric graph structure for the label inference phase with theoretical motivations. Unlike existing graph construction methods, we differentiate the distinct roles that labeled nodes and unlabeled nodes could play. Accordingly, we design an efficient block-wise graph learning algorithm with a global convergence guarantee. Other benefits induced by our method, such as enhanced robustness to noisy node features, are explored as well. Finally, we perform extensive experiments on synthetic and real-world datasets to demonstrate its superiority to the state-of-the-art graph construction methods in GSSL.

Optimal Block-wise Asymmetric Graph Construction for Graph-based Semi-supervised Learning

Abstract

Graph-based semi-supervised learning (GSSL) serves as a powerful tool to model the underlying manifold structures of samples in high-dimensional spaces. It involves two phases: constructing an affinity graph from available data and inferring labels for unlabeled nodes on this graph. While numerous algorithms have been developed for label inference, the crucial graph construction phase has received comparatively less attention, despite its significant influence on the subsequent phase. In this paper, we present an optimal asymmetric graph structure for the label inference phase with theoretical motivations. Unlike existing graph construction methods, we differentiate the distinct roles that labeled nodes and unlabeled nodes could play. Accordingly, we design an efficient block-wise graph learning algorithm with a global convergence guarantee. Other benefits induced by our method, such as enhanced robustness to noisy node features, are explored as well. Finally, we perform extensive experiments on synthetic and real-world datasets to demonstrate its superiority to the state-of-the-art graph construction methods in GSSL.

1 Introduction

Graph-based semi-supervised learning (GSSL) is a burgeoning research field [52, 9, 14, 63, 65, 62]. As a subclass of semi-supervised learning (SSL), GSSL exhibits promise since it encapsulates the smoothness or manifold assumption, where samples with similar features are likely to share the same label. GSSL methods begin by constructing an affinity graph, wherein nodes represent samples, and weighted edges denote similarity between pairs of nodes. This process aligns with the manifold assumption, implying that nodes connected by edges with large weights tend to have the same label. Upon obtaining the affinity graph, various label inference algorithms such as label propagation [85, 82, 25, 65] can be executed to predict labels for the unlabeled nodes.

Preliminary empirical studies [16] suggest that the quality of the affinity graph significantly influences label prediction performance. However, the graph construction phase in GSSL has received less scrutiny compared to the subsequent label inference phase. Constructing a high-quality graph presents a challenge as its quality can only be assessed indirectly through postmortem verification via label inference performance. Classic solutions such as the Radial Basis Function (RBF) Kernel [85] , kNN graph [17], and b -matching [26], while simple, may exhibit low robustness against noise due to their simplicity. More recent and complex methods, typically framed as optimization problems to find

the optimal graph under the smoothness assumption, suffer from inefficient optimization techniques without fast convergence guarantees [18, 29, 63] or lack a solid theoretical foundation [58, 14, 23].

Consequently, a series of research questions arise: 1) What is the optimal graph structure for label inference algorithms in GSSL? 2) How can we optimize this optimal graph efficiently? 3) What kinds of benefits can this optimal graph bring?

Most existing GSSL graph construction methods treat all nodes equally, disregarding label information, which results in a symmetric, undirected graph. However, we contend that an asymmetric, directed graph structure might be more beneficial for subsequent label inference due to the distinct roles labeled and unlabeled nodes can play. Intuitively, edges from labeled to unlabeled nodes would naturally facilitate supervision information propagation, while edges from unlabeled to labeled nodes may introduce inconsistency, potentially undermining the prediction accuracy for the labeled nodes.

Motivated by this key observation, we revisit a unified optimization framework that encompasses most label inference algorithms, assuming the affinity graph is already constructed. We then fix the downstream label inference result and position the graph weight matrix (or adjacency matrix) as the optimization variable. This shift allows us to define optimality in the graph construction step of GSSL concisely, addressing our first research question. We subsequently present a tailored optimal solution featuring an asymmetric graph structure that aligns with our proposed intuition. In response, we introduce the B lock-wise A ffinity G raph L earning (BAGL) algorithm by leveraging duality and the fast proximal gradient method, addressing our second research question. Finally, we demonstrate that BAGL ensures a sub-linear global convergence rate of O (1 /k ) and can alleviate issues of noisy node features, addressing our third research question.

In summary, this work offers four main contributions. First, we provide a succinct definition of the optimality of the affinity graph in GSSL, and through rigorous derivation, propose an ad-hoc solution with an asymmetric structure. Second, we design a block-wise graph learning framework, BAGL, to infer the weights in the optimal graph structure. Third, we prove that a global sub-linear convergence rate is guaranteed for BAGL and analyze other benefits. Fourth, we perform extensive experiments on synthetic and real-world datasets to demonstrate the effectiveness and efficiency of BAGL.

2 Preliminary

2.1 Problem Formulation

We present a formulation for the GSSL problem, comprising two steps: graph construction and label inference [52]. Our study primarily investigates the optimal construction of the affinity graph in the first phase to facilitate enhanced performance in the second label inference phase.

Given a set of data points { x i } n i =1 , where each x i ∈ R d is sampled from a d dimensional feature space. In this paper, we interchangeably use the terms node , point , and sample to refer to x i . Each sample x i has a label y i ∈ N c , where N c = { i ∈ N + | 1 ≤ i ≤ c } with c being the number of classes. Given the labels of the l samples { x i } l i =1 as { y i } l i =1 , the ultimate goal of general transductive SSL is to infer the labels { y i } l + u i = l +1 for the remaining u unlabeled samples { x i } l + u i = l +1 ( n = l + u ). As a subcategory of general SSL, GSSL methods first construct a graph G = {V , E , W } based on all the training samples { x i } n i =1 and partially given labels { y i } l i =1 . Here, V is the node set with |V| = n . Each node represents each sample x i . E is the edge set in which each edge ( i, j ) is assigned with a weight W ij (the i -th row, j -th column entry in W ∈ R n × n ) to reflect the affinity or similarity between the sample pair ( x i , x j ) . Generally, a larger weight indicates a higher level of similarity. W ij = 0 indicates no edge between node i and node j . Therefore, the key challenge in the first graph construction step is to generate W based on { x i } n i =1 and { y i } l i =1 so that the underlying manifold of the data is properly encoded. In the second step, various label inference algorithms can be performed on G to propagate the given labels { y i } l i =1 and make predictions { y i } l + u i = l +1 for unlabeled nodes.

However, the performance of the label inference step is significantly contingent on the quality of the weight matrix W from the first graph construction step. This paper aims to address three critical research questions pertaining to the graph construction step in GSSL. First, what constitutes the overall structure of the optimal weight matrix, W ∗ ? (Sect. 3.1). We define the optimal weight matrix, W ∗ , as the one that gives the best prediction results when used with the same label inference method

across different graphs W . Second, how to find an efficient method for optimizing the entries in W ∗ ? (Sect. 3.2). Third, what kinds of benefits can W ∗ bring from the theoretical perspective? (Sect. 3.3)

2.2 Recap on the Unified Framework for Label Inference Step in GSSL

Before we delve into the proposed structure for the optimal affinity graph, we first revisit the unified framework of the label inference step, allowing for a seamless definition of the optimal affinity graph.

If we assume the affinity graph has been constructed, then numerous influential label inference methods [85, 82, 83, 4, 5, 9] can be performed over this graph to infer the labels for the unlabeled nodes. However, the majority of them can be incorporated into the following framework.

We first define the node feature matrix X ∈ R n × d as X = [ x 1 , · · · , x n ] ⊺ and the predicted soft label matrix F ∈ R n × c as F = [ f 1 , · · · , f n ] ⊺ with f i ∈ R c (1 ≤ i ≤ n ) . Ground-truth label matrix is given as Y = [ y 1 , · · · , y n ] ⊺ ∈ { 0 , 1 } n × c with y i ∈ { 0 , 1 } c (1 ≤ i ≤ n ) . Here, for the labeled nodes { x i } l i =1 , y i is a one-hot vector in which Y ij = 1 if x i belongs to class j ( y i = j ), and Y ij = 0 , otherwise. For the unlabeled nodes { x i } l + u i = l +1 , y i is an all-zero vector for initialization.

Driven by the geometry of the affinity graph, we can unify these label inference algorithms as a minimizer of the optimization problem as Problem (1).

Note that the loss function Q ( F ) consists of a quadratic variation term Tr( F ⊺ SF ) as the graph smoothness regularizer, and a quadratic Frobenius error norm ∥ F -Y ∥ 2 F = Tr(( F -Y ) ⊺ ( F -Y )) , both of which should ideally be small subject to a trade-off parameter Λ between them. Here the smoothing matrix S = s ( W ) ∈ S n + in the first term, with s : R n × n → S n × n + , is positive semidefinite and determined by the weight matrix W of the graph to ensure the adjacent nodes share similar predictions. The second term measures the distance between predicted results and initial assignments, and restricts the output for labeled nodes from deviating too much compared with the ground-truth. Λ is a diagonal matrix with Λ ii ≥ 0 and S + Λ must be invertible to avoid trivial solutions.

By the first-order optimality condition, we can easily obtain the optimal solution for Problem (1).

Based on the optimal soft label matrix F ∗ , the final predicted label for each node is given as ˆ y i = ˆ y ( f i ) = arg max 1 ≤ j ≤ c F ∗ ij . It is worth noting that the unified optimization framework in Problem (1) can admit most of the mainstream GSSL methods. For instance, if we set the smoothing matrix as the normalized graph Laplacian matrix S = L = s ( W ) = D -1 2 ( D -W ) D -1 2 and Λ = λ I , we can easily recover one of the most popular label inference methods for GSSL,

λ ∑ n i =1 ∥ f i -y i ∥ 2 2 } . Here, L ∈ S n + is defined as L = D -1 2 LD -1 2 , where the combinatorial Laplacian matrix L ∈ S n + is given as L = D -W , and the degree matrix D ∈ R n × n is a diagonal matrix defined as D ii = ∑ n j =1 W ij . We summarize several representative works under this unified framework in Table 5 with explicit forms of S (or s ( W ) ) and Λ in Appendix B.2.

3 Methodology

3.1 Motivation: Optimal Affinity Graph Structure

3.1.1 Definition of the Optimality of the Affinity Graph

If we perform the same label inference algorithm from the above-mentioned generalized framework on all possible affinity graphs W , the optimal graph W ∗ will enable the label inference step to obtain the most accurate predictions. Under the above-mentioned unified framework for label inference (Problem (1)), the weight matrix W is fixed while the soft label matrix F is the optimization variable. This is due to our goal of executing label inference to attain the optimal F ∗ given the affinity graph W . Similarly, when we want to construct the optimal graph W ∗ given the label inference framework,

we consider the weight matrix W as the optimization variable, keeping the soft label matrix F fixed as the solution in Eq. (2). The form of Q ( · ) remains unchanged as it is related to the generalization bound for GSSL that would be discussed in Appendix D.2. This approach aids in identifying the 'optimal' affinity graph for the label inference algorithms under the unified framework.

Definition 1 (Optimality of the Affinity Graph) . The affinity graph G is optimal for label inference under the unified GSSL framework if its weight matrix W is the minimizer of Problem (3).

Although the optimization Problem (3) is intractable in general due to the various forms of s ( W ) in the constraint, we can present an ad-hoc optimal structure of W ∗ that is independent of s ( W ) , which motivates our proposed method in Sect. 3.2 from a theoretical perspective.

3.1.2 Structure of the Optimal Affinity Graph

We then present an equivalent proposition in Theorem 1, which provides a necessary and sufficient condition for the optimality of W ∗ in Problem (3). Motivated by some classic graph sharpening techniques [47, 15, 46], Theorem 1 helps to circumvent the challenges of directly dealing with Problem (3) and holds regardless of the forms of s ( W ) (Appendix E.1).

Theorem 1. W ∗ obtained by Definition 1 is optimal if and only if Y l = F l holds. Y l ∈ { 0 , 1 } l × c , F l ∈ R l × c are the ground-truth label matrix, and the soft label matrix for labeled nodes.

To put it in a simpler way, Theorem 1 tells us that if the affinity graph is optimal, then after we perform the label inference algorithm for GSSL, the predicted soft label for the labeled nodes would coincide with the ground truth precisely. The converse of this observation also holds true. Unfortunately, it remains an open question to solve Y l = F l by listing all possible classes of solutions due to the complex interconnection of F and W . However, we can provide a simple ad-hoc solution for Y l = F l in Proposition 1. For one thing, this asymmetric graph structure, in the theoretical sense, conforms to the intuition of the better affinity graph we discussed earlier. For another, it also sheds some light on the proposed optimization framework to infer the weights in this graph structure later.

Proposition 1. If W ∗ can be expressed as (4), W ∗ is an optimal solution given by (3) in Definition 1.

where W ul ∈ R u × l , and W uu ∈ R u × u can be non-zero submatrices with arbitrary entries.

Proposition 1 provides an ad-hoc solution to Problem (3) (Appendix E.2). If the constructed graph is asymmetrical with edges from unlabeled nodes to labeled nodes only, then it is optimal by Definition 1. This asymmetrical optimal graph structure answers the first research question in Sect. 2.1.

The optimal asymmetric graph structure presented in Proposition 1 offers meaningful interpretations and notable advantages. It effectively eliminates the influence from unlabeled nodes to labeled nodes by enforcing W ll and W lu to be zero matrices. This aligns with the intuition that label information should be propagated from labeled nodes to unlabeled nodes in the GSSL methods, rather than the other way around. More technically, when applying the classic label inference algorithm LGC [82] on this optimal graph structure, the soft label matrix for unlabeled nodes now becomes F u = ( I -µ W uu ) -1 W ul Y L with some constant 0 ≤ µ ≤ 1 . This formulation indicates that the LGC algorithm spreads supervision information from labeled nodes to unlabeled nodes once through W ul Y l , followed by propagating this information solely among unlabeled nodes through ( I -µ W uu ) -1 = I + µ W uu + µ 2 W 2 uu + · · · . By Theorem 1, this optimal asymmetric graph guarantees zero empirical risk on the labeled nodes. Moreover, many existing graph construction methods in GSSL primarily focus on node features while disregarding label information. Consequently, these methods may produce heterophilous edges that connect nodes with similar features but different labels. Such edges violate the manifold assumption in GSSL, where nodes with the same label tend to be linked. During the label inference step, the label information of these nodes connected by heterophilous edges confuses each other during propagation, resulting in misleading predictions. By setting W ll = O , our method eliminates these heterophilous edges completely. As a result, it increases the edge homophily ratio of the constructed graph and enhances the robustness of subsequent label inference algorithms, as validated in Appendix D.1.

3.2 Implementation: Block-wise Graph Learning Algorithm

3.2.1 Framework

By differentiating the roles that labeled and unlabeled nodes would play, we arrive at an optimal structure for the affinity graph in Proposition 1. However, an efficient algorithm to infer the exact weights or entries for blocks in Eq. (4) is still in demand. Motivated by previous works [29, 30, 19], we make the following reasonable restrictions or assumptions on W to obtain a more meaningful graph for GSSL. First, the features for two directly connected nodes should not vary too much, no matter whether they are labeled or not. This agrees with the manifold assumption in GSSL, where links tend to form between similar nodes. Violation of this fundamental assumption usually causes a significant performance drop in the label inference step. Second, the constructed graph should be well-connected in terms of both W ul and W uu to ensure the supervision information could propagate freely among all nodes. Otherwise, some disconnected nodes may never receive any label information. Third, it is desirable to control the sparsity of the graph, avoiding an overly sparse or overly dense graph. We will empirically demonstrate the effects of the sparsity control later. We propose two similar optimization frameworks for W ul and W uu in Eq. (4) as Problem (5) and (6).

⊙ is the Hadamard product and log( · ) is the element-wise logarithm operator. Since Problem (5) and (6) share the same form, we will only focus on Problem (5) as an example. The first term ∥ W ul ⊙ Z ul ∥ 1 = ∑ 1 ≤ i ≤ u, 1 ≤ j ≤ l W ij ∥ x i -x j ∥ 2 2 encourages similar nodes to be connected with larger weights, meeting the manifold assumption. The second and third logarithmic barrier term act on the out-degree and in-degree vectors to make sure that each unlabeled node is connected by at least one labeled node and vice versa, improving the overall connectivity of the graph. The last Frobenius norm term measures the sparsity of the graph. The parameters α 1 , α 2 , β 1 , β 2 are positive.

3.2.2 Optimization

For convenience of presentation, we view the entries in W ul ( W uu ) as a new vector w = vec[ W ul ] ∈ R ul + . Similarly, we have z = vec[ Z ul ] ∈ R ul + . Accordingly, a linear mapping matrix T 1 ∈ { 0 , 1 } u × ul transforms the edge weights to the corresponding out-degree vector (i.e. T 1 w = W ul 1 ). Similarly, we have T 2 w = W ⊺ ul 1 for the in-degree vector. Further, if we let T ⊺ = ( T ⊺ 1 , T ⊺ 2 ) ∈ { 0 , 1 } n × ul , α = α 1 = α 2 , and β = β 1 we can easily transform Problem (5) into Problem (7) (primal) in a more compact way with an extra linear constraint.

The state-of-the-art method [61] applies the linearized alternating direction method of multipliers (ADMM) algorithm [8] directly to the primal problem with a similar structure in Problem (7), which lacks the theoretical guarantee on its convergence rate since the objective function in Problem (7) has no Lipschitz gradient. Motivated by the recent work [45], we circumvent this critical issue by applying the FISTA algorithm [2], a proximal gradient method, to the dual problem of (7) instead. Motivated by recent advances [3, 61], we can now provide a better convergence rate with rigorous theoretical analysis so that our proposed graph construction method is much more efficient with guarantees.

Dual Problem Formation We construct the Lagrangian function by introducing the Lagrangian multipliers λ ∈ R n as L ( w , v , λ ) = f ( w ) + g ( v ) -⟨ λ , Tw -v ⟩ . We establish the corresponding dual problem as Problem (8) by introducing the conjugate functions f ∗ , g ∗ for simpler notation (Appendix E.3).

with F ( λ ) = f ∗ ( T ⊺ λ ) , G ( λ ) = g ∗ ( -λ ) . By Slater's condition, we know that strong duality holds as long as v resides in the range of T . Therefore, the optimal values for Problem (7) and (8) are identical, and the optimal solution for Problem (8) can be attained. Consequently, we can apply the FISTA algorithm to the dual problem to generate the dual sequence that converges to the optimal solution, and construct the corresponding optimal solution back for the primal problem.

Dual Problem with FISTA It is not hard to show that F ( λ ) is differentiable and ∇ F ( λ ) = Tx ∗ , where x ∗ = arg max x { ( T ⊺ λ ) ⊺ x -f ( x ) } (Appendix E.4). Therefore, the dual problem minimizes the sum of a differentiable convex function F and a closed proper convex function G . This structure of the dual Problem (8) immediately paves the way for applying proximal gradient methods. Here, for the sake of a better convergence rate, we apply the FISTA algorithm with a fixed step size to the dual Problem (8), and the following iteration schemes are performed. We first choose any λ 0 = λ -1 and fix the step size as t = 2 β l + u . Henceforth, k = 1 , 2 , · · · , we repeat the following two steps as

Hence, based on the above-mentioned properties of ∇ F ( µ k ) , we have ∇ F ( µ k ) = T ¯ w k with ¯ w k set as ¯ w k = arg max w { ( T ⊺ µ k ) ⊺ w -f ( w ) } = [ T ⊺ µ k -z 2 β ] + (Appendx E.5).

Let p k = µ k -t T ¯ w k . By the extended Moreau decomposition [43], prox γh ( z ) + γ prox γ -1 h ∗ ( z/γ ) = z, ∀ z . Wehave prox tG ( p k ) = p k -t prox t -1 G ∗ ( t -1 p k ) = µ k -t ( T ¯ w k -¯ u k ) . Here, ¯ u k = prox t -1 g ( T ¯ w k -t -1 µ k ) . Note that g ( v ) = -α 1 ⊺ log( v ) and by the definition of proximal mapping, it is easy to prove that prox t -1 g ( v ) = 1 2 ( v + √ v ⊙ v +4 αt -1 1 ) . Therefore, we can simplify the updating step (10) as the following three steps (Appendix E.6).

Finally, we arrive at Procedure GWBI , where the optimal graph weights are inferred for one block like W uu in the optimal graph structure suggested in Proposition 1. Here, instead of directly optimizing the primal variable of the block graph weight vector w , we consider its corresponding dual variable λ for a better convergence rate. In each iteration step, we first find an extrapolated point µ based on the points λ from two previous steps (line 3). We then perform the proximal gradient update on this extrapolated point (lines 4-6) to obtain λ for the next iteration. Note that lines 4-6 are the detailed instantiation of Eq. (10). Finally, we can convert the dual variable λ back to the desired primal variable w based on line 4 after its convergence. With this core procedure in hand, we plug it into the proposed Algorithm 1, B lock-wise A ffinity G raph L earning (BAGL) algorithm, in Appendix C.

3.3 Theoretical Analysis

̸

We prove that our proposed optimization method for Problem (7) in Procedure GWBI enjoys the guarantee of the global convergence rate, where the generated primal sequence converges to the global optimal solution at a rate of O ( 1 k ) with a fixed step size. To begin with, it is well known that the FISTA algorithm enjoys the global convergence rate of O ( 1 k 2 ) [2]. For simplicity, we focus on the results when optimizing Problem (5), which can be viewed as a general case of Problem (6) when l = u . Therefore, Q ( λ ) ≡ F ( λ ) + G ( λ ) converges to the dual optimal value Q ( λ ∗ ) at a rate of O ( 1 k 2 ) due to this well-known fact [2], which yields Theorem 2.

Theorem 2. Let { λ k } be the dual sequence generated by Procedure GWBI , then

Procedure GraphWeightBlockInference( z , T , α , β , t , ϵ )

Input: Distance vector z , linear mapping matrix T , balancing parameters α and β , step size t , error tolerance parameter ϵ .

Output: Block graph weight vector ˆ w

We then consider the corresponding primal sequence { w k } and its convergence rate. Since the strong duality holds and a dual optimal solution λ ∗ exists, any primal optimal point ( w ∗ , v ∗ ) is also a minimizer of L ( w , v , λ ∗ ) . This motivates us to construct the primal sequence { w k } based on the dual sequence { λ k } as w k = arg min w L ( w , v , λ k ) = arg max w {⟨ T ⊺ λ k , w ⟩ -f ( w ) } . Thanks to Theorem 3, this primal sequence { w k } is guaranteed to converge to the optimal primal solution w ∗ of Problem (7) at the rate of O ( 1 k ) .

Theorem 3. Let w k = arg max w {⟨ T ⊺ λ k , w ⟩ -f ( w ) } with the dual sequence { λ k } given by Procedure GWBI . w ∗ and λ ∗ are the optimal solution of Problem (7) and the optimal solution of Problem (8), respectively. We have,

Motivated by [45], Theorem 3 establishes that the proposed method exhibits a sub-linear convergence rate, which represents a state-of-the-art result for optimization-based graph construction methods, accompanied by a global convergence guarantee. This improved convergence rate is primarily attributed to the utilization of the FISTA algorithm applied to the dual problem. Time complexity analysis is included in Appendix G.5. More results regarding the robustness of our method are discussed in Appendix D.

4 Experiments

In this section, we conduct numerical experiments on both synthetic and real-world datasets to demonstrate the advantages of our proposed BAGL method in terms of efficacy and convergence. Robustness analysis (Appendix G.3) and more experimental results are included in Appendix G.

4.1 Baseline Models

We choose the following graph construction methods in GSSL for comparison. Radial basis function kernel (RBF) [85] and kNN graph [17] are two classic methods. Smooth graph learning (SGL) [29] is a popular method in the graph signal processing domain. RGCLI [7] is another label-informed graph construction method. Anchor Graph Regularization (AGR) [40] deals with large-scale graph construction. GraphEBM [14] and BCAN [63] are two state-of-the-art methods. The former exploits the energy-based model while the latter constructs a bipartite graph.

All the hyper-parameters are fine-tuned with the grid search method. We repeat the experiment 20 times for each case and report the average result with optimal parameter setting in the efficacy analysis. Unless otherwise specified, the default label inference algorithm is LGC, and the label rate is ten labeled samples per class. More details on the experimental settings can be found in Appendix F.

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

(a) Synthetic dataset

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

(b) RBF

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

(c) BAGL (Ours)

4.2 Synthetic Dataset

We generate a synthetic dataset as shown in Fig. 1 (a). The constructed dataset contains two clusters, a dense Gaussian cluster surrounded by a sparse ring-like cluster. With only one labeled sample given in each cluster, we compare the result of our proposed BAGL method (Fig. 1 (c)) with the result of the most popular method RBF (Fig. 1 (b)). We use the coordinates as the node feature and set the width in the RBF kernel as 0 . 7 . For visualization purposes, we show the adjacency matrix with yellow line segments connecting the node pairs if the weight associated with the edge is greater than 0.5 after normalization. The direction of the edge is ignored in Fig. 1 (c). For a fair comparison, we perform label propagation [85] on both constructed affinity graphs. We can see that BAGL can recover two ground-truth clusters much better. Unlike RBF, BAGL can improve the connectivity by the logarithm penalty term and reduce the inter-cluster links by the block-wise design.

4.3 Real-world Datasets

Classification tasks are implemented to assess the performance of BAGL against all graph construction baseline methods on six real-world datasets, listed in Table 1. ORHD (Optical Recognition of Handwritten Digits Data Set), USPS , MNIST , and EMNIST Letters are four popular digits image datasets. COIL100 is an object image dataset. TDT2 is a text dataset. We fix the number of anchor nodes as 1000 in four datasets (COIL100, USPS, ORHD and TDT2), while for the rest two datasets (MNIST, EMNIST-Letters), the number of anchors is fixed as 2000. We perform tf-idf and principal component analysis (PCA) as the pre-processing step on TDT2 dataset. The default label inference algorithm is LGC, with ten labels per class. Further details can be found in Appendix F.1.

4.3.1 Efficacy

We fix the number of labeled samples per class to ten and select three label inference methods for the second phase of GSSL. We report average results by performing 20 trials for each algorithm over all the settings in Table 2. Our algorithm outperforms all methods in the USPS, TDT2, and EMNISTLetters datasets, indicating that BAGL can learn an optimal graph for label inference algorithms in the unified framework. Moreover, we perform the Friedman test with the Bonferroni-Dunn post hoc test for statistical significance analysis. Fig. 2 illustrates the critical difference (CD) diagram on the accuracy, where the average rank is marked along the axis with lower (better) ranks to the left. If the average rank difference between two models is greater than one CD, the relative performance is believed to be different. Accordingly, BAGL significantly outperforms all other baselines by a large margin. We also conduct experiments under low label rates with LGC fixed as the label inference method. Fig. 3 (a) and (b) demonstrate that BAGL performs relatively well with low label rates. This phenomenon can be attributed to the utilization of label information in BAGL. (Appendix G.1)

accuracy

8

7

6

5

4

3

2

1

BAGL

GraphEBM

AGR

BCAN

RBF

SGL

RGCLI

1

2

3

4

5

6

7

8

9

10

Number of labeled samples per class

50

60

70

80

90

100

Accuracy (%)

RBF

SGL

RGCLI

AGR

GraphEBM

BCAN

BAGL

(a) ORHD

1

2

3

4

5

6

7

8

9

10

Number of labeled samples per class

50

60

70

80

90

100

Accuracy (%)

RBF

SGL

RGCLI

AGR

GraphEBM

BCAN

BAGL

(b) USPS

0

200

400

600

800

Number of iterations

10 -15

10 -10

10 -5

10 0

10 5

SGL

SGL w/ ADMM

BAGL

(c) COIL100

0

200

400

600

800

Number of iterations

10 -15

10 -10

10 -5

10 0

10 5

SGL

SGL w/ ADMM

BAGL

(d) TDT2

4.3.2 Convergence

As an essential part of the overhead for BAGL, Procedure GWBI needs to be efficient for practice. Compared with two other optimization-based methods sharing a similar objective, SGL and its accelerated version by ADMM [61], BAGL enjoys the fastest convergence rate on both datasets. We sample 1% nodes in each dataset for the convenience of presentation in Fig. 3 (c) and (d). Its outstanding performance confirms the theoretical analysis in Theorem 3. (Appendix G.2).

4.3.3 Ablation Study

To obtain a better understanding of why the proposed BAGL works, we perform some ablation studies to empirically show how the key design of BAGL will potentially affect performance. We create three variants based on the original version of BAGL. First, to demonstrate the significance of the optimal asymmetric structure, we now do not differentiate labeled nodes and unlabeled nodes, and let any graph structure be the potential optimal structure without the constraints of W ul = O and W uu = O . We call this variant BAGL w/o optimal structure . Second, to reveal the importance of the connectivity regularization term, we set α = 0 in Procedure GWBI to remove the connectivity consideration. This variant is termed BAGL w/o connectivity . Third, to investigate the effects of sparsity control, we set β = 10 e -5 ≈ 0 in Procedure GWBI to allow the sparsity of the constructed graph to vary arbitrarily. The last variant of BAGL is abbreviated as BAGL w/o sparsity . We conduct the experiments on the ORHD dataset with the same setting as Table 2 and report the classification accuracy results in Table 3. The proposed optimal asymmetric graph structure contributes most to the success of BAGL. The connectivity regularization term and the sparsity control term also matter since they together encourage a more sparse graph (the latter) but without disconnected components (the former), which is a more favorable graph for the second label inference step.

5 Conclusion

In this paper, we propose a novel approach to graph construction for graph-based semi-supervised learning. Building upon the optimal asymmetric graph structure derived from theoretical insights, we develop an efficient block-wise graph construction method that guarantees faster convergence. Our approach combines theoretical insights with practical considerations to provide a more effective and reliable framework for the graph construction step in graph-based semi-supervised learning.

6 Limitations

BAGL is an optimization-based method for the graph construction step in graph-based semisupervised learning. Graph Neural Networks (GNNs) excel in learning representations for graphstructured data [64, 54, 12, 49, 39, 78, 41, 75, 13, 53, 77, 51, 73, 52, 50, 38, 76, 11]. More recent graph structure learning methods aim to learn a clean graph structure from the given noisy graph so that the subsequent GNNs trained on this learned clean graph can obtain better performance. In GSSL, however, there is no given graph structure, and we need to learn the graph structure based on the node features only. Therefore, it is a more challenging task compared to graph structure learning. Therefore, we do not compare our method with other graph structure learning methods since their settings and goals are slightly different. We leave the investigation of graph structure learning for GSSL as future work since it is currently out of the scope of this work.

The other limitation of BAGL is it is only suitable for the transductive setting. If we have nodes or samples unseen in the training set, we have to construct the affinity graph again by executing BAGL again to infer their labels, which is often time-consuming and troublesome regarding efficiency. This is not desirable in real-world applications since we often come across new training samples after we build the affinity graph. We also leave the investigation of the inductive extension of BAGL as future work since this lack of inductive generalization is a well-known challenge in graph-based semi-supervised learning.

Even though BAGL is quite efficient in terms of convergence rate, it may still have computational issues when dealing with extremely large-scale datasets with billions of samples because the time spent on finishing one iteration during the optimization would increase dramatically when the number of training samples is extremely large. We leave the exploration of graph construction methods for extremely large-scale datasets as future work. Other potential applications of our method can be explored in the hyperbolic space [67, 70, 68, 71, 69, 66, 72] or in the natural language processing domain [32, 34, 33, 21, 20, 55, 42, 80, 81, 79].

References

Ahmed El Alaoui, Xiang Cheng, Aaditya Ramdas, Martin J. Wainwright, and Michael I. Jordan. Asymptotic behavior of ℓ p -based laplacian regularization in semi-supervised learning. CoRR , abs/1603.00564, 2016.

Amir Beck and Marc Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. , 2(1):183-202, 2009.

Amir Beck and Marc Teboulle. A fast dual proximal gradient algorithm for convex minimization and applications. Operations Research Letters , 42(1):1-6, 2014.

Mikhail Belkin, Irina Matveeva, and Partha Niyogi. Tikhonov regularization and semisupervised learning on large graphs. In ICASSP (3) , pages 1000-1003. IEEE, 2004.

Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. J. Mach. Learn. Res. , 7:23992434, 2006.

Lilian Berton and Alneu de Andrade Lopes. Graph construction based on labeled instances for semi-supervised learning. In ICPR , pages 2477-2482. IEEE Computer Society, 2014.

Lilian Berton, Thiago de Paulo Faleiros, Alan Valejo, Jorge Carlos Valverde-Rebaza, and Alneu de Andrade Lopes. RGCLI: robust graph that considers labeled instances for semi-supervised learning. Neurocomputing , 226:238-248, 2017.

Stephen P. Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. , 3(1):1-122, 2011.

Jeff Calder, Brendan Cook, Matthew Thorpe, and Dejan Slepcev. Poisson learning: Graph based semi-supervised learning at very low label rates. In ICML , volume 119 of Proceedings of Machine Learning Research , pages 1306-1316. PMLR, 2020.

Hongxu Chen, Hongzhi Yin, Tong Chen, Quoc Viet Hung Nguyen, Wen-Chih Peng, and Xue Li. Exploiting centrality information with graph convolutions for network representation learning. In ICDE , pages 590-601. IEEE, 2019.

Yankai Chen, Yixiang Fang, Yifei Zhang, and Irwin King. Bipartite graph convolutional hashing for effective and efficient top-n search in hamming space. arXiv preprint arXiv:2304.00241 , 2023.

Yankai Chen, Yifei Zhang, Menglin Yang, Zixing Song, Chen Ma, and Irwin King. WSFE: wasserstein sub-graph feature encoder for effective user segmentation in collaborative filtering. CoRR , abs/2305.04410, 2023.

Yankai Chen, Yifei Zhang, Menglin Yang, Zixing Song, Chen Ma, and Irwin King. WSFE: wasserstein sub-graph feature encoder for effective user segmentation in collaborative filtering. In SIGIR , pages 2521-2525. ACM, 2023.

Zhijie Chen, Hongtai Cao, and Kevin Chen-Chuan Chang. Graphebm: Energy-based graph construction for semi-supervised learning. In ICDM , pages 62-71. IEEE, 2020.

Inae Choi and Hyunjung Shin. Semi-supervised learning with ensemble learning and graph sharpening. In IDEAL , volume 5326 of Lecture Notes in Computer Science , pages 172-179. Springer, 2008.

Celso André R. de Sousa, Solange O. Rezende, and Gustavo E. A. P. A. Batista. Influence of graph construction on semi-supervised learning. In ECML/PKDD (3) , volume 8190 of Lecture Notes in Computer Science , pages 160-175. Springer, 2013.

Cheng-Hao Deng and Wan-Lei Zhao. Fast k-means based on k-nn graph. In ICDE , pages 1220-1223. IEEE Computer Society, 2018.

Xiaowen Dong, Dorina Thanou, Pascal Frossard, and Pierre Vandergheynst. Laplacian matrix learning for smooth graph signal representation. In ICASSP , pages 3736-3740. IEEE, 2015.

Xiaowen Dong, Dorina Thanou, Michael G. Rabbat, and Pascal Frossard. Learning graphs from data: A signal representation perspective. IEEE Signal Process. Mag. , 36(3):44-63, 2019.

Yifan Gao, Jingjing Li, Michael R Lyu, and Irwin King. Open-retrieval conversational machine reading. arXiv preprint arXiv:2102.08633 , 2021.

Yifan Gao, Chien-Sheng Wu, Jingjing Li, Shafiq Joty, Steven CH Hoi, Caiming Xiong, Irwin King, and Michael Lyu. Discern: Discourse-aware entailment reasoning network for conversational machine reading. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2439-2449, 2020.

Advitya Gemawat. Graphgem: Optimized scalable system for graph convolutional networks. In SIGMOD Conference , pages 2920-2922. ACM, 2021.

Fang He, Feiping Nie, Rong Wang, Haojie Hu, Weimin Jia, and Xuelong Li. Fast semisupervised learning with optimal bipartite graph. IEEE Trans. Knowl. Data Eng. , 33(9):32453257, 2021.

Fang He, Feiping Nie, Rong Wang, Xuelong Li, and Weimin Jia. Fast semisupervised learning with bipartite graph for large-scale data. IEEE Transactions on Neural Networks and Learning Systems , 31(2):626-638, 2020.

Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R. Benson. Combining label propagation and simple models out-performs graph neural networks. In ICLR . OpenReview.net, 2021.

Tony Jebara, Jun Wang, and Shih-Fu Chang. Graph construction and b -matching for semisupervised learning. In ICML , volume 382 of ACM International Conference Proceeding Series , pages 441-448. ACM, 2009.

Junteng Jia and Austin R. Benson. A unifying generative model for graph learning algorithms: Label propagation, graph convolutions, and combinations. SIAM Journal on Mathematics of Data Science , 4(1):100-125, 2022.

Rie Johnson and Tong Zhang. On the effectiveness of laplacian normalization for graph semi-supervised learning. J. Mach. Learn. Res. , 8:1489-1517, 2007.

Vassilis Kalofolias. How to learn a graph from smooth signals. In AISTATS , volume 51 of JMLR Workshop and Conference Proceedings , pages 920-929. JMLR.org, 2016.

Vassilis Kalofolias and Nathanaël Perraudin. Large scale graph learning from smooth signals. In ICLR (Poster) . OpenReview.net, 2019.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR (Poster) . OpenReview.net, 2017.

Jingjing Li, Yifan Gao, Lidong Bing, Irwin King, and Michael R Lyu. Improving question generation with to the point context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3216-3226, 2019.

Jingjing Li, Zichao Li, Tao Ge, Irwin King, and Michael R Lyu. Text revision by on-the-fly representation optimization. pages 10956-10964, 2022.

Jingjing Li, Zichao Li, Lili Mou, Xin Jiang, Michael Lyu, and Irwin King. Unsupervised text generation by learning from search. Advances in Neural Information Processing Systems , 33:10820-10831, 2020.

Qimai Li, Xiao-Ming Wu, Han Liu, Xiaotong Zhang, and Zhichao Guan. Label efficient semi-supervised learning via graph filtering. In CVPR , pages 9582-9591. Computer Vision Foundation / IEEE, 2019.

Yu-Feng Li, Shao-Bo Wang, and Zhi-Hua Zhou. Graph quality judgement: A large margin expedition. In IJCAI , pages 1725-1731. IJCAI/AAAI Press, 2016.

Jiye Liang, Junbiao Cui, Jie Wang, and Wei Wei. Graph-based semi-supervised learning via improving the quality of the graph dynamically. Mach. Learn. , 110(6):1345-1388, 2021.

Langzhang Liang, Zenglin Xu, Zixing Song, Irwin King, Yuan Qi, and Jieping Ye. Tackling long-tailed distribution issue in graph neural networks via normalization. IEEE Transactions on Knowledge and Data Engineering , pages 1-11, 2023.

Langzhang Liang, Zenglin Xu, Zixing Song, Irwin King, and Jieping Ye. Resnorm: Tackling long-tailed degree distribution issue in graph neural networks via normalization. CoRR , abs/2206.08181, 2022.

Wei Liu, Junfeng He, and Shih-Fu Chang. Large graph construction for scalable semi-supervised learning. In ICML , pages 679-686. Omnipress, 2010.

Yueen Ma, Zixing Song, Xuming Hu, Jingjing Li, Yifei Zhang, and Irwin King. Graph component contrastive learning for concept relatedness estimation. In AAAI , pages 1336213370. AAAI Press, 2023.

Yueen Ma, Zixing Song, Xuming Hu, Jingjing Li, Yifei Zhang, and Irwin King. Graph component contrastive learning for concept relatedness estimation. Proceedings of the AAAI Conference on Artificial Intelligence , 37(11):13362-13370, Jun. 2023.

Neal Parikh and Stephen Boyd. Proximal algorithms. Found. Trends Optim. , 1(3):127-239, jan 2014.

Michael G. Rabbat. Inferring sparse graphs from smooth signals with theoretical guarantees. In ICASSP , pages 6533-6537. IEEE, 2017.

Seyed Saman Saboksayr and Gonzalo Mateos. Accelerated graph learning from smooth signals. IEEE Signal Process. Lett. , 28:2192-2196, 2021.

Hyunjung Shin, N. Jeremy Hill, Andreas Martin Lisewski, and Joon-Sang Park. Graph sharpening. Expert Syst. Appl. , 37(12):7870-7879, 2010.

Hyunjung Shin, N. Jeremy Hill, and Gunnar Rätsch. Graph based semi-supervised learning with sharper edges. In ECML , volume 4212 of Lecture Notes in Computer Science , pages 401-412. Springer, 2006.

Anthony Man-Cho So. Moment inequalities for sums of random matrices and their applications in optimization. Math. Program. , 130(1):125-151, 2011.

Zixing Song and Irwin King. Hierarchical heterogeneous graph attention network for syntaxaware summarization. In AAAI , pages 11340-11348. AAAI Press, 2022.

Zixing Song, Yueen Ma, and Irwin King. Individual fairness in dynamic financial networks. In NeurIPS 2022 Workshop: New Frontiers in Graph Learning , 2022.

Zixing Song, Ziqiao Meng, Yifei Zhang, and Irwin King. Semi-supervised multi-label learning for graph-structured data. In CIKM , pages 1723-1733. ACM, 2021.

Zixing Song, Xiangli Yang, Zenglin Xu, and Irwin King. Graph-based semi-supervised learning: A comprehensive review. IEEE Transactions on Neural Networks and Learning Systems , pages 1-21, 2022.

Zixing Song, Yifei Zhang, and Irwin King. Towards an optimal asymmetric graph structure for robust semi-supervised node classification. In KDD , pages 1656-1665. ACM, 2022.

Zixing Song, Yuji Zhang, and Irwin King. Towards fair financial services for all: A temporal GNN approach for individual fairness on transaction networks. In CIKM , pages 2331-2341. ACM, 2023.

Xin Sun, Tao Ge, Shuming Ma, Jingjing Li, Furu Wei, and Houfeng Wang. A unified strategy for multilingual grammatical error correction with pre-trained cross-lingual language model. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 , pages 4367-4374. International Joint Conferences on Artificial Intelligence Organization, 7 2022. Main Track.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices , page 210-268. Cambridge University Press, 2012.

Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. In ICML , volume 148 of ACM International Conference Proceeding Series , pages 985-992. ACM, 2006.

Fei Wang and Changshui Zhang. Label propagation through linear neighborhoods. IEEE Trans. Knowl. Data Eng. , 20(1):55-67, 2008.

Hongwei Wang and Jure Leskovec. Combining graph convolutional neural networks and label propagation. ACM Trans. Inf. Syst. , 40(4), nov 2021.

Xiaolu Wang, Yuen-Man Pun, and Anthony Man-Cho So. Distributionally robust graph learning from smooth signals under moment uncertainty. IEEE Trans. Signal Process. , 70:6216-6231, 2022.

Xiaolu Wang, Chaorui Yao, Haoyu Lei, and Anthony Man-Cho So. An efficient alternating direction method for graph learning from smooth signals. In ICASSP , pages 5380-5384. IEEE, 2021.

Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, Juncheng Liu, and Bryan Hooi. Nodeaug: Semi-supervised node classification with data augmentation. In KDD , pages 207-217. ACM, 2020.

Zhen Wang, Long Zhang, Rong Wang, Feiping Nie, and Xuelong Li. Semi-supervised learning via bipartite graph construction with adaptive neighbors. IEEE Transactions on Knowledge and Data Engineering , pages 1-1, 2022.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu. A comprehensive survey on graph neural networks. IEEE Trans. Neural Networks Learn. Syst. , 32(1):4-24, 2021.

Tian Xie, Bin Wang, and C.-C. Jay Kuo. Graphhop: An enhanced label propagation method for node classification. IEEE Transactions on Neural Networks and Learning Systems , pages 1-15, 2022.

Menglin Yang, Zhihao Li, Min Zhou, Jiahong Liu, and Irwin King. Hicf: Hyperbolic informative collaborative filtering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2212-2221, 2022.

Menglin Yang, Ziqiao Meng, and Irwin King. FeatureNorm: L2 feature normalization for dynamic graph embedding. In 2020 IEEE International Conference on Data Mining (ICDM) , pages 731-740. IEEE, 2020.

Menglin Yang, Min Zhou, Marcus Kalander, Zengfeng Huang, and Irwin King. Discretetime temporal network embedding via implicit hierarchical learning in hyperbolic space. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1975-1985, 2021.

Menglin Yang, Min Zhou, Jiahong Liu, Defu Lian, and Irwin King. Hrcf: Enhancing collaborative filtering via hyperbolic geometric regularization. In Proceedings of the ACM Web Conference 2022 , pages 2462-2471, 2022.

Menglin Yang, Min Zhou, Lujia Pan, and Irwin King. κ hgcn: Tree-likeness modeling via continuous and discrete curvature learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2965-2977, 2023.

Menglin Yang, Min Zhou, Hui Xiong, and Irwin King. Hyperbolic temporal network embedding. IEEE Transactions on Knowledge and Data Engineering , 2022.

Menglin Yang, Min Zhou, Rex Ying, Yankai Chen, and Irwin King. Hyperbolic representation learning: Revisiting and advancing. arXiv preprint arXiv:2306.09118 , 2023.

Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering , pages 1-20, 2022.

Yuan Yuan, Xin Li, Qi Wang, and Feiping Nie. A semi-supervised learning algorithm via adaptive laplacian graph. Neurocomputing , 426:162-173, 2021.

Yifei Zhang, Yankai Chen, Zixing Song, and Irwin King. Contrastive cross-scale graph knowledge synergy. In KDD , pages 3422-3433. ACM, 2023.

Yifei Zhang, Hao Zhu, Ziqiao Meng, Piotr Koniusz, and Irwin King. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM Web Conference 2022 , 2022.

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. COSTA: covariancepreserving feature augmentation for graph contrastive learning. In KDD , pages 2524-2534. ACM, 2022.

Yifei Zhang, Hao Zhu, Zixing Song, Piotr Koniusz, and Irwin King. Spectral feature augmentation for graph contrastive learning and beyond. In AAAI , pages 11289-11297. AAAI Press, 2023.

Yuji Zhang and Jing Li. Time will change things: An empirical study on dynamic language understanding in social media classification. arXiv e-prints , pages arXiv-2210, 2022.

Yuji Zhang, Jing Li, and Wenjie Li. Vibe: Topic-driven temporal adaptation for twitter classification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Singapore, December 2023. Association for Computational Linguistics.

Yuji Zhang, Yubo Zhang, Chunpu Xu, Jing Li, Ziyan Jiang, and Baolin Peng. #HowYouTagTweets: Learning user hashtagging preferences via personalized topic attention. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 7811-7820, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In NIPS , pages 321-328. MIT Press, 2003.

Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf. Learning from labeled and unlabeled data on a directed graph. In Proceedings of the 22nd International Conference on Machine Learning , ICML '05, page 1036-1043, New York, NY, USA, 2005. Association for Computing Machinery.

Dengyong Zhou, Bernhard Schölkopf, and Thomas Hofmann. Semi-supervised learning on directed graphs. In NIPS , pages 1633-1640, 2004.

Xiaojin Zhu, Zoubin Ghahramani, and John D. Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In ICML , pages 912-919. AAAI Press, 2003.

Liansheng Zhuang, Zihan Zhou, Shenghua Gao, Jingwen Yin, Zhouchen Lin, and Yi Ma. Label information guided graph construction for semi-supervised learning. IEEE Trans. Image Process. , 26(9):4182-4192, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an optimal asymmetric graph structure for the label inference phase in graph-based semi-supervised learning (GSSL). The key motivation or intuition proposed by the authors is that we need to differentiate the roles of labeled and unlabeled nodes. Therefore, the authors design an efficient block-wise graph learning algorithm with a global convergence guarantee. The proposed method is shown to be superior to SOTA graph construction methods in GSSL through extensive experiments on synthetic and real-world datasets. The paper addresses the challenge of constructing a high-quality graph, which significantly influences label prediction performance, and proposes a solution with theoretical motivations and benefits, such as enhanced robustness to noisy node features.\n\nSTRENGTHS:\n1.\tQuality: The work is of high quality, with rigorous theoretical motivations and comprehensive experiments. First, the motivation is strongly supported by some theoretical analysis. The proposed asymmetric optimal graph structure is deduced from Definition 1 rigorously. The authors provide a comprehensive explanation of the optimization problem and the structure of the optimal affinity graph. Second, they also present a detailed implementation of the block-wise graph learning algorithm BAGL. The paper is well-referenced, indicating a thorough understanding of the existing literature. Third, the provided convergence analysis on BAGL is also rigorous. The global sublinear convergence rate in Theorem 3 makes sense to the reviewer. Forth, the experiments are comprehensive, including several comparisons from the aspects of prediction accuracy, efficiency, and convergence rate. \n2.\tClarity: The paper is well-structured and clear, with each section logically leading to the next. The authors provide clear definitions and explanations of complex concepts, making the paper accessible to readers with varying levels of expertise in the field. However, some parts of the appendix are suggested to move to the main body to give more background context on graph-based semi-supervised learning. The use of mathematical notation and diagrams further enhances the clarity of the paper.\n3.\tSignificance: The reviewer thinks the work makes some significant contributions to the GSSL field. First, the investigated problem is significant since most GSSL literature only focuses on the label inference step. This paper focuses on the neglected graph construction step instead, as the quality of the graph affects the subsequent step a lot. Second, the proposed method achieves the SOTA global convergence rate, contributing to the GSSL field significantly.\n\nWEAKNESSES:\nThere are several potential improvements in this paper.\n1.\tThe background knowledge of the unified label inference framework can be elaborated. The details behind Eq.(1) should be provided to give readers without GSSL backgrounds more context. For example, Appendix B.2 should be added to the main body. Table 5 is very informative.\n\n2.\tThe recent graph structure learning method should be discussed or compared since the goal of GSL is also similar to the task investigated in this paper.\n\n3.\tThe conclusion part is short. More future work can be added.\n\n4.\tOther comments in questions section.\n\nQUESTIONS:\n1.\tWhile the paper provides a detailed explanation of the block-wise graph learning algorithm, it would be beneficial to have a more explicit discussion on the computational complexity of the algorithm. How does the algorithm scale with the size of the dataset? What are the implications of using this algorithm on large-scale datasets?\n2.\tThe paper could benefit from a discussion on potential real-world applications of the proposed method. In what specific domains or scenarios would this method be particularly useful?\n3.\tIt would be interesting to hear the authors' perspectives on the limitations of their proposed method and how they plan to address these in future work. Are there any specific challenges or difficulties that arose during the development of the method?\n4.\tThe paper mentions that the proposed method enhances the robustness of subsequent label inference algorithms. Could the authors elaborate on this? How does the method handle noise? Some theoretical analysis may be provided.\n\nLIMITATIONS:\nPlease refer to the weaknesses and questions sections.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
The paper proposes a method for graph construction stage of graph based semi-supervised learning. They further evaluate their method with experimental results.

STRENGTHS:
The authors present strong theoretical results.

WEAKNESSES:
The experimental results for the proposed method in Table 2 are only marginally better than the baselines.

QUESTIONS:
NA

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an efficient and effective method for constructing affinity graphs in Graph-based Semi-supervised Learning (GSSL), with a focus on the distinct roles of labeled and unlabeled nodes. The authors present a formulation for the GSSL problem, comprising two steps: graph construction and label inference. They investigate the optimal construction of the affinity graph in the first phase to facilitate enhanced performance in the second label inference phase. The paper offers four main contributions: a succinct definition of the optimality of the affinity graph in GSSL, a block-wise graph learning framework to infer the weights in the optimal graph structure, proof of a global sub-linear convergence rate for the proposed method, and extensive experiments on synthetic and real-world datasets to demonstrate the effectiveness and efficiency of the proposed method.\n\nSTRENGTHS:\nOriginality:\nThe paper presents a novel approach to constructing affinity graphs in GSSL, with a focus on the distinct roles of labeled and unlabeled nodes. The proposed method is based on an asymmetric structure and a block-wise graph learning framework, which are different from existing methods. The paper also offers a succinct definition of the optimality of the affinity graph in GSSL, which is a unique contribution to the field. Overall, the paper is highly original in its approach to graph construction in GSSL.\n\nQuality:\nThe paper is well-written and presents a rigorous derivation of the proposed method. The authors provide a clear explanation of the problem formulation and the proposed solution, as well as a detailed analysis of the benefits of the proposed method. The experiments are extensive and well-designed, with results that demonstrate the effectiveness and efficiency of the proposed method. The paper also includes a thorough review of related work, which adds to the quality of the paper. Overall, the paper is of high quality.\n\nClarity:\nThe paper is well-organized and easy to follow. The authors provide clear explanations of the concepts and methods used in the paper, and the figures and tables are well-designed and easy to understand. The paper also includes a summary of the contributions and a conclusion that summarizes the main findings. Overall, the paper is highly clear and well-presented.\n\nSignificance:\nThe paper makes a significant contribution to the field of GSSL by proposing an efficient and effective method for constructing affinity graphs. The proposed method is based on an asymmetric structure and a block-wise graph learning framework, which are different from existing methods. The paper also offers a succinct definition of the optimality of the affinity graph in GSSL, which is a unique contribution to the field. The experiments demonstrate the effectiveness and efficiency of the proposed method, which has the potential to improve the performance of GSSL algorithms in a wide range of applications. Overall, the paper is highly significant in its contribution to the field of GSSL.\n\nWEAKNESSES:\n1. Lack of comparison with more recent state-of-the-art methods: While the paper compares the proposed method with several existing methods, some of these methods are relatively old and may not represent the current state-of-the-art in GSSL. It would be useful to compare the proposed method with more recent methods to provide a more comprehensive evaluation.\n\n2. Limited discussion of the limitations of the proposed method: While the paper discusses the benefits of the proposed method, there is limited discussion of its limitations. It would be useful to discuss the situations in which the proposed method may not be effective and to provide guidance on when to use the proposed method versus other methods.\n\n3. Lack of real-world applications: While the paper includes experiments on synthetic and real-world datasets, there is limited discussion of real-world applications of the proposed method. It would be useful to provide examples of how the proposed method could be applied in real-world scenarios and to discuss the potential impact of the proposed method on these applications.\n\nOverall, while the paper presents a novel approach to constructing affinity graphs in GSSL, there are some weaknesses that could be addressed to improve the paper's impact and relevance.\n\nQUESTIONS:\nQuestions:\n How does the proposed method compare to more recent state-of-the-art methods in GSSL?\n Can you provide more guidance on when to use the proposed method versus other methods?\n Can you provide examples of how the proposed method could be applied in real-world scenarios?\n\nSuggestions:\n1. Consider comparing the proposed method with more recent state-of-the-art methods in GSSL to provide a more comprehensive evaluation.\n2. Provide more discussion of the limitations of the proposed method and when it may not be effective.\n3. Provide examples of how the proposed method could be applied in real-world scenarios to demonstrate its potential impact and relevance.\n4. Consider including a discussion of the computational complexity of the proposed method and how it compares to other methods.\n5. Consider including a more detailed explanation of the block-wise graph learning framework used in the proposed method to help readers better understand the approach.\n\nLIMITATIONS:\nThe paper does not explicitly discuss the limitations and potential negative societal impact of the proposed method. While the paper does discuss the benefits of the proposed method, it does not provide a comprehensive discussion of its limitations or potential negative consequences. \n\nIt is important for authors to consider the potential limitations and negative consequences of their work, as this can help to ensure that the benefits of the work outweigh any potential negative impacts. In particular, authors should consider the ethical implications of their work and how it may impact society as a whole.\n\nTherefore, it can be said that the authors have not adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes a novel methodology for graph-based semi-supervised learning by leveraging a asymetric graph construction technique. The main contribution of the paper is the design of a block-wise graph learning framework to estimate the weights of a graph.

STRENGTHS:
The main strengths of the paper are:
- The derivation of the structure of the optimal affinity graph
- The derivation of an optimization algorithm for the implementation of the block-wise graph learning algorithm
- The thorough experimental evaluation to assess the performance of the proposed algorithm in different scenarios

WEAKNESSES:
- While the derivation of the optimal affinity graph is novel, the optimization algorithm for learning the graph weights are heavily influenced by prior works, and therefore not much novel.

- The plots (c) and (d) in Figure 3 are not as helpful as the x-axis represent number of iterations where instead the authors should use computational time. Therefore, from Figures 3c and 3d we cannot conclude much about the computational efficiency of the proposed algorithm.

QUESTIONS:
- I am not convinced regarding the novelty of the framework. Can the authors elaborate how the development of section 3.2 is different from the work of [23] apart from the well-known application of the FISTA step?

LIMITATIONS:
The authors properly addressed the limitations of their proposed framework.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

comments: SUMMARY:
The authors proposes to solve graph-based semi-supervised learning (GSSL) problems by first finding the "optimal graph" for SSL. The optimal graph has edges only from labeled to unlabeled nodes, or between unlabeled nodes. These edge weights are computed through FISTA algorithm in the dual space, and theoretical guarantees are provided for sub-linear convergence rates. Experiments are conducted using both synthetic and real-world datasets.

STRENGTHS:
This paper is very well written and nicely presented. The clear writing made it easy to follow.

I did not carefully read through the proofs in the appendix, but the motivation for the framework and derivation of the algorithm seem correct. I appreciate that this paper did solid work on all aspects - problem formulation, clever optimization algorithm, theoretical convergence analysis, and numerical experiments.

WEAKNESSES:
Please see the questions below.

QUESTIONS:
1. When comparing methods such as RBF and KNN to BAGL, were RBF and KNN used to also generate asymmetric (and not symmetric) graphs? This is not made immediately clear in the text, and I'm wondering if the superior performance mainly comes from keeping $W_{lu}, W_{ll}$ all zero matrices.
 
2. Figure 1 c is visually appealing, but I'm having trouble understanding why that solution is any better or more probable than the RBF solution in 1b. 

3. There's a mismatch between appendix section labels and their references in the main text. This should be fixed since the appendix includes important time complexity and runtime analysis.

4. How does BAGL react to class imbalance, either in the labeled nodes or in the entire classification problem?

LIMITATIONS:
Yes, in the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07