Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels

1 Introduction . Graph-based Semi-Supervised Learning ( SSL ) refers to classifying unlabeled data based on a handful of labeled data and a given graph structure indicating the connections between all data . Recently , graph-based SSL has attracted increasing attention due to its solid mathematical foundation , and satisfactory performance [ 1 , 2 , 3 ] . As the mainstream to solve graph-based SSL problems , Graph Neural Networks ( GNNs ) , which operate in the graph domain , have achieved impressive performance in recent years [ 4 , 5 , 6 , 7 ] . Nevertheless , current GNNs , such as Graph Convolutional Networks ( GCNs ) [ 8 ] and graph attention networks ( GATs ) [ 9 ] , require sufficient labeled data to obtain satisfactory generalization abilities . Unfortunately , the reliance on sufficient labeled data increases the burden of data collection , and the number of labels can be extremely limited in some real-world scenarios . The performance of most ∗This work was done when Sheng was a research intern at JD Explore Academy . †C . Gong is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . current GNNs seriously declines as the label size shrinks , since the scarce supervision signals are insufficient to train a model with satisfactory discriminative ability , see Figure 1 . To the best of our knowledge , few studies have focused on semi-supervised classification with GNNs at extremely low label rates [ 10 ] . In line with the aforementioned observations , this paper proposes a new framework termed Contrastive Graph Poisson Networks ( CGPN ) to address the problem of semi-supervised node classification under extremely low label rates . Deriving from the variational inference [ 12 , 13 ] , our proposed CGPN framework approximates the intractable posterior with a surrogate distribution , where two types of GNNs have been adopted for instantiation . Accordingly , we first design a new Graph Poisson Network ( GPN ) to propagate the limited labels to the entire graph effectively . Specifically , our GPN is motivated by Poisson learning [ 14 ] ; flexibly models the Poisson label propagation based on attention mechanism ; and leverages the structural information to better guide the propagation of labels . Meanwhile , we exploit another GNN , such as GAT , together with the proposed GPN to model the approximated posterior according to variational inference . On this basis , we acquire predictions from two comparable views , where a contrastive objective can be naturally incorporated to jointly refine the learning process of the GPN and GNN models . Moreover , the supervision signals implicitly contained in the massive unlabeled data can be exploited with the formulated contrastive loss . As a result , the model learning ability of our proposed framework can be lifted . Experimental results on benchmark datasets confirm the strong benefits of our proposed CGPN when dealing with semi-supervised node classification at very low label rates . In summary , the contributions of this paper lie in three folds : First , we propose a novel GNN framework termed CGPN to solve the semi-supervised node classification with extremely limited labels . CGPN significantly outperforms the existing GNNs . Second , we design a new Graph Poisson Network ( GPN ) . Different from the Poisson learning algorithm , our GPN incorporates graph-structure information and could be trained in an end-to-end manner to guide the propagation of labels more flexibly . Third , we integrate contrastive learning into the variational inference framework , so that extra supervision information can be explored from the massive unlabeled data to help train our CGPN framework . 2 Problem description . We start by formally introducing the problem of graph-based SSL . Given a set of n = l+ u examples Ψ = { x1 , · · · , xl , xl+1 , · · · , xn } , where the first l examples are provided with the labels { yi } li=1 and the remaining u examples constitute the unlabeled set with typically l≪ u . Let X ∈ Rn×d denote the feature matrix with the i-th row formed by the feature vector xi , and Y ∈ Rn×c denote the label matrix with its ( i , j ) -th element Yij = 1 if xi belongs to the j-th class and 0 otherwise . Here d is the feature dimension and c is the number of classes . The dataset Φ is represented by an undirected graph G = ⟨V , E⟩ , where V represents the node set containing all examples and E is the edge set modeling the similarity among the nodes/examples . The adjacency matrix of G is denoted as A , where Aij = 1 if there exists an edge between xi and xj and Aij = 0 , otherwise . In this paper , we target transductive graph-based SSL which aims to find the labels yl+1 , yl+2 , · · · , yn of the unlabeled examples xl+1 , xl+2 , · · · , xn based on Ψ . 3 Related work . 3.1 Graph-based semi-supervised learning . SSL methods focus on training models with small amounts of labeled data as well as relatively large amounts of unlabeled data for [ 15 , 16 , 17 , 18 , 19 ] . Graph-based SSL algorithms have been one of the most popular research topics during the past decades . The early graph-based techniques are designed based on the simple assumption that nearby nodes are likely to have the same label . This goal can be achieved through the low-dimensional embeddings with Laplacian eigenmaps [ 20 , 21 ] , Markov random walks [ 22 ] , etc . Meanwhile , graph partition [ 23 ] offers another important line in graph-based SSL . To further enhance the learning capacities , various techniques have been proposed to model the data features and graph structure jointly , such as Planetoid [ 24 ] , where the supervised classifier is regularized with a Laplacian regularizer or an embedding-based regularizer . Recently , a set of graph-based SSL approaches have been proposed to improve the performance of the above-mentioned techniques , including [ 25 , 14 , 26 ] . 3.2 Graph neural networks . In the past few years , increasing attention has been paid to GNN models [ 27 , 28 , 26 , 29 , 30 , 31 , 32 ] . Early-staged works aim to derive diverse types of graph convolution in spectral-domain based on the graph spectral theory [ 8 , 33 , 4 ] . In [ 33 ] , a general graph convolution framework based on graph Laplacian is first proposed . Afterwards , Defferrard et al . [ 4 ] approximate the convolutional filter via using a K-order Chebyshev polynomial , in order to avoid the intense calculations of eigendecomposition of the normalized graph Laplacian . In addition to this , Kipf and Welling [ 8 ] further simplify the graph convolution by a localized first-order approximation , which brings about more efficient filtering operations than spectral CNNs . Another line of research efforts focus on directly performing graph convolution in the spatial domain [ 5 , 27 , 34 , 35 ] . In spatial GNN models , the convolution operation is defined as a weighted average function over the neighbors of each node , which characterizes the impact exerting to the target node from its neighboring ones . For instance , Hamilton et al . [ 5 ] propose a general inductive framework called GraphSAGE , which is able to learn an embedding function generalizing to previously unseen graph nodes . Besides , in [ 9 ] , the graph attention network ( GAT ) is devised by utilizing the attention mechanism , which assigns different weights to the neighboring nodes and aggregates feature with discrimination . Although these models exploit the inter-dependencies among labeled and unlabeled nodes , their performances can still degrade dramatically when the number of labels is extremely limited . 3.3 Poisson learning . Poisson learning is motivated by the need to address the degeneracy of Laplacian SSL when label information is very limited . To be concrete , traditional Laplacian learning algorithm [ 15 ] aims at solving the following problem : { Lu ( xi ) = 0 , if l + 1 ≤ i ≤ n , u ( xi ) = yi , if 1 ≤ i ≤ l , , ( 1 ) where L represents the unnormalized graph Laplacian and u ( xi ) ∈ Rc is the label vector of xi . Here , the node xi belongs to the j-th class if the j-th component uj ( xi ) is the largest in u ( xi ) . Although Laplacian learning works very well for SSL tasks with a moderate number of labeled examples , the performance becomes quite poor at very low label rates . Different from Laplacian learning , where labels are imposed as boundary conditions , in Poisson learning , labels appear as the source term . The solution of a Poisson equation can be computed as Lu ( xi ) = l∑ j=1 ( yj − ȳ ) δij for i = 1 , ... , n ( 2 ) satisfying ∑n i=1 diu ( xi ) = 0 , where δij = 1 if i = j and δij = 0 , otherwise , ȳ = 1 lF1 with F ∈ Rc×l denoting the label matrix of the l labeled nodes , and di = ∑n j=1 Aij . The Poisson equation ( 2 ) can be solved efficiently with a simple iteration , and the result of the ( t+ 1 ) -th iteration can be obtained as U ( t+1 ) ← U ( t ) +D−1 ( B⊤ − LU ( t ) ) , ( 3 ) where D is the diagonal matrix with Dii = di , L is the Laplacian matrix , and B = [ F− ȳ , O ] is the source term . Here , O ∈ Rc× ( n−l ) indicates a zero matrix . After T iterations , the prediction can be acquired as U ( T ) . The specific interpretation of Poisson learning algorithm can be found in [ 14 ] . 4 Methodology . This section details our proposed CGPN framework . Specifically , we describe the critical components of CGPN by explaining the variational inference framework , presenting the instantiation with GPN and GNN models , and illustrating the contrastive label inference . 4.1 Inference framework . To infer the labels of the unlabeled nodes , i.e. , YU , we need to estimate the posterior distribution given the node features X , the observed labels YL , and the adjacency matrix A , namely pθ ( YU |A , X , YL ) with parameters θ. Computation of this posterior is usually analytically intractable , so we resort to approximate posterior inference methods . Inspired by the recent advances in scalable variational inference [ 36 , 37 ] , we introduce a distribution qϕ ( YU |A , X , YL ) parameterized by ϕ to approximate the true posterior pθ ( YU |A , X , YL ) . Afterwards , we can write the Evidence Lower BOund ( ELBO ) as LELBO ( θ , ϕ ) = log pθ ( YL|A , X ) −DKL ( qϕ ( YU |A , X , YL ) ||pθ ( YU |A , X ) ) , ( 4 ) where DKL ( ·||· ) represents the Kullback-Leibler divergence between two distributions . For practical use , it remains to specify the parametric forms of qϕ ( YU |A , X , YL ) and pθ ( Y|A , X ) with GNNs .

Summary: This paper focuses on semi-supervised node classification with extremely limited labels and the authors propose a new framework termed Contrastive Graph Poisson Networks (CGPN). Specifically, they design a Graph Poisson Networks to propagate the limited labels to the entire graph and employ a Graph Neural Network (GNN) to guide the propagation. Meanwhile, labeled data and unlabeled data are also considered in the final objective function. For the labeled data, they simply use a cross-entropy loss and for the unlabeled data, they apply a pairwise contrastive loss to explore extra supervision information.