Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels

1 Introduction . Graph-based Semi-Supervised Learning ( SSL ) refers to classifying unlabeled data based on a handful of labeled data and a given graph structure indicating the connections between all data . Recently , graph-based SSL has attracted increasing attention due to its solid mathematical foundation , and satisfactory performance [ 1 , 2 , 3 ] . As the mainstream to solve graph-based SSL problems , Graph Neural Networks ( GNNs ) , which operate in the graph domain , have achieved impressive performance in recent years [ 4 , 5 , 6 , 7 ] . Nevertheless , current GNNs , such as Graph Convolutional Networks ( GCNs ) [ 8 ] and graph attention networks ( GATs ) [ 9 ] , require sufficient labeled data to obtain satisfactory generalization abilities . Unfortunately , the reliance on sufficient labeled data increases the burden of data collection , and the number of labels can be extremely limited in some real-world scenarios . The performance of most ∗This work was done when Sheng was a research intern at JD Explore Academy . †C . Gong is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . current GNNs seriously declines as the label size shrinks , since the scarce supervision signals are insufficient to train a model with satisfactory discriminative ability , see Figure 1 . To the best of our knowledge , few studies have focused on semi-supervised classification with GNNs at extremely low label rates [ 10 ] . In line with the aforementioned observations , this paper proposes a new framework termed Contrastive Graph Poisson Networks ( CGPN ) to address the problem of semi-supervised node classification under extremely low label rates . Deriving from the variational inference [ 12 , 13 ] , our proposed CGPN framework approximates the intractable posterior with a surrogate distribution , where two types of GNNs have been adopted for instantiation . Accordingly , we first design a new Graph Poisson Network ( GPN ) to propagate the limited labels to the entire graph effectively . Specifically , our GPN is motivated by Poisson learning [ 14 ] ; flexibly models the Poisson label propagation based on attention mechanism ; and leverages the structural information to better guide the propagation of labels . Meanwhile , we exploit another GNN , such as GAT , together with the proposed GPN to model the approximated posterior according to variational inference . On this basis , we acquire predictions from two comparable views , where a contrastive objective can be naturally incorporated to jointly refine the learning process of the GPN and GNN models . Moreover , the supervision signals implicitly contained in the massive unlabeled data can be exploited with the formulated contrastive loss . As a result , the model learning ability of our proposed framework can be lifted . Experimental results on benchmark datasets confirm the strong benefits of our proposed CGPN when dealing with semi-supervised node classification at very low label rates . In summary , the contributions of this paper lie in three folds : First , we propose a novel GNN framework termed CGPN to solve the semi-supervised node classification with extremely limited labels . CGPN significantly outperforms the existing GNNs . Second , we design a new Graph Poisson Network ( GPN ) . Different from the Poisson learning algorithm , our GPN incorporates graph-structure information and could be trained in an end-to-end manner to guide the propagation of labels more flexibly . Third , we integrate contrastive learning into the variational inference framework , so that extra supervision information can be explored from the massive unlabeled data to help train our CGPN framework . 2 Problem description . We start by formally introducing the problem of graph-based SSL . Given a set of n = l+ u examples Ψ = { x1 , · · · , xl , xl+1 , · · · , xn } , where the first l examples are provided with the labels { yi } li=1 and the remaining u examples constitute the unlabeled set with typically l≪ u . Let X ∈ Rn×d denote the feature matrix with the i-th row formed by the feature vector xi , and Y ∈ Rn×c denote the label matrix with its ( i , j ) -th element Yij = 1 if xi belongs to the j-th class and 0 otherwise . Here d is the feature dimension and c is the number of classes . The dataset Φ is represented by an undirected graph G = ⟨V , E⟩ , where V represents the node set containing all examples and E is the edge set modeling the similarity among the nodes/examples . The adjacency matrix of G is denoted as A , where Aij = 1 if there exists an edge between xi and xj and Aij = 0 , otherwise . In this paper , we target transductive graph-based SSL which aims to find the labels yl+1 , yl+2 , · · · , yn of the unlabeled examples xl+1 , xl+2 , · · · , xn based on Ψ . 3 Related work . 3.1 Graph-based semi-supervised learning . SSL methods focus on training models with small amounts of labeled data as well as relatively large amounts of unlabeled data for [ 15 , 16 , 17 , 18 , 19 ] . Graph-based SSL algorithms have been one of the most popular research topics during the past decades . The early graph-based techniques are designed based on the simple assumption that nearby nodes are likely to have the same label . This goal can be achieved through the low-dimensional embeddings with Laplacian eigenmaps [ 20 , 21 ] , Markov random walks [ 22 ] , etc . Meanwhile , graph partition [ 23 ] offers another important line in graph-based SSL . To further enhance the learning capacities , various techniques have been proposed to model the data features and graph structure jointly , such as Planetoid [ 24 ] , where the supervised classifier is regularized with a Laplacian regularizer or an embedding-based regularizer . Recently , a set of graph-based SSL approaches have been proposed to improve the performance of the above-mentioned techniques , including [ 25 , 14 , 26 ] . 3.2 Graph neural networks . In the past few years , increasing attention has been paid to GNN models [ 27 , 28 , 26 , 29 , 30 , 31 , 32 ] . Early-staged works aim to derive diverse types of graph convolution in spectral-domain based on the graph spectral theory [ 8 , 33 , 4 ] . In [ 33 ] , a general graph convolution framework based on graph Laplacian is first proposed . Afterwards , Defferrard et al . [ 4 ] approximate the convolutional filter via using a K-order Chebyshev polynomial , in order to avoid the intense calculations of eigendecomposition of the normalized graph Laplacian . In addition to this , Kipf and Welling [ 8 ] further simplify the graph convolution by a localized first-order approximation , which brings about more efficient filtering operations than spectral CNNs . Another line of research efforts focus on directly performing graph convolution in the spatial domain [ 5 , 27 , 34 , 35 ] . In spatial GNN models , the convolution operation is defined as a weighted average function over the neighbors of each node , which characterizes the impact exerting to the target node from its neighboring ones . For instance , Hamilton et al . [ 5 ] propose a general inductive framework called GraphSAGE , which is able to learn an embedding function generalizing to previously unseen graph nodes . Besides , in [ 9 ] , the graph attention network ( GAT ) is devised by utilizing the attention mechanism , which assigns different weights to the neighboring nodes and aggregates feature with discrimination . Although these models exploit the inter-dependencies among labeled and unlabeled nodes , their performances can still degrade dramatically when the number of labels is extremely limited . 3.3 Poisson learning . Poisson learning is motivated by the need to address the degeneracy of Laplacian SSL when label information is very limited . To be concrete , traditional Laplacian learning algorithm [ 15 ] aims at solving the following problem : { Lu ( xi ) = 0 , if l + 1 ≤ i ≤ n , u ( xi ) = yi , if 1 ≤ i ≤ l , , ( 1 ) where L represents the unnormalized graph Laplacian and u ( xi ) ∈ Rc is the label vector of xi . Here , the node xi belongs to the j-th class if the j-th component uj ( xi ) is the largest in u ( xi ) . Although Laplacian learning works very well for SSL tasks with a moderate number of labeled examples , the performance becomes quite poor at very low label rates . Different from Laplacian learning , where labels are imposed as boundary conditions , in Poisson learning , labels appear as the source term . The solution of a Poisson equation can be computed as Lu ( xi ) = l∑ j=1 ( yj − ȳ ) δij for i = 1 , ... , n ( 2 ) satisfying ∑n i=1 diu ( xi ) = 0 , where δij = 1 if i = j and δij = 0 , otherwise , ȳ = 1 lF1 with F ∈ Rc×l denoting the label matrix of the l labeled nodes , and di = ∑n j=1 Aij . The Poisson equation ( 2 ) can be solved efficiently with a simple iteration , and the result of the ( t+ 1 ) -th iteration can be obtained as U ( t+1 ) ← U ( t ) +D−1 ( B⊤ − LU ( t ) ) , ( 3 ) where D is the diagonal matrix with Dii = di , L is the Laplacian matrix , and B = [ F− ȳ , O ] is the source term . Here , O ∈ Rc× ( n−l ) indicates a zero matrix . After T iterations , the prediction can be acquired as U ( T ) . The specific interpretation of Poisson learning algorithm can be found in [ 14 ] . 4 Methodology . This section details our proposed CGPN framework . Specifically , we describe the critical components of CGPN by explaining the variational inference framework , presenting the instantiation with GPN and GNN models , and illustrating the contrastive label inference . 4.1 Inference framework . To infer the labels of the unlabeled nodes , i.e. , YU , we need to estimate the posterior distribution given the node features X , the observed labels YL , and the adjacency matrix A , namely pθ ( YU |A , X , YL ) with parameters θ. Computation of this posterior is usually analytically intractable , so we resort to approximate posterior inference methods . Inspired by the recent advances in scalable variational inference [ 36 , 37 ] , we introduce a distribution qϕ ( YU |A , X , YL ) parameterized by ϕ to approximate the true posterior pθ ( YU |A , X , YL ) . Afterwards , we can write the Evidence Lower BOund ( ELBO ) as LELBO ( θ , ϕ ) = log pθ ( YL|A , X ) −DKL ( qϕ ( YU |A , X , YL ) ||pθ ( YU |A , X ) ) , ( 4 ) where DKL ( ·||· ) represents the Kullback-Leibler divergence between two distributions . For practical use , it remains to specify the parametric forms of qϕ ( YU |A , X , YL ) and pθ ( Y|A , X ) with GNNs .

Summary: For node classification tasks with extremely limited labeled data, this paper proposes a new GNN architecture GPN that extends Poisson learning to incorporate graph structure information and its contrastive learning method. The goal is to propagate the limited labels to the entire graph effectively to enhance classification performance. The framework consists of GNN for $p_\theta(Y|A, X)$ and GPN for $q_\phi(Y|A,X,Y_L)$ which are trained with ELBO loss and contrastive loss. GPN, inspired by GAN, is instantiated to learn attention-based edge weights used to construct a new diagonal matrix and graph Laplacian for Poisson learning.