Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels
Authors: Sheng Wan
Yibing Zhan
Liu Liu
Baosheng Yu
Shirui Pan
Chen Gong
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Machine Learning

Abstract: Graph Neural Networks (GNNs) have achieved remarkable performance in the task of semi-supervised node classification. However, most existing GNN models require sufficient labeled data for effective network training. Their performance can be seriously degraded when labels are extremely limited. To address this issue, we propose a new framework termed Contrastive Graph Poisson Networks (CGPN) for node classification under extremely limited labeled data. Specifically, our CGPN derives from variational inference; integrates a newly designed Graph Poisson Network (GPN) to effectively propagate the limited labels to the entire graph and a normal GNN, such as Graph Attention Network, that flexibly guides the propagation of GPN; applies a contrastive objective to further exploit the supervision information from the learning process of GPN and GNN models. Essentially, our CGPN can enhance the learning performance of GNNs under extremely limited labels by contrastively propagating the limited labels to the entire graph. We conducted extensive experiments on different types of datasets to demonstrate the superiority of CGPN. 

Contrastive Graph Poisson Networks: Semi-Supervised Learning with Extremely Limited Labels
Sheng Wan Yibing Zhan', Liu Liu", Baosheng Yu', Shirui Pan", Chen Gong
IPC Lab, Key Lab of Intelligent Perception and Systems
High-Dimensional Information of Ministry of Education
2 Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering. Nanjing University of Science and Technology
"JD Explore Academy 'The University of Sydney
5 Department of Data Science and AI, Faculty of IT, Monash University wansheng3150ho tmaiihhoommll....mmmm com, zhanyibing@jd com, {liu. liul, baosheng. yu @sydney edu au, shirui paniiiippmmmonaas edu,, chen. leen.gonenjjuse edu,
Abstract
Graph Neural Networks (GNNs) have achieved remarkable performance in the task semi-supervised node classification. However, most existing GNN models require sufficient labeled data for effective network training. Their performance can be seriously degraded when labels are extremely limited. To address this issue, we propose new framework termed Contrastive Graph Poisson Networks (CGPN) for node classification under extremely limited labeled data. Specifically, our CGPN derives from variational inference; integrates newly designed Graph Poisson Network (GPN) to effectively propagate the limited labels to the entire graph and normal GNN, such Graph Attention Network, that flexibly guides the propagation of GPN; applies a contrastive objective to further exploit the supervision information from the learning process of GPN and GNN models. Essentially, our CGPN can enhance the learning performance of GNNs under extremely limited labels by contrastively propagating the limited labels the entire graph. We conducted extensive experiments on different types of datasets to demonstrate the superiority CGPN.
1 Introduction
Graph-based Supi-Super Learning (SSL) refers to classifying unlabeled data based on handful of labeled data and given graph structure indicating the connections between all data. Recently, graph-based SSL. has attracted increasing attention due its solid mathematical foundation, and
satisfactory performance [ 2 3)
As the mainstream solve graph-based SSL problems, Graph Neural Networks (GNNs), which operate in the graph domain, have achieved impressive performance in recent years [4] 5 6 77. Nevertheless, current GNNs, such as Graph Convolutional Networks (GCNs) g and graph attention networks (GATs) [9. require sufficient labeled data to obtain satisfactory generalization abilities. Unfortunately, the reliance on sufficient labeled data increases the burden of data collection, and the number of labels can be extremely limited in some real-world scenarios. The performance of most
"This work was done when Sheng was research intern at JD Explore Academy.
fC. Gong is the corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

eurrent GNNs seriously declines as the label size shrinks, since the scarce supervision signals are insufficient to train model with satisfactory discriminative ability, see Figure [] To the best of our knowledge, few studies have focused on semi-supervised classification with GNNs at extremely low label rates 10].
In line with the aforementioned observations, this paper proposes a new framework termed Con- trastive Graph Poisson Networks (CGPN) to ad- dress the problem of semi-supervised node clas- sification under extremely low label rates. Deriv- ing from the variational inference [12 13. our
80
Accuracy (%
70
60
GCN
proposed CGPN framework approximates the in- tractable posterior with a surrogate distribution, where two types of GNNs have been adopted for instantiation.
50
GAT
CGPN (ours)
40
3
2
Accordingly, we first design new Graph Poisson Network (GPN) to propagate the limited labels the entire graph effectively. Specifically, our GPN motivated by Poisson learning [4]; flexibly models the Poisson label propagation based on attention mechanism; and leverages the structural information to better guide the propagation of labels. Meanwhile, we exploit another GNN, such as GAT, together with the proposed GPN model the approximated posterior according to variational inference.
Number of labels per class
Figure I: Classification performance of GCN, GAT, and our proposed CGPN with different sizes of labeled data Cora dataset.
On this basis, we acquire predictions from two comparable views, where contrastive objective can be naturally incorporated to jointly refine the learning process of the GPN and GNN models. Moreover, the supervision signals implicitly contained in the massive unlabeled data can be exploited with the formulated contrastive loss. As result, the model learning ability of our proposed framework can be lifted. Experimental results on benchmark datasets confirm the strong benefits of our proposed CGPN when dealing with remi-supervised node classification at very low label rates.
In summary, the contributions of this paper lie in three folds:
First, we propose a novel GNN framework termed CGPN to solve the semi-supervised node classifi- cation with extremely limited labels. CGPN significantly outperforms the existing GNNs.
Second, we design new Graph Poisson Network (GPN). Different from the Poisson learning algorithm, our GPN incorporates graph-structure information and could be trained in an end-to-end manner to guide the propagation of labels more flexibly.
Third, we integrate contrastive learning into the variational inference framework, so that extra supervision information can be explored from the massive unlabeled data to help train CGPN framework.
Problem description
We start by formally introducing the problem of graph-based SSL. Given a set of n = 1+ u examples V = (xi,-.., X1, X1+1, )... where the first 1 examples are provided with the labels { | and the remaining u examples constitute the unlabeled set with typically < u. Let X E Raxd denote the feature matrix with the i-th row formed by the feature vector x,, and Y guxe denote the label matrix with its (ih element Y., = 1 if x; belongs to the j-th class and 0 otherwise. Here d is the feature dimension and is the number of classes. The dataset & is represented by an undirected graph g = EV,E), where V represents the node set containing all examples and E is the edge set modeling the similarity among the nodes/examples. The adjacency matrix of G is denoted as A, where Aij = 1 if there exists an edge between x; and xj and Aij O. otherwise. In this paper, we target transductive graph-based SSL which aims to find the labels 1t+ 24 "., ya the unlabeled examples X1+1, X1+2."" Xn based on V.

---

3 Related work
3.1 Graph-based semi-supervised learning
SSL methods focus on training models with small amounts of labeled data as well as relatively large amounts of unlabeled data for [15 T6
17 18 19). Graph-based SSL algorithms have been one of the most popular research topics during the past decades. The early graph-based techniques are designed based on the simple assumption that nearby nodes are likely to have the same label. This goal can be achieved through the low-dimensional embeddings with Laplacian eigenmaps [20 21,, Markov random walks [22]. etc. Meanwhile, graph partition (23 offers another important line in graph-based SSL. To further enhance the learning capacities, various techniques have been proposed to model the data features and graph structure jointly, such Planetoid (4). where the supervised classifier is regularized with a Laplacian regularizer or an embedding sabbedding-based regularizer. Recently, set of graph-based SSL approaches have been proposed improve the performance of the above-mentioned techniques, including [25] 14 26
3.2 Graph neural networks
In the past few years, increasing attention has been paid to GNN models [27 28 26
29
30
31
32
Early-staged works aim to derive diverse types of graph convolution in
based on the graph spectral theory 8 33 4. In [33,, a general graph convolution framework based on graph Laplacian first proposed. Afterwards, Defferrard et al. 44 approximate the convolutional filter via using K-order Chebyshev polynomial, in order to avoid the intense calculations of cigendecomposition of the normalized graph Laplacian. In addition to this, Kipf and Welling [8] further simplify the graph convolution by localized first-order approximation, which brings about more efficient filtering operations than spectral CNNs. Another line of research efforts focus on directly performing graph convolution in the spatial domain [5] 27 34 (35] In spatial GNN models,
the convolution operation is defined as a weighted average function over the neighbors of each node, which characterizes the impact exerting to the target node from its neighboring ones. For instance, Hamilton el al. [5] propose general inductive framework called GraphSAGE, which is able to learn an embedding function generalizing to previously unseen graph nodes. Besides, in [9], the graph attention network (GAT) is devised by utilizing the attention mechanism, which assigns different weights to the neighboring nodes and aggregates feature with discrimination. Although these models exploit the inter-depente denendeeens among labeled and unlabeled nodes, their performances can still degrade dramatically when the number of labels is extremely limited.
3.3 Poisson learning
Poisson learning is motivated by the need to address the degeneracy of Laplacian SSL when label information is very limited. To be concrete, traditional Laplacian learning algorithm [15] aims at solving the following problem:
Lu(x,) = O, if+tkiin u(x.) ya. if
where L represents the unnormalized graph Laplacian and u(x;) E RRC the label vector of x; Here, the node x; belongs to the j-th class if the j-th component (x;) ) is the largest in u(x;). Although Laplacian learning works very well for SSL tasks with a moderate number of labeled examples, the performance becomes quite poor at very low label rates.
Different from Laplacian learning, where labels are imposed boundary conditions, in Poisson leamiing labels appear as the source term. The solution of Poisson equation can be computed as
Lu(x,)= = Σ (u, F)5. for i == 1.....
(2)
satisfying Ei d,u(x,) = 0, where d.j = 1 i j and 5.j = 0, otherwise, y = FF with F E Rexx denoting the label matrix of the I labeled nodes, and d, = Σÿ=1 Aij. The Poisson (2) can be solved efficiently with simple iteration, and the result of the (1 1)-th iteration can be obtained as
equation
((+11) ( D ( -UU(),
(3)
3

---

where D is the diagonal matrix with D; = d4 L is the Laplacian matrix, and B = [ y. O] is the source term. Here, O E [ex((--) indicates a zero matrix. After I' iterations, the prediction can be acquired as U The specific interpretation of Poisson learning algorithm can be found in [14].
4 Methodology
This section details our proposed CGPN framework. Specifically, we describe the critical components of CGPN by explaining the variational inference framework, presenting the instantiation with GPN and GNN models, and illustrating the contrastive label inference.
4.1 Inference framework
To infer the labels of the unlabeled nodes, i.e., Yu, we need to estimate the posterior distribution given the node features X, the observed labels YL. and the adjacency matrix A, namely PY [c|AA, X, YL) with parameters 0. Computation of this posterior is usually analytically intractable, so we resort to approximate posterior inference methods. Inspired by the recent advances in scalable variational inference [36 37, we introduce distribution (s Y& A, X, YL) parameterized by d to approximate the true posterior Po (Yyu X, Y2). Afterwards, we can write the Evidence Lower BOund (ELBO)
as
(ELBO(O, log po Y r/A, X) DaI A, X, Yr)l Yu],,
where (-I-) represents the Kullback-Ledd Leibler divergence between two distributions. For practical use, it remains to specify the parametric forms of qo (Yul A, X. Yr) and pe po AA with GNNs.
4.2
Instantiations
In this section, the instantiation of go( (Y|A, X, Yt) and (Y(AA, will be illustrated.
4.2.1
Instantiation of do (Y|A,, Y.) 1.) with Graph Poisson Networks
To approximate the posterior model go( (Y|A,, X, Yr.), we need a strong function with the inputs A, X, and Y, and outputs the probability of Y. Due to the extremely scarce label information in YL. most existing methods are ineffective here. Fortunately, the Poisson learning algorithm [14 is recently proposed to address the scenarios with very limited labels. The superiority of Poisson learning over traditional Laplacian learning has been proven both theoretically and experimentally at very low label rates [14]. However, the graph structure has not been fully leveraged to guide the propagation of labels in Poisson learning. Concretely, Poisson learning relies on a fixed graph which can be noisy in reality, and thus the intrinsic relationships among graph nodes cannot be well explored. Meanwhile, the structural information constituted by the neighboring node features has not been exploited, since Poisson learning mainly emphasizes the propagation of the input label information. As consequence, inaccurate label predictions can be accumulated with iterative propagation, which inevitably results in performance degradation. To handle these difficulties, propose more flexible GNN model called 'Graph Poisson Networks' (GPN).
Inspired by GAT 19), we intend to adaptively capture the importance of the neighbors exerting to the target node via attention mechanism. In this way, the graph information can be gradually refined via network training, which makes the propagation of labels more reasonable. To be specific, we first compute the attention coefficient Gij between nodes x; and x, as
Eij a [x,; Wxy],
where ä is a trainable weight vector, W is trainable weight matrix, and [: denotes the concatena- tion operation. The attention coefficient ei usually normalized across the neighbors of x; with softmax function to make it comparable across nodes:
exp(ci))
ai ΣREN, exp(ew)
(6)
where N; denotes the indices of x, neighbors. In our GPN, the normalized attention coefficient Oss is adopted to represent the edge weights of the input adjacency matrix A and can be further optimized

---

via network training. In this way, improved edge weights can help guide the propagation of labels in a more reasonable way. At this point, the output of our GPN can be obtained by the following iteration:
i -(ー)) + D-1 B LÚ
(7)
where D and L. represent the diagonal matrix and Laplacian matrix of the attention-based graph, respectively, and ( is the result of the f-th iteration.
Note that Eq. (7) fails to incorporate the structural information formed by the node features, as the prediction in each iteration mainly relies on the label information. To capture the meaningful structural information, a feature transformation module fFT is adopted to introduce feature information explicitly. Specifically, the feature transformation module fFT predicts labels based the node features and can be expressed as single-layer perceptron. Afterwards, by propagating the predictions of fET iteratively, the feature information within the neighborhood structure can be incorporated correspondingly and further improve the final predictions. Therefore, we modify the output of the (T 2)t iteration, namely U()-2), to
(TT-2) Ú(T-2) + fFT(X),
(8)
where 7' denotes the number of iterations. By doing so, the involved feature information can be further propagated and the extracted structural information can help refine the label predictions. In addition, the feature transformation module can also accelerate the convergence of the iterative process shown in Eq. (7). Note that the outcomes of fFT can only be propagated in the last two iterations to avoid performance degradation which is caused by over-smoothing.
4.2.2 Instantiation of po Y A, X) with graph neural networks
For po( A, X) in the ELBO obejective function, can flexibly instantiate it with a GNN model that takes A and X as the inputs and outputs the probability of Y. Note that most of the existing GNN models are applicable. In this paper, the well-known Graph Convolutional Networks (GCN) []] and GAT [9] are considered for instantiation. Note that when we employ GAT to instantiate Po || X)), the attention coefficients are shared across GPN and GAT, so that the scale of network parameters can be reduced. Additionally, the GAT can help guide the propagation of labels through the shared attention coefficients.
4.3 Contrastive label inference
Furthermore, we intend to leverage the supervision signals beyond the limited labels. In this paper, contrastive learning (3 39 is utilized explore extra supervision information from the massive unlabeled data for model training, which can improve the performance of label inference. To be specific, we maximize the agreement between the predictions of the same node that are generated from 4s (Y|A, X, Y)) and pa ||A,
(|A,X) X), i.e., z, and 2. Meanwhile, we pull the predictions of different node pairs away, As result, the pairwise contrastive loss between z, and ž; can be defined as
exp( (z, 2,) /T)
Cpc 2,)
=llog
exp( (zi, 2,) //T)
Σ
1 ロ,ز exp( (z, ž,) /T) Lax exp( 2,) /T)
where (,,-) denotes the inner product and is a tunable temperature parameter. Based on Eq. 9. the overall contrastive objective be minimized is
LCont Σ (Lpo(zi, Lpo(Zi,
(10)
In addition the contrastive loss, a standard multiclass softmax cross-entropy loss Lce should also be applied to penalize the difference between the outcomes of gs (Y A, X, YL) and the ground-truth labels, i.e., Zz and Yz. Hence, by assigning the weight hyperparameters A and X2 to Leg and Lcont correspondingly, we arrive at the total loss as
L(),0) LELBOO o) + ALL YL) + AgCConts
(11)
5

---

Variational inference
Contrastive label inference
Graph Poisson Networks
Graph Neural Networks BAY
ELBO loss
Contrastive
loss
q (Y (XXX)
Figure 2: The framework of our Contrastive Graph Poisson Networks (CGPN). do (Y), Yz) and po (Y|A X) are instantiated by GPN and GNN models, respectively. To approximate the true posterior, the GPN and GNN models are jointly optimized based the ELBO objective, where contrastive loss is also utilized to explore extra supervision signals for stable model training.
Table I: Dataset statistics
Datasets
Nodes
Edges
Features
Classes
Cora
2,708
5,429
1,433
7
CiteSeer
3,327
4,732
3,703
6
PubMed
19,717
44,338
500
3
Amazon Photo
7,650
119,081
745
8
where Z, generated by the instantiated model (o(Y|A, X, Y,) denoting the prediction of the labeled examples Xz. With the contrastive objective across the GPN and GNN models, the mutual information can be exploited to improve the ability of label inference. The overall framework is referred to Contrastive Graph Poisson Networks (CGPN), of which the overview is exhibited in Figure
Experiments
To reveal the effectiveness of our proposed CGPN framework, extensive experiments have been conducted the task of semi-supervised node classification. We mainly focus evaluating the model performance under label-scarce settings.
5.1 Experimental settings
Datasets. The experiments are conducted on four commonly used benchmark datasets, including three widely-used citation networks (i.e., Cora, CiteSeer, and PubMed) [II] 40), and one Amazon product co-purchase networks (i.e., Amazon Photo) [11. the citation networks, the nodes represent documents and their links refer to citations between documents, where each node is associated with a bag-of-wordh feature vector and a ground-truth label. In the product co-purchase networks, the nodes represent goods and the links indicate that two goods are frequently bought together, where node features are bag-of-words encoded product reviews, and class labels correspond to the product categories. The dataset statistics are summarized in Table
Baselines. When we evaluate the performance different methods in the label-scarce settings, six state-of-the-an models are used for comparison, including GCN (8) GAT (9, Bayesian Graph Convolutional Neural Networks (BGCN) [42). Multi-View Graph Representation Learning (MVGRL) 43 Generalized PageRank Graph Neural Networks (GPRGNN) 44 and Approximate Personalized Propagation of Neural Predictions (APPNP) 45). BGCN enhances the learning ability of GNNs by modeling the uncertainty the graph structure. MVGRL explores the supervision information contained in the unlabeled data by contrasting the encodings from two structural views of graphs. GPRGNN jointly optimizes node feature and topological information extraction to obtain excellent learning performance for label patterns. APPNP leverages large and adjustable neighborhood for convolution by using the relationship between GCN and PageRank. Besides, we also include the Multi-Layee Perceptron (MLP) without using any graph information competitor, where the

---

Table 2: Classification accuracy with different label rates on Cora dataset
# Labels per class
2
3
MLP
26.25:1.005
32.1941.94
36.20
38.92:22.59
GCN
44.73:19.16
58.73:17.29
62.38:15.59
67.4111.009
GAT
9
46.04:18.07
62.01:17.29
63.65:16.04
68.13
BGCN
42
49.92=#.72
64.55=6.88
64.98-6.61
71.69=6.62
MVGRL
43
56.0227.044
68.3014.86
71.39=5.08
73.79=4.29
GPRGNN
44
51.65#11.61
62.56:16.36
68.5447.28
71.69=6.62
APPNP
45
53.52+12.05 62.07+4.46
62.02=9.34
70.92
CGPN-GCN
60.6419.18
73.5412.76 69.1519.03
74.43+2.25
CGPN-GAT
61.17+7.77
69.93+7.01
73.19+4435
75.60=
Table 3: Classification accuracy with different label rates on CiteSeer dataset
Labels per class
MLP
26.10:1:.00
30.34=3.74
35.99:13.42
38.44:14.60
GCN
8
32.00+19.77
43.11
50.44+5.88
56.14
GAT
35.10:8.77
44.74-9.01
53.68=15.75
19.43=
BGCN
42
35.1648..04
46.48=5.70
55.49
58.99:5.26
MVGRL
43
42.65.17.89
56.66
61.70:1.70
63.70:+2.33
GPRGNN
44
32.30::11.41
46.36
52.60
59.59
APPNP
45
10.44
36.59-9.22
58.61+11.04
62.34
CGPN-GCN
50.49-99.72
58.45 7.05
3.707+3.76
64.79+2.11
CGPN-GAT
52.68
58.52 6.16
62.02+33888
6..21+2.76
number of layers set to two. For the proposed CGPN framework, we implement two model variants where GCN and GAT are used for instantiation, namely CPGN-GCN and CGPN-GAT.
Training details. For all the adopted datasets, we randomly choose one, two, three, and four labeled nodes per class for training, respectively, in order evaluate the model performance under label-scarce settings. The hyperparameters, such as the number of hidden units and the learning rate, determined via grid search. In experiments, the original architecture of GCN is adopted in both the baselines and CGPN-GCN. In CGPN-GAT, the attention coefficients are shared between GPN and GAT, where only the single-head attention mechanism utilized for simplicity. The experiments are conducted on a Linux server equipped with a Tesla P40 GPU.
5.2 Node classification results
Here, we present the classification results of our proposed CGPN framework (CGPN-GCN and CGPN-GAT) and the baseline methods at different label rates. The experimental results on Cora, CiteSeer, PubMed, and Amazon Photo datasets are shown in Tables 2 and 5 respectively, where the highest record at each label rate is highlighted in bold. We observe that both CGPN-GCN and CGPN-GAT achieve substantial performance gains at different label rates when compared with the baselines. In particular, the margin between our proposed framework and the best baseline method can exceed 4% on Cora and CiteSeer datasets given one labeled node per class, which demonstrates that the CGPN framework could effectively enhance the learning performance of GNNs. Although MVGRL utilizes the contrastive objective for graph representation learning, our proposed CGPN-GCN and CGPN-GAT exhibit better classification results. We hypothesize that the gap between MVGRL and our CGPN in accuracy is mainly due to that the devised GPN transfers as much knowledge as possible from the limited labeled nodes to the massive unlabeled ones. Notably, we could also find that the performance of the two variants, i.e., CGPN-GCN and CGPN-GAT seems quite similar since CGPN is a flexible framework and is insensitive to the choices of GNN instantiations.

---

Table 4: Classification accuracy with different label rates on PubMed dataset
# Labels per class
2
3
MLP
4..84
53.24114.73
57.09::4.82
60.29
GCN
55.40:16.69
62.31
64,48:1.557
69.34:1..19
GAT
9
56.49:1.81
63.03:15.77
64.43::.119
68.4742.98
BGCN
42
57.97=6.56
63.86=#..41
66.09:3.86
68.10+3.20
MVGRL
43
52.20+12.93
60.7948.74
64.5118.06
67.6947.67
GPRGNN
44
10.26
61.30:6.59
64.77:8.05
69.73+4.50
APPNP
45
48.89
63.7447.55
66.8219.76
69.27
CGPN-GCN
69.8416.76 5.66
64.6043.20 63.18
70.9414.11 67.0114.05
CGPN-GAT
68.67+4.36
70.58
Table 5: Classification accuracy with different label rates on Amazon Photo dataset
# Labels per class
2
3
4
MLP
36.8419.12
42.67=5.11
55.96:17.02
55.54
GCN 8
10.80
78.42:1..22
82.53:15.07
83.37#2.61
GAT
9
60.68:12.93
74.5916.96
78.7011..78
81.6413.04
BGCN
42
53.69=11.433
69.76=12.67
78.26:18.23
79.66:14.49
MVGRL
43
59.7118.29
71.35+7.12
75.56:15.29
76.7444.63
GPRGNN
44
14.92
75.28+8.20
81.5943.10
82.10
APPNP
45
67.11
70.03=11.51
78.9913.23
81.17=4.03
CGPN-GCN
71.1718.94
79.5415.118
84.0943.98
84.43
CGPN-GAT
11.25+11.25
78.58455.48
84.57+22.24
85.41+1.89
5.3 Parametric sensitivity
There are two important hyperparameters that should be manually tuned in our objective function Eq. (1T). The first is Å₁ used to regularize the cross-entropy loss of GPN. Another is X2 assigned to the contrastive objective. By varying a and X2 from 0.1 to 1.5 with interval of 0.2, the corresponding parametric sensitivity is shown in Figure 3 We find that the behavior of the proposed CGPN framework is relatively stable with the change of X and A2. We speculate that GPN and contrastive modules are mutually beneficial and can collaborate to obtain promising results. Generally, the two hyperparameters not difficult to tune in practical applications.
0.7
0.6
Accuracy
0.6
0.5
Accuracy
0.5
0.4
0.4
0.3
0.3
0.2
0.1
0.3
0.5
0.7
0.9
1.1 1.3
1.5
0.1
1.5
(a) Cora
(b) CiteScer
Figure 3: Parametric sensitivity of x and A2-
5.4 Ablation study
Inspired by the Poisson Learning (PL. algorithm, the devised GPN model learns to propagate limited labels to the entire graph with structural information in end-to-end way. To shed light on
this critical component, we report the classification results obtained by using GPN and PL. in Table
and 7 where the sizes of labeled nodes per class are kept identical to those in Sec. 5.2
The results

---

Table 6: Classification accuracy of reduced models on Cora dataset
# Labels per class
3
4
PL [4] GPN
46.23:17.96 89.26:18.66
60.8411.50 64.4511.009
63.24:1.584
65.59:15.36
66.90:15.85
67.40:15.81
CGPN-GCN CGPN-GAT
60.64:19.18 61.17
69.15:1..03 69.93:1.001
2.764 73.1914.35
74.43-12.25
1.660
Table 7: Classification accuracy of reduced models on Citeseer dataset
I Labels per class
3
4
PL 14
36.43++7.25 37.64
40.69++..87 48.2447.73
43.44-1-5.99
45.33++1.85
GPN
51.27+:5.56
53.40:+4.62
CGPN-GCN CGPN-GAT
50.49:19.72
58.45+7.05
62.07+3..76
64.79++.11
92.68
68.52
62.02-t:3.88
65.21
show that our GPN model can effectively boost the performance Poisson learning extremely limited label rates. Mcanwhile, the margin between GPN and CGPN frameworks cannot be ignored, which reveals the power of the extra supervision information explored from massive unlabeled data.
6 Conclusion
In this paper, propose novel CGPN framework for semi-supervised node classification with extremely limited labels. To alleviate the performance degeneracy of the existing GNNs, we devise model termed GPN, which can flexibly propagate the limited labels to the entire graph by exploiting the structural information. Meanwhile, a contrastive objective is employed to extract supervision information from massive unlabeled data. As consequence, the performance of our CGPN can be effectively enhanced by optimizing the overall objective function. Experimental results reveal the superiority of method when compared with various baseline methods. Note that how to generalize our proposed method to inductive settings will be an interesting focus of study in the future.
7 Broader impact
Our work could have the following positive impacts: (1) This paper provides a new framework for SSL on graph-structured data. (2) Our proposed framework can improve the performance of GNNs, especially when labeled data are extremely limited. (3) The proposed framework is compatible with different types of GNNs (e.g., GAT and GCN).
Similar to many other machine learning algorithms, the proposed framework can be used for good and also be used for harm at the same time. Note that GNNs and our method are not immune such misuse. Although we have no optimal solution to such problem, we believe that this can be solved in the future.
To sum up. we believe our proposed work can be beneficial to society since many important real-world applications stand to benefit from CGPN when label information is very limited.
8 Acknowledgments
This research was supported in part by NSF of China (Nos: 61973162 and 62002090), NSF of Jiangsu Province (No: BZ2021013), the Fundamental Research Funds for the Central Universities (Nos: 30920032202, 30921013114) CCF-Tencent Open Fund (No: RAGR20200101) Hong Kong Scholars Program (No: XI2019036), and Australian Research Council project (Nos: DP- OP-100033424 and FL- 1L-170100117).

---

References
[1] Xiaojin Zhu, John Lafferty, and Ronald Rosenfeld. Semi-supervised learning with graphs. PhD thesis, Carnegie Mellon University, language technologies institute, school of computer science, 2005.
[2] Chen Gong. Tongliang Liu, Dacheng Tao, Keren Fu, Enmei Tu, and Jie Yang. Deformed graph Laplacian for semisupervised learning. IEEE Transactions on Neural Networks and Learning Systems,
2015.
[3] Chen Gong, Dacheng Tao, Keren Fu, and Jie Yang. Fick's law assisted propagation for semisupervised learning. IEEE Transactions on Neural Networks and Learning Systems, 26(9):2148-216) 2014.
[4] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pages 3844-3852, 2016.
[5] Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pages 1024-1034, 2017.
[6] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint urXiv:1312.c0
2013.
[7] Sheng Wan, Chen Gong, Ping Zhong, Bo Du, Lefei Zhang, and Jian Yang. Multiscale dynamic graph convolutional network for hyperspectral image classification. IEEE Transactions Geoscience and Remote Sensing, 58(5):3162-3177, 2020.
[8] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017.
[9] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10900 2017.
[10] Wanyu Lin, Zhaolin Gao, and Baochun Li. Shoestring: Graph-based semi-supervised classifica- tion with severely limited labeled data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4174-4182, 2020.
[11] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93 93, 2008.
[12] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 14, 2013.
[13] Pantelis Elinas, Edwin Bonilla, and Louis Tiao. Variational inference for graph convolu- tional networks in the absence of graph data and adversarial settings. In Advances in Neural Information Processing Systems, volume 33, 2020.
[14] Jeff Calder, Brendan Cook, Matthew Thorpe, and Dejan Slepcev. Poisson learning: Graph based semi-supervised learning at very low label rates. In International Conference on Machine Learning, pages 1306-1316, 2020.
[15] Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. supersupe learning using Gaussian fields and harmonic functions. In International Conference on Machine Learning, pages 912-919, 2003.
[16] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint 1000,09000,,009,, 2020.
[17] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, volume 33, 2020.
10

---

[18] Viktor Olsson, Wilhelm Tranheden, Juliano Pinto, and Lennart Svensson. Classmix: Segmentation-baat based data augmentation for semi-supervised learning. Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1369-1378, 2021.
[19] Jing Zhang and Dacheng Tao. Empowering things with intelligence: A survey of the progress, challenges, and opportunities in artificial intelligence of things. IEEE Internet of Things Journal, 8(10))778--7817, 2020.
[20] Mikhail Belkin and Partha Niyogi. Semi-supervised learning on Riemannian manifolds. Ma- chine Learning, 56(1):209-22) 2004.
[21] Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 7(11), 2006.
[22] Martin Szummer Tommi Jaakkola and Martin Szummer. Partially labeled classification with Markov random walks. Advances in neural information processing systems, 14:945-955 -925, 2002.
[23] Michael Speriosu, Nikita Sudan, Sid Upadhyay, and Jason Baldridge. Twitter polarity classifi- cation with label propagation over lexical links and the follower graph. In Proceedings of the First workshop on Unsupervised Learning in NLP, pages 53-63, 2011.
[24] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning, pages 40-48, 2016.
[25] Chen Gong, Jian Yang, and Dacheng Tao. Multi-modal curriculum learning over graphs. ACM Transactions on Intelligent Systems and Technology. 10(4):1-25 2019.
[26] Sheng Wan, Shirui Pan, Jian Yang, and Chen Gong. Contrastive and generative graph con- volutional networks for graph-based semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10049-10057, 2021.
[27] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint vrXiv:/810.0000
2018.
[28) Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chenggi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[29] Yu-Chi Chen, Kuan-Tine -ing Lai, Dong Liu, and Ming-Syan Chen. Tagnet: Triplet-attention graph networks for hashtag recommendation. IEEE Transactions on Circuits and Systems for Video Technology, 2021.
[30] Hongyang Gao, Yi Liu, and Shuiwang Ji. Topology-aware graph pooling networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[31] Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip Yu. Graph self-supervised learning: A survey, arXiv preprint
2021.
[32] Haibo Wang, Chuan Zhou, Xin Chen, Jia Wu, Shirui Pan, and Jilong Wang. Graph stochastic neural networks for semi-supervised learning. Advances in Neural Information Processing Systems, 2020.
[33] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and deep locally connected networks on graphs. In International Conference on Learning Representations, 2014.
[34] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In International Conference on Machine Learning, pages 2014-2023, 2016.
[35] James Atwood and Don Towsley. Difflusion-convllutionn neural networks. In Advances in Neural Information Processing Systems, pages 1993-2001, 2016.

---

[36] Jiaqi Ma, Weijing Tang, Ji Zhu, and Qiaozhu Mei. A flexible generative framework for graph- based semi- sumii-spervised learning. In Advances in Neural Information Processing Systems, pages 3281-3290 2019.
[37] Diederik P Kingma, Danilo J Rezende, Shakir Mohamed, and Max Welling. Semi- suni-supereised learning with deep generative models. In Advances in Neural Information Processing Systems, pages 3581-3589 2014.
[38] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006. 04131, 2020.
[39] Ming Jin, Yizhen Zheng, Yuan-Fang Li, Chen Gong, Chuan Zhou, and Shirui Pan. Multi-scale contrastive siamese networks for self-supervised graph representation learning. In International Joint Conference on Artificial Intelligence, 2021.
[40] Aleksandar Bojchevski and Stephan Günnemann. Deep Gaussian embedding of graphs: Unsu- pervised inductive learning via ranking. In International Conference on Learning Representa- tions, 2018.
[41] Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.0586. 2018.
[42] Yingxue Zhang, Soumyasundar Pal, Mark Coates, and Deniz Ustebay. Bayesian graph con- volutional neural networks for semi-supervised classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5829-5836, 2019.
[43] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pages 4116 4126, 2020.
[44] Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference Learning Representations, 2021.
[45] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference Learning Representations, 2018.
12


Reviews:
[{"review_id": "znosZgcqT6", "reviewer": "Reviewer_62ca", "summary": "This paper focuses on semi-supervised node classification with extremely limited labels and the authors propose a new framework termed Contrastive Graph Poisson Networks (CGPN). Specifically, they design a Graph Poisson Networks to propagate the limited labels to the entire graph and employ a Graph Neural Network (GNN) to guide the propagation. Meanwhile, labeled data and unlabeled data are also considered in the final objective function. For the labeled data, they simply use a cross-entropy loss and for the unlabeled data, they apply a pairwise contrastive loss to explore extra supervision information. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Since few studies have focused on semi-supervised classification with extremely limited labels, this paper makes an interesting attempt. The authors propose Graph Poisson Networks (GPN) to avoid the performance degradation caused by the accumulation of inaccurate label prediction with iteration propagation.\n2. The proposed method contains several specially designed parts to play different roles, which well handle the given data. This method is novel to me.\n2. Experimental results demonstrate the effectiveness of the proposed method.\n4. This paper is clearly written and well-organized.\n\n\nCons:\n1. In section 3.3, the authors employ a feature transformation module $f_{FT}$ to modify the outputs of the last two iterations. However, it is not clear whether $f_{FT}$ works.\n2. In line 181, \"in the last two interations\". Should it be “… last two iterations …”? The authors may further re-check the typos of this paper.\n3. There are only tables on Page 8. The authors may avoid that by adding some words to this page.\n\nOverall, I think this is a decent paper with a novel method that can be accepted.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Hbt5OJR2i7I", "reviewer": "Reviewer_u74h", "summary": "In this paper, authors propose the Graph Poisson Network (GPN) to  effectively propagate the information of limited labels to the entire graph, which is developed for semi-supervised node classification. The main contributions are as follows.\n\nFirstly,  they integrate contrastive learning and variational inference framework into Poisson learning algorithm. This novel combination can propagate the label information more flexible.\n\nSecondly, as they claimed, they achieve a significant improvement on semi-supervised node classification tasks.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation and method of this paper is interesting. They consider the graph neural network, variational inference, and the thought of contrastive learning simultaneously, and unify them with poison learning algorithm for better semi-supervised node classification tasks.\n\nHowever, there are some problems in the paper that I think it still has a long way to publish in NeurIPS.\n\nThe first problem is writing. Although the authors use the mathematical views to describe their method, the notations are confused. For example, in Eq (1) in Sec. 2.3, what does $l$ and &n& means? It seems that they define them in 3.1, but I suggest that the authors can reorganize the mathematical descriptions for better reading. Also, what does $U$ in Eq. (3) means?\n\nThe second problem is the derivations. Eq. (4) is the most important loss in this paper. But the authors do not tell us how to derive the ELBO. You mean ELBO, I think you should point out what’s the ELBO? Does it the ELBO of label likelihood like $p(Y)$?\n\nThe third problem the contrastive label loss in Eq. (9). I didn’t find the definition of $z_i$ and $z_j$. Besides, the contrastive loss pull the predictions of different node pairs away. If different node pairs has the same label, does it make sense? I think you should pull the predictions of node pairs that has different labels.\n\nThe fourth problem is the experiments. I just see some simple numerical results about performance without any other explanations to show the advantages of their method. It is better to provide more insights about what does this model learn rather than simple numerical results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "Blr5QBBi-su", "reviewer": "Reviewer_yD2p", "summary": "For node classification tasks with extremely limited labeled data, this paper proposes a new GNN architecture GPN that extends Poisson learning to incorporate graph structure information and its contrastive learning method. The goal is to propagate the limited labels to the entire graph effectively to enhance classification performance. The framework consists of GNN for $p_\\theta(Y|A, X)$ and GPN for $q_\\phi(Y|A,X,Y_L)$ which are trained with ELBO loss and contrastive loss. GPN, inspired by GAN, is instantiated to learn attention-based edge weights used to construct a new diagonal matrix and graph Laplacian for Poisson learning. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main experimental results and ablation study show the proposed method effectively propagates limited labeled data.\nUsing both variational loss and the contrastive loss for the output of GNN $p_\\theta$ and GPN $q_\\phi$ is novel and interesting.\nAfter reading this paper, I found that the notion of propagating some information is similar to APPNP [1].  It would be nice to compare the performances of APPNP in the extreme case covered by this paper.  \nAdditionally, the datasets on which this paper experiments are assortative so that propagating label information to the neighbors can be effective. Providing the experimental results on disassortative datasets such as Texas, Wisconsin, Actor, Squirrel, Chameleon, and Cornell would also be useful. \n\n\nThe followings are some questions for several unclear things. \n- In Eq. (1) & (2), isn't  $u(x_i) \\in \\mathcal{R}^c$ and $L$ is an nxn matrix? How is their matrix multiplication defined?\n- How many iterations we need for Eq. (7)? Doesn't it take a long time if large T is required?\n- Ablation study for Eq. (8)\n- Qualitative analysis for learned attention would be interesting\n\n[1] PREDICT THEN PROPAGATE: GRAPH NEURAL NETWORKS MEET PERSONALIZED PAGERANK, ICLR2019", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The negative social impact of this paper does not seem to be a concern.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the issue of doing label propagation on a large graph where ground truth labels are extremely sparse.  The authors purpose a hybrid strategy using poisson networks and GNNs to learn a label propagation rule.  The reviewers agree that this is a well motivated and timely paper, however the original draft was light on experiments and the reviewers wanted to see a range of additional studies.  Several of these studies were provided during the rebuttal phase.  I have reviewed these studies myself, and I feel they add some insights to the paper and should be added in the camera ready.  There is still one remaining reject vote, but I think the criticisms were largely addressed in the rebuttal.  One criticism in particular, that the authors did not spent enough time introducing the ELBO, is reasonable but I do not consider it a fatal flaw.

Decision: Accept (Poster)