InfoGCL: Information-Aware Graph Contrastive Learning
Authors: Dongkuan Xu
Wei Cheng
Dongsheng Luo
Haifeng Chen
Xiang Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Representation Learning
Contrastive Learning
Information Bottleneck

Abstract: Various graph contrastive learning models have been proposed to improve the performance of tasks on graph datasets in recent years. While effective and prevalent, these models are usually carefully customized. In particular, despite all recent work create two contrastive views, they differ in a variety of view augmentations, architectures, and objectives. It remains an open question how to build your graph contrastive learning model from scratch for particular graph tasks and datasets. In this work, we aim to fill this gap by studying how graph information is transformed and transferred during the contrastive learning process, and proposing an information-aware graph contrastive learning framework called InfoGCL. The key to the success of the proposed framework is to follow the Information Bottleneck principle to reduce the mutual information between contrastive parts while keeping task-relevant information intact at both the levels of the individual module and the entire framework so that the information loss during graph representation learning can be minimized. We show for the first time that all recent graph contrastive learning methods can be unified by our framework. Based on theoretical and empirical analysis on benchmark graph datasets, we show that InfoGCL achieves state-of-the-art performance in the settings of both graph classification and node classification tasks.

InfoGCL: Information-Aware Graph Contrastive Learning
Dongkuan Xu
Wei Cheng"
Dongsheng Luo' Haifeng Chen² Xiang Zhang'
'he Pennsylvania State University
NEC Labs America
1 [dux19, du12622 x edu
{weicheee cheng, haifeng) (errffengnnneeeeeeeee -eeeeeeecllll com
Abstract
Various graph contrastive learning models have been proposed to improve the per- formance learning tasks on graph datasets in recent years. While effective and prevalent, these models are usually carefully customized. In particular, although all recent researches create two contrastive views, they differ greatly view aug- mentations, architectures, and objectives. It remains an open question how to build your graph contrastive learning model from scratch for particular graph learning tasks and datasets. In this work, we aim to fill this gap by studying how graph information is transformed and transferred during the contrastive learning process and proposing an information-awan graph contrastive learning framework called InfoGCI. The key point of this framework is follow the Information Bottleneck principle to reduce the mutual information between contrastive parts while keeping task-relevant information intact at both the levels of the individual module and the entire framework so that the information loss during graph representation learning can be minimized. We show for the first time that all recent graph contrastive learning methods can be unified by framework. We empirically validate theoretical analysis on both node and graph classification benchmark datasets, and demonstrate that our algorithm significantly outperforms the state-of-the-arte
Introduction
Inspired by their success in the vision and language domains, contrastive learning methods have been wildly adopted by recent progress in graph learning improve the performance of a variety of tasks 22 13 22]. In nutshell, these methods typically learn representations by creating two augmented views of graph and maximizing the feature consistency between the two views. Inheriting the advantages of self-supervised learning, contrastive learning relieves graph representation learning from its reliance label information in graph domain, where label information can be very costly or even impossible to collect while molabeled/partially labeled data is common, such as chemical graph data (88) Graph contrastive learning methods have achieved similar (and even better) performance as compared to the equivalent methods trained with labels on benchmark graph datasets 42 13
Despite being effective and prevalent, existing graph contrastive learning models differ mostly in augmented view design, encoding architecture, and contrastive objective (refer to Table in Appendix for more comparisons). For learning task, usually requires substantial degree of domain expertise to carefully design and customize these modules for the specific dataset. For example, while both DGI [33] and InfoGraph (28 seek to obtain graph representations by maximizing the mutual information between patch-level and graph-level representations, they adopt different graph encoders, GCN [5] and GIN 38 respectively. mvgrl applies graph diffusion convolution to construct the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

BXRO
GNN
fi(-)
MLP
Vi
Zi
Poxitive Pair for Contrasting
(A.
GNN
50
MLP
B***
Zj
View Augmentation
View Encoding
Representation Contrasting
Figure 1: Graph contrastive learning approaches consist of three stages: 1) a graph G undergoes
view augmentation, (+)). ()()), obtain two semantically similar views, Vi. Vj- 2) the views are fed into view encoder networks, fa(-), f)(,, to extract latent representations, 2;, 2j- 3) the feature consistency between representations maximized optimize the objective function based on contrastive mode (c)(),, (y))), where (.),, c-) are aggregation operations applied to representations.
augmented view, while GCC 22 and GRACE 45 adopt subgraph sampling and graph perturbation, respectively.
The main question this paper attempts to answer is: how to perform contrastive learning for your learning tasks on specific graph datasets? However, answering this question is challenging. First, contrastive learning consists of multiple components, such as view augmentation and information encoding. For each of them, there are various choices. Numerous variations make it difficult to design models that are both robust and efficient. Existing graph contrastive learning approaches are carefully designed for different learning tasks different datasets, however, none of them studies the guiding principles for choosing the best components. Second, graph data has unique properties that distinguish it from other types of data, such as rich structural information and highly diverse distribution [33] 28 [O]] Thus, is desirable to design the contrastive learning model that fits your graph data properties, even without any domain knowledge of the data.
We propose to address these challenges via Information Bottleneck (IB) B1,, which provides crucial principle for representation learning. Specifically, IB encourages the representations to be maximally informative about the target the downstream task, which helps keep task-relevant information. Concurrently, IB discourages the representation learning from acquiring the task- iusk-irrelevant informa- tion from the input data, which is related to the idea of minimal sufficient statistics (27). However, different from the typical representation learning, there are two information flows involved in the two augmented views contrastive learning. Therefore, we extend the previous work [37 43 and propose InfoGCL, information contrastive learning framework for graph data.
To study how information is transformed and transferred, we decouple typical graph contrastive learning model into three sequential modules (as shown in Figure I]: view augmentation, view encoding, and representation contrasting. We further formalize how to find the optimal of the three modules into three optimization problems. To build the optimal graph contrastive learning model for the particular dataset and task, we argue that it necessary and sufficient to minimize the mutual information between contrastive representations while maximizing task-relevant information at the levels of both individual module and entire framework. Our work is also motivated by the InfoMin theory [30], which suggests that good set of views for contrastive learning in the vision domain should share the minimal information necessary to perform well at the downstream task. Beyond view selection, work extends InfoMin suggest principles of selecting view encodings and contrastive modes for graph learning considering the unique properties of graph data.
We suggest practically feasible principles to find the optimal modules in graph contrastive learning and show that all recent graph contrastive learning methods can be unified by these principles: the augmented views should contain as much relevatl information as possible, while they should share as little information as possible; ii) the view encoder should be task-relevant and simple as much possible; iii) the contrastive mode should keep task-relevant information as much as possible after contrasting. Besides, we also investigate the role of negative samples in graph contrastive learning and argue that negative samples are not necessarily required, especially when graph data

---

is not extremely sparse. Our proposed method, InfoGCL. is validated on a rich set of benchmark datasets for both node-level and graph-level tasks, where we analyze its ability to capture the unique structural properties of the graph data. The results demonstrate that our algorithm achieves highly competitive performance with up to 5.2% relative improvement in accuracy on graph classification task and competitive results on node classification task over the state-of-the-an unsupervised methods.
2 Related Work
Graph Contrastive Learning. Some recent research efforts in graph domain have been attracted by the success of contrastive learning in vision and language domains
A number of graph Despite all of them creating two views and targeting at maximizing the feature disagreement between the two views, these methods are carefully designed and differ in various aspects. Deep graph Infomax (DGI) [33] applied the InfoMax principle [8]] to graph data by contrasting the representations of node-level and graph-level for node classification tasks. Different from DGI, InfoGraph (28) aims at node classification tasks and it contrasts the representations of graph-level and substructure-level of different granularity. In addition, DGI and InfoGraph use different graph encoders to extract latent representations. mvgri [ studies both node and graph classification tasks. It transforms the adjacency matrix to a diffusion matrix and treat the two matrices as two congruent views. However, in GCC [22] and GRACE [45). subgraph sampling and graph perturbation are used to create the augmented views. GraphCL 42 explores the view augmentations approaches for graph contrastive learning. Specifically. it studies the approaches of node dropping, edge perturbation, attribute masking, and subgraph sampling. A recent work [44] also studies graph data augmentations. However, it focuses graph neural networks for node classification and does not study the contrastive learning framework. Our method differs from them. We aim to answer the question how perform contrastive learning for your graph data and tasks. Instead of carefully designing the architectures, we decouple typical graph contrastive learning into three stages and provide our InfoGCL principles analyze the optimality theoretically and practically.
contrastive learning approaches have been proposed [28 22 42 13]
Information Bottleneck. Our method is related to the Information Bottleneck (IB) theory 3I) which aims to find the best trade-off between accuracy and complexity when summarizing a random variable. IB has been recently used to study the deep learning approaches [26] 37 43). Specifically, IB expresses the trade-off between the mutual information measures I(D, Z) and I(Z. y) as
max IB 3 = (DD; Z) + (I(
where D. Z, y are the input, the latent representation and the task label, respectively. is a hyperparameter. In other words, IB aims to learn representation Z that is maximally expressive about y, while being minimally expressive about D. More recently, there are some efforts applying the IB theory to graph representation learning. [37] aims to generate both expressive and robust graph representations, while [43] studies the subgraph recognition problem. In contrast, our work focuses on graph contrastive learning, where the two augmented views make the optimization objective (information trade-off) different. Our work is also related to the idea of minimal sufficient statisties (27), which has been recently studied in the vision domain [30], claiming that good set of image views should share the minimal information necessary to perform well at the downstream task. Different from B0), we focus on the graph domain and propose three stages considering the unique properties graph data.
3 Preliminaries and Notations
Graph Representation Learning. A graph is denoted by 9= (A, X). A E RRxx is the adjacency matrix. X = is the node attribute matrix, where is the attribute dimension. In this work, we focus on both node-level and graph-level tasks. For node-level task, given graph G and the labels of subset of nodes, denoted by Y., the goal is learn the latent representation z, for each node v such that Zu preserves both network structures and node attributes, which can be further used to predict Y,. For graph-level task, given set of graphs G = (G), g², ... } and the labels of some graphs, denoted by Y, g. the goal is to learn the latent representation 2, for each graph such that 2g can be used predict Yg. Typically, the graph data is fed into graph neural networks (GNNs) to generate the representations, such Z, GNNs(G).
3

---

y
y
Vi
Vj
Vi
A
D
C
C
g
Figure 2: Illustration of optimal views. (Left) The relationships between graph G. two views, Vi. V,, and task y in terms of information entropy. A, B, C, D. E are overlapping areas. Two views are contained by graph because views are functions of graph. (Middle) A, B, become null when Eqs. (5) hold, which indicates the views and graph share the same amount of task-relevant information. (Right) C further becomes null when Eq. (3) holds, which indicates all the shared information between views is task-relevant, i.e., the views optimal.
Graph Contrastive Learning. Given an input graph, graph contrastive learning aims to learn the representations of graph nodes (for graph-level or node-level tasks respectively) through maximizing the feature consistency hetween two augmented views of the input graph via contrastive loss in the latent space. We decouple a typical graph contrastive learning model into three sequential modules.
(i View augmentation. Graph G undergoes data augmentation a(-) to obtain two views V,j, i.e. V; ~ 9:( and V, ~ (g()). A view is represented as graph data, such as V, = (A.,. X.,), where A, R2 and X,, Rnxd In practice, view augmentation approaches include node dropping, edge perturbation, subgraph sampling, etc.
(ii) View encoding. Graph-level or node-level latent representation extracted from views Vi Vj by using the view encoder networks f()) (a GNN backbone plus projection MLP), i.e., Z; (v; and 2j ~ fj (v₂). The two encoders might or might not share parameters depending on whether they are from the same domain.
(iii) Representation contrasting. Given the latent representations, contrastive loss is optimized to score the positive pairs 2, 2j higher compared to other negative pairs. Typically, the negative pairs are constructed from the augmented views of other graphs the same minibatch. The InfoNCE loss 21 has been adopted as one of popular contrastive losses, which is defined as:
exp(h(Z 2)
LNCE = -E log
(2)
N Ew= (1((, Zj,ne)
where h(-) is a contrasting operation score the agreement between two representations. Theoreti- cally, minimizing the InfoNCE loss equivalently maximizes lower bound on the mutual information between the views positive pairs. In other words, [(z,, 22) > log (N) LNCE, where I(-) measures the mutual information.
4 Information-Aware Graph Contrastive Learning
In this paper, we study how to perform contrastive learning for specific graph tasks and datasets. In particular, attempt answer the following questions for graph contrastive learning: (i) What the optimal augmented views? (ii) What is the optimal view encoder? (iii) What the optimal contrastive mode?
4.1 View Augmentation
The goal view augmentation is to create realistically rational data via the transformation approaches that do not affect the semantic label. Compared to the augmentation in other domains, graph view augmentation needs to consider the structural information of graph data, such as the node, the edge, and the subgraph. There are various graph view augmentation methods proposed recently. We follow

---

similar definition used in 42 to categorize four kinds of view augmentation approaches for graph data. Node dropping discards a certain part nodes along with their edges the input graph to create new graph view. Edge perturbation perturbs the connectivity in the graph via adding or dropping partial edges. Attribute masking masks part of node attributes and assumes that the missing attributes can be well predicted by the remaining ones. Subgraph sampling samples subgraph from the input graph. The rationale behind these approaches is that the semantic meaning of graph has certain robustness to graph perturbation.
The augmented views generated in the graph contrastive framework are typically used separate downstream task. To characterize what views are optimal for downstream task, we define the optimality of views. The main motivation is: the optimal augmented views should contain the most task-relevant information, and the information shared between views should only be task-relevane
Corollary 1. (Optimal Augmented Views) For downstream task T whose goal is to predict semantic label y, the optimal views, v., V3, generated from the input graph G are the solutions to the following optimization problem:
(Vi vJ) = arg mig I V2) VaVi
(3)
s.l. T(v::))= I(vjit)
(4)
I(v.y)) = [(::))
(5)
This says that for the optimal graph views, the amount of information shared between them is minimized (Eq. (3), while the two views contain the same amount of information with respect to
y (Eq. (4). ). which also the amount of information that the input gprah contains about the task
(Eq. 5 ). The illustration of the optimal views is shown in Figure 2 and the proof is in the Appendix.
4.2
View Encoding
View encoding aims to extract the latent representations of nodes or graphs via feeding the data of two views into view encoder networks such that the generated representations preserve both structure and attribute information in the views. The view encoders are quite flexible in graph contrastive learning and typically they are GCN [5), GAT 32), or GIN 38 etc.
The representations extracted via view encoding are further utilized to optimize the objective function of contrastive learning. After well trained, the view encoders are used to generate the graph/node representations for downstream task. To characterize what encoders are optimal, optimality of view encoders for graph contrastive learning. The main motivation is: the representation generated by the optimal encoder for a view should keep all the shared information by the two contrastive views, meanwhile the kept information is all task-relevant.
define the
Corollary 2. (Optimal View Encoder) Given the optimal views, vi, vj. for downstream task T whose goal is to predict a semantic label y, the optimal view encoder for view vł is the solution to the following optimization problem:
J arg min [(J((V)):v))
(6)
ft s.t. [(f(v():vz) = 1(v]:
indicates that for the optimal view encoder, the amount of information shared between the optimal view and the extracted representation is minimized (Eq. (6)) while the information shared between the two optimal views kept after the encoding process of one view (Eq. 77. The illustration of the
optimal encoder is shown in Figure 3 and the proof is illustrated in the Appendix.
4.3 Representation Contrasting
To allow flexible contrasting for graph data, we consider contrastive modes similar to 133 contrastive mode is denoted by (c;(-), c-)),, where c;(+), cy(-) are the aggregation operations applied to the representations extracted by view encoders, The contrastive modes are unique to graph data because of the structural information inside a graph. Specifically, we consider five contrastive modes. In global-global mode, the graph representations from two views are contrasted. Thus, c((+), (.)+) are averaging aggregation operations in this mode. In local-global mode, contrast the node
5

---

y
y
D
D
Zi
Zi
Z
Figure 3: Illustration of optimal view encoding. (Left) The relationships between two (optimal) views, vi. vj. task y, and representation z, in terms of information entropy. A, B, are null because the two views are optimal here. 2; is contained by view V because representations are functions of views. (Middle) z, covers D when Eq. 7 holds, which indicates the shared information between views is kept after encoding. (Right) View z, further exactly covers D and the view encoding becomes optimal, i.c., zi, which indicates all the information shared between view Vi and representation z is task-relevant.
representations from one view with the graph representations from the other view. Thus, ca(-), c+()) are the identical transformation and averaging aggregation operations, respectively. In local-local mode, the node representations from two views are contrasted. In multi-scale mode, we contrast graph representation of one view with the intermediate representation from the other. In hybrid mode, both global-global and local-global are applied.
To characterize which mode is optimal, we define the optimality of contrastive mode for graph contrastive learning. The main motivation is: the optimal contrastive mode keeps the most task- relevant information after the representations are aggregated. The proof is included in the Appendix.
Corollary 3. (Optimal Contrastive Mode) Given the latent representations, z;, zj,, extracted by the optimal view encoders, i.e., z = (v?). =;= [v]( and a downstream lask T with label y, the optimal contrastive mode is the solution to the following optimizution problem, where Co Oj are the aggregation operations applied to the latent representattons:
(c) = arg min -((((2)))(()))
(8)
(er,ey)
4.4 InfoGCL Principle
According to our proposed corollaries, we can theoretically design the optimal contrastive learning approach for our specific graph data and task. However, in real-world scenarios, the conditions to meet the exact optimality of contrastive learning hard or even not practically possible to reach because of data noise and limited model capability. Therefore, we propose to achieve the optimal for each stage independently and practically, which is an approximation to achieve the original optimality. Specifically, we make the following propositions to address the questions of the optimal views, optimal view encoder, and optimal contrastive mode.
Proposition 1. For task T with label given a bunch of graph view augmentation methods, {ęı (-), (e(-).... ... }, that create two views Vs, Vj. the recommended augmentation methods are the ones, qi(-), (-).), that maximize T(V;;y) + I(V;: I(v;; v)), i.e., the area of A+ B+D in Figure D
Proposition Given a task T with label y and set of view encoders, (.f(),, f2((). ... }. that
generate representation via taking view vi input, the recommended view encoder the that maximizes the mutual information between v., 2; and Symmetrically the same for view V
Proposition 3. Given a task T with label y. the extracted representations, 2y Zi. and a set of aggregation operations, (ci(-), cz(.), ... }. the recommended contrastive mode is the one, (c),0y), that has the largest amount of mutual information between cz(2,). cz(2,) and y.
The qualitative and quantitative evaluation of these propositions are shown in Section 5.3
4.5 Role of Negative Samples
Current graph contrastive learning approaches heavily depend on negative samples. However, recent progresses of contrastive learning in vision domain indicate that negative samples are not necessarily

---

Graph Task Datasets
Node Task Datasets
MUTAG
PTC-MR
IMDB-B
IMDB M
NCI1
COLLAB
Cora
Citeseer
Pubmed
# Graphs
188
344
1000
1500
4110
5000
1
# Nodes
17.9
14.3
19.8
13.0
29.9
74.5
3327
2708 5429
19717 44338
# Edges
19.8
14.7
193.1
65.9
1.1
33.0
4732
# Classes
2
2
6
3
Table I: Dataset statistics.
required [8] 4) of which the main benefit is to avoid careful treatment to retrieve the negative pairs. To study the influence of negative samples on graph contrastive learning, we follow the framework
of SimSiam 4 to revise the loss function Eq. A very recent work [29] also studies graph contrastive learning without negative samples. Different from it, we focus on both node and graph classification tasks.
N
Zi,n
Zjjm
N
Zi,n
|2
5
Experiments
In this section, we evaluate our InfoGCL with number experiments. We first describe datasets, evaluation protocol, and experimental setup. Then, we present the experimental results on both node and graph classification. Last, we analyze our proposed principles via ablation study.
5.1 Setup
We use both graph classification and node classification benchmark datasets that are widely used in the existing graph contrastive learning approaches. The graph classification datasets include MUTAG [17), PTC-MR [17], IMDB-B B [40], IMDB-M [40]. NCI1 [34), and COLLAB [40]. MUTAG a collection of nitroaromatic compounds represented as graphs, where vertices stand for atoms and edges represent bonds between atoms. PTC-MR is a collection of 344 chemical compounds which report the careinogenicity for rats. IMDB-B and IMDB-M are two movie collaboration datasets, where nodes represent actors/actress and there is an edge between them if they appear in the same movie. NCII, graphs are the representation of chemical compounds, where vertices stand for atoms and edges represent bonds between atoms. COLLAB is collaboration dataset, where researchers are nodes and an edge indicates collaboration between two researchers. The node classification datasets include Citeseer, Cora, and Pubmed P3 All of them are citation networks, where nodes are documents and edges are citation links. These datasets are summarized in Table
We closely follow the evaluation protocol of previous state-of-the-ane graph contrastive learning approaches. For graph classification, we report the mean 10-fold cross validation accuracy after runs followed by a linear SVM. The linear SVM is trained by applying cross validation on training data folds and the best mean accuracy is reported. For node classification, we report the mean accuracy on test set after 50 runs of training followed by a linear neural network model. To make comparison fair, we adopt the basic setting of InfoGraph for graph classification. We conduct experiment with the values of the number of GNN layers, the number of epochs, batch size, the parameter C of SVM the sets (2, 4, 8, 12), (10, 20, 40, 100), (32, 64, 128, 256) and (10-3, 10-2, 102. 10x J. respectively. We adopt the basic setting of DGI for node classification. Specifically, we set the number of GNN layers to and experiment with the batch size in the set (2, 4, 8). The hidden dimension of representations is set to 512. We also apply the early stopping strategy.
5.2 Experimental Results
To evaluate our method InfoGCL on graph classification, we use thhree categories of baselines.
The kernel approaches include shortest path kernel (SP) (2), Graphlet kernel (GK) 22], Weisfeiler-
Lehman sub-tree kernel (WL) [24, deep graph kernels (DGK) (39, and multi-scale Laplacian kernel
(MI.) [16]. The supervised baselines include GraphSAGE [1). GCN [5], GIN [38] GAT [32
We also compare with the unsupervised approaches, including Random Walk 7 node2vec [9].
sub2vec [14], graph2vec (20), InfoGraph (28), GraphCL [2). and mvgrl [3] Table shows the
7

---

Method
MUTAG
PTC-MR
IMDB-B
IMDB-M
NCI1
COLLAB
Kernel Approaches
85.2 2.4
2.4
55.6 0.2
38.0 0.3
73.3 0.1
GK
81.7 + 2.1
57.3
1.4
65.9
1.0
43.9
0.4
66.0
72.8 0.3
WL
80.7 3.0
58.0
0.5
72.3 1 3,4
47.0
0.5
8..0
78.9
1.9
DGK
87.4222
60.1 2.6
67.0 + 0.6
44.6 0.5
0.5
73.1 = 0.3
MLG
87.9 + 1.6
63.3 + 1.5
66.6 + 0.3
41.2 = 0.0
80.8 1.3
Supervised Approaches
GraphSAGE
85.1
63.9
72.3 5.3
50.9 2.2
77.7
1.5
68.3
4.2
GCN
85.6 5.8
4.3
74.0
80.2
2.0
79.0
1.8
GIN-O
89.4 5.6
64.6
75.1
5.1
52.3 2.8
82.7
1.7
80.2
1.9
GIN-e
89.0 + 6.0
6.2
74.3
5.1
52.1 3.6
82.7
1.6
80.1
1.9
GAT
89.4 6.1
66.7 5.1
70.5 2.3
4.1
66.6
2.2
67.4
17.+25 2.9
Unsupervised Approaches
Random Walk
83.7 + 1.5
+7.9
50.7 0.3
34.7 + 0.2
64.3
0.3
node2vec
72.6
10.2
8.0
50.2
36.0 0.7
54.9
56.1 0.2
sub2vec
61.1
15.8
6..0
1.5
36.7
52.8
graph2vec
83.2 9.6
6.9
71.1
5.94
75.4 1.2
InfoGraph
89.0 1.1
1.47
73.0
49.7
76.2 1.4
70.7 1.1
GraphCL.
86.8 1.3
6.13
71.1
49.2
77.9 0.4
71.4
1.2
mvgrl
89.7 1: 1.1
62.5
74.2 0.7
51.2 1 0.5
77.0
0.8
76.0
1: 1.2
InfoGCL
91.2 = 1.3
1.5
75.1 0.9
51.4 0.8
80.2 = 0.6
80.0 - 1.3
Table 2: Graph classification results (%).
Method
Cora
Citeseer
Pubmed
Supervised Approaches
MLP
55.1
46.5
71.4
ICA
75.1
69.1
73.9
LP
68.0
45.3
63.0
ManiReg
59.5
60.1
70.7
SemiEmb
59.0
59,6
71.7
Planetoid
75.7
64.7
77.2
Chebyshev
81.2
69.8
74.4
GCN
81.5
70.3
79.0
JKNNtt
82.7 0.4
73.0 + 0.5
77.9 + 0.4
GAT
83.0 = 0.7
72.5 = 0.7
79.0 0.3
Unsupervised
Approaches
Linear
47.9
49.3 0.2
69.1
DeepWalk
70.7
$1.4
74.3
GAE
71.5
65.8
72.1
VERSE
72.5
5.4
DGI
83.8 0.5
72.0
77.9 17.9-00 0.3
GraphCL
82.5 1 0.1
0.2
mvgrl
86.8 0.5
13.3+0.3
80.1 + 0.7
InfoGCL
83.5 - 0.3
73.5 0.4
79.1 0.2
Table 3: Node classification results (%).
graph classification results. We observe that our approach achieves the best results compared to other unsupervised approaches. Our approach also outperforms or matches the best kernel approaches across the datasets. Even compared with the supervised ones, approach achieves the best in 2 out of datasets and the results of our approach on other 4 dataset are among the top.
For node classification tasks, we compare InfoGCL with some supervised approaches and unsu- pervised approaches. The supervised baselines include a simple MLP model, iterative classifi- cation algorithm (ICA) [19], manifold regularization (ManiReg) [. semi-supervised embedding (SemiEmb) (35, Planetoid [41, Chebyshev (5), GCN, JKNet (36), GAT. Table shows the node classification results. It is observed that our approach achieves the state-of-the-ar results and com- petes the best one with respect to the existing unsupervised approaches. Compared to supervised baselines, our approach outperforms all the basclines.

---

Method
MUTAG
IMDB-B
COLLAB
Cora
Citeseer
Pubmed
InfoGCL (w/ neg)
91. + 1.4 91.2 1.3
75.1 = 0.5
80.2 + 10
78.6 0.4 83.5
70.4 + 0.6
77.4 + 0.7 79.1 + 0.2
InfoGCL. (w/ neg)
75.1 + 0.9
80.0 + 1.3
+ 0.3
73.5 + 0.4
Table 4: Comparison between InfoGCL with negative samples and without negative samples.
5.3
Evaluation of InfoGCL Principle
We can unify the existing graph contrastive learning methods through the perspective of InfoGCL principle: all recent graph contrastive learning methods can be decoupled into three stages that implicitly follow the InfoGCL principle, though being different in model architecture design and optimization strategies. Below, we analyze some observations from several recent work. Because of the limited space, please refer to the Appendix for more results of the quantitative analysis.
Obs. Composing a graph and its augmentation benefits downstream performance 42) Compared to composing graph and the graph itself, augmentation leads to smaller I (v;; v,) (Proposition 1).
Obs. ii Composing different augmentations benefits more [42] Compared to composing graph and its augmentations, two augmentations further decrease I (v: (Proposition 1).
When Obs. iii. Node dropping and subgraph sampling are generally beneficial across datasets 42
compared to attribute masking and edge perturbation, they change the semantic meaning of the graph relatively slightly, which leads to higher T(v;; y). I(yy; y) (Proposition 1).
Obs. iv. Edge perturbation benefits social networks but hurts some biochemical molecules (42) The
semantic meaning of social networks are robust to edge perturbation. However, the semantic meaning of some biochemical molecules are determined by local connection pattern, where edge perturbation decreases I(v;; y) (Proposition 1).
Obs. v. Contrasting node and graph representations consistently performs better than other contrastive modes across benchmarks [13. Compared to other contrastive modes, node-graph (i.e., local-global) mode generally extracts more graph structure information, which benefits predicting task label (Proposition 3).
5.4
Effect of Negative Samples
To study the effect of negative samples on graph contrastive learning, we follow SimSiam 4 and design the objective as Eq. (9). We conduct experiments on three graph task and three node task datasets. The results are reported in Table 4 observed that the negative samples show little influence on the three graph task datasets, while performance drops on the three node task datasets, especially the Cora dataset.
According to the dataset statistics summarized in Table we see the networks of Cora, Citeseer, Pubmed much sparser (in terms of network topology). Furthermore, know the node features of these three datasets are also much sparser (one-hot encoding with high dimensionality). We speculate this because the contrastive learning models tends to collapse easier if negative samples especially when data is too sparse. Therefore, we make the hypothesis: negative samples benefit graph modeling, especially when i) network topology, and ii) node features are extremely sparse.
used,
Conclusion and Limitations
We propose InfoGCL, an information-awanl graph contrastive learning framework for graph con- trastive learning. Existing graph contrastive learning approaches are usually carefully designed. We aim answer how to perform contrastive learning for your learning tasks specific graph data. Our method decouples the typical contrastive learning approaches into three sequential modules and provides the theoretical analysis for reaching the optimality. To address the questions of optimality in a practical way, we propose the InfoGCL principle, which implicitly followed by all recent graph contrastive learning approaches. In addition, explore the role of negative samples in graph contrastive learning and find negative samples are not necessarily required. Experiments on both node and graph benchmark datasets demonstrate the effectiveness of our method. Note that method is not without limitations. We can further improve our method by designing better practical approximations the theoretical optimality of graph contrastive learning.

---

Funding Transparency Statement
This project was partially supported by NSF projects IIS-1707548 and CBET-1638320.
References
[1] Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. J. Mach. Learn. Res, 7, 2006.
[2] Karsten M. Borgwardt and Hans-Peter Kriegel. Shortest-patts kernels on graphs. In ICDM, 2005.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
[4] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CoRR, abs/2011.10566 2020.
[5] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS, 2016.
[6] Yuanqi Du, Shiyu Wang. Xiaojie Guo, Hengning Cao, Shujie Hu, Junji Jiang, Aishwarya Varala, Abhinav Angirekula, and Liang Zhao. Graphgt: Machine learning datasets for deep graph generation and transformation. 2021.
[7] Gartner, Flach, and Wrobel. On graph kernels: Hardness results and efficient alternatives. In COLT Workshop, 2003.
[8] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Pires, Zhaohan Guo, Mohammad Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. 2020.
[9] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD, 2016.
[10] Xiaojie Guo, Yuanqi Du, and Liang Zhao. Property controllable variational autoencoder via invertible mutual dependence. In International Conference on Learning Representations, 2020.
[11] Xiaojie Guo, Yuanqi Du, and Liang Zhao. Deep generative models for spatial networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 505-515, 2021.
[12] William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NeurlPS, 2017.
[13] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In ICML, pages 3451-3461. 2020.
[14] Kexin Huang and Marinka Zitikk. Graph meta learning via local subgraphs. CoRR, abs/2006.07889, 2020.
[15] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In /CLR, 2017.
[16] Risi Kondor and Horace Pan. The multiscale laplacian graph kernel. In NIPS, 2016.
[17] Nils Kriege and Petra Mutzel. Subgraph matching kernels for attributed graphs. In ICML, 2012.
[18] R. Linsker. Self-organization in a perceptual network. Computer, 21:105, 1988.
[19] Qing Lu and Lise Getoor. Link-based classification. In ICML, 2003.
10

---

[20] Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. CoRR. 2017.
[21] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03307 2018.
[22] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. KDD, 2020.
[23] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data, Al magazine, 29(3):93-93, 2008.
[24] Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn. and Karsten M. Borgwardt. Weisman graph kernels. J. Mach. Learn. Res, pages 2539-2561, 2011.
[25] Nino Shervashidze, S. V. N. Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten M. Borgwardt. Efficient graphlet kernels for large graph comparison. AISTATS, 2009.
[26] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:/703.008: 2017.
[27] Stefano Soatto and Alessandro Chiuso. Modeling visual representations: Defining properties and deep approximations. In ICLR, 2016.
[28] Fan- Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICLR, 2019.
[29] Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. Bootstrapped representation learning on graphs. arXiv preprint arXiv:2102.06514 2021.
[30] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. NeurIPS, volume 33, 2020.
[31] Naftali Tishby, Fernando C. N. Pereira, and William Bialek. The information bottleneck method. CoRR, physics/000405,, 2000.
[32] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. ICLR, 2018.
[33] Petar Veličković, William Fedus, William L. Hamilton, Pictro Liò, Yoshua Bengio, and R Devon Hjelm. Deep Graph Infomax. In ICLR, 2019.
[34] Nikil Wale, Ian A Watson, and George Karypis. Comparison descriptor spaces for chemical compound retrieval and classification. Knowledge and Information Systems, 14(3)3347-375, 2008.
[35] Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi-supervised embedding, 2013.
[36] Felix Wu, Tianyi Zhang 0007, Amauri H. Souza Jr., Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. ICML, 2019.
[37] Tailin Wu, Hongyu Ren, Pan Li, and Jure Leskovec. Graph information bottleneck. CoRR, 2020.
[38] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
[39] Pinar Yanardag and S. V. N. Vishwanathan. Deep graph kerness. In KDD, 2015.
[40] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In KDD, pages 1365-1374, 2015.

---

[41] Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi- supeeviied learning with graph embeddings. ICML, abs/1603.0555 60.66660..8.,..... 2016.
[42] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS, volume 33, 2020.
[43] Junchi Yu, Tingyang Xu, Yu Rong. Yatao Bian, Junzhou Huang, and Ran He. Graph information bottleneck for subgraph recognition. ICLR, 2021.
[44] Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. arXiv preprint arXiv:2006. 06830, 2020.
[45] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep Graph Contrastive Representation Learning, In ICML Workshop, 2020.
12


Reviews:
[{"review_id": "szxKsTA5rZF", "reviewer": "Reviewer_U1jZ", "summary": "This paper proposes InfoGCL, a general framework for graph contrastive learning framework design. More specifically, InfoGCL investigates the optimal (and practically optimal) option for three modules in graph contrastive learning, i.e. view augmentation, view encoding and representation contrasting. Experimental results have proven effective over various benchmarks compared with groups of the kernel, supervised and unsupervised. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. Good formulation and clear identification of the three-stage graph contrastive learning process. \n2. Technically sound through the angle of information bottleneck of optimal graph contrastive learning options, with improved explainability.\n3. Extensive experimental on InfoGCN over a large group of baseline through fair comparison with important observations.\n\nWeaknesses:\n1. A few terms and concepts that may not be generally recognized are not formally defined or fully discussed.\n2. IB-based InfoGL seems to be only applicable for a single (given) task.\n3. Some experimental results are less convincing and somewhat lack sufficient explanation. \n\nDetailed comments and questions:\n1. As pointed out in the weaknesses, it is assumed that all the proposition 1-3 are applied when only one task T is pre-given, while many GNN frameworks are intended for multiple downstream tasks while graph contrastive learning is served for pre-training. One drawback is that, once the views are optimized over “task-related” information, the InfoGCL would have to perform independently (or train from scratch) for multiple downstream tasks. \n2. Regarding experiment: \n2.1  The variances of Node2vec and sub2vec on the MUTAG dataset are extremely high which seems strange. \n2.2 The performance drop without negative samples on the Cora dataset and others is insufficient and not convincing (even though the authors only hypothesize the sparsity is one factor). \n3. Many terms that are not universally recognized or self-explained are not formally defined and may need clarification. Examples are: \n3.1 The term “view” i.e. generated variations (subgraphs) from the original graph which may have confusing meanings in other graph communities. \n3.2 The Venn figures in Figures 1 and 2 need revision for better visibility and explanation. \n3.3 The detailed definition of 5 modes in Section 4.3\n3.4 “Negative pairs” in InfoNCE loss vs “Negative samples” in Section 4.5\n\nMinor corrections or suggestions:\n1. The Algorithm 1 block in the appendix is relatively important to understand the practical implementation of InfoGCL, which is recommended to be added in the main section. Otherwise, reviewers may be lost about how InfoGCL is working in practice.\n2. Line 172: Typo “gprah”\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "cnMkL9CQWaL", "reviewer": "Reviewer_PEad", "summary": "The paper provides a systematic approach to building graph contrastive learning models for\nany particular graph learning taks and dataset. The suggested approach builds on the Information Bottleneck principle and unifies all recent graph representation learning methods based on contrastive learning.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think this is a good paper, trying to present a unifying view on graph contrastive learning. A typical graph contrastive learning model is decoupled into three sequential modules: View augmentation, view encoding and representation contrasting. Based on the principle of the Information Bottleneck, the paper suggest feasible objectives to identify optimal modules for each of these three parts:\n* The augmented views should contain as much task-relevant information as possible, while they shoul share as little information as possible.\n* The view encoder should be task-relevant and simple as much as possible.\n* Thhe contrastive mode should keep task-relevant information as much as possible70after contrasting\n\nThe paper also argues that negative samples are not necessarily required for graph contrastive learning.\n\nOriginality: The paper unifies existing approaches, and provides a novel systemic perspective on graph contrastive learning.\n\nQuality: The paper is technically sound, corollaries are proven in the the appendix, the work is complete as such.\n\nClarity: The paper is written well. \n\nSignificance: The results are important to researches in the area of graph representation learning. It's holistic approach to graph contrastive learning through the lens of the Information Bottleneck principle provides novel insights for this area of research. The presented empirical results look convincing.\n\nA minor detail is Eq.9: I think it would be nice to explain with a few sentences what the principle behind SimSiam is.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are mentioned in the final section, but not explicitly explained.\nSocietal Impact is missing.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5h"}, "extra_text": {}}, {"review_id": "R9jw1bdVc5", "reviewer": "Reviewer_gMsc", "summary": "In this paper, the authors study the graph contrastive learning problem and propose an information-aware approach called InfoGCL. The motivation for this work is that all recent graph contrastive learning approaches are customized, and it is an open question how to build the graph contrastive learning method for particular tasks and datasets. The authors' method decouples the typical graph contrastive learning into three modules and formalizes how to find the optimal of the three modules into three optimization problems. Experimental results on both node-level and graph-level tasks demonstrate the advantages of the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is pretty interesting overall. Different from existing work, this work tries to unify the various contrastive learning approaches for graph-structure data. It extends the information bottleneck theory to optimize the representation learning in the graph contrastive learning framework.\n\nOne main contribution of this work is to understand graph contrastive learning from the optimization view. From the observations in section 5.3, it seems that the proposed InfoGCL is general and well supported by existing work. Personally, I like this work because instead of simply proposing a method to achieve better results, it digs into the hidden reasoning behind graph contrastive learning.\n\nThis paper contains solid theoretical content, which is well organized and easy to follow. The motivation, contribution, and methodology are well written. The experimental results support the claims. Overall, this paper is with good writing and clarity.\n\nHowever, there are some specific questions/concerns. \n\n1. In the Introduction, “Second, graph data has unique properties that distinguish it from other types of data, such as rich structural information and highly diverse distribution”. I see the proposed method considers graph structure, but not clear how it could generalize to cover various graph properties.\n\n2. In the Introduction, “However, different from the typical representation learning, there are two information flows involved in the two augmented views in contrastive learning.”. I am not an expert in the information theory area. Could the authors elaborate more on the difference/significance in terms of the two flows?\n\n3.  Corollary 1 (Section 4.1). Are you sure to achieve optimal conditions in real data scenarios? It looks very constrained.\n\n4. In Section 4.3, “The main motivation is: the optimal contrastive model keeps the most task-relevant information after the representations are aggregated”. This problem appears to be a discrete optimization problem, and the authors have used the method of obtaining the optimum by traversal. Can there be a more optimal solution here?\n\n5. In Section 5.4, “Therefore, we make the hypothesis: negative samples benefit graph modeling, especially when i) network topology, and ii) node features are extremely sparse.”. I am interested in the study of negative samples. I have not seen these conclusions in the cv domain. Can the authors provide more analytical or experimental supports?\n\n6. In Section 5 (Experiments). The datasets of Cora Citeseer Pubmed are very small. I suggest the authors experiment with larger datasets.\n\n7. The propositions in Section 4.4 are interesting and seem reasonable. More evidence/support is preferred.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a systematic approach to contrastive learning on graphs which builds on ideas from information bottleneck theory and unifies multiple graph representation learning methods in a single framework. The paper is written well and relevant to the NeurIPS community. All reviewers support its acceptance, especially due to the novel and promising approach which unifies existing methods and provides new perspectives for graph learning. Reviewers highlighted also that the approach is technically sound, that claims in the paper are theoretically supported, and convincing experimental results. Please take the feedback on concerns and limitations from reviewers into account when preparing the camera ready version of the manuscript.
Decision: Accept (Poster)