InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
Authors: Fan-Yun Sun
Jordan Hoffman
Vikas Verma
Jian Tang
Venue:  (2020)
Keywords: graph-level representation learning
mutual information maximization

Abstract: This paper studies learning the representations of whole graphs in both unsupervised and semi-supervised scenarios. Graph-level representations are critical in a variety of real-world applications such as predicting the properties of molecules and community analysis in social networks. Traditional graph kernel based methods are simple, yet effective for obtaining fixed-length representations for graphs but they suffer from poor generalization due to hand-crafted designs. There are also some recent methods based on language models (e.g. graph2vec) but they tend to only consider certain substructures (e.g. subtrees) as graph representatives. Inspired by recent progress of unsupervised representation learning, in this paper we proposed a novel method called InfoGraph for learning graph-level representations. We maximize the mutual information between the graph-level representation and the representations of substructures of different scales (e.g., nodes, edges, triangles). By doing so, the graph-level representations encode aspects of the data that are shared across different scales of substructures. Furthermore, we further propose InfoGraph*, an extension of InfoGraph for semisupervised scenarios. InfoGraph* maximizes the mutual information between unsupervised graph representations learned by InfoGraph and the representations learned by existing supervised methods. As a result, the supervised encoder learns from unlabeled data while preserving the latent semantic space favored by the current supervised task. Experimental results on the tasks of graph classification and molecular property prediction show that InfoGraph is superior to state-of-the-art baselines and InfoGraph* can achieve performance competitive with state-of-the-art semi-supervised models.

Published as a conference paper at ICLR 2020
INFOGRAPH: UNSUPERVISED AND SEMI-SUPERVISED GRAPH-LEVEL REPRESENTATION LEARNING VIA MU- TUAL INFORMATION MAXIMIZATION
Fan- Yun 1.2 Jordan Hoffmann Vikas Verma 2,3 Jian Tang2.5,6
1 National Taiwan University,
2 Mill- Sutaaquueeec Institute for Learning Algorithms, Canada
3 Aalto University, Finland
"Harvard University, USA
'HEC Montreal, Canada
CCIFAR AI Research Chair
b04902045entu. saaaanggharrvrd.ee.
jhof fmanneg. harvard. waaavveemm@aalto. fi jian. tang@hec..cc.....
ABSTRACT
This paper studies learning the representations of whole graphs in both unsuper- vised and semi-supervised scenarios. Graph-level representations variety of weal-world applications such as predicting the properties of molecules and community analysis in social networks. Traditional graph kernel based methods are simple, yet effective for obtaining fixed-length representations for graphs but they suffer from poor generalization due to hand-crafted designs. There are also recent methods based on language models (e.g. graph2vec) but they tend to only consider certain substructures (e.g. subtrees) as graph representatives. Inspired by recent progress of unsupervised representation learning, in this paper we proposed a novel method called InfoGraph for learning graph-level representations. We maximize the mutual information between the graph-level representation and the representations of substructures of different scales (e.g., nodes, edges, triangles). By doing so, the graph-level representations encode aspects of the data that are shared across different scales substructures. Furthermore, we further propose InfoGraph*, an extension of InfoGraph for semi-supervised scenarios. InfoGraph* maximizes the mutual information between unsupervised graph representations learned by InfoGrapp and the representations learned by existing supervised meth- ods. As a result, the supervised encoder learns from unlaheled data while preserving the latent semantic space favored by the current supervised task. Experimental results on the tasks of graph classification and molecular property prediction show that InfoGraph superior to state-of of-te-------arrr baselines and InfoGraph* can achieve performance competitive with state-of-the art semi- sumi-sueeevieed models.
critical in a
INTRODUCTION
Graphs have proven to be an effective way to represent very diverse types of data including social networks Newman Girvan (2004). biological reaction networks(Pvvoooollss et al. (201T), protein- protein interactions Krogan et al. (2006). the quantum mechanical properties of individual molecules Xie Grossman (0188): Jin et al. (018)), and many more. Graphs provide explicit information about the coupling between individual units in larger part along with well defined framework for assigning properties to the nodes and the edges connecting them. There has been a significant amount of previous work done studying many aspects of graphs including link prediction Gao et al.) (201T); Wang et al. (201T) and node prediction Blei et al. (2003). Due to its flexibility, graph-like data structures capture rich information which is critical in many applications.

---

Published as a conference paper at ICLR 2020
At the lowest level, much work has been done on learning node representations- low-dimension
vector embeddings of individual nodes Perozzi et al. (2014); Tang et al. (2015); Grover & Leskoved
(2016), Another field that has attracted large amount of attention recently is learning representations entire graphs. Such problem is critical in a variety of applications such as predicting the properties of molecular graphs in both drug discovery and material science Chen et al. (2019h a). There has
been some recent progress based on neural message passing algorithms Gilmer et al. (2017); Xic
& Grossman 2 which learn the representations of entire graphs in supervised way. These methods have been shown achieving state-of ofate-of-hhe-rr results on a variety of different prediction tasks Kipf et al. (2018); Xie & Grossman (2018); Gilmer al. (2017): Chen al. (2019a).
However, one of the most difficult obstacles for supervised learning on graphs is that it is often very costly or even impossible to collect annotated labels. For example, in the chemical domain labels are typically produced with costly Density Functional Theory (DFT) calculation. One option is to use semi-supervised methods which combine a small handful of labels with a larger, unlabeled, dataset. In real-world applications, partially labeled datasets are common, making tools that are able to efficiently utilize the present labels particularly useful.
Coming up with methods that are able to learn unsupervised representations of an entire graph, as opposed to nodes, is an important step in working with unlabeled or partially labeled graphs Narayanan et al. (2017); Hu et al. 2019); Nguyen et al. (2017). For example, there exists work that explores pre-training techniques for graphs to improve generalization Hu et al. (2019). Another common approach to unsupervised representation learning on graphs is through graph kernels Pržulj (2007); Kashima et al. (2003); Orsini et al. (2015). However, many of these methods do not provide explicit graph embeddings which many machine learning algorithms operate on. Furthermore, the handcrafted features of graph kernels lead to high dimensional, sparse or non-smooth representations and thus result in poor generalization performance, especially on large datasets Narayanan al.) (2017).
Unsupervised learning of latent representations is also an important problem in other domains, such as image generation Kingma& & Welling (2013); Kim & (2018) and natural language processing Mikolov et al.] (2013a). recent work introduced Deep Infomax, method that maximizes the mutual information content between the input data and the learned representation [Hjelm et al. (2018). This method outperforms other methods on many unsupervised learning tasks. Motivated by Deep InfoMax Hjelmm et al. (2018), aim to use mutual information maximization for unsupervised representation learning on the entire graph. Specifically, our objective is to maximize the mutual information between the representations of entire graphs and the representations of substructures of different granularity. We name our model InfoGraph.
We also propose semi-supervised learning model which we name InfoGraph We employ a student-teacher framework similar to Mean-Teacher method Tarvainen & Valpola (2017). We maximize the mutual information between intermediate representations of the two models so that the student model learns from the teacher model. The student model is trained on the labeled data using a supervised objective function while the teacher model trained on unlabeled data with InfoGraph. Using InfoGraph*, we achieve performance competitive with state-of-the-as methods on molecular property prediction.
We summarize our contributions as follows:
We propose InfoGraph, an unsupervised graph representation learning method based on Deep InfoMax (DIM) [Hjelm et al. (2018).
We show that InfoGraph can be extended to semi-supervised prediction tasks graphs.
We empirically show that InfoGraph surpasses state-of-ther oftth-rf- performance graph classi- fication tasks with unsupervised learning and obtains performance comparable with state-of- art methods molecular property prediction tasks using semi-supervised learning.
2 RELATED WORK
Representation learning for graphs has mainly dealt with supervised learning tasks. Recently, however, researchers have proposed algorithms that learn graph-level representations in an unsupervised manner Narayanan al. (2017): Adhikari et al. (2018)
2

---

Published as a conference paper at ICLR 2020
Concurrently to this work, information maximizing graph neural networks (IGNN) was introduced which uses mutual information maximization between edge states and transform parameters to achieve state-of-the-ar predictions on variety of supervised molecule property prediction tasks Chen et al. (2019b). In this work, our focus is on unsupervised and semi-supervised scenarios.
Graph Kernels. Constructing graph kernels is common unsupervised task in learning graph representations. These kernels are typically evaluated on node classification tasks. In graph kernels, a graph G is decomposed into (possibly different) {Ga} sub-structures. The graph kernel K ( G2) is defined based on the frequency of each sub-structure appearing in G and G2 respectively. Namely, K(1 G2) (fa.,' fG...) where fc. is the vector containing frequencies of {Gs} sub-structures. and (,) an inner product in an appropriately normalized vector space. Much work has been devoted to deciding which uub-structures are more suitable than others (refer to appendix A. Instead of defining hand crafted similarity measures between substructures, InfoGraph adopts a more principled metric- mutual information.
Contrastive methods. An important approach for unsupervised representation learning is to train an encoder to be contrastive between representations that capture statistical dependencies of interest and those that do not. For example, a contrastive approach may employ a scoring function, training the encoder to increase the score "real" input (a.k.a, positive examples) and decrease the score on "fake" input (a.k.a., negative samples). For more detailed discussion, refer to appendix A.2.
Deep Graph InfoMax (DGI) Veličković et al. (2018) also belongs to this category, which aims to train a node encoder that maximizes mutual information between node representations and the pooled global graph representation. Although we built upon similar methodology, our aim is different than theirs as our goal is to obtain embeddings at the whole graph level for unsupervised and semi- supervised learning whereas DGI only evaluates node level embeddings. In order to differentiate our method with Deep Graph Infomax (Veličković et al. (2018), ), we term our model InfoGraph.
Semi-supervised Learning. A comprehensive overview of semi-supervised learning (SSL) methods is out of the scope of this paper. We refer readers to Appendix B for short overview or Zhu al. 2003);; Chapelle al. (006)); Oliver al. (2018) for more comprehensive discussions. Here. we discuss a state-of-the art method applicable for regression tasks Mean Teacher (arvainen & Valpola (2017) Meaan Teacher adds loss term which encourages the distance between the original network's output and the teacher's 's output to be small. The teacher's predictions are made using an exponential moving average of parameters from previous training steps. Inspired by the "student- teacher" framework in Mean Teacher model, our semi-supervised model (InfoGraph*) deploys two separate encoders but instead of explicitly encouraging the output of the student model to be similar to the teacher model's output, enable the student model to learn from the teacher model by maximizing mutual information between intermediate representations learned by two models.
METHODOLOGY
Most recent work on graphs focus on supervised learning tasks or learning node representations. However, many graph analytic tasks such as graph classification, regression, and clustering require representing entire graphs as fixed-length feature vectors. Though graph-level representations can be obtained through the node-level representations implicitly, explicitly extracting the graph can be more straightforward and optimal for graph-oriented tasks.
Another scenario that is important, yet attracts comparatively less attention in the graph related literature semi-supervised learning. One of the biggest challenges prediction tasks in biology Yan ct al. (2017); Yang et al. (2014) or molecular machine leamiiggDveenuud Duvenaud et al. (2015); Gilmer et al. (2017): Jia & Liang (0177) is the extreme scarcity of labeled data. Therefore, semi-supervised learning, in which large number of unlabeled samples are incorporated with a small number of labeled samples to enhance accuracy of models, will play a key role in these areas.
In this section, we first formulate unsupervised whole graph representation learning problem and a sumi-super prediction task on graphs. Then, we present our method to learn graph-level representations. Afterwards we present our proposed model for the semi-supervised learning scenario.
3

---

Published as a conference paper at ICLR 2020
Gruph Convolution Encoder
Caagph-rre oegrcamnakkn Gaah.A
Ardiitect
Cropa
READOUT
Tiazing data Goagh A vus nodes
Training dass of Goophe alllaadie midee
CINCAT (iap
Crogth Convolrzan Excodie
Pecc leal
ppesrrctiona of mode &
Distimirsttr
Graph
Figure| 1: Illustration of InfoGraph. N.A. denotes neighborhood aggregation. An input graph is encoded into a feature map by graph convolutions and jumping concatenation. The discriminator takes (global representation, patch representation) pair as input and decides whether they are from the same graph. InfoGraph uses batch-wise fashion to generate all possible positive and negative samples. For example, consider the toy example with 2 input graphs in the batch and 7 nodes (or patch representations) in total. For the global representation of the blue graph, there will be 7 input pairs to the discriminator and same for the red graph. Thus, the discriminator will take 14 (global representation, patch representation) pairs as input in this case.
3.1 PROBLEM DEFINITION
Unsupervised Graph Representation Learning. Given set of graphs G [G1, ...... and positive integer (the expected embedding size), our goal is to learn a ö-dimensional distributed representation of every graph G, E G. We denote the number of nodes in G, as |G:|. We denote the matrix of representations of graphs as & e R G|
Semi-supervied Graph Prediction Tasks. Given a set of labeled graphs G [G ........ , with corresponding output for, o(g4)), and set of unlabeled samples G [ [ goal is to learn model that can make predictions for unseen graphs. Note that in most cases |G |G4]
{G|G=++1, Gia=++
3.2 INFOGRAPH
We focus graph neural networks (GNNs) flexible class of embedding architectures which generate node representations by repeated aggregation over local node neighborhoods. The represen- tations of nodes are learned by aggregating the features of their neighborhood nodes, refer to these as patch representations. GNNs utilize a READOUT function to summarize all the obtained patch representations into a fixed length graph-level representation.
Formally, the k-th layer of a GNN is
(N) (OMBINE(A) (R((--)) AGGREGATE! (((15-11) (1-11), evo) : N(()))))))))))))))))))))))))(N
where he is the feature vector of node at the k-th iteration/layer (or patch representation centered at node i), Eve is the feature vector of the edge between u and v, and ((v) are neighborhoods to node he is often initialized node features. READOUT can be simple permutation invariant function such as averaging or a more sophisticated graph-level pooling function Ying et al. (2018); Zhang al. 2018).
We seek to obtain graph representations by maximizing the mutual information between graph-level and patch-level representations. By doing so, the graph representations can learn to encode aspects of the data that are shared across all substructures. Assume that we are given a set of training samples
G : (G, E G) N with empirical probability distribution IP the input space. Let denote the set of parameters of a K-layer graph neural network. After the first &e layers of the graph neural network,
the input graph will be encoded into set of patch representations (x(()),Nr Next, summarize feature vectors at all depths of the graph neural network into a single feature vector that captures patch information at different scales centered every node. Inspired by Xu al. (2018b),
4

---

Published as a conference paper at ICLR 2020
concatenation. That is,
h4 - CONCAT( (R) ( JEm)
(2)
HG) = (MADOUT((MA))))
(3)
where hi the summarized patch representation centered node i and H²(G) the global representation after applying READOUT. Note that here we slightly abuse the notation of h.
We define our mutual information (Ml) estimator on global/local pairs, maximizing the estimated MI over the given dataset G {Gj E G ز
3. 6 arg maxx
Σ GEG
2
D Hc()).
(4)
0.V
NEG
La.e is the mutual information estimator modeled by discriminator T3 and parameterized by a neural network with parameters Øb. We use the Jensen-She -haenSShnnnon MI estimator (following the formulation of Nowozin et al. (2016),
To. (G); H
o. H&(x))) -Exx))))))xessl
Eexa (sp(TT.TTTTTTTTT (h,, H&()))
(5)
where is an input sample, x' (negative sample) is an input sampled from PP = P. distribution identical to the empirical probability distribution the input space, and sp(z) = log + e*) the
softplus function. In practice, generate negative samples using all possible combinations of global and local patch representations across all graph instances in batch.
Since H (G') is encouraged have high MI with patches that contain information all scales, this favours encoding aspects the data that are shared across patches and aspects that are shared across scales. The algorithm is illustrated in Fig. [
should be noted that our model similar to Deep Graph Infomax (DGI) Veličković et al. 2), a model proposed for learning unsupervised node embeddings. However, there important design differences due to the different problems that we are focusing on. First, in DGI they use random sampling obtain negative samples due the fact that they mainly focusing learning node embeddings on graph. However, contrastive methods require a large number of negative samples be competitive Hjelm et (2018), thus the use of batch-wise generation of negative samples is crucial as we trying to learn graph embeddings given many graph instances. Second, the choice of graph convolution encoders also crucial. We use GIN Xu et al./ (2018a) while DGI uses GCN GiNK & Welling (2016) as GIN provides better inductive bias for graph level applications. Graph neural network designs should be considered carefully so that graph representations can be discriminative towards other graph instances. For example, we use sum over mean for READOUT and that can provide important information regarding the size of the graph.
3.3
SEMI-SUPERVISED INFOGRAPH
Based on the previous unsupervised model, a straightforward way to do semi-supervised property prediction on graphs is to combine the purely supervised loss and the unsupervised objective function which acts regularization term. In doing so, the model is trained predict properties for the labeled dataset while keeping rich discriminative intermediate representation learned from both the labeled and the unlabeled dataset. That is, we try minimize the following objective function:
|GL
G Σ
Tiuotal
Isupervived (G.)),
(haspervieedhhssG.))) HG))
where supervised Mas (G,), 0;) defined as the loss function of graph G, that measures the discrepancy between the classifier output yo (G:) and the true output 0;. Taussupervised (G,); HG(G,) is the
5

---

Published as a conference paper at ICLR 2020
Label
Prediction
Unsupervised Loss (second term Eq.
Supervised Less Eirst term Eq.
Maximize
READOUT
Discriminator
Graph Convolution Encoder
Maximize MI (Eiq.
Graph Convolution Encoder
Maximize
Input Graphs
Figure 2: Illustration of the semi-supervised version of InfoGraph (InfoGraph*)) There are two separate encoders with the same architecture, for the supervised task and the other trained using both labeled and unlabeled data with an unsupervised objective (eq. equation 4 We encourage the mutual information of the representations learned by the two encoders to be high by deploying discriminator that takes pair of representation as input and determines whether they are from the same input graph.
unsupervised InfoGraph loss term as defined in eq. equation 4 that can be optimized using both labeled and unlabeled data. The hyper-parameter À controls the relative weight between the purely supervised and the unsupervised loss. The intuition behind this is that the model will benefit from learning good representation from the large amount of unlabeled data while learning to predict the corresponding supervised label.
However, supervised tasks and unsupervised tasks may favor different information or different semantic space. Simply combining the two loss functions using the same encoder may lead to "negative transfer" (Pan Yang 2009 Rosenstein et al. 2005). We propose a simple way to alleviate this problem: we deploy two encoder models: the encoder the labelled data (supervised encoder) and the encoder on the unlabelled data (unsupervised encoder). For transferring the learned representations from the unsupervised encoder the supervised encoder, encourages the representations learned by the two encoders to have high mutual information, at all levels of representations (third term of Eq. (8). Formally, let jo denote the set of parameters of another K K-layered graph neural network, identical to the one parameterized by Φ, and let À be a tunable hyper-parameted H²(G), H be global encoder representations of the graph G at encoder layer k, then total loss function can be defined follows:
define loss term that
G²,
KC 13"++1""
Tiotal Σ
suppervised (G:)
Σ
Teunsupervised (hr (G,); HpG,))
|0 10"1+16")
K
(8)
special instance of the student-teacher framework. However, unlike the recent student-teacher methods for semi-supervised learning (Laine & Aila 2016 Tarvainen & Valpolal 2017 Verma et al. (011b)), which enforce the predictions of the student model to be similar to the teacher model, we enforce the transfer of knowledge from the teacher model the student model via cnntual-iffommation maximization various levels of representations. In practice, to reduce the computation overhead introduced by the third term of Eq g instead of enforeing the mutualiinformaioon maximization over all the layers the encoders, each training
Notice that this formulation be
WWe slightly abuse this term in this paper usually refers to transferring knowledge from less related source and thus may hurt the target performance.

---

Published as a conference paper at ICLR 2020
update, we enforce innorumatiioooooiionnnnn maximization randomly chosen layer of the encoder (Verma et al., (019a).
In our semi-supervised experiments, refer to the naive method using the objective function given in eq. equation as InfoGraph. We refer to the method that uses two separate encoders and employ the objective function given in eq. equation 8 as InfoGraph". InfoGraph is fully summarized in Figure 3
4 EXPERIMENTS
We evaluate the effectiveness of the graph-level representation learned by InfoGraph downstream graph classification tasks and on semi-supervised molecular property prediction tasks.
4.1 DATASETS
For graph classification,, we conduct experiments on well-known benchmark datasets: MUTAG, PTC, BINARY, REDDIT-MULTI-SK
AMDB-BINARE and IMDB-MULTI (Yanardag & Vishwanathan (2015). ). For semi supervise learning tasks, we use the publicly available QM9 dataset
Ramakrishnan et al. 2014). Additional details of the datasets be found in Appendix B.
4.2 BASELINES
For graph classification, used 6 state-of-the-are graph kernels for comparison: Random Walk (RW)
Gärtner et al.] (2003). Shortest Path Kernel (SP) Borgwardt & Kriegel (2005). Graphlet Kernel (GK)
Shervashidze et al. (2009), Weisfeiler-Lehman Sub-tree Kernel
Shervashhes et al.) (201
Deep Graph Kemels (DGK) Yanardag & Vishwanathan (2015), and Multi-Scale Laplacian Kernel
(MLG) Kondor & Pan (016). Aside from graph kernels, we also compare with 3 unsupervised
graph-level representation learning methods: node2vec Grover & Leskoved (2016), sub2vee Adhikari
al.] (2018), and graph2vec Narayanan et al. (2017). Node2vec neural embedding framework that learns feature representations of individual nodes in graphs and we aggregate node embeddings to obtain graph embeddings.
For semi-supervised tasks, aside from comparing the results with the fully supervised results, we also compare our results with a state-of-the-art semi-supervised method: Mean Teachers Tarvainen & Valpola (0077)
4.3 EXPERIMENT CONFIGURATION
For graph classification tasks, we adopt the same procedure of previous works Niepert et al. (016)); Verma & Zhang (2017); Yanardag & Vishwanathan (2015); Zhang et al. (2018) to make a fair comparison and used 10-fold cross validation accuracy to report the classification performance. Experiments are repeated 5 times. We report results from previous papers with the same experimental setup if available. If results are parameter search according to the original paper. For node2vec Grover & Leskoved (2016), we took the result from Narayanan et al. (2017) but we did not run it on all datasets as the implementation details are not clear in the paper. For Deep Graph Kernels, we report the best result out of Deep WL Kernels, Deep GK Kernels, and Deep RW Kernels. For sub2vec, we report the best result out of its two variants: sub2vec-N and sub2vec-S. For all methods, the embedding dimension is set to 512 and parameters of downstream classifiers are independently tuned using cross validation on training folds of data. The best average classification accuracy is reported for each method. The classification accuracies are computed using LIBSS Chang Lin (201T). and the C' parameter was selected from {10-3 10-2 10a. 10
previously reported,
implement them and conduct hyper-
The QM9 dataset has 130462 molecules it. We adopt similar experimental settings as traditional semi-supervised methods Tarvainen & Valpola (2017); Laine & Aila (2016); Miyato et al. (2018) 2018). We randomly chose 5000 samples as labeled samples for training and another 10000 as validation samples, 10000 samples for testing, and use the rest unlabeled training samples. Note that use the exact same split when running the supervised model and the semi superviid model. We use the validation set do model selection and we report scores on the test set. All targets were normalized
7

---

Published as a conference paper at ICLR 2020
Dataset
MUTAG
PTC-MR
RDT-B
RDT-M5K
IMDBB-B
IMDB-M
(No. Graphs)
188
344
2000
4999
1000 2
1500
(No. classes)
2
2
2
3
(Avg, Graph Size)
17.93
14.29
429.63
508.32
19.77
13.00
Graph Kernels
RW
83.72 1: 1.50
57.85
1.30
OMR
OMR
50.68
1. 0.26
34.65
0.19
SP
85.22 1: 2.43
58.24
2.44
64.11 1 0.14
39.55 di 0.22
55.60
1: 0.22
37.99
0.30
GK
81.66
2.11
57.26
1.41
77.34
0.18
41.01 = 0.17
65.87 + 0.98
13.89
0.38
WL
80.72 +: 3.00
57.97
0.49
68.82
= 0.41
46.06 :: 0.21
72.30
:: 3.44
46.95
0.46
DGK
87.44 2.72
60.08
2.55
78.04
0.39
41.27 = 0.18
06.96
= 0.56
44.55
MLG
87.91 1.61
63.26
1.48
> Day
Day
0.25 06.55
11.17
Other Unsupervised Methods
node2vee
72.03 + 10.20
58.08 8.00
1.
59.99
graphZvec
83.15
60.17
75.78
0.26
71.1
0.54
InfoGraph
53.46
19.69
Table 1: Classification accuracy on 6 datasets. The result in bold indicates the best reported classification accuracy. The top half of the table compares results with various graph kernel approaches while bottom half compares results with other Late-of-the-ar unsupervised graph representation learning methods. 1 day' represents that the computation exceeds 24 hours. 'OMR' out of memory error.
Target MAE
Mu (0) 0.3201
Alpha (1)
HOMO
LUMO (3)
Gap (4)
R2
ZPVE16)
(3
(8)
H
Gr10)
Cx(I
0.5792
0.0060
0.0062
0.0091
10.0469
0.0007
0.32M
0.2934
0.2722
0.2948
0.2368
Semi- Supervised
Teachers
1.09
0.93
InfoGraph
1,02
0.97
1.02
0.99
1.01
0.71
0.96
0.85
0.93
0.93
0.99
1,00
InfoCrrphh"
0.99
0.94
0.99
0.99
0.98
0.49
0.52
0.44
0.58
0.57
0.54
0,83
Table 2: Results of semi-supervised experiments on QM9 dataset. The result in bold indicates the best performance. The top half of the table shows the mean absolute error (MAE) of the supervised model. The bottom half shows the eтrоr ratio (with respect to supervised result) the semi-supervised models using the same underlying model. Lower scores better and values less than 1.0 indicate better performance than the supervised baseline.
to have mean 0 and variance 1. We minimize the mean squared error between the model output and the target, although we evaluate mean absolute error.
4.4 MODEL CONHIGRRATION
For the unsupervised experiments, we use the Graph Isomorphism Network (GIN) Xu et al. (2018a) 8a). For the semi-supervised experiments, adopt the same model in Gilmer et al. (2017) (enn-s2s). As recommended in Oliver et al. (2018), we use the exact same underl anderlying model architecture when comparing cemi-supervised learning approaches but instead to provide a rigorous comparative analysis in a common framework. In both scenarios, models were trained using SGD with the Adam optimizer. We
goal is not to produce kate-of-the--a results,
Pytorch Paszke al. (2017) and the Pytorch Geometric Fey & Lenssen (2019) libraries for all our experiments. For detailed hyper-parameter settings and architecture detail of the discriminator, see Appendix C.
RESULTS
The results evaluating unsupervised graph level representations using downstream graph classifica- tion tasks are presented in Table We show results from six methods including three -taee-of-the-art graph kernel methods: WL Shervashidze et al. (2011). DGK Yanardag & Vishwanathan (2015), and MLG Kondor & Pan (2016). While these kernel methods perform well on individual datasets. none of them are competitive across all of the datasets. Additionally, MLG suffers from a long run time and take more than 24 hours to run the two larger benchmark datasets. We find that InfoGraph
8

---

Published as a conference paper at ICLR 2020
outperforms all of these baselines on 4 out of 6 of the datasets. In the other datasets, InfoGraph still has very competitive performance.
The results of the semi-supervised learning experiments on the molecular property prediction task are presented in Table 2 We observe that by simply combining the supervised objective with the unsupervised infomax objective (InfoGraph) obtains better performance compared to the purely supervised models on out of 12 of the targets. However, in out of 12 targets it does not obtain better performance and 4 out of 12 targets, results in poorer performance. This "negative transfer" effect may be caused by the fact that the supervised objective and the unsupervised objective favor different information or different latent semantic space. This effect alleviated with InfoGraph* our modified version of InfoGraph for semi-supervised learning. InfoGraph* improves over the supervised model in all the 12 targets. InfoGraph* obtains the best result on 11 targets while the Mean Teacher method obtains the best results on 2 targets (with one overlap). However, the Mean Teacher model yields worse performance on 2 targets when compared the supervised result.
6 CONCLUSION AND FUTURE WORK
In this paper, we propose InfoGraph to learn unsupervised graph-level representations and InfoGraph* for semi-supervised learning. We conduct experiments graph classification and molecular property prediction tasks to evaluate these two methods. Experimental results show that InfoGraph and InfoGraph* are both very competitive with state-of-the-arr methods. There are many research works on semi-supervised learning on image data, but few of them focus on semi-supervised learning for graph structured data. the future, we aim to explore semi-supervised frameworks designed specifically for graphs.
ACKNOWLEDGMENTS
We would like to thank Shengchao Liu and Weihua Hu for the extremely helpful discussions and comments.
REFERENCES
Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learning for subgraphs. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 170-182. Springer, 2018.
David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):9931102. 2003.
Karsten M Borgwardt and Hans-Peter Kriegel. Shortest-path kernels graphs. In Fifth IEEE international conference on data mining (ICDM'05), pp. 8-pp. IEEE, 2005.
Chih-Chune Chang and Chih-Jen Lin. Libsvm: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):27, 2011.
Olivier Chapelle. Bernhard Schölkopf, Alexander Zien, et al. Semi-supervisee learning, vol. 2. Cambridge: MIT Press. Cortes, C., & Mohri, M.(2014). Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103126, 2006.
Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks universal machine learning framework for molecules and crystals. Chemistry of Materials, 31(9):3564-355 3572, 2019a.
Pengfei Chen, Weiwen Liu, Chang Yu Hsieh, Guangyong Chen, and Shengyu Zhang. Utilizing edge features in graph neural networks via variational information maximization. arXiv preprint 196.v:1906 2019b.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. Proceedings of the 25th international conference on Machine learning, pp. 160-167. ACM, 2008

---

Published as a conference paper at ICLR 2020
Gerben KD de Vries. A fast approximation of the weiseeiler-ehmman graph kernel for rdf data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 606-621. Springer, 2013.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional networks graphs for learning molecular fingerprints. In Advances in neural information processing systems, pp. 2224-2232, 2015.
Aasa Feragen, Niklas Kasenburg, Jens Petersen, Marleen de Bruijne, and Karsten Borgwardt. Scalable kernels for graphs with continuous attributes. In Advances in Neural Information Processing Systems, pp. 216 216-224, 2013.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint Xrv: 1.Xiv:1003.00 903.02428, 2019.
Sheng Gao, Ludovic Denoyer, and Patrick Gallinari. Temporal link prediction by integrating content and structure information. In Proceedings of the 20th ACM international conference on Information and knowledge management, pp. 1169-1174. ACM, 2011.
Thomas Gärtner, Peter Flach, and Stefan Wrobel. On graph kemels: Hardness results and efficient alternatives. In Learning theory and kernel machines, pp. 149-143. Springer, 2003.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Baariing-Volume Volume 70, pp. 1263-1272. JMLR. org, 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pp. 529-536, 2005.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855-864. ACM, 2016.
Devon Hjelm, Alex Fedorov, Samuel MarchilMa,,,,,,,,,, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint w0w.11888.06700 2018
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Pre-training graph neural networks. arXiv preprint arXiv: 1905. 12265, 2019.
Sergey Ivanov and Evgeny Burnaev. Anonymous walk embeddings. arXiv preprint arXiv: 1805. 11911, 2018.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv: 1707.0738. 2017.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. arXiv preprint arXiv: 100.1043224444 2018.
Hisashi Kashima, Koji Tsuda, and Akihiro Inokuchi. Marginalized kernels between labeled graphs In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 321-328, 2003.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: A.1i:1802..0988 2018.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
Xrv preprint
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. arXiv preprint arXiv: 2018
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint aivi 1609.660222 609.02907, 2016.
10

---

Published as a conference paper at ICLR 2020
Risi Kondor and Horace Pan. The multiscale laplacian graph kemel.. In Advances in Neural Information Processing Systems, pp. 2990-2998, 2016.
Nils M Kriege, Pierre-Louis Giscard, and Richard Wilson. On valid optimal assignment kernels and applications to graph classification. In Advances in Neural Information Processing Systems, pp. 1633-1631, 2016.
Nevan Krogan, Gerard Cagney, Haiyuan Yu, Gouqing Zhong. Xinghua Guo, Alexandr Ignatchenko, Joyce Li, Shuye Pu, Nira Datta, Aaron P Tikuisis, et al. Global landscape of protein complexes in the yeast saccharomyces cerevisiae. Nature, 440(7084):637, 2006.
Samuli Laine and Timo Aila. Temporal ensembling for semi- comi-supervised learning. arXiv preprint arXiv: 11w.00210.2222 2016.
Quoc Le and Tomas Mikolov. Distributed representations of sentences and documents. In International conference on machine learning, pp. 1188-11999 1196, 2014.
Kong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop Challenges in Representation Learning, ICML, volume 3, pp. 2, 2013.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint arXiv: 1icKv:1301.33: 1301.3781, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado. and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, pp. 3111-3119, 2013b.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and supi-supee learning. IEEE transactions on pattern analysis and machine intelligence, 11(8):1979-199) 2018.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-con estimation. In Advances in neural information processing systems, pp. 2265-2273, 2013.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen andvYang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. CoRR, abs/1707. 0500000005 2017. /ttp::tt////rxxx..or////////// hatpgg//b//1700 05005
Marion Neumann, Novi Patricia, Roman Garnett, and Kristian Kersting. Efficient graph kernels by randomization. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 378-393. Springer, 2012.
Mark EJ Newman and Michelle Girvan. Finding and evaluating community structure in networks. Physical review E, 69(2):0261 13, 2004.
Hai Nguyen, Shin-ichi Maeda, and Kenta Oono. Semi-supervised learning of hierarchical representa- tions of molecules using neural message passing. arXiv preprint arXiv: 2017.
Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In International conference on machine learning, pp. 2014 20233, 2016.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep sunissupre learning algorithms. In Advances Neural Information Processing Systems, pp. 3235-3246. 2018.
Francesco Orsini, Paolo Frasconi, and Luc De Raedt. Graph invariant kernels. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345111 2009.
11

---

Published as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Georgios A Paylopoulos, Maria Secrier, Charalampos Moschopoulos, Theodoros Soldatos, Sophia Kossida, Jan Aerts, Reinhard Schneider, and Pantelis G Bagos. Using graph theory to analyze biological networks. BioData mining, 4(1):10, 2011.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710. ACM, 2014.
Nataša Pržulj. Biological network comparison using graphlet degree distribution. Bioinformaticc, 23 12)x177--183, 2007.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1:140022, 2014.
Michael Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas Dietterich. To transfer or not to transfer. In NIPS 2005 workshop transfer learning, volume 898, pp. 3, 2005.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Mutual exclusivity loss for semi-supervised deep learning. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 1908-1912.
IEEE, 2016.
Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. Efficient graphlet kernels for large graph comparison. Artificial Intelligence and Statistics, pp. 488-495, 2009.
Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lchman graph kernels. Journal of Machine Learning Research, 12(Sep): 2539-2561, 2011.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In Proceedings of the 24th international conference on world wide web, pp. 1067-1077. International World Wide Web Conferences Steering Committee, 2015.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve superviid deep learning results. In Advances in neural information processing systems, pp. 195--1204 2017.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint arXiv: 10341, 2018.
Saurabh Verma and Zhi-Li Zhang. Hunt for the unique, stable, sparse and fast feature learning on graphs. Advances in Neural Information Processing Systems, pp. 88-98, 2017.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings Machine Learning Research, pp. 6438 6447, Long Beach, Califomia, USA, 09-15 Jun 2019a. PMLR. URL http::/proceedinnee mlr. www.::awww....p....a...mm.me... wwtpp///e//eeeennneeee 9a. html
Vikas Verma, Alex Lamb, Kannala Juho, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for sumi-super learning. In Sarit Kraus (ed.), Proceedings of the Twenty- International Joint Conference Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019. ijcai.org, 2019b. doi: 10.24963//jcai.2019 URL httpps:/ddo org/ #rg/10.c 24963/ Jeai 2019
Fei Wang, Tao Li, Xin Wang, Shenghuo Zhu, and Chris Ding. Community discovery using nonnega- tive matrix factorization. Data Mining and Knowledge Discovery, 22(3):493-521, 2011.
12

---

Published as a conference paper at ICLR 2020
Boris Weisfeiler and Andrei A Lehman. A reduction of graph to canonical form and algebra arising during this reduction. Nauchno- Nauchno-Techhicee Technicheskaya Informatsia, 2(9):12-16, 1968.
Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14): 145301, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv: 180.008800 2018a.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanic Jegelka. Representation learning on graphs with jumping knowledge networks. arXiv preprint arXiv:1806.03536, 2018b.
Yan Yan, Shangzhao Qiu, Zhuxuan Jin, Sihong Gong, Yun Bai, Jianwei Lu, and Tianwei Yu. Detecting subnetwork-level dynamic correlations, Bioinformatics, 33(2):256-265, 2017.
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1363-1374. ACM, 2015.
Rendong Yang, Yun Bai, Zhaohui Qin, and Tianwei Yu. Egonet: identification of human disease ego-network modules. BMC genomics, 15(1):314, 2014.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In Advances in Neural Information Processing Systems, pp. 4800-4810, 2018
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference on Machine learning (ICML-03), pp. 912 912-919, 2003.
13

---

Published as a conference paper at ICLR 2020
A RELATED WORK
A. GRAPH KERNELS
Shervashidze et al. (009)) random walk and shortest path kernels Kashima et al. (2003); Borgwardt & K (2005), and the Weisfeiler-Lehman
Popular graph kernels are graphlets Pržulj
(2077);
subtree kernel Shervashidze et al. [201]] lp. Furthermore, deep graph kernels Yanardag & Vishwanathan
(2015), graph invariant kernels Orsini et al. (2015), optimal assignment graph kernels Kriege et al. (0016) and multiscale Laplacian graph kernels Kondor & Pan (2016) have been proposed with the goal to redefine kemel functions to appropriately capture sub- structural similarity at different levels. Another line of research in this area focuses on efficiently computing these kernels either through exploiting certain structural dependencies, or via approxination//aaddmiaiiii Ferraeememaaiiteoeeeeeeee al. (2013) de Vries (2013) Neumann et al. (2012).
A.2 CONTRASTIVE METHODS
Contrastive methods are central many popular word-embedding methods Collobert & Weston (2008);
Mnih & Kavukcuoglu (2013); Mikolov et al. (2013b). Word2vec Mikolov et al. (2013a) is an unsupervised algorithm which obtains word representations by using the representations to predict context words (the words that surround it). Doc2vee Le & Mikolov (2014) is an extension of the continuous Skip-gram model that predicts representations of words from that of a document containing them. Researchers extended many of these unsupervised language models to learn representations of graph-structured input Adhikari et al. (2018); Narayanan et al. (017)). For example, graph2vec Narayanan al. (017)) extends Doc2vec arbitrary graphs. Intuitively, for graph2vec a graph and the rooted subgraphs in it correspond to a document and words in paragraph vector, respectively. One of the technical contributions of the paper is using the Weisfeiler-Lchman relabelling algorithm Weisfeiler & Lehman (1968); Shervashidze et al. (001 I) to enumerate all rooted subgraphs up to specified depth. AWE (Anonymous Walk Embeddings) [vanov & Burnaev (2018) another method based CBOW framework. instead of using rooted subgraphs as words like graph2vec, AWE considers anonymous walk embeddings for the same source node as co-occurring words. InfoGraph has the two advantages when compared with these methods. First, InfoGraph learns representations directly from data instead of utilizing hand-crafted procedures (i.e. Weiisfelle-Lehmane relabelling algorithm in graph2vec and random walkw in AWE). Second. InfoGraph has clear objective that can be easily combined with other objectives. For example, InfoGraph*, the semi-supervised method that we proposed.
B SEMI-SUPERVISED LEARNING
Here we discuss the most common class of SSL methods which involve adding an additional loss term to the training of a neural network as they are pragmatic and are currently the state-of-the-art on image classification datasets.
Entropy Minimization (EntMin): EntMin Grandvalet & Bengio (2055 adds loss term applied that encourages the network make "confident" (low-entropy) predictions for all unlabeled examples, regardless of their class.
Pseudo-labeling Lee (2033) proceeds by producing "pseudo-labels" for unlabeled input data points using the prediction function itself over the course of training. Pseudo-labels which have a corresponding class probability that is larger than a predefined threshold are used as targets for a the standard supervised loss function.
TI-odel: Neural networks can produce different outputs for the same input while common regu- larization techniques such as data augmentation, dropout, and adding noise are applied. 11-Model Laine & Aila (2016); Sajjadi et al. (2016) adds a loss term which encourages the distance between a network's output for different passes of unlabeled data through the network to be small.
Virtual Adversarial Training: Instead of relying on the built-in stochasticity as in II-Model, Virtual Adversarial Training (VAT) Miyato et al. (2018) directly approximates tiny perturbation to add to the input which would most significantly affect the output of the prediction function. This pertubation can be approximated with an extra back-propagation for each optimization step.
14

---

Published as a conference paper at ICLR 2020
Mean Teacher: A difficulty with the II-model approach is that it relies potentially unstable "target" prediction, namely the second stochastic network prediction which can rapidly change over the course of training. As result. Tarvainen & Valpola (2017) proposed to obtain stable target output for unlabeled data by setting the target to predictions made using an exponential moving average of parameters from previous training steps.
C DATASETS
C.1 GRAPH CLASSIFICATION DATASETS
MUTAG contains 188 mutagenic aromatic and heteroaromatic nitro compounds with 7 different discrete labels. PTC is a dataset of 344 different chemical compounds that have been tested for carcinogenicity in male and female rats. This dataset has 19 discrete labels. HIDB-BINARY and IMDB-MULTI are movie collaboration datasets. Each graph corresponds to an ego-network for each actor/actress, where nodes correspond to actors/actresses and an edge is drawn between two actors/actressess they appear in the same movie. Each graph is derived from a pre-specified genre of movies, and the task to classify the genre graph it is derived from. REDDIT-BINARY and REDDIT-MULTISK are balanced datasets where each graph corresponds to an online discussion thread and nodes correspond to users. An edge drawn between two nodes if least of them responded to another's ss comment. The task is to classify each graph to the community or subreddit that it belongs to.
C.2 QM9
All molecules in the dataset consist of Hydrogen (H), Carbon (C), Oxygen (O), Nitrogen (N), and Flourine (F) atoms and contain up to non-Hydrogen atoms. all, this results in about 134,000 drug-like organic molecules that span wide range of chemical compositions and properties. A total of 12 interesting and fundamental chemical properties are pre-computed for each molecule. For a detailed description of the properties in the QM9 dataset, section 10.2 of Gilmer al. (2017).
D MODEL CONFIGURATION
For the unsupervised experiments, (2018a). GNN layers are chosen from {4, 8, 12). Initial learning rate is chosen from the set The number of epochs chosen from {10, 20, 100). The batch size is
the Graph Isomorphism Network (GIN) Xu et al.
210-2,10 10-3, 110-,,10-,,10-11 10-1).
set to 128.
For the remi-supervisedd experiments, the number of set2set computations is set to 3. Model were trained with an initial learning rate 0.001 for 500 epochs with a batch size 20. For the supervised case, the weight decay is chosen from {0, 10-3 10-4), For InfoGraph and InfoGraph*, À chosen from {10 10 10 53.
The discriminator scores global-patch representation pairs by passing two representations different non-linear transformations and then takes the dot product of the two transformed representations. Both non-linear transformations are parameterized by 3-laerred beed-fowward neural networks with jumping connections. Following each linear layer is a ReLU activation function.
15

---

Published as a conference paper at ICLR 2020
E CONVERGENCE PLOT
To prove that the objective Bq with multiple loss terms can be optimized, we provide a convergence plot of InfoGraph*. We can see that the three loss terms all converge after around 150 epochs of training.
InfoGraph* Convergence Plot
Supervised
Eq.
1.0
0.5
0.0
Unsupervised (second
-29
-31
Mutual Information Loss
-60
W
25
50
75
100
125
150
175
Figure 3: Convergence plot of InfoGraph on QM9 target 7.
16


Reviews:
[{"review_id": "rkl8SAaQ5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a graph-level representation, which extends the existing node-level representation learning mechanism. Besides, both unsupervised and semi-supervised learning are leveraged for InfoGraph and InfoGraph*, receptively. The authors naturally apply Deep Graph Infomax, a contrastive representation learning method, for the whole graph level instead of the previous node embedding learning. The experiments on graph classification and molecular property prediction indicate the effectiveness, even compared to the supervised methods.\n\nThe learned graph-level representation looks good to me. Instead of some heuristics based graph-level pooling, the proposed method automatically figure out the best way to produce the fixed-length feature vector in a data-driven approach. Such motivation is pretty reasonable and natural for me. Also, the usage of both unsupervised and semi-supervised learning procedure is well-motivated. Overall speaking, the paper is well written. The definitions of problems, the details of methods, and the settings of experiments are clear to me.\n\nI have some questions and suggestions for the authors:\n1. The overall writing looks good to me. Besides, It could be much better if the authors could provide more and better illustrations for the method. Both figures 1 and 2 are not that informative to me, to be honest. I know it could hard to visualize the graph-level representation, but it worth it. There are many steps and equations in the paper, and the illustration could play an essential role in putting all the steps together to demonstrate the big picture.\n\n2. The authors carefully discuss the difference between this submission and concurrent work (Information Maximizing Graph Network) or existing work (Deep Graph Infomax), which helps a lot for the reader to understand the literature better. It could be much better if this submission could be more self-contained. For example, for the semi-supervised learning setting, the authors let the reader read some external papers. I am suggesting that a small section in the appendix could make life much more comfortable.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJljp73E9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an unsupervised method for graph embedding. The authors seek to obtain graph representations by maximizing the mutual information between graph-level and patch-level\nrepresentations. They also consider a semi-supervised task when the Mutual Information-based criterion has an additional term which quantifies a classification error, obtained when constructing a classifier based on the obtained graph representations. \n\nDespite having good experimental results, the proposed approach is rather a mix of previous works and hence not novel. \n\nIn particular, the main building block of the embedding algorithm, the target functional based on mutual information, was borrowed from Deep Graph Informax paper. The differences, listed by the authors, are only of technical nature. Advantage of using it for unlabeled data is poorly motivated: why we can learn smth useful when maximizing the mutual information between graph-level and patch-level representations obtained via GNN? What if patch-level representations are not sufficiently characteristic to have anything in common with the graph?\n\nThere is no discussion of [1], which uses CBOW framework, has theoretical properties, and produces good results in experiments. There is no comparison with GNN models such as [2]. \n\nMinor comments: please, correct fonts - they are different in formulas 6,7 and 5\n\nI would be more interested to see explanation of the obtained results for each particular dataset (e.g. why MUTAG has 89% accuracy and PTC 61%); what so different about dataset and whether we reached a limit on most commonly used datasets. \n\n[1] Anonymous Walk Embeddings? ICML 2018, Ivanov et. al. \n[2] How Powerful are Graph Neural Networks? ICLR 2019, Xu et. al.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlXq_au9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new graph representation learning method for the whole graph under both unsupervised and semi-supervised setting. Different from existing ones using graph kernel, or graph2vec, the proposed InfoGraph is able to extract graph-level representation with fixed-length features that are generalized well. Basically, InfoGraph is parameterized by graph neural networks, but guided by mutual information loss. Experiments on both unsupervised and semi-supervised experiments on popular benchmarks demonstrate the effectiveness of InfoGraph and InfoGraph*\n\n*  The paper is well written and easy to follow, and the research problem is of great value in different fields.\n\n* In general, the novelty of this paper is ok, but it’s partially based on Deep InfoMax (DIM) published recently. This may undermine the novelty of this paper somehow.\n\n* Authors change the fonts in equation from italic to non-italic in Eq. (6), please make sure to use one format throughout the paper.\n\n* Why Jensen-Shannon MI estimator is used in Eq. (5) instead of other estimators for MI, and any more explanations or motivations here?\n\n* Eq. (7) and (8) in facts jointly optimize between two networks \\phi and \\varphi, but little details about the optimization have been exposed. Also, we are not sure if the loss from the two models can converge finally. Better to show some qualitative results and analysis.\n\n* READOUT seems to play a critical role in building the global representation, however, it is unclear if other READOUT function will work well, and why the current one is feasible. Please explain.\n\n* In semi-supervised setting, it seems InfoGraph* is comparable to the SOTA MeanTeachers model on 12 targets. I am not sure if this can reflect the true performance of the proposed model. Maybe another dataset will be able to highlight the superiority.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a graph embedding method for the whole graph under both unsupervised and semi-supervised setting. It can extract a fixed length graph-level representation with good generalization capability. All reviewers provided unanimous rating of weak accept. The reviewers praise the paper is well written and is value to different fields dealing with graph learning. There are some discussions on the novelty of the approach, which was better clarified after the response from the authors. Overall this paper presents a new effort in the active topic of graph representation learning with potential large impact to multiple fields. Therefore, the ACs recommend it to be an oral paper.
Decision: Accept (Spotlight)