GraphMix: Regularized Training of Graph Neural Networks for Semi-Supervised Learning
Authors: Vikas Verma
Meng Qu
Alex Lamb
Yoshua Bengio
Juho Kannala
Jian Tang
Venue:  (2020)
Keywords: Regularization
Graph Neural Networks
Mixup
Manifold Mixup
Semi-supervised Object Classification over graph Data

Abstract: We present GraphMix, a regularization technique for Graph Neural Network based semi-supervised object classification, leveraging the recent advances in the regularization of classical deep neural networks. Specifically, we propose a unified approach in which we train a fully-connected network jointly with the graph neural network via parameter sharing, interpolation-based regularization and self-predicted-targets. Our proposed method is architecture agnostic in the sense that it can be applied to any variant of graph neural networks which applies a parametric transformation to the features of the graph nodes. Despite its simplicity, with GraphMix we can consistently improve results and achieve or closely match state-of-the-art performance using even simpler architectures such as Graph Convolutional Networks, across three established graph benchmarks: Cora, Citeseer and Pubmed citation network datasets, as well as three newly proposed datasets : Cora-Full, Co-author-CS and Co-author-Physics.

Under review as a conference paper at ICLR 2020
GRAPHMIX: REGULARIZED TRAINING OF GRAPH NEURAL NETWORKS FOR SEMI-SUPERVISED LEARN- ING
Anonymous authors Paper under double-blind review
ABSTRACT
We present GraphMix, regularized training scheme for Graph Neural Network based semi-supervised object classification, leveraging the recent advances in the regularization of classical deep neural networks. Specifically, we propose a unified approach in which we train a fully-connected network jointly with the graph neural network via parameter sharing, interpolation- regularization and self- predicted-targeeeeee -targess.... Our proposed method is architecture agnostic in the sense that it can applied to any variant of graph neural networks which applies a parametric transformation to the features of the graph nodes. Despite its simplicity, with GraphMix we can consistently improve results and achieve or closely match state-of-ther tt-hhe-rr art performance using even simpler architectures such Graph Convo- lutional Networks, across three established graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, well three newly proposed datasets: Cora-Full, Co-outhor- CS and De-author-Physice
INTRODUCTION
Due to the presence of graph structured data across wide variety of domains, such as biological networks, social networks and telecommunication networks, there have been several attempts to design neural networks that can process arbitrarily structured graphs. Early work includes (Gori)
et al. Scarselli et al. 2009) which propose neural network that can directly process most type of graphs e.g., acyclic, cyclic, directed, and undirected graphs. More recent approaches include (Bruna)
et al. 2013 Henaff et al. 20155 Defferrard et al.) 2016 Kipf & Welling 2016] Gilmer et al. 2017 Hamilton et al. 2017] Veličković et al. 2018 2019 Qu et al.) 2019 Gao & Ji, 2019 Ma et al. 2019), among others. Many of these approaches of Semi-supervised learning over graph structured data (Zhou et al. 2018). Much of this research effort has been dedicated to developing novel architectures.
designed for addressing the important problem
Unlike many existing works which try to come up with the new architectures, we re-synthesize ideas from the recent advances in the regularization of the classical neural network, and propose an architcttur--gnosticc framework for regularized training of graph neural network based semi- supervised object classification. Recently, Pata-Augmentatiin based regularization has been shown to be very effective in other types of neural networks but how to apply these techniques in graph neural networks still under-explored. Our proposed method GraphMix [s a unified framework that utilizes interpolation based data augmentation (Zhang et al. 2018 Verma et al. 2019a) and earfettree-- prediction based data- aitia-augmentatioon techniques (Laine & Aila 2016) Tarvainen & Valpola 2017] Verma et a [019b] Berthelot et al. 2019). We show that with our proposed unified framework, we can achieve state-of-the-an performance even when using simpler graph neural network architectures
such Graph Convolutional Networks (Kipf & Welling 2017) and without incurring any significant additional computation cost.
code available

---

Under review as a conference paper at ICLR 2020
2
PROBLEM DEFINITION AND PRELIMINARIES
2.1 PROBLEM SETUP
We are interested in the problem of semi-supervised object classification using graph structured data. We can formally define such graph structured data as g (V. A), where y represents the set of nodes fei...., V, and is the adjacency matrix representing the edges between the nodes of
Each node in the graph has a corresponding d-dimensional feature vector x; E R.. The feature vectors of all the nodes X [x1..... ....... x,] are stacked together to form the entire feature matrix X E Rexd Each node belongs to one out of C' classes and can be labeled with a C-dimensional
one-hot vector y: E {0. 1)c. Given the labels of Y for few of the labeled nodes V2 V, the task is to predict the labels YU of the remaining nodes Vv = V Vt.
2.2 GRAPH NEURAL NETWORKS
Graph Neural Networks (GNN) learn the 4h layer representations of a sample by leveraging the representations of the samples N B(i in the neighbourhood of This is done by using aggregation function that takes as an input the representations of all the samples and the graph structure and outputs the aggregated representation. The aggregation function can be defined using the Graph Convolution layer (iipf & Welling 2017), Graph Attention Layer (Veličković et al. 2018), or any general message passing layer (Gilmer et al. 2017). Formally, let H; R*** be matrix containing the k-dimensional representation of n nodes in the 4A layer, then:
H = (H,WW, A)
where W E R is linear transformation matrix, k' is the dimension of (1 1)th layer and a is the aggregation function that utilizes the graph adjacency matrix A.
2.3 INTERPOLATION BASED REGULARIZATION TECHNIQUES
Recently, interpolation-basee techniques have been proposed for regularizing neural networks. We briefly describe some of these techniques here. Mixup (Zhang et al. 2018) trains a neural network on the convex combination of input and targets, whereas Manifold Mixup (Verma et al. (2019a) trains neural network on the convex combination the hidden states randomly chosen hidden layer and the targets. While Mixup regularizes neural network by enforcing that the model output should change linearly in between the examples in the input space, Manifold Mixup regularizes the neural network by learning better (more discriminative) hidden states.
Formally, suppose g + is a function that maps input sample to hidden states, f h + y is function that maps hidden states to predicted output, is a random variable drawn from Beta(a, a) distribution, Mix b) À * + (1 A) * b is an interpolation function, 'D is the data distribution, (x, y) and (x', yy)) pair of examples sampled from distribution and be loss function such as cross-entropy loss, then the Manifold Mixup Loss is defined as:
C
E
E (x' y'>D Reta(
E
(L(x) Mix,
(x,Y)
We use above Manifold Mixup loss for training auxiliary Fully-ccomeeted-neewwwk as described in Section and Line 6 and 12 of Algorithm [
GRAPHMIX
MOTIVATION
Data Augmentation is arguably the simplest and most efficient technique for regularizing neural network. In some domains, such as computer vision, speech and text, there exist efficient data augmentation techniques, for example, random cropping, translation Cutout (Devries & Taylor
2

---

Under review as a conference paper at ICLR 2020
Lrex
CNNN
FCN
GNN
YEes
YeNy
W2
Shared Parameters
W2.
HpoN
HaNy
Wi
Shared Parametars
Wi, A
X
X
Predicted targets generated by GNN
Figure 1: The procedure for training with GraphMix The Colly-Conneeted Network (FCN) and the Graph Neural Network (GNN) share linear transformation matrix (W) applied on the node features. The FCN is trained using Manifold Mixup by interpolating the hidden states HECN and the corresponding labels Y. This leads to better features which are transferred to the GNN via parameter sharing. The predicted targets generated by the GNN for unlabeled data used augment the input data for the FCN. The FCN and the GNN losses are minimized jointly by alternate minimization.
2017) for computer vision, Ko et al. (2015) Park et al. (2019) for speech and Xie et al. (2017)
for text domain. However, data augmentation for the graph-structured data remains under-explored. There exists some recent work along these lines but the prohibitive additional computation cost (see
Section (5.) introduced by these methods make them impractical for real-world large graph datasets. Based on these limitations, our main objective is to propose an efficient data augmentation technique for graph datasets.
Recent work based on interpolation-based data augmentation (Zhang et al. 2018 Verma et al. 2019a has seen sizable improvements in regularization performance across number tasks. However, these techniques are not directly applicable to graphs for an important reason: Although we can create additional nodes by interpolating the features and corresponding labels, remains unclear how these new nodes must be connected to the original nodes via synthetic edges such that the structure of the whole graph is preserved. To alleviate this issue, we propose to train an auxiliary Fully-commectdd-net (FCN) using Manifold Mixup as discussed in Section 3.2 Note that the FCN only uses the node features ( not the graph structure), thus the Manifold mixup loss in Eq. 2 can be directly used for training the FCN. Furthermore, drawing inspiration from the success of self- supervised semi-supervised learning algorithms (self- seelf-rredicte-ttargeeee be also interpreted 2019), we explore self-supervision in the training of GNNs. We note that self-supervision has already been explored for unsupervised representation learning from graph structured data (Veličković et al.) 2019), but not for semi-supervised object classification over graph structured data. Based on these challenges and motivations we present our proposed approach GraphMix for training Graph Neural Networks in the following Section.
based algorithms which can form of data-augmentationn techniques) (Verma al. 2019b; Berthelot et al.
3.2 METHOО
GraphMix augments the vanilla GNN with a Fully Connected Network (FCN). The FCN loss is computed using the Manifold Mixup discussed in Section 2.3 and the GNN loss is computed in the standard way. The Manifold Mixup training of FCN facilitates learning more discriminative node representations. How to transfer these discriminative node representations to the GNN? We apply parameter sharing between FCN and GNN to facilitate this. Using these more discriminative representations of the nodes, well as the graph structure, GNN loss is computed in the usual way.
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 GraphMix A procedure for improved training of Graph Neural Networks (GNN)
1: Input: A GCN: 9XX, 0), a FCN: f(X, 0) which shares parameters with the GCN. Beta distribution parameter a for Manifold Mixup Number of random perturbations K, Sharpening lemperature T. Consistency parameter yy Number epochs N. r(t): rampup function increasing the importance of consistency regularization. (Xz, YL) represents labeled samples and Xy represents unlabeled samples.
2: for t = to N do
3:
== random(00,) // generate randomly 0 or i= then
4:
5:
A~ Beta(a, a) /1 Sample mixing coefficient from Beta distribution Leop L (X, Vt. // supervised loss from FCN using the Manifold Mixup
6:
7:
for k- 1 to K do SaandomaPPrturraaioons.
8:
XU.A=
// Apply 8th round of random perturbation X
9:
end
10:
YU = K Ev | Xu.u; // Compute average predictions across K perturbations of Xu
using the GCN
11:
Yu Sharpen(Yu, (U,T), T) 11 Apply temperature sharpening to the average prediction
12:
Luoup (XX, 0, 0,A), /1 unsupervised loss from FCN using the Manifold Mixup
S=Cupp + (()) Luuup
1/ Total loss is weighted sum of supervised and unsupervised FCN
loss
14:
else
15:
L= 0(,(Xx,,,A)) A1) // Loss using the vanilla GCN
end
end for
18:
return C
Both the FCN loss and GNN loss are optimized in an alternating fashion during training. Furthermore, the predicted targets from the GNN are used to augment the training set of the FCN. In this way, both made
FCN and GNN facilitate each other's 's learning process. At inference time, the predictions using only GNN. The diagrammatic representation of GraphMix is presented in Figure I and the full algorithm is presented in Algorithm
The GraphMix framework can be applied to any underlying GNN as long as the underlying GNN ap- plies parametric transformations to the node features. In our experiments, we show the improvements over GCN (Kipf & Welling 2016). GAT (Veličković et al. 2018)) and Graph U-Net using GraphMix. Furthermore, (raphMix(GCN] framework does not add any significant computation cost over the underlying GNN, because the underlying GNN is trained in the standard way and the FCN training requires trivial additional computation cost for computing the predicted-taree ttrgetss (Section 3.2.1 and 3.2.1 and the interpolation function Mix (a, b) in Section 2.3 There are no additional memory requirements for GraphMix(GCN), since FCN and GNN share the parameters.
Some implementation considerations. For Manifold Mixup training of FCN, we apply mixup only in the hidden layer. Note that in Verma al. (2019)), the authors recommended applying mixing in a randomly chosen layer (which also includes the input layer) at each training update. However, we observed under-fittin fitting when applying mixpp randomly at the input layer or hidden layer. Applying mixup only in the input layer also resulted in underfitting and did not improve test accuracy.
The performance of self-supervision based algorithms such as GraphMix is greatly affected by the accuracy of the predicted targets. To improve the accuracy of the predicted targets, we applied the average of the model prediction on random perturbations of an input sample as discussed in Section 3 and sharpening as described in Section 3.2.2 Further, we draw similarities and difference of GraphMix w.r.t. Co-training framework in the Section 3.2.3
3.2.1 ACCURATE TARGET PREDICTION FOR UNLABELED DATA
Recent state-of-the-as semi-supervised learning methods use a teacher model to accurately predict targets for the unlabeled data. These predicted targets on the unlabeled data are used as "true labels" for further training of the model. The teacher model can be realized as temporal ensemble of the student model (the model being trained) (Laine & Aila 2016) or by using an Exponential Moving Average (EMA) of the parameters of the student model (Tarvainen & Valpola 2017). Another recently proposed method for accurate target predictions for unlabeled data is to use the average of the predicted targets across K random augmentations of the input sample (Berthelot al. 2019).
4

---

Under review as a conference paper at ICLR 2020
Along these lines, in this work, compute the predicted target as the average of predictions on K drop-out versions of the input sample. We also used the EMA of the student model but it did not improve test accuracy across all the datasets (see Section 4.4 for details).
3.2.2 ENTROPY MINIMIZATION
Many recent vemi--uperrisse learning algorithms (Laine & Aila 2016 Miyato et al. 2018 Tarvainen
& Valpola 2017 Verma et al. 2019b) are based on the cluster assumption (Chapelle al. 2010))
which posits that the class boundary should pass through the low-density regions of the marginal data distribution. One way to enforce this assumption is to explicitly minimize the entropy of the model's predictions p(y 2. 8) on unlabeled data by adding an extra loss term to the original loss term (Grandvalet & Bengio 2005). The entropy minimization can also achieved implicitly by modifying the model's prediction
the unlabeled data such that the prediction has low entropy and using these low-entropy predictions as targets for the further training of the model. Examples include "Pscudolabels" (Lee 2013) and "Sharpening" (Berthelot et al.) (019). Pscudolabeling constructs hard(1-hot) labels for the unlabeled samples which have "high-confidens many of the unlabeled samples may have "low-confidence predictions", they can not be used in the Pseudolabeling technique. On the other hand, Sharpening does not require "high-confidence predictions". and thus it can be used for all the unlabelled samples. Hence in this work, we use Sharpening for entropy minimization. The Sharpening function over the model prediction p(y|æ,, 0) can be formally defined as follows (Berthelot et al. 2019), where T the temperature hyperparameter and C' is the number of classes:
predictions". Since
Sharpen(p,, T) = p.¹
Σ
(3)
3.2.3 CONNECTION TO CO-TRAINING
The GraphMix approach can be seen as a special instance of the Co-training framework (Blum & Mitchell 1998). Co-training assumes that the description of an example can be partitioned into two distinct views and either of these views would be sufficient for learning if we had enough labeled data. In this framework, two learning algorithms are trained separately on each view and then the prediction of each learning algorithm on the unlabeled data is used to enlarge the training set of the other. Our method has some important differences and similarities to the Co-training framework. Similar to Co-training, we train two neural networks and the predictions from the GNN are used to enlarge the training set of FCN. The important difference is that instead of using the predictions from the FCN to enlarge the training set for the GNN, we employ parameter sharing for passing the learned information from FCN to GNN. In our experiments, directly using the predictions of the FCN for GNN training resulted in reduced accuracy. This is due to the fact that the number of labeled samples for training the FCN is sufficiently low and hence the FCN does not make accurate enough predictions. Another important difference is that unlike the co-training framework, FCN and GNN do use completely distinct views of the data: the FCN uses feature vectors X and the GNN uses the feature vector and adjacency matrix (X, A).
4 EXPERIMENTS
We present results for the GraphMix algorithm using standard benchmark datasets and the standard architecture in Section 4.2 and 4.3 We also conduct an ablation study on GraphMix in Section 4.4 to understand the contribution of various components to its performance. Refer to Appendix A.5 for implementation and hyperparameter details.
4.1 DATASETS
We use three standard benchmark citation network datasets for semi-supervised node classification, namely Cora, Citeseer and Pubmed. In all these datasets, nodes correspond to documents and edges correspond to citations. Node features correspond to the bag-of-words representation of the document. Each node belongs to of C classes. During training, the algorithm has access to the feature vectors and edge connectivity of all the nodes but has access to the class labels of only a few of the nodes.
5

---

Under review as a conference paper at ICLR 2020
For semi-ssupervised link classification, (Kumar et al., 2016; 2018). The nodes in these datasets correspond to the bitcoin users and the edge weights between them correspond to the degree of trust between the users. Following (Qu et al.) 2019), we treat edges with weights greater than 3 as positive instances, and edges with weights less than -3 are treated negative ones. Given a few labeled edges, the task is to predict the labels of the remaining edges. The statistics of these datasets as well as the number of sainiing/vaidaaioonnssst nodes is presented in Appendix A.I
two datasets Bitcoin Alpha and Bitcoin OTC from
4.2 SEMI-SUPERVISED NODE CLASSIFICATION
For basclines, we choose GCN (Kipf & Welling 2017 and the recent sfate-of-the-art methods GAT (Veličković al. 2018), GMNN (Qu et al. 2019) and Graph U-Net (Gao & Jil 2019). We additionally
use two self-traiming based baselines: in the first one, we trained a GCN with self-generated predicted targets, and in the second one, we trained FCN with self-generated predicted targets, named "GCN (with predieted-targets)" and "FCN (with predicted-targets)" respectively in Table 1 For generating the predicted-targets above two baselines, we followed the procedure of Section 3.2. I and Section 3.2.2 GCCphM GraphMix(GAT) and GraphMix U-Net) refer to the methods where underlying GNNs are GCN, GAT and Graph U-Net respectively. Refer to Appendix Section A.3 for implementation and hyperparameter details.
Shchur et al. (2188) has demonstrated that the performance of the current state-of-the-art Graph Neural Networks on the standard rrain/validation/tee split of the popular benchmark datasets (such as Cora, Citeseer, Pubmed, etc) is significantly different from their performance on the random splits. For fair evaluation, they recommend using multiple random partitions of the datasets. Along these lines, we created 10 random splits of the Cora, Citeseer and Pubmed with the same train/ validation/test number of samples as in the standard split. We also provide the results for the standard train/ fuin/valldation/tes: split in Table 5 in Appendix A.2
The results are presented in Table We observe that GraphMix always improves the accuracy of the underlying GNNs such as GCN, GAT and Graph U-Net across all the dataset, with GraphMix(GCN) achieving the best results.
We further present results with fewer labeled samples and results on larger datasets (Cora-Full Co-author-CS and Co-author-Physics) Section A.3 and Section A.4 respectively.
Table I: Results of node classification (% test accuracy) using 10 random Train/Validatie Testtto////ettttee split of datasets. [*] means the results are taken from the corresponding papers. We conduct 100 trials and report mean and standard deviation over the trials (refer to Table 5 in the Appendix for comparison with other methods on standard Train/Validations
split).
Algorithm
Cora
Citeseer
Pubmed
GCN
72.56+12.46 1..4
78.7440.99
GAT
1.86
70.41+1.81
78.98:00.9
Graph U-Net
1.65
67.55+0.69
76.79+2.45
GCN(with predicted-targent targets)
1.74
73.62+2.11
79.81
FCN(with predicted-targets)
75.19-13.53
70.49+11.9
73.40
GraphMix (GCN)
82.07=1.17
76.45+1155
80.72+1108
GraphMix (GAT)
80.63+1.31
1..0
10.14+1.51
GraphMix (Graph-U-Net)
8..1
72.85+1.711
78.47+0064
4.3 SEMI-SUPERVISED LINK CLASSIHICATION
In the Semi-supervised Link Classification problem, the task is to predict the labels of the remaining links, given a graph and labels of few links. Following (Taskar et al. 2004). we can formulate the link classification problem as a node classification problem. Specifically, given an original graph G. construct dual Graph G". The node set V of the dual graph corresponds to the link set E' of the original graph. The nodes in the dual graph G' are connected their corresponding links in the graph G share node. The attributes of node in the dual graph are defined as the index of the
6

---

Under review as a conference paper at ICLR 2020
nodes of the corresponding link in the original graph. Using this formulation, present results link classification on Bit OTC and Bit Alpha benchmark datasets the Table 2 As the numbers of the positive and negative edges show that GraphMix(GGN)) improves the performance over the haseline GCN method for both the datasets. Furthermore, the results of GraNh comparable with the recently proposed state-of-the-art method GMNN QQu et al. 2019)
strongly imbalanced, report the Fl score. Our results
Table 2: Results on Link Classification (%F1 score).
[*]
means the results are taken from the
corresponding papers
Algorithm
Bit OTC
Bit Alpha
DeepWalk Walk (Perozzi et al. 2014)
63.20
62.71
GMNN (Qu et 2019)
66.93
65.86
GCN GCN(with predicted-targets)
65.72:10.38
64.00=t0.19
65.15::0.29
64.56:10.21
FCN(with predicted-targets
60.13:10.40
59.74
GraphMix (GCN)
66.35:10.41
65.34+0.19
4.4 ABLATION STUDY
Since GraphMix consists of various components, some of which common with the existing literature of semi-supervised learning, we set out to study the effect of various components by systematieally removing or adding component from GraphMix We measure the effect of the following:
Removing the Manifold Mixup and predicted targets from the FCN training.
Removing the predicted targets from the FCN training.
Removing the Manifold Mixup from the FCN training.
Removing the Sharpening of the predicted targets.
Removing the Average of predictions for K random perturbations of the input sample
Using the EMA Tarvainen & Valpola 2017 of GNN for target prediction.
The ablation results for semi-supervised node classification are presented in Table 3 We did not do any hyperparameter tuning for the ablation study and used the best performing hyperparameters found for the results presented in Table We observe that all the components of GraphMix contribute to its performance, with Manifold Mixup training of FCN contributing possibly the most. Furthermore, observe that using the EMA model (which is an emsemble model) (Tarvainen & Valpola (017) for computing the predicted- targets could improve the performance of GraphMix for all the datasets.
Table 3: Ablation study results using 10 labeled samples per class (% test accuracy). We report mean and standard deviation over ten trials.
Ablation
Cora
Citeseer
Pubmed
GraphMix
79.30+
1..78
77.13+3.60
-without Manifold Mixup, without predicted targets
3..58
61.01
72.56
-without predicted-targets
72.85-3..79
64.40+12.20
74.74
-without Manifold Mixup
69.08 5.003
6..66.+1.80
74.1 +.0.4
-no Sharpening
73.2543.41
64.65+4.21
74.97+1.47
Averaging of predictions
74.17+
65.52+
75.5942.63
-iith EMA
79.8442.228
71.21+1.32
77.46+3..13
7

---

Under review as a conference paper at ICLR 2020
4.5 VISUALIZATION OF THE LEARNED FEATURES
In this section, we present the analysis of the features learned by GraphMix for Cora dataset. Specifically. we present the 2D visualization of the hidden states using the t-SNE (van der Maaten & Himton] 2008) in Figure 2a 2b and 2c We observe that GraphMix learns hidden states which are better separated and condensed than GCN and preederteddtttrreeeeee
We further evaluate the Soft- rank (refer to Appendix A.7 ) of the class-specific hidden states to demonstrate that GraNhMix(GGNN) makes the class-specific hidden states more concentrated than GCN and (prreddcted-targettee as shown in 2d Refer to Appendix A.8 for 2D representation of other datasets.
(a) GCN
(b) GCNppredicted-trgget
(c) GraphMix(GCN)
(d) Class-specific Soft- Rank
Figure 2: 2D representation of the hidden states of Citeseer dataset using (a) GCN, (b) GCN(predicted- targets), (c) GraphMix, and (d) Ronk-Rank of Class-specific hidden states (lower Soft-Rand reflects more concentrated class-specific hidden states)
RELATED WORK
5.1 SEMI-SUPERVISED LEARNING OVER GRAPH DATA
There exists a long line of work for Semi-supervised learning over Graph Data. Earlier work included using Graph Laplacian Regularizer for enforcing local smoothness over the predicted targets for
the nodes (Zhu Ghahramani 2002, Zhu et al. 2003 Belkin et al. 2006). Another line of work learns node embedding in an unsupervised way (Perozzi et al. 2014) which can then be used as an input to any classifier, or learns the node embedding and target prediction jointly (Yang et al., 2016). Many of the recent Graph Neural Network based approaches (refer to Zhou et al. (2.8)) for review of these methods) are inspired by the success of Convolutional Neural Networks in image and text domains, defines the convolutional operators using the neighbourhood information of the nodes (Kipf & Welling 2017 Veličković et al., 2018: Defferrard et al., 2016). These convolution operator based method exhibit saate-of-the-reulle for semi-supervised learning over graph data, hence much of the recent attention is dedicated to proposing architectural changes to these methods (Qu et al.) 2019 Gao & Ji, 2019 Ma et al.) 2019). Unlike these methods, we propose a regularization technique that can be applied to any of these Graph Neural Networks which uses parameterized transformation on the node features.
5.2 DATA AUGMENTATION
It is well known that the generalization of learning algorithm can be improved by enlarging the training data size. Because labeling more samples is labour-intensive and costly, Data-augmentation has become de facto technique for enlarging the training data size, especially in the computer vision applications such as image classification. Some of the notable Data Augmentation techniques include Cutout (Devries & Taylor 2017) and DropBlock (Ghiasi et al., 2018). Cutout, a contiguous part of the input is zeroed out. DropBlock further extends Cutout to the hidden states. In another line of research, such as Mixup and BC-learning (Zhang et al. 2018 Tokozume et al.] 2017). additional training samples are generated by interpolating the samples and their corresponding targets. Manifold Mixup Veerma et al. 2019a) proposes to augment the data in the hidden states and shows that learns more discriminative features for supervised learning. Furthermore, ICT (Verma et al. (019b) and MixMatch (Berthelot et al. 2019) extend the Mixup technique to semi-supervised learning, by computing the predicted targets for the unlabeled data and applying the Mixup on the unlaheled data
8

---

Under review as a conference paper at ICLR 2020
and their corresponding predicted targets. Even further, for unsupervised learning, ACAI (Berthelot*
eet al. 2019) and AMR (Beckham et al. 2019) explore the interpolation techniques for autoencoders. ACAI interpolates the hidden states of an autoencoder and uses critic network to constrain the reconstruction of these interpolated states to be realistic. AMR explores different ways of combining the hidden states of autoencoder other than the convex combinations of the hidden states. Unlike, all of these techniques which have been proposed for the fixed topology datasets, in this work, we propose interpolation based data-augmentation techniques for graph structured data.
5.3 REGULARIZING GRAPH NEURAL NETWORKS
Regularizing Graph Neural Networks has drawn some attention recently. GraphSGAN (Ding et al. 2018) first uses an embedding method such as DeepWalk (Perozzi et al. 2014) and then trains generator-classiffer networks in the adversarial learning setting generate fake samples in the low-density region between sub-graphs. BVT (Deng et al.) (019)) and/Feng et al. (2019) generate adversarial perturbations to the features of the graph nodes while taking graph structure into account. While these methods improve generalization in graph-structured data, they introduce significant additional computation cost: GraphScan requires computing node embedding preprocessing step. B and Feng et al. (2019) require additional gradient computation for computing adversar- ial perturbations. Unlike these methods, GraphMix does introduce any significant additional computation since it is based on interpolation- based techniques and self self-generated predicted targets.
6 DISCUSSION
We presented GraphMix a simple and efficient regularized training scheme for graph neural net- works. GraphMix is general scheme that can be applied to any graph neural network that uses parameterized transformation on the feature vector of the graph nodes. Through extensive experi- ments, we demonstrated state-of-the-ar performances or close state-of-the-an performance using this simple regularization technique on various benchmark datasets, more importantly, GraphMix improves test accuracy over vanilla GNN across all the datasets, even without doing any extensive hyperparameter search. Further, conduct systematic ablation study to understand the effect of different components in the performance of GraphMix This suggests that in parallel to designing new architectures, exploring better regularization for graph structured data is promising avenue for research.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Christopher Beckham, Sina Honari, Vikas Verma, Alex Lamb, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Christopher Pal. On Adversarial Mixup Resynthesis. arXiv e-prints, art. arXiv:1903.02700 Mar 2019.
Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric frame- work for learning from labeled and unlabeled unlabeledexamplerr examples. Mach. Learn. Res., 7:2399-24533 Decem- ber 2006. ISSN 1532-4435. URLhttp: http://d...ccm..r/iiiacaaaa/a//e
acm
1248632
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A Holistic Approach to Semi-Supervised Learning. arXiv e-prints, art. utXiv:1905.02249 May 2019.
David Berthelot*. Colin Raffel*, Aurko Roy, and Ian Goodfellow. Understanding and improving in- terpolation in autoencoders via an adversarial regularizer. In International Conference on Learning net Representations, 2019. URL [ttps::::/p// http:e..e//oooooeeeeeeeeeeeeeee forum -id-
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the Eleventh Anmual Conference Computational Learning Theory, COLT' 98, pp. doi: 10. 14/1279944322799922
92-100, New York, NY, USA, 1998. ACM. ISBN 1
URL http: . acm.
11e.:.....11999111111 279962
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised
inductive learning via ranking. In International Conference on Learning Representations, 201 8.
URL
net.ewwwweeeeeeeeeeeeeeeeeee ftre:/ooummmmmm id- OW
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. CoRR, abs/1312.6203, 2013.
Olivier Chapelle, Bernhard SchIkopf, and Alexander Zien. Semi-Supervised Learning. The MIT Press, 1st edition, 2010. ISBN 0262514125, 9780262511225
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3844-3852. 2016.
Zhijie Deng, Yinpeng Dong, and Jun Zhu. Batch virtual adversarial trainin for graph convolutional networks. CoRR, abs/1902.00912, 2019. URL http::p//aaa.....////aba//////// 0919229..9992
Terrance Devries and Graham W. Taylor. Improved with cutout. CoRR, htls/1708.000 705.2552, 2017. URL URL http:p//aaax.w..cc///a//////////
ofmooiiaaaiiiooooooooooooccccala neural networks nettorsss
Ming Ding, Jie Tang, and Jie Zhang. Semi-supervised learning on graphs with generative adversarial nets. In Proceedings of the 27th ACM International Conference on Information and Knowledge
Management, CIKM 18, pp. 913 913-922, New York, NY, USA, 2018. ACM. ISBN 978-1-4503-6014 URL http:://oii acm, org wtt../....1///2599999
2. doi: 0211445/22992062277776
3271768
Fuli Feng, Xiangnan He, Jie Tang, and Tat-Seng Chua. Graph adversarial training: Dynamically regularizing based on graph structure. CoRR, abs//990..,226, 2019. URL http://arxxii
org/ atg/abs/1900 08226
Hongyang Gao and Shuiwang Ji. Graph u-nets. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2083-2092 2092, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL. |RI
ngs m www.srrsssvv... v97 97/ /gaol 9a .html
Golnaz Ghiasi, Tsung-Yt Lin, and Quoe V Le. Dropblock: A regularization method for convolutional networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp.
10

---

Under review as a conference paper at ICLR 2020
10727-10733 Curran Associates, Inc., 2018. URL http::/pppprs...i......... -- wwt.:p/www.......aa.........ooea
pdf
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural
message passing for quantum chemistry. In ICML, 2017.
Marco Gori, Gabriele Monfardini, and Scarselli Franco. A new model for learning in graph domains.
IEEE International Joint Conference on Neural Networks.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. K. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems 17, pp. 529-536. MIT Press, 2005. URL hittp://papers... mipeerrs..ppis............. ccc cc/paper/ 2740- semi supervised learning by enttropy-miimmmaatiin pdf
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, 2017.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks graph-structured data. ArXiv, a0s/1506.,5163, 2015.
Thomas N Kipf and Max Welling, Variational graph auto-encoders. arXiv preprint arXiv:1611.03300 2016.
Thomas Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. Audio augmentation for speech recognition. In INTERSPEECH, 2015.
Srijan Kumar, Francesca Spezzano, VS Subrahmanian, and Christos Faloutsos. Edge weight predic- tion in weighted signed networks. ICDM, 2016.
Srijan Kumar, Bryan Hooi, Disha Makhija, Mohit Kumar, Christos Faloutsos, and VS Subrahmanian. Rev2: Fraudulent user prediction in rating platforms. In WSDM, 2018.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. CoRR, abs/1610.02242, 2016. URL
wrw.iivvorrr//eooe attp//bs//11l00 02242]
Dong-Hyun Lee. Pseudo-label: : The simple and efficient semi-supervised learning method for deep neural networks. 2013.
Qing Lu and Lise Getoor. Link-based classification. In Proceedings of the Twentieth International
Conference on International Conference Machine Learning, ICML'03, pp. 496-503, AAAI Press, 2003. ISBN 1-57735-189-4. URL http: peltp://dl... //tl://lll. acm. http:///a..accooo.iiiiitttttee wtt.::/ccccaaa.t..ooooooooo cfmidd
55441834 3041901
Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu, Disentangled graph convolutional networks. In ICML, 2019.
Takeru Miyato, Shin ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. CoRR, ab8/1611.08402, 2016. lttp::///xxx
abs/1611. 08402
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin Cubuk, and Quoc Le. Apementment: A Simple Data Augmentation Method for Automatic Speech Recogni- tion. arXiv e-prints, art. arXiv:1904.088
Apr 2019.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In KDD, 2014.
11

---

Under review as a conference paper at ICLR 2020
Meng Qu, Yoshua Bengio, and Jian Tang. GMNN: Graph Markov neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5241-5250, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfar- dini. The graph neural network model. Trans. Neur. Netw., 20(1))61-80, January 2009. ISSN 1045-9227. doi: 10. 1190/09NN 2008. 2005605. URL htRLbttp://dx.coo doi 1100///T...1111 2008. 2005605)
Oleksandr Shchur, Maximilian Mumme, Alcksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. CoRR, abs/1811.05580 2018. URL http://aaxxvv.or/
abs/ abs/ 0tte/8811.055 05868
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve semi--uperrvie deep learning results. In Advances in Neural Information Processing Systems 30, pp. 1195-1204, 2017.
Ben Taskar, Ming-Fai Wong, Pieter Abbeel, and Daphne Koller. Link prediction in relational data. In NIPS, 2004.
Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-class learning for image classifica- tion. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5486-5494, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma- chine Learning Research, 9:2579-2600 2008. URL httt:://ww/w jmlr wwtppppprg//..///.o.
wandermaaten0882 htm.m.........m.mmmm
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learn- Long Beach, California. USA, 09-15 Jun 2019a. PMLR. URL ing Research, pp. www.iingsssr.....mmmmmmme....me. mlr wwwss/esssssee. v 1tt9////eal9/a... 1t..ma199.hhc... html ltttp://proceediingg
Vikas Verma, Alex Lamb, Kannala Juho, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In Sarit Kraus (ed.), Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019. ijcai.org, 2019b. doi: 10.24963/jcci..019 URL htthtttpp:///o/ orrg/00 24963/1jcas 2019
Jason Weston, Frédéric Ratle, Hossein Mobahi, and Ronan Collobert. Deep learning via semi- supervised embedding. In Grégoire Montavon, Geneviève Orr, and K. R. Müller (eds.), In Neural Networks: Tricks of the Trade. Springer, second edition, 2012.
Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Daniel Jurafsky, and Andrew Y. Ng. Data noising as smoothing in neural network language models. ArXiv, abs/1703.02573, 2017.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. ICML, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018. URL https
//openrevis ew net. mett//orumm..-rrleeee.
12

---

Under review as a conference paper at ICLR 2020
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neural networks: A review of methods and applications. CoRR, abs/1812.08434, 2018. URL
http::/arxiv/ wrt./bs/////....... 08434
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propaga- tion. Technical report, 2002.
Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In ICML, 2003.
A
APPENDIX
A.1 DATASETS
The statistics of these datasets as well as the number of training'validationnnes nodes is presented in
Table
Table 4: Dataset statistics.
Dataset
Nodes
Edges
Features
Classes
Training
Validation
# Test
Cora
2,708
5,429
1,433
7
140
500
1,000
Citeseer
3,327
4,732
3,703
6
120
500
1,000
Pubmed
19,717
44,338
500
3
60
500
1,000
Bitcoin Alpha
3,783
24.186
3,783
2
100
500
3,221
Bitcoin OTC
5,881
35,592
5,881
2
100
500
5,947
A.2 COMPARISON WITH STATE-OF-THE-ART METHODS
We present the comparion of GraphMix with the recent state-of-the-art methods well earlier methods using the standard Train/Validationn split in Table
A.3 RESULTS WITH FEWER LABELED SAMPLES
We further evaluate the effectiveness of GraphMix the learning regimes where fewer labeled samples exist. For each class, we randomly sampled K E (5, 10) samples for training and the same number samples for the validation. We used all the remaining labeled samples as the test set. We repeated this process for 10 times. The results in show that GraphMix achieves even better improvements when the labeled samples are fewer Refer to Table I for results with 20 training samples per class).
A.4 RESULTS ON LARGER DATASETS
In this section, we provide results three recently proposed datasets which relatively larger than standard benchmark datasets Saarr/Ciieeee//ummeell Specifically, we use Cora-Full dataset proposed in [Bojchevki & Günnemann (2118) and Coauthor-CS and Coauthor-Physics datasets proposed in Shchur et al. (2018). We took processed versions of these dataset available here We did 10 random splits the the data into vrain/validationteest split. For the classes which had more than 100 samples, we choose 20 samples per class for training, 30 samples per class for validation and the remaining samples as test data. For the classes which had less than 100 samples, we chose 20% samples, per class for training. 30% samples for validation and the remaining for testing. For each split we run experiments using 100 random seeds. The statistics of these datasets in presented in Table g and the results presented in Table 7 We observe that GraphMix(GCN) improves the results over GCN for all the three datasets. We note that we did minimal hyperparameter search for GraphMix(GCN) mentioned in A.8.. and doing more rigorous hyperparameter search further improve the performance of GraphMix.
13

---

Under review as a conference paper at ICLR 2020
Table 5: Comparison of GraphMix with other methods (% test accuracy ), for Cora, Citeseer and Pubmed.
Method
Cora
Citeseer
Pubmed
Results reported from the literature
MLP
55.1%
46.5%
71.4%
ManiReg (Belkin et al.
2006)
59.5%
60.1%
70.7%
SemiEmb Weston et
2012)
59.0%
59.6%
71.7%
LP (Zhu et al. 2003)
68.0%
45.3%
63.0%
DeepWalk (Perozzi et
2014
67.2%
43.2%
65.3%
ICA (Lu & Getoor, 2003)
75.1%
69.1%
73.9%
Planetord (Yang et al. 20166
75.7%
64.7%
77.2%
Chebyshev (Detterrard al.
2016
81.2%
69.8%
74.4%
GCN (Kipf & Welling (211
81.5%
70.3%
79.0%
MoNet (Monti et al.
20166
81.7 + 0.5%
78.8 + 0.3%
GAT (Velcckvii et al. 2018
83.0 0.7%
72.5 + 0.7%
79.0 + 0.3%
GraphScan (Ding al.
2018
83.3 1.3
1.8
GMNN (Qu cl al. 2019)
83.7%
73.1%
81.8% %
DisenGCN (Ma al. 2019)
83.7%
73.4%
80.5%
Graph U-Net Gao & 2019
84.4%
73.2%
79.6%
BVAT Deng et al.
83.6++0.5
74.0
79.9+0.4
Our Experiments
GCN
81.3010.66
70.61=0.22
79.8611.34
GAT
82.70+0022
70.40=0.35
79.05+0.64
Graph U-Net
81.74+0.54
67.69=1.10
77.73 +0.98
GCN (with predicted-targets)
82.03
73.38=0.35
82.42.10.36
FCN (with predicted-targets
80.30:10.75
71.50=0.80
77.40
GraphMix (GCN)
83.9410.57
7..52-0.59
80.98+0..5
GraphMix (GAT)
83.32
73.08-0.23
01.10
GraphMix (Graph U-Net)
82.18
69.00 +1.32
78.76
Table 6: Results using less labeled samples (% test accuracy). K referes to the number of labeled samples per class.
Cora
Citeseer
Pubmed K
Algorithm
K
GCN
66.39:14.26
72.91=3.10
55.61
64.19-3.89
66.06:33.85
75.57+11.58
GAT
68. 17.15.55.54
73.88 4.35
55.54.1..2
61.63.10.42
64.24.14.79
73.60.11.85
Graph U-Net
64.42-45..44
71.48-3.03
49.43++5.8
61.16=3.47
65.05=+4.69
68.65++3.69
GraphMix (GCN)
6..99:16.46
79.30 1.36
58.55
70.78
6..66:13.90
77.13+13.60
GraphMix (GAT)
72.01+66.68
75.82-2.73
57.6=0.64
62.24=2.90
66.61+3.69
7,.96=+1.70
GraphMix (Graph U-Net)
66.84.165510
73.14
54.3915.07
64.36-3.48
67. 67.40.15.33
70.4331.75
A.5 IMPLEMENTATION AND HYPERPARAMETER DETAILS
We use the standard benchmark architecture used in GCN (Kipf & Welling 2017). GAT (Veličkovid cet al. 2018) and GMNN (Qu et al. 2019), among others. This architecture has one hidden layer and the graph convolution is applied twice the input layer and on the output of the hidden layer. The FCN in GraphMix shares the parameters with the GCN.
GraphMix introduces four additional hyperparameters, namely the parameter of Beta distribution used in Manifold Mixup training the FCN, the max-con consisseet coefficient Ymaz which controls the trade-off between the supervised loss and the unsupervised loss (loss computed using the pseudolables)
14

---

Under review as a conference paper at ICLR 2020
Table 7: Comparison of GraphMix with other methods (% test accuracy for Cora-Full, Coauthor-CS, Coauthor- and NELL. * refers to the results reported in Shchur et al. (2018).
Method
Cora-Full
Coauthor-CS
Coauthor-Physies
NELL
GCN
0.6
91.1=0.5
92.8+1.0
GAT
11.9
90.5+0.6
92.5+0.9
MoNer
5..
90.8+10.6
92.5+10.9
GS-Mean*
18.6
91.3+2.8
93.0:10.8
GCN (Kipf & Welling 2017
66.0
GCN
60.13+00.57
91.27+00.56 91.83++0.51
92.90++0.92
63.65++1.11
GraphMix (GCN)
61.80++0.54
04.49-00.44
66.32-+1.04
Table 8: Dataset statistics
Datasets
Classes
Features
Nodes
Edges
Cora-Full
67
8710
18703
62421
Coauthor-CS
15
6805
18333
81894
Saauthor-Phssiee
5
8415
34493
247962
NELL
210
5414
65755
266144
of FCN, the temparature T in sharpening and the number of random perturbations K applied to the input data for the averaging of the predictions.
We conducted minimal hyperparameter seach over only a and Tmar and fixed the hyperparameters T and K to 0.1 and 10 respectively. The other hyperparameters were set the best values for underlying GNN (GCN or GAT), including the learning rate, the L2 decay rate, number of units in hidden layer etc. We observed that GraphMix is not very sensitive the values of a and Tmaz and similar values of these hyperparameters work well across all the benchmark datasets. Refer to Appendix [A.5] and AA66 for the details about the hyperparameter values and the procedure used for the best hyperparameters selection.
A.5.1 FOR RESULTS REPORTED IN SECTION
For GCN and GraphMix(GCN), we used Adam optimizer with learning rate 0.01 and L2-decay 5e-4, the number of units in the hidden layer 16, dropout rate in the input layer and hidden layer was set to and 0.0, respectively. For GAT and GraThMix(GAT).. we used Adam optimizer with learning rate 0.005 and 1.2-decay 5e-4, the number of units in the hidden layer 8 and the dropout rate in the input layer and hidden layer was searched from the values {0.2, 0.5,, 0.8].
For and Tmoz of Graph and GraTh [0.0, 0.1, 1.0, 2.0] and [0.1, 1.0, 10.0, 20.0) respectively.
searched over the values in the set
For GrCDh and Citeseer and Mmaz = 10.0 works best for Pubmed.
1.0 works best across all the datasets. Tmean: 1.0 works best for Cora
1.0 works best for Cora and Citeseer and 0.1 works best for Pubmed. Tmaz = 1.0 works best for Cora and Citeseer and Tmaz 10.0 works best for Pubmed. Input droputrate=0.5 and hidden dropout rate=0.5 work best for Cora and Citeseer and Input dropout rate=0.2 and hidden dropout rate =0.2 work best for Pubmed.
For GAT)
We conducted all the experiments for 2000 epochs. The value of consistency coefficient (line in Algorithm|] TD is increased from 0 to its maximum value Ymar from epoch 500 to 1000 using the sigmoid ramp-up of Mean-Teacher (Tarvainen & Valpola 2017)
15

---

Under review as a conference paper at ICLR 2020
A.5.2 FOR RESULTS REPORTED IN SECTION
For a of GraphMix(GCN), we searched over the values in the set [0.0, 0.1, 0.5, 1.0] and found that 0.1 works best across all the datasets. For Ymaz- searched over the values in the set [0.1, 1.0, 10.0] and found that 0.1 and 1.0 works best across all the datasets. Rest of the details for GraphMix(GCN) and GCN same as Section A.5.1
A.5.3 FOR RESULTS REPORTED IN SECTION
searched over the values in the set [0.0, 0.1, 1.0] and found that 0.1 works best for both the datasets. For Tmans we searched over the values the set [0.1, 1.0, 10.0] and found that 0.1 works best for both the datasets. We conducted all the experiments for 150 epochs. The value of consistency coefficient y (line 13 in Algorithm[[] increased from to its maximum value "maz from epoch 75 to 125 using the sigmoid ramp-up of Mean-Teacher (Tarvainen & Valpola 2017].
For of GraphMix(GCN),
Both for GraphMix(GCN) and GCN, we use Adam optimizer with learning rate 0.01 and L2-decay 0.0, the number of units in the hidden layer 128, dropout rate in the input layer was set to 0.5.
A.6 HYPERPARAMETER SELECTION
For each configuration of hyperparameters, we run the experiments with 100 random seeds. We select the hyperparameter configuration which has the best validation accuracy averaged over these 100 trials. With this best hyperparameter configuration, for 100 random seeds, we train the model again and use the validataion set for model selection (i.e. report the test accuracy at the epoch which has best validation accuracy.)
A.7 SOFT-RANK
Let H be matrix containing the hidden states of the samples from particular class. The Roft-Rank of matrix H is defined by the sum of the singular values of the matrix divided by the largest singular value. A lower Soft-Rank implics fewer dimensions with substantial variability and provides continuous analogue to the notion of rank from matrix algebra. This provides evidence that the concentration class-specific states observed when using GraphMix in Figure B can be measured directly from the hidden states and is not artifact of the T-SNE visualization.
A.8 FEATURE VISUALIZATION
We present the 2D visualization of the hidden states learned using GCN and GraphMix(GCN) for Cora, Pubmed and Citeseer datasets in Figure 3 We observe that for Cora and Citeseer, GraphMix learns substantially better hidden states than GCN. For Pubmed, we observe that although there is no clear separation between classes, "Green" " and "Red" classes overlap less using the GraphMix, resulting in better hidden states.
A.8.1 HYPERPARAMETER DETAILS FOR RESULTS IN TABLE
For all the experiments we use the standard architecture mentioned in Section A.5 and used Adam optimizer with learning rate 0.001 and 64 hidden units the hidden layer. For Coauthor-CS and Seauthor-Physice we trained the network for 2000 epochs. For Cora-Full, we trained the network for 5000 epochs because we observed the training loss of Cora-Full dataset takes longer to converge.
For Coauthor-CS CS and Seauthor-Physiie We set the input layer dropout rate to 0.5 and weight-decay to 0.0005, both for GCN and Graph.MiX(GCN).. We did not conduct any hyperparameter search over the GraphMix hyperparameters a, Xmaas temparature T and number of random permutations K applied to the input data for (GCN) for these two datasets, and set these values to 1.0, 1.0, 0.1 and 10 respectively.
For Cora-Full dataset: We found input layer dropout rate 0.2 and weight-decay 0.0 to the best for both GCN and GraphMix(GCN). For GraphMix(GCN) we fixed a, temparature 7' and
16

---

Under review as a conference paper at ICLR 2020
10
10
-10
0-
-10
-20
-30
-30
-50
30
15
10-
-10
-20
-20
-30
30
20
1.0
1.0
-40
-20
-40
-20
20
Figure 3: T-SNE of hidden states for Cora (left), Pubmed (middle), and Citeseer (right). Top row is GCN baseline, bottom is GraphMix.
number of random permutations K to 1.0 0.1 and 10 respectively. For Amazi did search over (1.0, 10.0, 20.0) and found that 10.0 works best.
For all the GraphMix(GCN) experiments, the value of consistency coefficient y (line 13 in Algorithm I is increased from to its maximum value "maz from epoch 500 to 1000 using the sigmoid ramp-up of Mean-Teacher Tarvainen & Valpola 2017).
17


Reviews:
[{"review_id": "SyxexpRpqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents a data augmentation procedure for semi-supervised learning on graph structured data. In general it is not clear how augmented observations can be incorporated into an existing graph while still preserving the graph structure. Instead of trying to incorporate an augmented dataset into the graph, this paper uses a seperate fully-connected network (FCN) that shares weights with a graph neural net (GNN). The final method also incorporates various components from the literature, such as Mixup and Sharpening. While the literature appears to have been adequately cited, the position of GraphMix relative to previous work could be better dilineated. In my opinion, the main contribution of this paper is the suggestion of using weight-sharing to sidestep the issue of incorporating augmented datapoints into graph-structured data. I am not sufficiently versed in the literature to assess whether the idea is sufficiently novel to warrant acceptance. The idea is simple and appears to improve the performance of Graph Convolutional Nets (GCNs), so I am leaning towards acceptance.\n\nRating Justification:\nEfficient data-augmentation procedures are an important area of research. The relative simplicity and generality of GraphMix is appealing. I would give the paper a higher score if the authors showed that GraphMix(Graph U-net) was an improvement over Graph U-net, or if it was made more clear that some substantial benefit is derived from using Mixup features.\n\nAdditional Comments:\n1. Based on the ablation study it seems that Mixup actually plays a very minor role in the overall success of the procedure. I would be curious to see the t-SNE visualization of the GraphMix(GCN) \\ Mixup features in order to determine how much of the cluster separation is due to Mixup specifically. I understand that previous work has suggested that Mixup features are superior for discrimination than normal features, but in this work specifically the evidence for this assertion is fairly weak. \n\n2. A clearer distinction between GraphMix(GCN) and GCN (with predicted targets) would be very helpful, especially since GCN (with predicted targets) actually performs the best on the standard PubMed splits. Why were the results for GCN (with predicted targets) not included in Table 2?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HyemZINEcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a regularization method for graph neural networks that exploit unlabeled examples. The authors claim that the proposed method for semi-supervised graph learning is based on data augmentation which is efficient and can be applied to different graph neural architectures, that is an architecture-agnostic regularization technique is considered in the paper.\n\nStrength\n•\tSimple yet effective regularization technique to neural networks for graph-structured data\n\nWeaknesses\n•\tWeak technical contribution to the problem\n•\tThe performance gain is from mostly the use of a semi-supervised learning approach based on entropy minimization, which has been developed without consideration of graph-structured data.\n\n\nThe paper does not compare the computational complexity of the proposed approach to the baseline approaches. Although it seems the additional computation cost for the proposed approach compared to the baseline such as GCN does not increase dramatically, it would be better if the authors could provide its computational cost as well as performance given the number of perturbed samples.\n\nAccording to Table 1, self-learning approaches, i.e., GCN (with predicted targets), works comparable to GraphMix. I do not find the benefit of using such a more complicated approach other than simple approaches such as self-learning in the experiment section.\n\nManifold Mixup is not explained properly. For example, the authors may need to explain two random samples, i.e., (x, y) and (x’, y’) are drawn from the data distribution D, and how it is used in GraphMix more clearly.\n\nAdjacency matrix A is used without its definition.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1g6Dm86FH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors developed GNN by integrating an interpolation based regularization. In particular, the proposed GraphMix model has two components, one GNN and one MLP. For the MLP, the manifold mixup loss is used for training. The two components share parameters so that the GCN training can be regularized by the manifold mixup loss. Moreover, the predicted samples of GCN are used to augment the data of MLP for training in a self-supervised manner. In the experiments, GraphMix was compared with several recent GNN models on several benchmark datasets and demonstrates effectiveness for node classification and link classification.\n\nThere are several concerns on this paper.\n1. The motivation to integrate interpolation based regularization for GNN is not intuitively clear. The authors may want to further clarify on how can Eq. (2) help improve the classification power of GNN in an intuitive manner.\n2. Some algorithm design choices, such as sharpening for entropy minimization, are not well justified. It is not clear why sharpening is selected over psudolabels.\n3. The overall technical contribution is somewhat incremental based on previous work on GNN and manifold mixup based regularizations. The model design is heuristic, and lack theoretic analysis, for example, on the parameter sharing based regularization over other choices of regularizations, if any.\n4. From the experimental results, the proposed model improves GCN and GAT, but the improvements on other baseline methods are a little bit subtle. The authors may want to further evaluate their framework on those GNNs such as GMNN and Graph U-Net to make the experiments more comprehensive, or justify on why they cannot be implemented using the proposed framework.\n5. In Table 2, 3, and 4, it is not clear on why GAT and GraphMix (GAT) are missing. This inconsistency with regard to Table 1 should be justified.\n6. In the ablation analysis, if the reason for the suboptimal performance when using EMA is the lack of hyperparameter search, then it is suggested to perform comprehensive hyperparameter search so that the results are more solid. Otherwise, the design choice cannot be well justified.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors integrate an interpolation based regularization to develop a graph neural network for semi-supervised learning. While reviewers enjoyed the paper, and the authors have provided a thoughtful response, there were remaining questions about clarity of presentation and novelty remaining after the rebuttal period. The authors are encouraged to continue with this work, accounting for reviewer comments in future revisions.
Decision: Reject