Title: GRAPHMIX: REGULARIZED TRAINING OF GRAPH NEURAL NETWORKS FOR SEMI-SUPERVISED LEARN-ING

Abstract: We present GraphMix, a regularized training scheme for Graph Neural Network based semi-supervised object classification, leveraging the recent advances in the regularization of classical deep neural networks. Specifically, we propose a unified approach in which we train a fully-connected network jointly with the graph neural network via parameter sharing, interpolation-based regularization and self-predicted-targets. Our proposed method is architecture agnostic in the sense that it can be applied to any variant of graph neural networks which applies a parametric transformation to the features of the graph nodes. Despite its simplicity, with GraphMix we can consistently improve results and achieve or closely match state-of-the-art performance using even simpler architectures such as Graph Convolutional Networks, across three established graph benchmarks: the Cora, Citeseer and Pubmed citation network datasets, as well as three newly proposed datasets : Cora-Full, Co-author-CS and Co-author-Physics.

1. INTRODUCTION:

Due to the presence of graph structured data across a wide variety of domains, such as biological networks, social networks and telecommunication networks, there have been several attempts to design neural networks that can process arbitrarily structured graphs. Early work includes (Gori et al.; Scarselli et al., 2009) which propose a neural network that can directly process most type of graphs e.g., acyclic, cyclic, directed, and undirected graphs. More recent approaches include (Bruna et al., 2013; Henaff et al., 2015; Defferrard et al., 2016; Kipf & Welling, 2016; Gilmer et al., 2017; Hamilton et al., 2017; Veličković et al., 2018; 2019; Qu et al., 2019; Gao & Ji, 2019; Ma et al., 2019) , among others. Many of these approaches are designed for addressing the important problem of Semi-supervised learning over graph structured data (Zhou et al., 2018) . Much of this research effort has been dedicated to developing novel architectures.

Unlike many existing works which try to come up with the new architectures, we re-synthesize ideas from the recent advances in the regularization of the classical neural network, and propose an architecture-agnostic framework for regularized training of graph neural network based semisupervised object classification. Recently, Data-Augmentation based regularization has been shown to be very effective in other types of neural networks but how to apply these techniques in graph neural networks is still under-explored. Our proposed method GraphMix1 is a unified framework that utilizes interpolation based data augmentation (Zhang et al., 2018; Verma et al., 2019a) and self-targetprediction based data-augmentation techniques (Laine & Aila, 2016; Tarvainen & Valpola, 2017; Verma et al., 2019b; Berthelot et al., 2019) . We show that with our proposed unified framework, we can achieve state-of-the-art performance even when using simpler graph neural network architectures such as Graph Convolutional Networks (Kipf & Welling, 2017) and without incurring any significant additional computation cost.

2. PROBLEM DEFINITION AND PRELIMINARIES 2.1 PROBLEM SETUP:

We are interested in the problem of semi-supervised object classification using graph structured data. We can formally define such graph structured data as G = (V, A), where V represents the set of nodes {v 1 , . . . , v n }, and A is the adjacency matrix representing the edges between the nodes of V.

Each node v i in the graph has a corresponding d-dimensional feature vector x i ∈ R d . The feature vectors of all the nodes X = [x 1 , . . . , x n ] are stacked together to form the entire feature matrix X ∈ R n×d . Each node belongs to one out of C classes and can be labeled with a C-dimensional one-hot vector y i ∈ {0, 1} C . Given the labels of Y L for few of the labeled nodes V L ⊂ V, the task is to predict the labels Y U of the remaining nodes V U = V \ V L .

2.2. GRAPH NEURAL NETWORKS:

Graph Neural Networks (GNN) learn the l th layer representations of a sample i by leveraging the representations of the samples N B(i) in the neighbourhood of i. This is done by using an aggregation function that takes as an input the representations of all the samples and the graph structure and outputs the aggregated representation. The aggregation function can be defined using the Graph Convolution layer (Kipf & Welling, 2017) , Graph Attention Layer (Veličković et al., 2018) , or any general message passing layer (Gilmer et al., 2017) . Formally, let H l ∈ R n×k be a matrix containing the k-dimensional representation of n nodes in the l th layer, then:

EQUATION

where W ∈ R k×k is a linear transformation matrix, k is the dimension of (l + 1) th layer and a is the aggregation function that utilizes the graph adjacency matrix A.

2.3. INTERPOLATION BASED REGULARIZATION TECHNIQUES:

Recently, interpolation-based techniques have been proposed for regularizing neural networks. We briefly describe some of these techniques here. Mixup (Zhang et al., 2018) trains a neural network on the convex combination of input and targets, whereas Manifold Mixup (Verma et al., 2019a ) trains a neural network on the convex combination of the hidden states of a randomly chosen hidden layer and the targets. While Mixup regularizes a neural network by enforcing that the model output should change linearly in between the examples in the input space, Manifold Mixup regularizes the neural network by learning better (more discriminative) hidden states.

Formally, suppose g : x -→ h is a function that maps input sample to hidden states, f : h -→ ŷ is a function that maps hidden states to predicted output, λ is a random variable drawn from Beta(α, α) distribution, Mix λ (a, b) = λ * a + (1 -λ) * b is an interpolation function, D is the data distribution, (x, y) and (x , y ) is a pair of examples sampled from distribution D and be a loss function such as cross-entropy loss, then the Manifold Mixup Loss is defined as:

L = E (x,y)∼D E (x ,y )∼D E λ∼Beta(α,α) (f (Mix λ (g(x), g(x ))), Mix λ (y, y )).

(2)

We use above Manifold Mixup loss for training an auxiliary Fully-connected-network as described in Section 3 and Line 6 and 12 of Algorithm 1.

3. GRAPHMIX:



3.1. MOTIVATION:

Data Augmentation is arguably the simplest and most efficient technique for regularizing a neural network. In some domains, such as computer vision, speech and text, there exist efficient data augmentation techniques, for example, random cropping, translation or Cutout (Devries & Taylor, 2017) for text domain. However, data augmentation for the graph-structured data remains under-explored.

There exists some recent work along these lines but the prohibitive additional computation cost (see Section 5.3) introduced by these methods make them impractical for real-world large graph datasets.

Based on these limitations, our main objective is to propose an efficient data augmentation technique for graph datasets.

Recent work based on interpolation-based data augmentation (Zhang et al., 2018; Verma et al., 2019a) has seen sizable improvements in regularization performance across a number of tasks. However, these techniques are not directly applicable to graphs for an important reason: Although we can create additional nodes by interpolating the features and corresponding labels, it remains unclear how these new nodes must be connected to the original nodes via synthetic edges such that the structure of the whole graph is preserved. To alleviate this issue, we propose to train an auxiliary Fully-connected-net (FCN) using Manifold Mixup as discussed in Section 3.2. Note that the FCN only uses the node features ( not the graph structure), thus the Manifold mixup loss in Eq. 2 can be directly used for training the FCN. Furthermore, drawing inspiration from the success of selfsupervised semi-supervised learning algorithms (self-predicted-targets based algorithms which can be also interpreted as a form of data-augmentation techniques) (Verma et al., 2019b; Berthelot et al., 2019) , we explore self-supervision in the training of GNNs. We note that self-supervision has already been explored for unsupervised representation learning from graph structured data (Veličković et al., 2019) , but not for semi-supervised object classification over graph structured data. Based on these challenges and motivations we present our proposed approach GraphMix for training Graph Neural Networks in the following Section.

3.2. METHOD:

GraphMix augments the vanilla GNN with a Fully Connected Network (FCN). The FCN loss is computed using the Manifold Mixup as discussed in Section 2.3 and the GNN loss is computed in the standard way. i = random(0,1) // generate randomly 0 or 1 4:

if i=0 then 5:

λ ∼ Beta(α, α) // Sample a mixing coefficient from Beta distribution 6:

Lsup = L f (XL, θ, λ), YL // supervised loss from FCN using the Manifold Mixup 7:

for k = 1 to K do 8:

XU,k = RandomP erturbations(XU ) // Apply k th round of random perturbation to XU 9:

end for 10: Both the FCN loss and GNN loss are optimized in an alternating fashion during training. Furthermore, the predicted targets from the GNN are used to augment the training set of the FCN. In this way, both FCN and GNN facilitate each other's learning process. At inference time, the predictions are made using only GNN. The diagrammatic representation of GraphMix is presented in Figure 1 and the full algorithm is presented in Algorithm 1.

ȲU = 1 K k g(Y | XU,k ; θ, A) //

The GraphMix framework can be applied to any underlying GNN as long as the underlying GNN applies parametric transformations to the node features. In our experiments, we show the improvements over GCN (Kipf & Welling, 2016) , GAT (Veličković et al., 2018) and Graph U-Net using GraphMix . Furthermore, GraphMix(GCN) framework does not add any significant computation cost over the underlying GNN, because the underlying GNN is trained in the standard way and the FCN training requires trivial additional computation cost for computing the predicted-targets (Section 3.2.1 and 3.2.1) and the interpolation function ( Mix λ (a, b) in Section 2.3). There are no additional memory requirements for GraphMix(GCN) , since FCN and GNN share the parameters. Some implementation considerations. For Manifold Mixup training of FCN, we apply mixup only in the hidden layer. Note that in Verma et al. (2019a) , the authors recommended applying mixing in a randomly chosen layer (which also includes the input layer) at each training update. However, we observed under-fitting when applying mixup randomly at the input layer or hidden layer. Applying mixup only in the input layer also resulted in underfitting and did not improve test accuracy.

The performance of self-supervision based algorithms such as GraphMix is greatly affected by the accuracy of the predicted targets. To improve the accuracy of the predicted targets, we applied the average of the model prediction on K random perturbations of an input sample as discussed in Section 3.2.1 and sharpening as described in Section 3.2.2. Further, we draw similarities and difference of GraphMix w.r.t. Co-training framework in the Section 3.2.3.

3.2.1. ACCURATE TARGET PREDICTION FOR UNLABELED DATA:

Recent state-of-the-art semi-supervised learning methods use a teacher model to accurately predict targets for the unlabeled data. These predicted targets on the unlabeled data are used as "true labels" for further training of the model. The teacher model can be realized as a temporal ensemble of the student model (the model being trained) (Laine & Aila, 2016) or by using an Exponential Moving Average (EMA) of the parameters of the student model (Tarvainen & Valpola, 2017) . Another recently proposed method for accurate target predictions for unlabeled data is to use the average of the predicted targets across K random augmentations of the input sample (Berthelot et al., 2019) .

Along these lines, in this work, we compute the predicted target as the average of predictions on K drop-out versions of the input sample. We also used the EMA of the student model but it did not improve test accuracy across all the datasets (see Section 4.4 for details).

3.2.2. ENTROPY MINIMIZATION:

Many recent semi-supervised learning algorithms (Laine & Aila, 2016; Miyato et al., 2018; Tarvainen & Valpola, 2017; Verma et al., 2019b) are based on the cluster assumption (Chapelle et al., 2010) , which posits that the class boundary should pass through the low-density regions of the marginal data distribution. One way to enforce this assumption is to explicitly minimize the entropy of the model's predictions p(y|x, θ) on unlabeled data by adding an extra loss term to the original loss term (Grandvalet & Bengio, 2005) . The entropy minimization can be also achieved implicitly by modifying the model's prediction on the unlabeled data such that the prediction has low entropy and using these low-entropy predictions as targets for the further training of the model. Examples include "Pseudolabels" (Lee, 2013) and "Sharpening" (Berthelot et al., 2019) . Pseudolabeling constructs hard(1-hot) labels for the unlabeled samples which have "high-confidence predictions". Since many of the unlabeled samples may have "low-confidence predictions", they can not be used in the Pseudolabeling technique. On the other hand, Sharpening does not require "high-confidence predictions" , and thus it can be used for all the unlabelled samples. Hence in this work, we use Sharpening for entropy minimization. The Sharpening function over the model prediction p(y|x, θ) can be formally defined as follows (Berthelot et al., 2019) , where T is the temperature hyperparameter and C is the number of classes:

Sharpen(p i , T ) := p 1 T i C j=1 p 1 T j (3)

3.2.3. CONNECTION TO CO-TRAINING:

The GraphMix approach can be seen as a special instance of the Co-training framework (Blum & Mitchell, 1998) . Co-training assumes that the description of an example can be partitioned into two distinct views and either of these views would be sufficient for learning if we had enough labeled data. In this framework, two learning algorithms are trained separately on each view and then the prediction of each learning algorithm on the unlabeled data is used to enlarge the training set of the other. Our method has some important differences and similarities to the Co-training framework. Similar to Co-training, we train two neural networks and the predictions from the GNN are used to enlarge the training set of FCN. The important difference is that instead of using the predictions from the FCN to enlarge the training set for the GNN, we employ parameter sharing for passing the learned information from FCN to GNN. In our experiments, directly using the predictions of the FCN for GNN training resulted in reduced accuracy. This is due to the fact that the number of labeled samples for training the FCN is sufficiently low and hence the FCN does not make accurate enough predictions. Another important difference is that unlike the co-training framework, FCN and GNN do not use completely distinct views of the data: the FCN uses feature vectors X and the GNN uses the feature vector and adjacency matrix (X, A).

4. EXPERIMENTS:

We present results for the GraphMix algorithm using standard benchmark datasets and the standard architecture in Section 4.2 and 4.3. We also conduct an ablation study on GraphMix in Section 4.4 to understand the contribution of various components to its performance. Refer to Appendix A.5 for implementation and hyperparameter details.

4.1. DATASETS:

We use three standard benchmark citation network datasets for semi-supervised node classification, namely Cora, Citeseer and Pubmed. In all these datasets, nodes correspond to documents and edges correspond to citations. Node features correspond to the bag-of-words representation of the document. Each node belongs to one of C classes. During training, the algorithm has access to the feature vectors and edge connectivity of all the nodes but has access to the class labels of only a few of the nodes.

For semi-supervised link classification, we use two datasets Bitcoin Alpha and Bitcoin OTC from (Kumar et al., 2016; 2018) . The nodes in these datasets correspond to the bitcoin users and the edge weights between them correspond to the degree of trust between the users. Following (Qu et al., 2019) , we treat edges with weights greater than 3 as positive instances, and edges with weights less than -3 are treated as negative ones. Given a few labeled edges, the task is to predict the labels of the remaining edges. The statistics of these datasets as well as the number of training/validation/test nodes is presented in Appendix A.1.

4.2. SEMI-SUPERVISED NODE CLASSIFICATION:

For baselines, we choose GCN (Kipf & Welling, 2017) , and the recent state-of-the-art methods GAT (Veličković et al., 2018) , GMNN (Qu et al., 2019) and Graph U-Net (Gao & Ji, 2019) . We additionally use two self-training based baselines: in the first one, we trained a GCN with self-generated predicted targets, and in the second one, we trained a FCN with self-generated predicted targets, named "GCN (with predicted-targets)" and "FCN (with predicted-targets)" respectively in Table 1 . For generating the predicted-targets in above two baselines, we followed the procedure of Section 3.2.1 and Section 3.2.2. GraphMix(GCN) , GraphMix(GAT) and GraphMix(Graph U-Net) refer to the methods where underlying GNNs are GCN, GAT and Graph U-Net respectively. Refer to Appendix Section A.5 for implementation and hyperparameter details. Shchur et al. (2018) has demonstrated that the performance of the current state-of-the-art Graph Neural Networks on the standard train/validation/test split of the popular benchmark datasets (such as Cora, Citeseer, Pubmed, etc) is significantly different from their performance on the random splits. For fair evaluation, they recommend using multiple random partitions of the datasets. Along these lines, we created 10 random splits of the Cora, Citeseer and Pubmed with the same train/ validation/test number of samples as in the standard split. We also provide the results for the standard train/validation/test split in Table 5 in Appendix A.2.

The results are presented in Table 1 . We observe that GraphMix always improves the accuracy of the underlying GNNs such as GCN, GAT and Graph U-Net across all the dataset, with GraphMix(GCN) achieving the best results.

We further present results with fewer labeled samples and results on larger datasets (Cora-Full, Co-author-CS and Co-author-Physics) Section A.3 and Section A.4 respectively. 

4.3. SEMI-SUPERVISED LINK CLASSIFICATION:

In the Semi-supervised Link Classification problem, the task is to predict the labels of the remaining links, given a graph and labels of a few links. Following (Taskar et al., 2004) , we can formulate the link classification problem as a node classification problem. Specifically, given an original graph G, we construct a dual Graph G . The node set V of the dual graph corresponds to the link set E of the original graph. The nodes in the dual graph G are connected if their corresponding links in the graph G share a node. The attributes of a node in the dual graph are defined as the index of the nodes of the corresponding link in the original graph. Using this formulation, we present results on link classification on Bit OTC and Bit Alpha benchmark datasets in the Table 2 . As the numbers of the positive and negative edges are strongly imbalanced, we report the F1 score. Our results show that GraphMix(GCN) improves the performance over the baseline GCN method for both the datasets. Furthermore, the results of GraphMix(GCN) are comparable with the recently proposed state-of-the-art method GMNN (Qu et al., 2019) . Since GraphMix consists of various components, some of which are common with the existing literature of semi-supervised learning, we set out to study the effect of various components by systematically removing or adding a component from GraphMix . We measure the effect of the following:

• Removing the Manifold Mixup and predicted targets from the FCN training.

• Removing the predicted targets from the FCN training.

• Removing the Manifold Mixup from the FCN training.

• Removing the Sharpening of the predicted targets.

• Removing the Average of predictions for K random perturbations of the input sample

• Using the EMA (Tarvainen & Valpola, 2017) of GNN for target prediction.

The ablation results for semi-supervised node classification are presented in Table 3 . We did not do any hyperparameter tuning for the ablation study and used the best performing hyperparameters found for the results presented in Table 1 . We observe that all the components of GraphMix contribute to its performance, with Manifold Mixup training of FCN contributing possibly the most. Furthermore, we observe that using the EMA model (which is an emsemble model) (Tarvainen & Valpola, 2017) for computing the predicted-targets could improve the performance of GraphMix for all the datasets. Under review as a conference paper at ICLR 2020

4.5. VISUALIZATION OF THE LEARNED FEATURES:

In this section, we present the analysis of the features learned by GraphMix for Cora dataset. Specifically, we present the 2D visualization of the hidden states using the t-SNE (van der Maaten & Hinton, 2008) in Figure 2a , 2b and 2c . We observe that GraphMix learns hidden states which are better separated and condensed than GCN and GCN(predicted-targets). We further evaluate the Softrank (refer to Appendix A.7) of the class-specific hidden states to demonstrate that GraphMix(GCN) makes the class-specific hidden states more concentrated than GCN and GCN(predicted-targets), as shown in 2d. Refer to Appendix A.8 for 2D representation of other datasets. 5 RELATED WORK

5.1. SEMI-SUPERVISED LEARNING OVER GRAPH DATA:

There exists a long line of work for Semi-supervised learning over Graph Data. Earlier work included using Graph Laplacian Regularizer for enforcing local smoothness over the predicted targets for the nodes (Zhu & Ghahramani, 2002; Zhu et al., 2003; Belkin et al., 2006) . Another line of work learns node embedding in an unsupervised way (Perozzi et al., 2014) which can then be used as an input to any classifier, or learns the node embedding and target prediction jointly (Yang et al., 2016) . Many of the recent Graph Neural Network based approaches (refer to Zhou et al. (2018) for a review of these methods) are inspired by the success of Convolutional Neural Networks in image and text domains, defines the convolutional operators using the neighbourhood information of the nodes (Kipf & Welling, 2017; Veličković et al., 2018; Defferrard et al., 2016) . These convolution operator based method exhibit state-of-the-results for semi-supervised learning over graph data, hence much of the recent attention is dedicated to proposing architectural changes to these methods (Qu et al., 2019; Gao & Ji, 2019; Ma et al., 2019) . Unlike these methods, we propose a regularization technique that can be applied to any of these Graph Neural Networks which uses a parameterized transformation on the node features.

5.2. DATA AUGMENTATION:

It is well known that the generalization of a learning algorithm can be improved by enlarging the training data size. Because labeling more samples is labour-intensive and costly, Data-augmentation has become de facto technique for enlarging the training data size, especially in the computer vision applications such as image classification. Some of the notable Data Augmentation techniques include Cutout (Devries & Taylor, 2017) and DropBlock (Ghiasi et al., 2018) . In Cutout, a contiguous part of the input is zeroed out. DropBlock further extends Cutout to the hidden states. In another line of research, such as Mixup and BC-learning (Zhang et al., 2018; Tokozume et al., 2017) , additional training samples are generated by interpolating the samples and their corresponding targets. Manifold Mixup (Verma et al., 2019a) proposes to augment the data in the hidden states and shows that it learns more discriminative features for supervised learning. Furthermore, ICT (Verma et al., 2019b) and MixMatch (Berthelot et al., 2019) extend the Mixup technique to semi-supervised learning, by computing the predicted targets for the unlabeled data and applying the Mixup on the unlabeled data and their corresponding predicted targets. Even further, for unsupervised learning, ACAI (Berthelot* et al., 2019) and AMR (Beckham et al., 2019) explore the interpolation techniques for autoencoders. ACAI interpolates the hidden states of an autoencoder and uses a critic network to constrain the reconstruction of these interpolated states to be realistic. AMR explores different ways of combining the hidden states of an autoencoder other than the convex combinations of the hidden states. Unlike, all of these techniques which have been proposed for the fixed topology datasets, in this work, we propose interpolation based data-augmentation techniques for graph structured data.

5.3. REGULARIZING GRAPH NEURAL NETWORKS:

Regularizing Graph Neural Networks has drawn some attention recently. GraphSGAN (Ding et al., 2018) first uses an embedding method such as DeepWalk (Perozzi et al., 2014) and then trains generator-classifier networks in the adversarial learning setting to generate fake samples in the low-density region between sub-graphs. BVAT (Deng et al., 2019) and Feng et al. (2019) generate adversarial perturbations to the features of the graph nodes while taking graph structure into account. While these methods improve generalization in graph-structured data, they introduce significant additional computation cost: GraphScan requires computing node embedding as a preprocessing step, BVAT and Feng et al. (2019) require additional gradient computation for computing adversarial perturbations. Unlike these methods, GraphMix does not introduce any significant additional computation since it is based on interpolation-based techniques and self-generated predicted targets.

6. DISCUSSION:

We presented GraphMix , a simple and efficient regularized training scheme for graph neural networks. GraphMix is a general scheme that can be applied to any graph neural network that uses a parameterized transformation on the feature vector of the graph nodes. Through extensive experiments, we demonstrated state-of-the-art performances or close to state-of-the-art performance using this simple regularization technique on various benchmark datasets, more importantly, GraphMix improves test accuracy over vanilla GNN across all the datasets, even without doing any extensive hyperparameter search. Further, we conduct a systematic ablation study to understand the effect of different components in the performance of GraphMix . This suggests that in parallel to designing new architectures, exploring better regularization for graph structured data is a promising avenue for research.

Jie Zhou, Ganqu Cui, Zhengyan Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In ICML, 2003.

A APPENDIX:

A.1 DATASETS

The statistics of these datasets as well as the number of training/validation/test nodes is presented in Table 4 . We present the comparion of GraphMix with the recent state-of-the-art methods as well as earlier methods using the standard Train/Validation/Test split in Table 5 .

A.3 RESULTS WITH FEWER LABELED SAMPLES:

We further evaluate the effectiveness of GraphMix in the learning regimes where fewer labeled samples exist. For each class, we randomly sampled K ∈ {5, 10} samples for training and the same number of samples for the validation. We used all the remaining labeled samples as the test set. We repeated this process for 10 times. The results in Table 6 show that GraphMix achieves even better improvements when the labeled samples are fewer ( Refer to Table 1 for results with 20 training samples per class).

A.4 RESULTS ON LARGER DATASETS:

In this section, we provide results on three recently proposed datasets which are relatively larger than standard benchmark datasets (Cora/Citeseer/Pubmed). Specifically, we use Cora-Full dataset proposed in Bojchevski & Günnemann (2018) and Coauthor-CS and Coauthor-Physics datasets proposed in Shchur et al. (2018) . We took processed versions of these dataset available here2 . We did 10 random splits of the the data into train/validation/test split. For the classes which had more than 100 samples, we choose 20 samples per class for training, 30 samples per class for validation and the remaining samples as test data. For the classes which had less than 100 samples, we chose 20% samples, per class for training, 30% samples for validation and the remaining for testing. For each split we run experiments using 100 random seeds. The statistics of these datasets in presented in Table 8 and the results are presented in Table 7 . We observe that GraphMix(GCN) improves the results over GCN for all the three datasets. We note that we did minimal hyperparameter search for GraphMix(GCN) as mentioned in Section A.8.1, and doing more rigorous hyperparameter search can further improve the performance of GraphMix . of FCN, the temparature T in sharpening and the number of random perturbations K applied to the input data for the averaging of the predictions.

We conducted minimal hyperparameter seach over only α and γ max and fixed the hyperparameters T and K to 0.1 and 10 respectively. The other hyperparameters were set to the best values for underlying GNN (GCN or GAT), including the learning rate, the L2 decay rate, number of units in the hidden layer etc. We observed that GraphMix is not very sensitive to the values of α and γ max and similar values of these hyperparameters work well across all the benchmark datasets. Refer to Appendix A.5 and A.6 for the details about the hyperparameter values and the procedure used for the best hyperparameters selection.

A.5.1 FOR RESULTS REPORTED IN SECTION 4.2

For GCN and GraphMix(GCN), we used Adam optimizer with learning rate 0.01 and L2-decay 5e-4, the number of units in the hidden layer 16 , dropout rate in the input layer and hidden layer was set to 0.5 and 0.0, respectively. For GAT and GraphMix(GAT), we used Adam optimizer with learning rate 0.005 and L2-decay 5e-4, the number of units in the hidden layer 8 , and the dropout rate in the input layer and hidden layer was searched from the values {0.2, 0.5, 0.8}.

For α and γ max of GraphMix(GCN) and GraphMix(GAT) , we searched over the values in the set [0.0, 0.1, 1.0, 2.0] and [0.1, 1.0, 10.0, 20.0] respectively.

For GraphMix(GCN) : α = 1.0 works best across all the datasets. γ max = 1.0 works best for Cora and Citeseer and γ max = 10.0 works best for Pubmed.

For GraphMix(GAT) : α = 1.0 works best for Cora and Citeseer and α = 0.1 works best for Pubmed. γ max = 1.0 works best for Cora and Citeseer and γ max = 10.0 works best for Pubmed. Input droputrate=0.5 and hidden dropout rate=0.5 work best for Cora and Citeseer and Input dropout rate=0.2 and hidden dropout rate =0.2 work best for Pubmed.

We conducted all the experiments for 2000 epochs. The value of consistency coefficient γ (line 13 in Algorithm 1) is increased from 0 to its maximum value γ max from epoch 500 to 1000 using the sigmoid ramp-up of Mean-Teacher (Tarvainen & Valpola, 2017) . number of random permutations K to 1.0 0.1 and 10 respectively. For λ max , we did search over {1.0, 10.0, 20.0} and found that 10.0 works best.

For all the GraphMix(GCN) experiments, the value of consistency coefficient γ (line 13 in Algorithm 1) is increased from 0 to its maximum value γ max from epoch 500 to 1000 using the sigmoid ramp-up of Mean-Teacher (Tarvainen & Valpola, 2017) .

:

code available at https://github.com/anon777000/GraphMix

https://github.com/shchur/gnn-benchmark

Figures and Tables:

Figure 1: The procedure for training with GraphMix . The Fully-Connected Network (FCN) and the Graph Neural Network (GNN) share linear transformation matrix (W ) applied on the node features. The FCN is trained using Manifold Mixup by interpolating the hidden states H F CN and the corresponding labels Y . This leads to better features which are transferred to the GNN via parameter sharing. The predicted targets generated by the GNN for unlabeled data are used to augment the input data for the FCN. The FCN and the GNN losses are minimized jointly by alternate minimization.

Figure 2: 2D representation of the hidden states of Citeseer dataset using (a) GCN, (b) GCN(predictedtargets), (c) GraphMix, and (d) Soft-Rank of Class-specific hidden states (lower Soft-Rank reflects more concentrated class-specific hidden states)

Figure 3: T-SNE of hidden states for Cora (left), Pubmed (middle), and Citeseer (right). Top row is GCN baseline, bottom row is GraphMix.

The Manifold Mixup training of FCN facilitates learning more discriminative node representations. How to transfer these discriminative node representations to the GNN? We apply parameter sharing between FCN and GNN to facilitate this. Using these more discriminative representations of the nodes, as well as the graph structure, GNN loss is computed in the usual way. Input: A GCN: g(X, A, θ), a FCN: f (X, θ, λ) which shares parameters with the GCN. Beta distribution parameter α for Manifold Mixup . Number of random perturbations K, Sharpening temperature T . Consistency parameter γ. Number of epochs N . γ(t): rampup function for increasing the importance of consistency regularization. (XL, YL) represents labeled samples and XU represents unlabeled samples. 2: for t = 1 to N do 3:

Table content: <table><tr><td>Algorithm 1 GraphMix : A procedure for improved training of Graph Neural Networks (GNN)</td></tr></table>

Compute average predictions across K perturbations of XU using the GCN 11:YU = Sharpen( ȲU , T ) // Apply temperature sharpening to the average prediction 12:Lusup = L f (XU , θ, λ), YU // unsupervised loss from FCN using the Manifold Mixup 13: L = Lsup + γ(t) * Lusup // Total loss is the weighted sum of supervised and unsupervised FCN

Table content: <table><tr><td/><td>loss</td><td/></tr><tr><td>14:</td><td>else</td><td/></tr><tr><td>15:</td><td>L = L g(XL, θ, A), YL</td><td>// Loss using the vanilla GCN</td></tr><tr><td>16:</td><td>end if</td><td/></tr><tr><td colspan="2">17: end for</td><td/></tr><tr><td colspan="2">18: return L</td><td/></tr></table>

Results of node classification (% test accuracy) using 10 random Train/Validation/Test split of datasets. [*] means the results are taken from the corresponding papers. We conduct 100 trials and report mean and standard deviation over the trials (refer to Table5in the Appendix for comparison with other methods on standard Train/Validation/Test split).

Table content: <table><tr><td>Algorithm</td><td>Cora</td><td>Citeseer</td><td>Pubmed</td></tr><tr><td>GCN</td><td colspan="3">77.84±1.45 72.56±2.46 78.74±0.99</td></tr><tr><td>GAT</td><td colspan="3">77.74±1.86 70.41±1.81 78.48±0.96</td></tr><tr><td>Graph U-Net</td><td colspan="3">77.59±1.60 67.55±0.69 76.79±2.45</td></tr><tr><td colspan="4">GCN(with predicted-targets) 80.41±1.78 73.62±2.11 79.81±2.85</td></tr><tr><td colspan="4">FCN(with predicted-targets) 75.19±3.53 70.49±1.91 73.40±2.48</td></tr><tr><td>GraphMix (GCN)</td><td colspan="3">82.07±1.17 76.45±1.57 80.72±1.08</td></tr><tr><td>GraphMix (GAT)</td><td colspan="3">80.63±1.31 74.08±1.26 80.14±1.51</td></tr><tr><td>GraphMix (Graph-U-Net)</td><td colspan="3">80.18±1.62 72.85±1.71 78.47±0.64</td></tr></table>

Results on Link Classification (%F1 score). [*] means the results are taken from the corresponding papers

Table content: <table><tr><td>Algorithm</td><td>Bit OTC</td><td>Bit Alpha</td></tr><tr><td>DeepWalk (Perozzi et al., 2014)</td><td>63.20</td><td>62.71</td></tr><tr><td>GMNN*(Qu et al., 2019)</td><td>66.93</td><td>65.86</td></tr><tr><td>GCN</td><td colspan="2">65.72±0.38 64.00±0.19</td></tr><tr><td>GCN(with predicted-targets)</td><td colspan="2">65.15±0.29 64.56±0.21</td></tr><tr><td>FCN(with predicted-targets)</td><td colspan="2">60.13±0.40 59.74±0.32</td></tr><tr><td>GraphMix (GCN)</td><td colspan="2">66.35±0.41 65.34±0.19</td></tr><tr><td>4.4 ABLATION STUDY</td><td/><td/></tr></table>

Ablation study results using 10 labeled samples per class (% test accuracy). We report mean and standard deviation over ten trials.

Table content: <table><tr><td>Ablation</td></tr></table>

Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neural networks: A review of methods and applications. CoRR, abs/1812.08434, 2018. URL http://arxiv.org/abs/1812.08434. Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical report, 2002.

Table content: <table/>

Dataset statistics.

Table content: <table><tr><td>Dataset</td><td colspan="7"># Nodes # Edges # Features # Classes # Training # Validation # Test</td></tr><tr><td>Cora</td><td>2,708</td><td>5,429</td><td>1,433</td><td>7</td><td>140</td><td>500</td><td>1,000</td></tr><tr><td>Citeseer</td><td>3,327</td><td>4,732</td><td>3,703</td><td>6</td><td>120</td><td>500</td><td>1,000</td></tr><tr><td>Pubmed</td><td>19,717</td><td>44,338</td><td>500</td><td>3</td><td>60</td><td>500</td><td>1,000</td></tr><tr><td>Bitcoin Alpha</td><td>3,783</td><td>24,186</td><td>3,783</td><td>2</td><td>100</td><td>500</td><td>3,221</td></tr><tr><td>Bitcoin OTC</td><td>5,881</td><td>35,592</td><td>5,881</td><td>2</td><td>100</td><td>500</td><td>5,947</td></tr><tr><td colspan="2">A.2 COMPARISON</td><td/><td/><td/><td/><td/><td/></tr></table>

Comparison of GraphMix with other methods (% test accuracy ), for Cora-Full, Coauthor-CS, Coauthor-Physics, and NELL. * refers to the results reported inShchur et al. (2018).

Table content: <table><tr><td>Method</td><td>Cora-Full</td><td colspan="4">Coauthor-CS Coauthor-Physics NELL</td></tr><tr><td>GCN*</td><td>62.2±0.6</td><td colspan="2">91.1±0.5</td><td colspan="2">92.8±1.0</td><td>-</td></tr><tr><td>GAT*</td><td>51.9±1.5</td><td colspan="2">90.5±0.6</td><td colspan="2">92.5±0.9</td><td>-</td></tr><tr><td>MoNet*</td><td>59.8±0.8</td><td colspan="2">90.8±0.6</td><td colspan="2">92.5±0.9</td><td>-</td></tr><tr><td>GS-Mean*</td><td>58.6±1.6</td><td colspan="2">91.3±2.8</td><td colspan="2">93.0±0.8</td><td>-</td></tr><tr><td colspan="2">GCN (Kipf &amp; Welling, 2017) -</td><td>-</td><td/><td>-</td><td>66.0</td></tr><tr><td>GCN</td><td colspan="3">60.13±0.57 91.27±0.56</td><td colspan="2">92.90±0.92</td><td>63.65±1.17</td></tr><tr><td>GraphMix (GCN)</td><td colspan="3">61.80±0.54 91.83±0.51</td><td colspan="2">94.49±0.84</td><td>66.32±1.04</td></tr><tr><td/><td colspan="3">Table 8: Dataset statistics</td><td/></tr><tr><td>Datasets</td><td colspan="4">Classes Features Nodes</td><td>Edges</td></tr><tr><td>Cora-Full</td><td/><td>67</td><td colspan="2">8710 18703</td><td>62421</td></tr><tr><td>Coauthor-CS</td><td/><td>15</td><td colspan="2">6805 18333</td><td>81894</td></tr><tr><td colspan="2">Coauthor-Physics</td><td>5</td><td colspan="3">8415 34493 247962</td></tr><tr><td>NELL</td><td colspan="2">210</td><td colspan="3">5414 65755 266144</td></tr></table>

Peer review (iclr2020):
In this paper, the authors developed GNN by integrating an interpolation based regularization. In particular, the proposed GraphMix model has two components, one GNN and one MLP. For the MLP, the manifold mixup loss is used for training. The two components share parameters so that the GCN training can be regularized by the manifold mixup loss. Moreover, the predicted samples of GCN are used to augment the data of MLP for training in a self-supervised manner. In the experiments, GraphMix was compared with several recent GNN models on several benchmark datasets and demonstrates effectiveness for node classification and link classification. There are several concerns on this paper. 1. The motivation to integrate interpolation based regularization for GNN is not intuitively clear. The authors may want to further clarify on how can Eq. (2) help improve the classification power of GNN in an intuitive manner. 2. Some algorithm design choices, such as sharpening for entropy minimization, are not well justified. It is not clear why sharpening is selected over psudolabels. 3. The overall technical contribution is somewhat incremental based on previous work on GNN and manifold mixup based regularizations. The model design is heuristic, and lack theoretic analysis, for example, on the parameter sharing based regularization over other choices of regularizations, if any. 4. From the experimental results, the proposed model improves GCN and GAT, but the improvements on other baseline methods are a little bit subtle. The authors may want to further evaluate their framework on those GNNs such as GMNN and Graph U-Net to make the experiments more comprehensive, or justify on why they cannot be implemented using the proposed framework. 5. In Table 2, 3, and 4, it is not clear on why GAT and GraphMix (GAT) are missing. This inconsistency with regard to Table 1 should be justified. 6. In the ablation analysis, if the reason for the suboptimal performance when using EMA is the lack of hyperparameter search, then it is suggested to perform comprehensive hyperparameter search so that the results are more solid. Otherwise, the design choice cannot be well justified.