Improving Subgraph Representation Learning via Multi-View Augmentation
Authors: Yili Shen
Cheng-Wei Ju
Jiaxu Yan
Jun Yi
Zhou Lin
Hui Guan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Learning
Subgraph Representation Learning
Graph Data Augmentation
Multi-view Augmentation

Abstract: Subgraph representation learning based on Graph Neural Network (GNN) has exhibited broad applications in scientific advancements, such as predictions of molecular structure-property relationships and collective cellular function. In particular, graph augmentation techniques have shown promising results in improving graph-based and node-based classification tasks. Still, they have rarely been explored in the existing GNN-based subgraph representation learning studies. In this study, we develop a novel multi-view augmentation mechanism to improve subgraph representation learning models and thus the accuracy of downstream prediction tasks. Our augmentation technique creates multiple variants of subgraphs and embeds these variants into the original graph to achieve highly improved training efficiency, scalability, and accuracy. Benchmark experiments on several real-world biological and physiological datasets demonstrate the superiority of our proposed multi-view augmentation techniques in subgraph representation learning.

Under review as a conference paper at ICLR 2023
IMPROVING SUBGRAPH REPRESENTATION LEARNING VIA MULTI-VIEW AUGMENTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Subgraph representation learning based on Graph Neural Network (GNN) has exhibited broad applications in scientific advancements, such as predictions of molecular pructtre--rroperyy relationships and collective cellular function. In particular, graph augmentation techniques have shown promising results in im- proving graph-based and node-based classification tasks. Still, they have rarely been explored the existing GNN-based subgraph representation learning stud- ies. In this study, we develop novel multi-view augmentation mechanism improve subgraph representation learning models and thus the accuracy of down- stream prediction tasks. Our augmentation technique creates multiple variants of subgraphs and embeds these variants into the original graph to achieve highly im- proved training efficiency, scalability, and accuracy. Benchmark experiments on several real-worls biological and physiological datasets demonstrate the superior- ity of our proposed multi-view augmentation techniques subgraph representa- tion learning.
INTRODUCTION
Subgraph representation learning using Graph Neural Networks (GNNs) can be broadly applied to various subgraph-related tasks in many fields of science and technology. As an outstanding example, the PPI Prootein-Proeete Interaction) network (Zitmik et al. 2018) uses nodes, edges, and subgraphs to represent single proteins, their interactions, and the set of interacting proteins, respectively. GNNs can be used to predict the biological processes (PPI-BP), cell component (PPI-CC), and molecular function (PPI-MF) by classifying the functionality of subgraph (i.e., a group of proteins) in the PPI network. Another example is to apply GNN to fragment-ber based quantum chemical theory where each fragment in a crystal or aggregate is a subgraph and subgraph representation learning can predict the quantitative interactions between different fragments. Although applying GNNs to subgraph-relane tasks (Alsentzer et al., 2020 Kim & Oh 2022 Wang & Zhang 2021) starts to draw some attention, none of them have implemented graph augmentation techniques to improve task accuracy.
This work presents a novel multi-view approach to augment graphs for improving accuracy of sub- graph classification tasks. Inspired by the effectiveness of graph contrastive learning Hassani & Khasahmadi (2020); [Zhu et al.] (2020): You et al. (2020), our basic idea is to create multiple views of subgraph by augmenting learn the embedding for each of the view, and then combine the rep- resentations for predicting the label of the subgraph. The rationale behind it is that the augmented subgraphs (i.e., the multiple views) essentially form ensemble, which could provide more robust signal in determining the properties of the subgraph.
The basic idea poses a fundamental challenge in how to efficiently create augmented subgraphs. Augmenting the entire graph to produce different views of the same subgraph is not scalable because the size of the augmented graph will grow linearly with the number of views. Figure [c) illustrates the problem. With only one additional view, GNNs need to conduct forward and backward propaga- tions on two independent graphs (i.e., the original graph and the augmented graph) during training, doubling the training cost. We address the efficiency issue by embedding augmented subgraphs in the original graph, significantly decreasing the demand for GPU resources. In this case, the compu- tation of the embeddings for the augmented subgraphs can share intermediate representations within their neighborhood. Figure]] d()) illustrates an alternative efficient design where the augmented sub- graphs embedded into an augmented graph, instead of the original graph. We empirically validate

---

Under review as a conference paper at ICLR 2023
that preserving the original view of subgraphs is essential for multi-vied augmentation improve task accuracy.
w/ Augmentation
My Augreeentiinn AlMim
OTTO
EDOO
XOO0
C20B
$4200
2288
60
Figure 1: Illustration of graph augmentation approaches. (a) The original graph 9 contains two subgraphs (colored in orange and blue). (b) Augmented subgraphs created by randomly dropping some edges in G. The new graph 9' is called the augmented graph. (c) A graph with two independent components where g is the original graph and G' is the augmented graph. Learning on this graph doubles the training cost. (d) Augmented subgraphs g" are embedded into an augmented graph g'.
In summary, this work makes the following contributions:
This work proposes a novel multi-view augmentation strategy to improve the accuracy of subgraph- based learning tasks. This study is the first to explore the benefits of graph augmentation techniques in subgraph representation learning.
The proposed multi-view augmentation strategy dynamically binds augmented subgraph views to the whole graph to drop exaggerated GPU resource consumption in order to achieve highly-improved training efficiency and task accuracy.
Empirical evaluations on three subgraph datasets demonstrate that our augmentation ap- proach can improve existing subgraph representation learning by 0.3% 2.9%% in accuracy, which is on average 1.1% higher than general graph augmentation techniques DropEdge and GAug-M.
RELATED WORKS
Subgraph Representation Learning Subgraph representation learning using GNNs has gained substantial attention these years (Meng et al. 2018) due to its broad applications in scientific domains. Outstanding examples include SubGNN (SubGraph Neural Network) (Alsentzer et al. 2020). which routes messages for internal and border properties within sub-channels of each chan- nel, including neighborhood, structure, and position. After that, the anchor patch is sampled and the features of the anchor patch are aggregated to the connected components of the subgraph through six sub-channels. GLASS (Wang & Zhang 202I) employs a labeling trick (Zhang et al. (02I) and labels nodes belonging any subgraph to boost plain GNNs on subgraph tasks. S2N (Subgraph- To-Node) (Kim & Oh 2022) translates subgraphs into nodes and thus reduces the scale of the input graph. These approaches focus developing novel subgraph-based GNNs improve task accu- racy, but they have never implemented graph augmentation techniques.
Graph Augmentation Data augmentation is a vital part of deep learning. Many general graph augmentation techniques have been proposed to improve task accuracy recently. For node classifi- cation tasks, Rong et al. (2020) proposes DropEdge to randomly drop the edges a graph to enlarge the support of the training distribution. DGI (Deep Graph Infomax) (Veličković et al. 2019)) perturbs the nodes by performing row-wise swap of the input feature matrix while the adjacency matrix re- mains unchanged, generating negative samples for comparison learning and maximizing the mutual information of input and output. GAug (Zhao et al. (02I) generates and removes edges of the graph by training an edge predictor to finally achieve the effect of high connectivity between nodes
2

---

Under review as a conference paper at ICLR 2023
within the same class and low connectivity between nodes from different classes. NeuralSparse (Neural Sparsification) (Zheng et al., 2020) proposes a supervised graph sparsification technique that improves generalization by learning to remove potentially task-irrelevant edges from the input graph. GraphCI. (You et al. 2020) points out that different data augmentation techniques introduce different advantages in graph learning tasks in different domains. For example, edge perturbation can enhance learning in social network graphs, but can be counterproductive in compound graphs learning by destroying the original information. SUBG-CON (SUBGraph CONtrast) (Jiao et al., 2020) samples series of subgraphs containing regional neighbors from the original graph as train- ing data serve as an augmented node representation. Although these methods show promising results for augmenting graphs for node- and graph-based downstream tasks, they are not designed for augmenting subgraphs for subgraph- tasks.
Multi-view Graph Learning Multi-view representation learning on graphs has attracted signifi- cant attention because they capture different properties on the same graph. Hassani et al. (Hassani & Khasahmadi 2020) introduce a multi-view graph learning manner perform contrastive learn- ing. O2MAC (One2Multi graph AutoenCoder) (Fan et al. (020) proposes a multi-view- -based auto- encoder to promote self-supervised learning. MV-GNN Multi-View Graph Neural Network) (Ma) iet al., 2020) utilizes two MPNNs (Message Passing Neural Networks) (Gilmer et al. (017) to en- code atom and bond information respectively via multi-view graph construction. They construct multi-view graphs to express different levels of information in a graph, which is an intuitive and ef- ficient way of building augmented graphs. Our work also leverages multi sitii-veewbbbar augmentation but focuses subgraph-based tasks.
3 NOTATIONS AND PRELIMINARIES
3.1 NOTATIONS
Let g = (V, E, X) denote a graph, where V = 21, represents the node set, E S V x V represents the edge sets, and X is the matrix that represents the corresponding node feature. X,, the 7th row of X, represents the features associated with the 7th node. Let U; denote a node in g. The adjacency matrix A (0. 1|NN where ai denotes that (ra, E E. Gs (Vs, Es, Xs) denotes a subgraph of G. where Vs C V. Eg S E n (Vs x V)s, and Xs stacks the rows of X belonging Vs. The adjacency matrix subgraph Gs is As.
3.2 SUBGRAPH REPRESENTATION LEARNING
Given the set of subgraphs $ = [Gs,: Gs,,.., Gs., and their labels T = (1.s, As......,.e Is..) }. the goal of Subgraph Representation Learning is to learn a representation embedding hs, for each subgraph Gs. to predict the corresponding ts..
3.3 GRAPH AUGMENTATION
In the present work, we illustrate our multi-view augmentation scheme based on two typical existing graph augmentation strategies, DropEdge (Rong al. 2020)) and GAug-M (Zhao al. 2021).
DropEdge is graph data perturbation strategy that randomly drops edges in a graph (Rong et al. 2020) that it enlarges the training support improve the performance of GNNs on node-level tasks. We employ DropEdge for each subgraph to generate an augmented subgraph, by generating stochastic boolean mask M, E IRT m where m is the number of nodes in the subgraph and p represents the rate of dropping edges. The new adjacency matrix becomes A's As - M, O As. where ( means the element-wise wise product.
GAug-M (Zhao et al.) 2021 is graph data augmentation strategy that leverages neural edge predic- tors to promote intra-class and demote inter-class edges so as to form new edge weights. It contains a two-stage training schema. In the first step, we use VGAE (Variational Graph Auto-Encoders) (Kipl) & Welling 2016) as the edge predictor to get an probaabityy matrix M, which describes the graph's probabilistic connectivity, M = a alZ ). where Z GNN (A. X)). Denote [ as the number edges in graph G. In the second step. we use the probability matrix M, to make the top i | non-edges with the highest edge probabilities be connected, and the least, j edges with the
3

---

Under review as a conference paper at ICLR 2023
lowest edge probabilities to be disconnected to produce augmented graph from g to G', where ? E {0, 1}.
4
METHODS
In this section, we present our proposed multi- augmentation approach, following by analyses of the computational complexity and discussions the shortage of the alternatives shown in Figure and the advance of our multi-view approach.
4.1 METHODOLOGGY
Figure iillustates the basic idea of the multi-view augmentation that implemented in the present study. At each forward step, we generate augmented views of subgraphs in this batch by randomly perturbing original subgraphs with a particular graph data augmentation strategy. After that, we add the augmented subgraphs to the original graph and feed the new graph into subgraph-spect neural network. Here, we obtain subgraph embeddings of both the original subgraph and the aug- mented subgraph. These embeddings are fed into pooling function to generate single subgraph embedding for each subgraph, which is used for downstream subgraph-baser tasks. Meanwhile, to maximize the agreement between the original subgraph embeddings and the augmented subgraph embeddings, we utilize the contrastive loss between the original graph and augmented graphs.
Subgraph porturbation
Original griph
Poolig
WROSS
VOOUT
0200
Figure 2: Overview of our proposed subgraph augmentation approach. The two subgraphs in the original graph are colored in green and orange. We first generate multi-subgraph views via stochastic augmentation. Following that we connect the augmented subgraph to the remaining part of the original graph, by adding edges that link the augmented subgraph and the whole graph. After feeding forward the whole graph into aubgraph-specifff GNNs, we extract the subgraph embeddings of different views, respectively (triangles and squares). Ultimately, fuse the embeddings of different views by pooling function and obtain the augmented subgraph embeddings (diamonds).
obtain augmented subgraphs 9'S (V's 1's, X's). We enrich the original graph to include both the augmented subgraph and the original subgraph. The enriched graph is thus called Multi- View Graph. Mathematically, the multi-view graph g' (V E','', where V = -UU W's. The consequent adjacency matrix becomes
After subgraph augmentation,
A
Al: V'sl As
A' =
(1)
A A
Feeding forward the multi-view graph into subgraph-specific neural networks, by selecting the sub- graph and augmented subgraph nodes by masks Mso, Ms.., we can get the embeddings of both the augmented subgraph and the original subgraph and denote them as hs, and hs respectively. We fuse different subgraph embeddings into one embedding by applying pooling function (e.g., MaxPool or AvgPool):
H GNN(G")
(2)
hso NSA = H Ms. HMM.. hg Pool(hs. hg,)
(3)
(4)
4

---

Under review as a conference paper at ICLR 2023
With the learned subgraph embeddings. predict the subgraph properties by applying MLP (Multi-Layer Perception), (Hastic et al. 2009)
is softmax (MLP (hs)).
(5)
Meanwhile, we implement the contrastive loss between the augmented views, which enforces the embeddings of the original subgraph and the generated subgraph embedding to be close and those of augmented subgraphs not generated by this subgraph to be distant. Let À be the coefficient to control the contrastive loss contribution, the total loss function becomes
L Las(ts ts) co (1so hgg)).
4.2 COMPUTATIONAL COMPLEXITY ANALYSIS
The proposed subgraph multi-view augmentation is independent to augmentation strategies or subgraph-specific GNNs. Thus, we can train the model there's neither modification at the beginning of each epoch nor after the forward. We can sim- ply analyze the additional computational complexity made by the augmentation, the corresponding augmented graph inference at each forward step.
end-to-end fashion, which means
Let the number of nodes in the original graph be /V] the number of edges be |E], the number of the 7th subgraph node be V l, the number of the jt subgraph node be | and the training setting
is going to be B[[Vs]] Σ, Vs. b
for batch size be Therefore, the expectation of (Vsl. E[[s sl] |]
Likewise, the expectation of [ss. [[[ is going to be Es|]] Σ, [Es,1
b
Considering a graph neural network for which the computational complexity is O(JONN (VV, [E)), and the memory complexity is ((gNNN( [E]), the computational complexity for augmen-
tation strategy is O Off (1V), E)). and the memory complexity is Og (1V), [B]). Because
augment on subgraphs only during the augmentation stage, the computational complexity is
O(fau (E[[Vsl Is], E[JEEs]]] []), and the memory complexity is Oggau (E sll. (E With regard to the forward step, the total computational complexity of the whole graph is [ + E[ and
that of edges is [E] + BE EEsl]] In this way. the inference computational complexity is O (faNN [E] +
E||
R Es))) ; and the memory complexity O (FE) + E Il.
[ + E [Es]))
To summarize our analysis. the overall computational complexity is O(Jaug (E[Vsl), [E|])] + JCNN( +
Es [F]] + [[Ess and the overall memory complexity
Oggaug [EI (B[[Vsl]] sl), E + 9GNN( Bs]] [B] B[[Bs))))
4.3
STRENGTHS OF OUR MULTI- VIEW AUGMENTATION SCHEME
4.3.1
MUL.TI-IIEE ON SUBGRAPH
Augmentation on whole-graph vs. Augmentation on subgraph The augmentation strategies are supposed directly modify the whole-graph edges and nodes. However, the time complexity be- comes unaffordable for the augmentation strategy when it is greater than O(VI). Because the nodes of the subgraphs only accept messages from their k-hop neighbors, it's more efficient to augment subgraph-related edges.
Single-view vs. Multi-view DropEdge (Rong al. 2020) and GAug-M Zhao al. 2021 take the strategy of single-view augmentation. One way to augment the subgraph-bases data is to augment the original graph using the single-view augmentation on subgraph edges to perturb the message- passing flow of the subgraph, which will reduce the amount of the information contained in the original graph. Our multi-view method, on the other hand, can keep the original subgraph structures because it just modifies the augmented subgraph.
Copy graph vs. Subgraph multi-view GraphCL (You et al. 2020) creates another graph view to allow contrastive samples to perform the self-supervised training. It's intuitive create another whole graph by using the augmentation strategies. Neverthless, such an augmentation flow doubles the memory and time consumptions, which is not affordable for large-scale graph learning tasks like social networks or protein- protein interactions. In this way, multi-view graph learning improves the hardware resource requirements and accelerates the computations.
5

---

Under review as a conference paper at ICLR 2023
Augmentation all views vs. Preserving one original view Although augmentation all views will provide wider training support than keeping the original subgraph structures, however, there is a loss of the information from the original subgraphs. Because perform the augmentation on the newly-created view, it's enough for perturbing so that keeping one original view can help the model to learn lossless graph-based information. In this way, our proposed multi-view augmentation can show more generalizability and efficiency in subgraph representation learning.
4.3.2 STRAIGHT INFERENCE VS. JOINING IN CONTRASTIVE LOsS
Comparing with straight inference, a contrastive loss can effectively make the source subgraphs and the augmented subgraphs closer and the other augmented subgraphs more distant. This strategy will prevent the increased disagreement of views after epochs of training.
EXPERIMENTS
In this section, we evaluate the performance of our proposed subgraph multi-view augmentation strategy actoss variety of architectures, datasets, and graph augmentation strategies, and compare them with other strategies to show the effectiveness of our multi-view subgraph augmentation.
5.1 EXPERIMENT SETTINGS
5.1.1 DATASETS
Table 0 summarizes statistics of the datasets obtained from Sub- GNN (Alsentzer et al. 2020). We follow the split reported in Wang & "hhang (2021) Specifically, PPI- BP (Zitink et al. 2018) aims pre- dict the collective cellular function of given set of genes known to be associated with specific BP (Biological Processes) in common. The graph shows the correlation of the human PPI (Protein- Protein Interaction) network where nodes represent proteins and edges represent the interaction between proteins. A subgraph is defined by the collaboration of proteins and labeled according to cellular functions from six categories (metabolism, development, signal transduction, stress/death, cell organization, and transport). HPO-METAB and HPO-NEURO (Splinter et al. 2018 Hartley et al. 2020) simulate rare disease diagnosis with the task of predicting subcategories of metabolic and neurological disorders that are the most consistent with these phenotypes. The graph is a knowl- edge graph containing phenotypic and genotypic information for rare diseases. A subgraph consists of a collection of phenotypes associated with rare monogenic diseases.
Table Statistics of four real-world datasets.
DATASET
NODES
EDGES
SUBGRAPHS
PPI-BP HPO-METAB
17,080
316,951
1,591
14,587
3,238,174
2,400
HPO-NEURO 14,587
3,238,174
4,000
5.1.2 GNN MODELS
The proposed augmentation technique is model-agnostic because it does not alter the GNN model. In the experimental part, we evaluate the proposed multi-view augmentation using a subgraph-speciffe model GLASS (Wang& & Zhang 2021) and two widely-used GNN architectures: GCN (Graph Con- volutional Networks) (Kipf & Welling, 2017) and GSAGE (GraphSAGE) (Hamilton et al. 2017). We chose DropEdge (Rong et al. 2020) and GAug-M (Zhao et al. 2021) to create subgraph views that are different from the orignal subgraph.
5.1.3 BASELINES FOR COMPARISON
We compare our proposed approach w/ Augmentation MV with the following baselines.
Original: It directly trains the GNN model (i.e., GLASS, GCN, and GSAGE) to establish baseline without any augmentation (see Figure Ta)).
w/ Augmentation: It applics DropEdge or GAug-M on the original graph G to get an augmented graph g' (see Figure (()).. The GNNs will learn the augmented graph g'.

---

Under review as a conference paper at ICLR 2023
w/ Augmentation Copy: It applies DropEdge GAug-M the original graph G to get an augmented graph g' (see Figure (igure(ec). The GNNs will learn on both & and g". The final embedding is generated after a pooling operation in Equation 4
w/ Augmentation AlIView: It applies DropEdge or GAug-M on both original subgraphs and the augmented subgraphs, to generate g' and g", which fails to preserve the subgraph structure (see Figure [d).. The final embedding is generated after a pooling operation as in Equation
5.1.4 IMPLEMENTATION DETAILS
We perform hyperparameter search for all of the baselines and our approach and report the best mean Fl score on all test datasets. The searching space of hyperparameters and details are provided in in the Appendix A Because GAug-M (Zhao et al., 2021) requires an initial edge probabilistic distribution, we use VGAE (Kipf & Welling 2016) to perform the self-supervised learning on edge prediction to generate the distribution. For the GNN model GLASS (Wang & Zhang [02I], we first train the model in an unsupervised manner, and then use supervision from downstream tasks to fine-tune the model hyperparameters, following the procedure provided by the original paper. We perform 10 independent training and validation processes with 10 distinct random seeds.
5.2
RESULTS
5.2.1
OVERALL RESULTS
The empirical performance is summarized in Tables 2 and 3 Our proposed subgraph augmentation improves most of the task accuracy and improves across all three datasets where the whole-graph augmentation strategy succeeded in improving the accuracy, we applied our strategy into GCN, GSAGE, and GLASS. It performs better than most baseline approaches, mainly because it inhibits over-smoothing and over-fitting problems. Specifically, our approach improves the Micro-F1 scores by 2.9%, 0.5%%, and 0.3%% compared to plain GLASS, GCN, and GSAGE, respec- tively. Specifically, when we apply multi-view augmentation (MV) to GCN and GSAGE, we find that such backbones combined with multi-view augmentation show a great increase in performance.
Table 2: Mean Micro-F1 scores with standard deviations of the mean on three real-world datasets. Results are provided from runs with 10 random seeds on DropEdge.
BACKBONE
METHOD
PPI-BP
HPO-METAB
HPO-NEURO
Original
0.610 + 0.006
0.600 1 0.003
0.678 1 0.004
w/ DropEdge
0.626 0.006
0.607 0.008
0.678 = 0.004
w/ DropEdge copy
0,605 + 0.006
0.593 0.013
0.676 + 0.003
GLASS
DropEdge AllView
0.613 0.007
0.577 0.008
0.672 + 0.006
w/ DropEdge MV
0.628 + 0.005
0.607 + 0.008
0.685 + 0.003
Original
0.613 0.008
0.553 + 0.018
0.658 + 0.007
w/ DropEdge
0.618 - 0.006
0.556 0.006
0.651 + 0.006
w/ DropEdge copy
0.553 0.005
1349+00.00
0.317 + 0.018
GCN
w/ DropEdge AllView
0.595
0.552 0.019
0.613 + 0.010
w/ DropEdge MV
0.619 + 0.006
0.567 + 0.006
0.622 + 0.007
Original
0.621 + 0.006
0.581
+ 0.008
0.684 0.002
w/ DropEdge
0.618 + 0.006
0.556 0.006
0.651 + 0.006
w/ DropEdge copy
0.593 0.010
0.555 - 0.011
0.676 - 0.005
GSAGE
w/ DropEdge AllView
0.618 + 0.006
0.567 0.011
0.682 + 0.002
w/ DropEdge MV
0.623 + 0.003
0.591 = 0.006
0.687 + 0.001
On augmentation on subgraphs The augmentation should take place in the k-hop neighbors of subgraph to avoid the requirement of extreme fine tunes on hyperparameters,
demonstrated
from Tables 2 and 3 Comparing fields of w/ Augmentation with w/ Augmentation MV we see that the performance of our approach at least maintains the performance of implementing augmentation on whole graph. On the GSAGE backbone, we can see that DropEdge and GAug- M promote the performance of HPO-NEURO by 3.6% and 2.7%, respectively, compared with the
7

---

Under review as a conference paper at ICLR 2023
Table 3: Mean Micro-FI scores with standard deviations of the mean three real-world datasets. Results are provided from runs with 10 random seeds on GAug-M.
BACKBONE
METHOD
PPI-BP
HPO-METAB
HPO-NEURO
Original
0.610 + 0.006
0.600 + 0.003
0.678 = 0.004
w/ GAug-M
0.609 0.003
0.596 = 0.007
0.681 0.003
GLASS
w/ GAug-M copy
0.621 + 0.006
0.593 = 0.016
0.679 0.003
w/ GAug-M AllView
0.617 0.004
0.579 + 0.005
0.672 0.006
w/ GAug-M MV
0.625 + 0.004
0.598 + 0.005
0.682 + 0.003
Original
0.613 + 0.008
0.553 + 0.018
0.658 = 0.007
w/ GAug-M
0.603 + 0.008
0.557 + 0.011
0.641
0.006
w/ GAug-M copy
0.556 + 0.007
0.335 + 0.025
0.306 = 0.016
GCN
w/ GAug-M AllView
0.591 + 0.006
0.544 0.008
0.593
0.007
w/ GAug-M M MV
0.616 +: 0.008
0.564 + 0.008
0.640
0.003
Original
0.621 0.006
0.581 + 0.008
0.684
0.002
w/ GAug-M
0.603 + 0.017
0.588 + 0.0T2
0.672
0.004
w/ GAug-M copy
0.606 + 0.006
0.546 = 0.007
0.684 = 0.003
GSAGE
w/ GAug-M AllView
0.622 + 0.005
0.583 = 0.001
0.682 0.003
w/ GAug-M M MV
0.628 + 0.005
0.591 0.008
0.689 + 0.003
direct augmentation. This result shows that our approach is able to get stronger results within limited hyperparameter space because we perform the augmentation on neighbor regions, which is more important region for augmentation.
On creating subgraph multi-view only As discussed in Section 4.3. one way to perform aug- mentation is to duplicate graph. From Tables [and comparing helds of w/ Augmentation copy with w/ Augmentation MV, our approach always performs better than copying the original graph. We observe sharp decrease by over 21.8% the GCN backbone with HPO datasets. This result indicates the importance of the addition of the contrastive loss, and proves that duplicating the original graph may take potential failures aligning the original graph and the augmented graph.
On preserving one view The argument that augmenting all views can bring more training support models but lose the information from the original graph, as discussed in Section|4.3 can also be provided by Sables[aan]] Comparing fields of w/ Augmentation AllView with w/ Augmentation MV, we can find that our approach also brings some improvements. These results echo that it's necessary to keep at least one view to make the subgraphs lossless in information. w/ Augmentation copy with w/ Augmentation MV field,
5.2.2 A ABLATION STUDY ON THE CONTRASTIVE Loss
In addition to the comparison on different augmentation strategies. we also perform the ablation on the importance of the contrastive loss, as reported in Tables 4 [nd Overall, the results show that the contrastive loss can greatly improve the learning process of subgraph embeddings, which utilizes maximizing the agreement between the original subgraph and the augmented subgraph. Also, the decrease on w/ Augmentation copy compared with our approach echoes this point of view.
5.2.3 COMPUTATIONAL TIME
We compare the computational time for the direct augmentation on original graphs w/ Augmentation, copying the original graph w/ Augmentation Copy and our approach w/ Augmentation MV, based on the same experimental settings and hyperparameters. We generate the records using GAug-M as the aug- mentation approach, GLASS as the backbone, and PPI-BP the dataset. The results are shown in Table 6 We can that using multi-view does not greatly increase the computational complexity and the memory complexity in practice. Meanwhile, directly duplicating the graph will almost double the training time, which means unaffordable
Table 6: The training time records gener- ated by the experiments.
METHOD TIME EPOCH (s)
w/ GAug-M
0.439
w/ GAug-M MV
0.465
w/ GAug-M copy
0.788
8

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation Study on DropEdge.
BACKBONE
METHOD
PPI-BP
HPO-METAB
HPO-NEURO
Original 0.610 + 0.006
0.600 = 0.003
0.678 0.004
GLASS
W/ No Contrast
0.618:
0.597 1 0.005
0.685
1.0003
Contrast
0.628 = 0.005
0.607 = 0.008
0.683 0.004
Original
0.613 + 0.008
0.553 + 0.018
0.658 0.007
GCN
w/ No Contrast
0.616 = 0.006
0.506 = 0.030
0.606 0.001
w/ Contrast
0.619 = 0.006
0.567 = 0.006
0.622 + 0.007
Original
0.621 + 0.006
0.581 = 0.008
0.684 + 0.002
GSAGE
w/ No Contrast
0.616 + 0.007
0.583 = 0.008
0.684 + 0.004
w/ Contrast
0.623 + 0.003
0.591 = 0.006
0.687 + 0.001
Table 5: Ablation Study on GAug-M.
BACKBONE
METHOD
PPI-BP
HPO-METAB
HPO-NEURO
Original
0.610 = 0.006
0.600 = 0.003
0.678 + 0.004
GLASS
w/ No Contrast
0.611 0.006
0.591
0.007
0.682 + 0.003
w/ Contrast
0.625 + 0.004
0.598 + 0.005
0.680 + 0.003
Original
0.613 = 0.008
0.553
0.018
0.658 E 0.007
GCN
w/ No Contrast
0.613 + 0.008
0.513 # 0.019
0.580 + 0.015
w/ Contrast
0.616 + 0.008
0.564 + 0.008
0.640 + 0.003
Original
0.621 + 0.006
0.581 0.008
0.684 + 0.002
GSAGE
w/ No Contrast
0.621 = 0.005
0.582 = 0.008
0.684 = 0.003
Contrast
0.628 0.005
0.591 + 0.008
0.689 + 0.003
increase on the computational complexity. This result shows that our approach gains both efficiency and effectiveness in subgraph augmentation comparing with other baseline approaches.
CONCLUSION
We propose a novel model-agnostic subgraph augmentation strategy to facilitate subgraph-baser GNNs. By creating new subgraph and link to the original graph, it will include more diversity in message passing to the graph to enhance model training support. This subgraph-specific augmenta- tion strategy can improve the performance and the robustness of graph neural network. A bunch of improved experiments on both GAug-M and DropEdge on different datasets show the general- izability models and augmentation strategies. It's expected that our research will empower the subgraph representation learning to go further and broader.
REFERENCES
Emily Alsentzer, Samuel G. Finlayson, Michelle M. Li, and Marinka Zitnik. ral networks. In Advances in Neural Information Processing Systems, 2020. URL https:
Subgraph
Shaohua Fan, Xiao Wang, Chuan Shi, Emiao Lu, Ken Lin, and Bai Wang. One2Multi graph autoen- coder for multi-view graph clustering. In Proceedings of The Web Conference 2020 (WWW '20), 2020.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neu- ral message passing for quantum chemistry. International Conference on Machine Learning, volume 70, pp. 1272-1272, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. volume 30, pp. 1034-1034, 2017.

---

Under review as a conference paper at ICLR 2023
Taila Hartley, Gabrielle Lemire, Kristin D. Kernohan, Heather E. Howley, David R. Adams, and Kym M. Boycott. New diagnostic approaches for undiagnosed rare genetic diseases. Annu. Rev. Genomics. Hum. Genet., 21(1):351-372, 2020.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning graphs. International Conference on Machine Learning, volume 385, pp. 4116-4126, 2020.
Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2. Springer, 2009.
Yizhu Jiao, Yun Xiong, Jiawei Zhang, Yao Zhang, Tianqi Zhang, and Yangyong Zhu. Sub-graph contrast for scalable self-supervised graph representation learning. 2020 IEEE International Conference on Data Mining, pp. 222 231, 2020.
Dongkwan Kim and Alice Oh. Efficient representation learning of subgraphs by subgraph-to-node translation. In International Conference on Learning Representations 2022 Workshop on Ge- ometrical and Topological Representation Learning, 2022. URL https:/ httpps://opearerieee
net//oorm2iid-ggeee--
Thomas N Kipf and Max Welling. arXiv:1611.07308, 2016.
Variational graph auto-encoders.
arXiv preprint
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional
networks. In International Conference Learning Representations, 2017. URL https:
7/ //openreveww net. wttp://oooorrmm---ssSuaaaaa
Hehuan Ma, Yatao Bian, Yu Rong, Wenbing Huang, Tingyang Xu, Weiyang Xie, Geyan Ye, and Junzhou Huang. Multi-view graph neural networks for molecular property prediction. arXiv preprint arXiv:2005. 10000055.36677. 13607, 2020.
Changping Meng. Chandra Mouli, Bruno Ribeiro, and Jennifer Neville. Subgraph pattern neural networks for high-order graph evolution prediction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks node classification. In International Conference Learning Repre- sentations, 2020. URL http:///pppeeeeeeeeeeeeeetottttm
Kimberly Splinter, David R. Adams, Carlos A. Bacino, Hugo Bellen, Jonathan A. Bernstein, Alys M. Cheatle-Jarvela, Christine M. Eng. Cecilia Esteves, William A. Gahl, Rizwan Hamid, Howard J. Jacob, Bijal Kikani, David M. Koeller, Isaac S. Kohane, Brendan H. Lec, Joseph Loscalzo, Xi Luo, Alexa T. McCray, Thomas O. Metz, John J. Mulvihill, Stanley F. Nelson, Christina G.S. Palmer, John A. Phillips, Leslie Pick, John H. Postlethwait, Chloe Reuter, Van- dana Shashi, David A. Sweetser, Cynthia J. Tifft, Nicole M. Walley, Michael F. Wangler, Monte Westerfield, Matthew T. Wheeler, Anastasia L. Wise, Elizabeth A. Worthey, Shinya Yamamoto. and Euan A. Ashley. Effect of genetic diagnosis patients with previously undiagnosed disease. N. Engl. J. Med., 379(2)):213 1-19999 2018.
Petar Veličković, William Fedus, William I.. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In International Conference Learning Representations, 2019. URL htt.httts///ppeeeiiicoo
Xiyuan Wang and Muhan Zhang. GLASS: GNN with labeling tricks for subgraph representation learning. In International Conference Learning Representations, 2021. URL https://
openreview.nee www.nneet///ruteeee - XLxhE,KNbX1
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812- 5823, 2020.
10

---

Under review as a conference paper at ICLR 2023
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: theory of using graph neural networks for multi-node representation learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL httpttts///ppeereviiieeeee ew. net/ forum? i d= 9mg
Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data aug- mentation for graph neural networks. In Proceedings of the AAAI Conference on Artificial Intel- ligence, volume 35, pp. 11015-11023, 2021.
Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In International 2020.
Conference Machine Learning, volume 119, pp.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint 2006:00000.41333 2020.
Marinka Zitnik, Rok Sosič, Sagar Maheshwari, and Jure Leskovec. BioSNAP Datasets: Stan- ford biomedical network dataset collection, 2018. URL http::/snap.... staford.edd. biodata
A APPENDIX
A.1 IMPLEMENTATION DETAILS
A.1.1 HYPERPARAMETER TUNING
By solidating the hyperparameters by GLASS on all backbone settings, we implement the grid hyperparameter searching on augmentations. For X, we select searching space (0, 0.01, 0.05, 0.1, 0.5, 1,1.25, 1.5, 2.5). For DropEdge-related tasks, our searching space for p is (0.1, 0.2, 0.3, 0.4, 0.5). For GAug-related tasks, our searching space for (rm-pct, add pct) is {(0. 50), (15, 35), (30, 20), (45, 5)}. Since we used VGAE to generate different probabilistic edge distribution, we selected i from (1.
11


Reviews:
[{"review_id": "Fge6QSE-WW", "reviewer": "Reviewer_7uEX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a multi-view approach applied to both the original and augmented subgraphs to learn subgraph embeddings/tasks. They used existing graph augmentation techniques, and then align the augmented new graph with the original one and extract subgraph embeddings using available subgraph GNNs. ", "summary_of_the_review": "A well-executed, good quality paper, yet I would like to see a bit more depth and novelty. ", "strength_and_weaknesses": "The paper is easy to follow and well-presented, with a good set of well-conducted experiments and ablation studies. \n\n1) My biggest concern is the technical depth of this work. It would be helpful if the author would further characterize the augmentations and demonstrate to us how multiple-view + augmentation affects subgraph learning. And what exact benefits would we get? Under what conditions? \n2) Related to 1), it would be helpful if we can see how this method work on a larger set of various tasks. \n3) As for the performance, a question is how significantly augmentation helps. It looks like in most cases, the margin between the original and the augmented multiple views is small and likely not statistically significant. It seems to help more in some of the GCN cases. \n4) For computational time comparison in table 6, please add the original, i.e., w/o augmentation \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DKB-Ty1zW1r", "reviewer": "Reviewer_mpjc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a multi-view strategy for subgraph representation learning. The multi-view strategy aims to achieve better performance by combining with arbitrary graph data augmentation methods and GNN models.  Experimental results are provided to demonstrate the effectiveness of the proposed strategy.", "summary_of_the_review": "Major concerns are: (1) the technical contribution of the proposed strategy is limited; (2) Baseline models are weak. More comparisons are needed. (3) Some implementation details are missing.", "strength_and_weaknesses": "Strengths:\n1. The paper is clearly written, and the proposed framework is well articulated. \n2. The idea of this paper is straightforward and easy to understand.\n3. Experimental evaluation shows that the multi-view strategy works well when collaborating with two graph augmentation methods on 3 GNN models.\n\nWeaknesses:\n1. The technical contribution of the paper is limited.\nThe main idea behind the proposed multi-view strategy is to generate augmented samples by adding perturbations to the original (sub)graph, and to maximize the similarity between the representations of these (sub)graphs. However, similar ideas have been around in the literature of contrastive learning. Although most graph contrastive learning methods are designed for node- or graph-level tasks, they can be easily applied to subgraphs [1] [2].\n\n2. The application potential of the proposed method is limited. \nThe paper only adopts the proposed multi-view strategy to subgraph tasks, which could limit its application potential. A natural question is: can this multi-view strategy also improve node- and/or graph-level tasks (e.g., when combined with graph pooling methods [3] [4]).\n\n3. Some implementation details are missing, which could harm the reproducibility.\n   (1) For the pooling function Pool() in Eq. (4), which one was chosen in the experiments? Is there any option other than MaxPool and AvgPool? \n   (2) What is the design of the contrastive loss in Eq. (6)?\n\n4. Experimental study should be enhanced.\n   (1) The authors only conduct experiments with basic GNNs, such as GLASS, GCN and GraphSAGE. More SOTA GNN methods should be added to the comparison as well. \n   (2) No comparison with Graph Contrastive Learning methods was performed.\n   (3) The choice of the hyperparameter λ should be further explored.\n\n5. Some typos.\nIn the caption of Figure 1, blue -> green, G\" -> G’’\n\n\n[1] Suresh S, Li P, Hao C, et al. Adversarial graph augmentation to improve graph contrastive learning. Advances in Neural Information Processing Systems, 2021, 34: 15920-15933.\n[2] Yin Y, Wang Q, Huang S, et al. Autogcl: Automated graph contrastive learning via learnable view generators. Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(8): 8892-8900.\n[3] Lee J, Lee I, Kang J. Self-attention graph pooling. International conference on machine learning. PMLR, 2019: 3734-3743.\n[4] Zhang Z, Bu J, Ester M, et al. Hierarchical graph pooling with structure learning. arXiv preprint arXiv:1911.05954, 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "11CzUN2M6W", "reviewer": "Reviewer_qUyv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes multi-view augmentation for learning representations on subgraphs. The proposed method improves the efficiency of existing standard graph augmentation method.", "summary_of_the_review": "Overall, I am leaning towards rejection. My major concern is the novelty and performance of proposed methods. If you address my concerns, I will raise my score. ", "strength_and_weaknesses": "**Strengths**\n\n(1) This paper greatly improves the efficiency of graph augmentation for subgraphs.\n\n**Weakness**\n\n(1) I think that the novelty of this paper is limited. To augment subgraphs, this paper uses existing graph augmentation methods such as DropEdge and GauG. Also, the contrastive loss of proposed method is from MV-GNN. Even though embedding augmented subgraphs into original graph is somewhat novel, I think it would be better to propose subgraph specific graph augmentation methods.\n\n(2) The proposed methods have marginal performance gain compared to baselines. For example On PPI-BP, GLASS w/ DropEDGE MV has 0.002 performance gain compared to GLASS w/ DropEDGE.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject