A Fair Comparison of Graph Neural Networks for Graph Classification
Authors: Federico Errica
Marco Podda
Davide Bacciu
Alessio Micheli
Venue:  (2020)
Keywords: graph neural networks
graph classification
reproducibility
graph representation learning

Abstract: Experimental reproducibility and replicability are critical topics in machine learning. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of a wide research community, which resulted in a large stream of works.
As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimental procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in a controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with structure-agnostic baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.

Published as a conference paper at ICLR 2020
A FAIR COMPARISON OF GRAPH NEURAL NETWORKS FOR GRAPH CLASSIFICATION
Federico Errica"
Marco Podda"
Department of Computer Science
Department of Computer Science
University of Pisa
University of Pisa
federico. Federico.eerica@phhd unipi
ma marco. porcao.ooddedi... unipi. it
Davide Bacciu"
Alessio Micheli
Department of Computer Science University of Pisa ba baccivêdi. .undiinnnppp.. .unipi...
Department of Computer Science University of Pisa mi chel @di. redi.unigg unipi. it
ABSTRACT
Experimental reproducibility and replicability are critical topics in machine learn- ing. Authors have often raised concerns about their lack in scientific publications to improve the quality of the field. Recently, the graph representation learning field has attracted the attention of wide research community, which resulted large stream of works. As such, several Graph Neural Network models have been developed to effectively tackle graph classification. However, experimen- tal procedures often lack rigorousness and are hardly reproducible. Motivated by this, we provide an overview of common practices that should be avoided to fairly compare with the state of the art. To counter this troubling trend, we ran more than 47000 experiments in controlled and uniform framework to re-evaluate five popular models across nine common benchmarks. Moreover, by comparing GNNs with ttructure-agnosii baselines we provide convincing evidence that, on some datasets, structural information has not been exploited yet. We believe that this work can contribute to the development of the graph learning field, by providing a much needed grounding for rigorous evaluations of graph classification models.
INTRODUCTION
Over the years, researchers have raised concerns about several flaws in scholarship, such as exper- imental reproducibility and replicability in machine learning (McDermoti, 1976 Lipton Stein- hardi, (0188) and science in general (National Academies of Sciences & Medicine 2019). These issues are not easy address, as a collective effort is required avoid bad practices. Examples include the ambiguity of experimental procedures, the impossibility of reproducing results and the improper comparison of machine learning models. As a result, it can difficult to uniformly assess the effectiveness of one method against another. This work investigates these issues for the graph representation learning field, by providing uniform and rigorous benchmarking of state-of-the-art models.
Graph Neural Networks (GNNs) Micheli 2009; Scarselli et al. 2008) have recently become the standard tool for machine learning graphs. These architectures effectively combine node features and graph topology to build distributed node representations. GNNs can be used to solve node classification (Kipf & Welling 2017) and link prediction (Zhang & Chen 2018) tasks, they can be applied to downstream graph classification (Bacciu et al. (018). In literature, such models are usually evaluated on chemical and social domains (Xu al. 2019).
Given their appeal, an ever increasing number of GNNs is being developed (Gilmer et al. 2017). However, despite the theoretical advancements reached by the latest contributions in the field, find that the experimental settings are in many cases ambiguous or not reproducible.
Equull contribution.

---

Published as a conference paper at ICLR 2020
Some of the most common reproducibility problems encounter in this field concern hyper- parameters selection and the correct usage of data splits for model selection versus model assess- ment. Moreover, the evaluation code is sometimes missing or incomplete, and experiments are not standardized across different works in terms of node and edge features.
These issues easily generate doubts and confusion among practitioners that need fully transparent and reproducible experimental setting. As a matter of fact, the evaluation of a model goes through two different phases, namely model selection on the validation set and model assessment on the test set. Clearly, to fail in keeping these phases well separated could lead to over-optimistic and biased estimates of the true performance of a model, making it hard for other researchers to present competitive results without following the same ambiguous evaluation procedures.
With this premise, our primary contribution is to provide the graph learning community with fair performance comparison among GNN architectures, using a standardized and reproducible experi- mental environment. More in detail, we performed a large number of experiments within a rigorous model selection and assessment framework, in which all models were compared using the same features and the same data splits.
Secondly, we investigate if and to what extent current GNN models can effectively exploit graph structure. To this end, we add two domain-specific and atructuee-agnostic baselines, whose purpose is to disentangle the contribution of structural information from node features. Much to our sur- prise, we found out that these baselines can even perform better than GNNs on some datasets; this calls for moderation when reporting improvements that do not clearly outperform structure-agnostic competitors.
Our last contribution is a study on the effect of node degrees features in social datasets. Indeed, we show that providing the degree can be beneficial in terms of performances, and it has also impli- cations in the number of GNN layers needed to reach good results.
We publicly release code and dataset splits to reproduce our results. in order to allow other re- searchers carry out rigorous evaluations with minimum additional effor
Disclaimer Before delving into the work, we would like to clarify that this work does not aim at pinpointing the best (or worst) performing GNN, nor it disavows the effort researchers have put in the development of these models. Rather, it is intended to be an attempt to set up a standardized and uniform evaluation framework for GNNs, such that future contributions can be compared fairly and objectively with existing architectures.
2 RELATED WORK
Graph Neural Networks At the core of GNNs is the idea to compute state for each node in graph, which is iteratively updated according to the state of neighboring nodes. Thanks to layering (Micheli, 2009) or recursive (Scarselli et al. 2008) schemes, these models propagate information and construct node representations that can be "aware" of the broader graph structure. GNNs have recently gained popularity because they can efficiently and automatically extract relevant features from a graph; in the past, the most popular way to deal with complex structures was to use kernel functions (Shervashidze al. 2011) to compute task-agnostic features. However, such kernels are non-adaptive and typically computationally expensive, which makes GNNs even more appealing.
Even though in this work we specifically focus on architectures designed for graph classification, all GNNs share the notion of "convolution" over node neighborhoods, as a generalization of con- volution on grids. For example, GraphSAGE (Hamilton et al.] 2017) first performs sum, mean or max-pooling neighborhood aggregation, and then it updates the node representation applying lin- car projection on top of the convolution. It also relies on neighborhood sampling scheme to keep computational complexity constant. Instead, Graph Isomorphism Network (GIN) (Xu et al. 2019) builds upon the limitations of GraphSAGE, extending with arbitrary aggregation functions on multi-sets. The model is proven to be as theoretically powerful as the Weisfeiler-Lehman graph isomorphism. Very recently, Wagstaff et al.] (2019) gave an upper bound to the number of hidden units needed to learn permutation-invarian functions over sets and multi-sets. Differently from the above methods, Edge- CondiiiCondi Convolution (ECC) (Simonovsky & Komodakis 2017)
test of
Code available at: https:://githu..ccmmm// http::/ghhhhh..ccc.///////hhh//h
2

---

Published as a conference paper at ICLR 2020
learns different parameter for each edge label. Therefore, neighbor aggregation is weighted ac- cording to specific edge parameters. Finally, Deep Graph Convolutional Neural Network (DGCNN)
(Zhang et al. 2018) proposes a convolutional layer similar to the formulation of Kipf & Welling (017).
Some models also exploit pooling scheme, which is applied after convolutional layers in order to reduce the size of a graph. For example, the pooling scheme of ECC coarsens graphs through a differentiable pooling map that can be pre-computed. Similarly, DifiPool (Ying al. (018 pro- poses an adaptive pooling mechanism that collapses nodes on the basis of a supervised criterion. In practice, DiffPool combines differentiable graph encoder with its pooling strategy, so that the ar- chitecture is end-to-end trainable. Lastly, DGCNN differs from other works in that nodes are sorted and aligned by a specific algorithm called SortPool (Zhang et al. 2018).
Model evaluation The work of [Shchur et al. 2018 shares similar purpose with our contribu- tion. In particular, the authors compare different GNNs on node classification tasks, showing that results are highly dependent where changing splits leads to dramatically different performance rankings. Thus, they recommend to evaluate GNNs on multiple test splits to achieve fair comparison. Even though operate in different setting (graph instead of node classification), we follow the authors' suggestions by evalu- ating models under controlled and rigorous assessment framework. Finally, the work of Dacrema et al. (019) criticizes a large number of neural recommender systems, most of which are not repro- ducible, showing that only of them truly improves against simple baseline.
the particular cainn/alidatiion/esst split of choice, up to the point
3 RISK ASSESSMENT AND MODEL SELECTION
Here, we recap the risk assessment (also called model evaluation or model assessment) and model selection procedures, clearly layout the experimental procedure followed in this paper. For space reasons, the overall procedure is visually summarized in Appendix A.I
3.1 RISK ASSESSMENT
The goal of risk assessment is to provide an estimate of the performance of a class of models. When a test set is not explicitly given. a common way to proceed is to use k-fold Cross Validation (CV) (Stone) 1974 Varma & Simon 2006, Cawley & Talbot 2010). k-fold CV uses different training/test splits to estimate the generalization performance of model; for each partition, an internal model selection procedure selects the way, test data is never used for model selection. As model selection is performed independently for each training/test split, we obtain different "best" syper-parameter configurations; this is why we refer to the performance of class of models.
using the training data only. This
3.2 MODEL SELECTION
The goal of model selection, or parameter configurations the one that works best a specific validation set. If validation set is not given, one can rely on a holdout training/validation split or an inner k-fold, Nevertheless, the key point to remember is that validation performances are biased estimates of the generaliza- tion capabilities. Consequently, model selection results are generally over-optimistic; this issue is thoroughly documented in Cawley & Talbot (2010). This is why the main contribution of this work is to clearly separate model selection and model assessment estimates, something that is lacking or ambiguous in the literature under consideration.
tuning, is to choose among a set of candidate hyper-
OVERVIEW OF REPRODUCIBILITY ISSUES
To motivate our contribution, we follow the approach of Dacrema et al. (2019) and briefly review recent papers describing five different GNN models, highlighting problems in the experimental se- tups as well as reproducibility of results. We emphasize that our observations are based solely on the contents of their paper and the available code? Suitable GNN works were selected according to
As of the date of this submission.
3

---

Published as a conference paper at ICLR 2020
the following criteria: i) performances obtained with 10-fold CV; ii) peer reviewed; iii) strong archi- tectural differences; iv) popularity. In particular, we selected DGCNN (Zhag et al. 2018). DiffPool Ying et al. 20188) ECC (Simonovsky & Komodakis 2017), GIN (Xu al. 2019) and GraphSAGE Hamilton et al. 2017). For a detailed description each model we refer to their respective papers. Our criteria to assess quality of evaluation and reproducibility are: code for data preprocessing. model selection and assessment is provided; ii) data splits are provided; iii) data is split by means of stratification technique, to preserve class proportions across all partitions: iv) results of the 10-fold CV are reported correctly using standard deviations, and they refer to model evaluation (test sets) rather than model selection (validation sets). Table [ summarizes our findings.
Table Criteria for reproducibility considered this work and their compliance among considered models. (Y) indicates that the criterion is met, (N) indicates that the criterion is not satisfied, (A) indicates ambiguity (i.e. it is unclear whether the criteria met not), (-) indicates lack of infor- mation (i.e. no details are provided about the criteria). Note that GraphSAGE is excluded from this comparison, was not directly applied by authors to graph classification tasks.
DGCNN
DiffPool
ECC
GIN
Data preprocessing code
Y
Y
Y
Model selection code
N
N
N
Model evaluation code
Y
Y
Data splits provided
Y
N
N
Y
Label Stratification
N
Report accuracy on test
Y
A
A
N
Report standard deviations
N
N
The authors evaluate the model on 10-fold CV. While the architecture is fixed for all dataset, learning rate and epochs are tuned using only one random CV fold, and then reused on all the other folds. While this practice is still acceptable, it may lead to sub-optimal performances. Nonetheless, the code to reproduce model selection is not available. Moreover, the authors run CV 10 times, and they report the average of the 10 final scores. As result, the variance of the provided estimates is reduced. However, the same procedure was not applied to the other competitors as well. Finally, CV data splits are correctly stratified and publicly available, making it possible reproduce at least the evaluation experiments.
DGCNN
DiffPool From both the paper and the provided code, it is unclear if reported results obtained on a test set rather than validation set. Although the authors state that 10-fold CV is used, standard deviations of DiffPool and its competitors are not reported. Moreover, the authors affirm to have applied early stopping on the validation set to prevent overfitting; unfortunately. neither model se- lection code nor validation splits are available. Furthermore, according to the code, data is randomly split (without stratification) and no random seed is set, hence splits are different each time the code executed.
ECC The paper reports that ECC is evaluated on 10-fold CV, but results do not include standard deviations. Similarly to DGCNN, pareme-ars are fixed in advance, hence it is not clear if and how model selection has been performed. Importantly, there are references in the code repository to data pre-processing, data stratification, data splitting, and model selection.
The authors correctly list all the hyper- parameerrssssss tuned. However, as stated explicitly in the paper and in the public review discussion, they report the validation accuracy of 10-fold CV. In other words, reported results refer to model selection and not model evaluation. The code for model selection is not provided.
GIN
The original paper does not test this model on graph classification datasets, but GraphSAGE is often used in other papers as strong baseline. It follows that GraphSAGE results on graph classification should be accompanied by the code to reproduce the experiments. Despite that, the two works which report results of GraphSAGE (DiffPool and GIN) fail to do so.
GraphSAGE
4

---

Published as a conference paper at ICLR 2020
Summary Our analysis reveals that GNN works rarely comply with good machine learning prac- tices as regards the quality of evaluation and reproducibility of results. This motivates the need to re-evaluate all models within rigorous, reproducible and fair environment.
5 EXPERIMENTS
In this section we detail our main experiment, in which we re-evaluate the menteoe models on 9 datasets (4 chemical, 5 social), using a model selection and assessment framework that closely follows the rigorous practices described in Section 3] In addition, implement two baselines whose purpose is to understand the extent to which GNNs are able to exploit structural information. All models have been implemented by means of the Pytorch Geometrics library (Fey & Lenssen 2019), which provides graph pre-processing routines and makes the definition of graph convolution easter to implement. We sometimes found discrepancies between papers and related code; in such cases, we complied with the specifications in the paper. Because GraphSAGE was not applied to graph classification in the original work, we opted for a max-pooling global aggregation function to classify graph instances; further, we do not use the sampled neighborhood aggregation scheme defined in Hamilton et al. (2017). order to allow nodes to have access to their whole neighborhood.
Datasets All graph datasets are publicly available (Kersting et al. 2016) and represent rele- vant subset of those most frequently used in literature to compare GNNs. Some collect molecular graphs, while others contain social graphs. In particular, we used D&D (Dobson & Doig 2003) PROTEINS (Borgwardt et al., 2005), NCI1 (Wale et al. 2008) and ENZYMES (Schomburg et al. 2004) for binary and multi-clast classification of chemical compounds, whereas IMDB-BINAL IMDB-
REDDIT-5K and COLLAB (Yanardag & Vishwanathan 2015)
are social datasets. Dataset statisties are reported in Table A.2
Features In GNN literature, it is common practice to augment node descriptors with structural features. For example, DifiPool adds the degree and clustering coefficient to each node feature vector, whereas GIN adds one-hot representation of node degrees. The latter choice trades off an improvement in performances (due to injectivity of the first sum) with the inability to generalize to graphs with arbitrary node degree.
In general, good experimental practices suggest that all models should be consistently compared to the same input representations. This is why we re-evaluate all models using the same node features. In particular, we use one common setting for the chemical domain and two alternative settings as regards the social domain. As regards the chemical domain, nodes are labeled with a one-hot encoding of their atom type, though on ENZYMES we follow the literature and use 18 additional features available. As regards social graphs, whose nodes do not have features, we use either an uninformative feature for all nodes or the node degree. As such, we are able to reason about the effectiveness of the structural inductive bias imposed by the model; that is if the model is able to implicitly learn structural features or not. The effect of adding structural features to general machine learning models for graphs has been investigated in Gallagher & Eliassi-Ran (2008); here, we focus on the impact of node degrees performances for social datasets.
Baselines We adopt two distinct baselines, one for chemical and one for social datasets. On all chemical datasets but for ENZYMES, we follow Ralaivola et al. (2005): Luzhnica et al.] (2019) and implement the Molecular Fingerprint technique, which first applies global sum pooling (i.e.. counts the occurrences of atom types in the graph by summing the features of all nodes in the graph together) and then applies a single-layer MLP with ReLU activations. On social domains and ENZYMES (due to the presence of additional features), we take inspiration from the work of cormutation-invariann
Zaheer et al. (2077) to learn
functions over sets of nodes: first, we apply a single-layer MLP on top of node features, followed by global sum pooling and another single- layer MLP for classification. Note that both baselines do not leverage graph topology. Using these baselines as a reference is of fundamental importance for future works, as they can provide feedback on the effectiveness of GNNs a specific dataset. As a matter of fact, if GNN performances are close to the ones of a structure-agnostic baseline, one can draw two possible conclusions: the task does not need topological information to be efffettvely solved, or the GNN is not exploiting graph structure adequately. While the former can be verified through domain specific human expertise, the second is more difficult to assess, as multiple factors come into play such the amount of training
5

---

Published as a conference paper at ICLR 2020
data, the structural inductive bias imposed by the architecture and the haper-parameters used for model selection. Nevertheless, significant improvements with respect to these baselines are a strong indicator that graph topology has been exploited. Therefore, structure-agnostic baselines become vital to understand if and how a model can be improved.
Table 2: Pseudo-code for model assessment (left) and model selection (right). In Algorithm 1, "Select" refers to Algorithm whereas "Train" and "Eval" represent training and inference phases, respectively. After each model selection, the best configuration best,, is used to evaluate the external test fold. Performances are averaged across training runs, where our case is set to 3.
Algorithm Model Assessment (k-fold CV)
1: Input: Dataset D, set of configurations +
Algorithm Model Selection
2: Split D into & folds FR
1: Input: traing, θ
3: for i + I,..., k do
4: traing, test, < (U)#) Fj), F;
2: Split traing into train and valid
3: po =
5: best& f- Select(traing, θ)
4: for each 0 θ do
6: for r f I...., R do
5: model <- Train(trainn 0)
7: model, Train(traing, best,)
6: Pg 4- Po U Eval(model, valid)
8: Pr <- Eval(model,, test;)
7: end for
9: end for
10: perf& +- EF= P./R
8:
bestg t argmaxe Pe
9: return best,
11: end for
12: return Σi=
Experimental Setting Our experimental approach is to use 10-fold CV for model assessment and an inner holdout technique with 90%/10% training/validation split for model selection. After each model selection, we train three times on the whole training fold, holding out random fraction (10%) of the data to perform early stopping. These three separate runs are needed to smooth the effect of unfavorable random weight initialization on test performances. The final test fold score is obtained as the mean these three runs; Table 2 reports the pseudo-code of the entire evaluation process. To be consistent with literature, implement carly stopping with patience parameter n, where training stops if n epochs have passed without improvement on the validation set. A high value of can favor model selection by making it less sensitive to fluctuations in the validation score at the cost of additional computation. Importantly, all data partitions have been pre-computed, so that models are selected and evaluated the same data splits. Moreover, all data splits are stratified, i.e. class proportions are preserved inside each k-fold split as well as in the holdout splits used for model selection.
Hyper-parameters Hyper-pane apper-parameter tuning is performed via grid search. For the sake of con- ciseness, we list all hyper-parameters in Section A.4 Notice that we always include those used by other authors in their respective papers. We select the number of convolutional layers, the embed- ding space dimension, the learning rate, and the criterion for early stopping (either based on the validation accuracy or validation loss) for all models. Depending on the model, we also selected regularization terms, dropout, and other model-specific parameters.
Computational considerations Our experiments involve a large number of training runs. For all models, grid sizes range from 32 72 possible configurations, depending on the number of ryper-parametery to choose from. However, we tried to keep the upper bound on the number of parameters as similar as possible across models. The total effort required, terms of the number of single training runs, to complete model assessment procedures exceeded 47000. Such large number required extensive use of parallelism, both in CPU and GPU, to conduct the experiments in reasonable amount of time. We emphasize that in some cases (e.g. ECC in social datasets), training on a single parameee configuration required more than 72 hours, which would have made the sequential exploration of one single grid last months. Therefore. due to the large amount of experiments to conduct and to the computational resources available, we limited the time to complete single training to 72 hours.

---

Published as a conference paper at ICLR 2020
Table 3: Results chemical datasets with mean accuracy and standard deviation reported. Best performances are highlighted in bold.
D&D
NCTI
PROTEINS
ENZYMES
Baseline
78.4 4.5
69.8 + 2.2
75.8 3.7
65.2 6.4
DGCNN
76.6 4.3
1..41.7
T2.9
2.9 3.5
38.9 + 5.7
DiftPool
3.5
76.9
59.5
ECC
7.16
76.2 1.4
72.3 +
29.5
8.2
GIN
2.9
80.0 1.4
T3.3 -
59.6 1.5
GrahSAAEE
2.0
1.8
73.0
58.2 1 6.0
6 RESULTS AND DISCUSSION
Tables and 4 show the results of our experiments. Overall, GIN seems to be effective on social datasets. Importantly, we discover that on D&D, PROTEINS and ENZYMES none of the GNNs are able to improve over the baseline. On the contrary, on NCII the baseline is clearly outperformed: this result suggests that the GNNs we analyzed can actually exploit the topological information of the graphs in this dataset. Moreover, we observe that an everly-parameteriin to overfit the NCII training data completely. To this, consider that baseline with 10000 hidden units and no regularization reaches around 67% training accuracy, while GIN can easily overfit (<s 100%) the training data. This indicates that structural information hugely affects the ability to fit the training set. On social datasets, we observe that adding node degrees as features is beneficial, but such effect is more noticeable for REDDIT-BINARY REDDIT-5K and COLLAB.
baseline is not able
6.1 THE IMPORTANCE OF BASELINES
Our results also show that pructure-agnostie baselines are an essential tool to understand the effec- tiveness of GNNs and extract useful insights. As example, since none of the GNNs surpasses the baseline on D&D, PROTEINS and ENZYMES, we argue that the state-of-the-art GNN models analyzed not able to fully exploit the structure on such datasets yet: indeed, in chemistry, structural features are known to correlate with molecular properties (van Rossum] 1963). For all these reasons, suggest putting small performance gains these datasets into the right perspec- tive, at least until the baseline will clearly be outperformed. Currently, small average fluctuations on these datasets are likely to be caused by other factors, such random initializations, rather than a successful exploitation of the structure. In conclusion, we warmly recommend GNN practitioners include baseline comparisons future works, in order to better characterize the extent of their contributions.
6.2 THE EFFECT OF NODE DEGREE
Based on results, using node degrees input features is almost always beneficial increase performances on social datasets, sometimes by large amount. As an example, degree information sufficient for our baseline improve performances of 15%, hence being competitive many datasets; in particular, the baseline achieves the best performance on IMDB-BINARY. In contrast, adding node degrees is less relevant for most GNNs, since they can automatically infer such infor- mation from the structure. One notable exception DGCNN, which explicitly needs node degrees perform well on all datasets. Moreover, we observe that the ranking of all models, after the addi- tion of the degrees, drastically changes; this raises the question about the impact of other structural features (such clustering coefficient) on performances, which we leave future works.
However, one may also wonder whether the addition of the degree has an influence on the number of layers that are necessary to solve the task or not. We therefore investigated the matter by computing the median number of layers across the 10 different folds. We observed general trend across mod- els, with GraphSAGE being the only exception, where the addition of the degree reduces the number of layers needed by 1 as shown in Table A.3 This may be due to the fact that most architectures find useful to compute the degree at the very first layer, as such information seems useful to the overall performances.

---

Published as a conference paper at ICLR 2020
Table 4: Results social datasets with mean accuracy and standard deviation reported. Best performances are highlighted in bold. OOR means Out of Resources, either time (> 72 hours for single training) or GPU memory.
IMDB-B
IMDB-M
REDDIT-B
REDDIT-SK
COLLAB
Baseline
50.7 2.4
36.1 3.0
72.1 1 7.8
35.1 15.11D
55.0
DGCNN
53.3 = 5.0
38.6 2.2
77.1 2.9
35.7
57.1
DillPool
68.3 1: 6.1
45.1 3.2
76.6 2.4
34.6 + 2.0
67.7 1.9
ECC
67.8 1: 4.8
44.8 3.1
OOR
0OR
OOR
No
GIN
6..8
42.2
87.0 1000444
53.8 5.9
75.9 1.9
GraphSAGE
1.22::3..
2.0
Baseline
70.8 5.0
49.1
82.2 3.0
52.2 1.5
70.2 1.5
DGCNN
69.2 3.0
45.6 3.4
87.8 2.5
49.2
71.2 1.9
DifiPool
45.6
89.1
1.4 OOR
68.9
ECC
07.7
3.1
OOR
OOR
GIN
71.2 1: 3.9
48.5 3.3
89.9 1
56.1 L 1.7
75.6 2.3
GrihSAGGE
68.8 + 4.5
47.6 4 3.5
84.3 + 1.9
50.0 + 1.3
73.9 : 1.7
96.0
85.0
85.0
82.5
80.0
80.0
27.5
72.5
72.5
70.0
70.0
ENZYMES
IMDB-BINARY
IMDB-MULTI
80.0
57.5
77.5
60.0
52.5
50.0
47.5
70.0
67.5
28.0
40.0
REDDIT-BINARY
COLLAB
95.0
60.0
60.0
92.5
Accurcyy
90.0
56.0
76.0
67.5
54.0
85.0
72.0
82.5
80.0
68.0
46.0
GraphSAGE
Figure 1: Chemical and social (with degree) benchmark results are shown together with published results (when available). For each of them, we report validation and test accuracies the evaluated models, together with published results if available.
6.3 COMPARISON WITH PUBLISHED RESULTS
Figure compares the average values of our test results with those reported in literature. In addition, we plot the average of our validation results across the 10 different model selections. The plots show how our test accuracies in most cases different from what reported in the literature, and the gap
8

---

Published as a conference paper at ICLR 2020
between the two estimates is usually consistent. In contrast, average validation accuracies always higher or equal to our test results; this is expected, as discussed Section 3.2 Finally, emphasize once again that our results are i) obtained within the framework of rigorous model selection and assessment protocol; ii fair with respect of data splits and input features assigned to all competitors; üii reproducible. In contrast, saw in Section 4 how published results rely unclear or poorly documented experimental settings.
7 CONCLUSIONS
In this paper, we wanted to show how a rigorous empirical evaluation GNNs can help design future experiments and better reason about the effectiveness of different architectural choices. To this aim, we highlighted ambiguities in the experimental settings of different papers, and we proposed a clear and reproducible procedure for future comparisons. We then provided complete re-evaluation of five GNNs nine datasets, which required significant amount of time and computational resources. This uniform environment helped us reason about the role of structure, as we found that structure-agnostic baselines outperform GNNs some chemical datasets, thus suggesting that structural properties have not been exploited yet. Morcover, we objectively analyzed the effect of the degree feature performances and model selection in social datasets, unveiling effect on the depth of GNNs. Finally, we provide the graph learning community with reliable and reproducible results to which GNN practitioners can compare their architectures. We hope that this work, along with the library we release, will prove useful to researchers and practitioners that want to compare GNNs more rigorous way.
ACKNOWLEDGMENTS
D. Bacciu would like acknowledge support from the Italian Ministry of Education, University, and Research (MIUR) under project SIR 2014 LIST-IT (grant n. RBSI14STDE).
REFERENCES
Davide Bacciu, Federico Errica, and Alessio Micheli. Contextual graph Markov model: A deep and generative approach to graph processing. In Proceedings of the International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pp-
294-303. PMLR, 2018.
Karsten M Borgwardt, Cheng Soon Ong, Stefan Schönauer, SVN Vishwanathan, Alex Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21(suppl_1 1): i47-i56, 2005.
Gavin C. Cawley and Nicola L. C. Talbot. On over-fitting in model selection and subsequent se- lection bias in performance evaluation. Journal of Machine Learning Research, 11:2079-2107, 2010.
Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are really making much progress? worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference Recommender Systems (RecSys), pp. 101-109, 2019. 2019.
Paul Dobson and Andrew Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330(4))771-783, 2003.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. In Representation Learning on Graphs and Manifolds Workshop, International Conference on Learning Representations (ICLR), 2019.
Brian Gallagher and Tina Eliassi-Rad Leveraging label-independent features for classification in sparsely labeled networks: An empirical study. In International Workshop Social Network Mining and Analysis, pp. 1-19. Springer, 2008.
Justin Gilmer, Samuel S. Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neu- ral message passing for quantum chemistry. In Proceedings of the International Conference on

---

Published as a conference paper at ICLR 2020
Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pp. 1263 1272. PMLR, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems (NIPS), pp. 1024-1034. Curran Associates, Inc., 2017.
Kristian Kersting, Nils M. Kriege, Christopher Moriis, Petra Mutzel, and Marion Neumann. Bench- mark data sets for graph kernels, 2016. URL htt.hh:::/ggggphkeeeeeeee cs. tu tu-dortmund.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Zachary C Lipton and Jacob Steinhardt. Troubling trends in machine learning scholarship. arXiv preprint arXiv: 187.03110 2018
Enxhell Luzhnica, Ben Day, and Pietro Liò. On graph classification networks, datasets and baselines. Workshop on Learning and Reasoning with Graph-Structun Data, International Conference on Machine Learning (ICML), 2019.
Drew McDermott. Artificial intelligence meets natural stupidity. ACM SIGART Bulletin, 57:4-,, 1976.
Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions on Neural Networks, 20(3):498-51) -11, 2009.
Engineering National Academies of Sciences and Medicine. Reproducibility and replicability in science. National Academies Press, 2019.
Liva Ralaivola, Sanjay Swamidass, Hiroto Saigo, and Pierre Baldi. Graph kernels for chemical informatics. Neural networks, 180 1893)11093-1100, 2005.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions Neural Networks, 20(1):61-80, 2008
Ida Schomburg, Antje Chang, Christian Ebeling, Marion Gremse, Christian Heldt, Gregor Huhn, and Dietmar Schomburg. Brenda, the enzyme database: updates and major new developments. Nucleic acids research, 1):4431
2004.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnnemann. Pitfalls of graph neural network evaluation. In Relational Representation Learning Workshop, Advances in Neural Information Processing Systems (NeurIPS), 2018.
Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mchlhorn, and Karsten M Borg- wardt. Weisfeiler-lemman graph kernels. Journal of Machine Learning Research, 12(Sep):2539- 2561, 20
Martin Simonovsky and Nikos Komodakis. Dynamic conditiond filters in convolutional neu- ral networks on graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29-38, 2017.
Mervyn Stone. Cross-validates choice and assessment of statistical predictions. Journal of the royal statistical society. Series B (Methodological), pp. 111-147, 1974.
Jacques van Rossum. The Relation Between Chemical Structure and Biological Activity. Journal of Phurmacy and Pharmacology, 15(1):285-316 1963.
Sudhir Varma and Richard Simon. Bias in error estimation when using cross- validtttn for model selection. BMC bioinformatics, 7:91, 02 2006.
10

---

Published as a conference paper at ICLR 2020
Edward Wagstaff, Fabian Fuchs, Martin Engelcke, Ingmar Posner, and Michael A. Osbome. On the limitations of representing functions on sets. In Proceedings of the International Conference Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp- 6487-6494, PMLR, 2019.
Nikil Wale, Ian A Watson, and George Karypis. Comparison of descriptor spaces for chemical com- pound retrieval and classification. Knowledge and Information Systems, 14(3):347-375, 2008.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations (ICLR), 2019.
Pinar Yanardag and S. N. Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), pp. 1365-1374, 2015.
Zhitao Ying. Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hi- erarchical graph representation learning with differentiable pooling. In Advances in Neural Infor- mation Processing Systems (NeurIPS), pp.
Curran Associates, Inc., 2018
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poezos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in Neural Information Processing Systems (NIPS), pp. 3391-3401. Curran Associates, Inc., 2017.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in Neural Information Processing Systems (NeurlPS), pp- 5165-5175. Curran Associates, Inc., 2018.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI Conference on Artificial Intelli- gence (AAAI), pp. 4438 4445, 2018.

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1 VISUALIZATION OF THE EVALUATION FRAMEWORK
Dataset
split
fold
foldz
folds
fold
Test
Train
repeat
Train
Test
Train
Model Assessment:
Kout
Average Results
times
Train
Test
Model Selection: find the
best hyperparametere
Train Dataser
Hold-oou (this work)
OR
Inner k-fold CV
Trainan
Valid.
fold
fold2
folds
solde
Select best hyperparameters according to Validation performance
Valid.
Train
repeat
Train
Valid.
Train
Kmm
times
Train
Valid.
Select the best hyperparameters according average Validation performance
Figure 2: We give a visual representation of the evaluation framework. We apply an external kout- fold CV to get an estimate of the generalization performance of a model, and we use an hold-out technique (bottom-left) to select the best hyper-parametres For completeness, we show that it also possible to apply inner kin fold CV (implementing complete Nested Cross Validation). which obviously amounts to multiplying the computational costs of model selection by a factor kinn-
12

---

Published as a conference paper at ICLR 2020
A.2 DATASET STATISTICS
Table Dataset Statistics. Note that, when node labels are not present, we either assigned the same feature of the degree to all nodes in the dataset. Moreover, following the literature, the 18 additional node attributes for ENZYMES.
Graphs
# Classes
# Nodes
Edges
# Node labels
DD
1178
2
284.32
715.66
89
CHEM.
ENZYMES
600
32.63
64.14
NCII
4110
2
29.87
32.30
37
PROTEINS
1113
39.06
72.82
COLLAB
5000
3
74.49
2457.78
SOCIAL
IMDB-BINARY
1000
2
19.77
96.53
IMDE-MULTI
1500
3
13.00
65.94
REDDIT-BINARY
2000
2
429.63
497.75
REDDIT-5K
4999
5
508.82
594.87
A.3 EFFECT OF NODE DEGREE ON LAYERING
Table 6: The table displays the median number of selcted layers in relation to the addition of node degrees as input features on all social datasets. indicates that uninformative feature is used as node label.
IMDB-B
IMDB-M
REDDIT-B
REDDIT-M
COLLAB
1
DEG
DEG
1
DEG
1
DEG
DEG
DGCNN
3
3
3.5
3
4
3
3
2
4
2
DiffPool
1
2
2
2
1.5
ECC
2
GIN
3
2
2
4
4
3
4
GraphSAGE
4
3
4
3
3
5
3
5
13

---

Published as a conference paper at ICLR 2020
Neighbors Aggregation
Sum
Sum
sum
mccn
mccan
sum
mcan
max
sum
Embed. dim
64
128
Dense dim
128
gamma: 0.5)
Scheduler
ECC-LR
Step-LR
(step: 50,
Optimizer
Adam
Adam
Adam
Adam
Adam
Adam
SGD
Adam
Adam
Patience
500, loss 500, ace
500, loss 500, ace
500, loss
500, acc
1000. loss 1000, ace
500, loss 500, acc
500. loss 500, acc
500, loss 500, ace
500, loss 500, ace
500, loss 500, ace
Table 7: Hyper parameters used for model selection.
Dropout
0.5
0.05
0.25
0.5
L2
1e-2
1e-3
1e-4
le-2
1e-3
1e-4
le-2
le-3
le-4
1c-2
1e-3
1c-4
Epochs
5000
3000
3000
5000
1000
3000
1000
1000
1000
32 layers)
64 (5 layers)
64 (2 layers)
32 layers)
Hidden units
32
128
256
32
128
256
32
128
64
128
256
32
64
32
64
32
64
32
64
Learning rate
le-1
1e-3
le-6
le-1
le-3
le-6
lc-1
1c-3
lc-1
lc-3
Tc-6
Ic-4
1c-5
le-3
le-5
Tc-1
lc-2
le-2
1c-2
Ic-3
lc-4
Batch size
50 (cpu)
16 (gpu)
20 (cpu)
(guu)
32 (cpu)
8 (gpu)
32 (cpu) 16 (cuda)
32
128
32
128
32
128
32
128
A.4 HYPER PAR AMETERS TABLE
Convs per layer
Layers
hidden units
2
Baseline IMDB
Base, COLLAB and REDDIT
Baseline ENZYMES
GraphSAGE
Baseline chemical
DGONN
DifiPool
ECC
GIN
14


Reviews:
[{"review_id": "HJlHur8QYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n********** Post-Rebuttal Update **********\n\nI appreciate that authors provided comments on all the raised concerns and updated the paper accordingly. I think the revised paper is of a higher quality. \n\nAlthough the paper's experimental setup could be done in a more proper way -- nested CV; lack of which degrades the conclusiveness of the comparisons, I still think the paper is better to be accepted than not, given that 1) as pointed out by the authors the computational cost of the current experiments was quite high already, 2) the issues of replicability and reproducibility are important and GNNs are quite popular for various applications and 3) the results are interesting and important to be considered for future GNN works. So, I am happy to change my rating to weak accept. However, for the record, I would like to mention the following point:\n\nThe authors removed the word nested from the paper which is good since I still believe it does not correctly reflect what’s been done in the work. However, in the reply, they imply that several works including Varma & Simon, 2006 [1] would consider their method a nested CV. They attribute this different viewpoint to the reviewer’s misunderstanding of CV for *k-fold* CV. That is not true. CV requires “cross” testing which is absent from the inner loop of this work. I refer the interested readers to actually the mentioned paper by the authors: Varma & Simon, 2006 [1], sectio:background. Page 2 clearly defines cross validation requiring a for-loop for several evaluation across different training/test sets and page 3 defines nested CV as requiring an inner loop and  outer loop of this kind. So, as far as I understand, based on [1], the proposed method is not nested CV since the inner (model selection) part uses the classic single train/test split without any loop.\n\n\n********** Summary **********\n \nThe paper conducts an empirical study of 5 recently-proposed graph neural networks (GNN). Three questions are studied:\n1) For some selected set of classification tasks (chemistry and social networks) and datasets (9 different datasets), how do the considered GNN models perform, relative to each other, given the same hyperparameter search strategy and a “nested” cross validation scenario.\n2) How much does the structure (graph edges) bring on top of a multi-layer perceptron (MLP) operating on merely the node features?\n3) Do the considered GNN models exploit structural information of the input graphs beyond that of the node degrees?\n \n\n********** Strengths and Weaknesses **********\n \n+ it is shown that the inclusion of nodes’ degree in its feature representation brings a very large performance boost. This is very interesting.\n+ it is shown that on some datasets, the results of a simple baseline -- only operating on the node features, can achieve similar results to the elaborate GNNs. This result is very informative and suggests that this baseline should always be included in GNN works. \n+ some of the results in table 4 contradicts the corresponding papers which can be informative for the practitioners of the field.\n+ the reproducibility and replicability problems, that is the motivation of this work, are important concerns of the field.\n \n- reproducibility and replicability problems is not specific to research done on graph neural network and is a caveat for the general machine learning research as discussed by Lipton and Steinhardt 2018., and in fact for the current scientific practice at large as pointed in the recent NSA report [Reproducibility and Replicability in Science, 2019]. So, it may be important to raise and investigate this in different subfields of machine learning, such as GNN, individually. However, the current abstract and introduction of the paper (before the last sentence of intro’s second paragraph), associates this problem specifically to the research conducted in GNN. I strongly believe the better formulation is to refer to Lipton and Steinhardt 2018 as the troubling trend in ML as well as the NSA report in general and then mention that in the current work the authors focus on the subfield of GNN *classification*.\n- page 1: “For instance, it is often unclear how hyper-parameters have been selected or which validation splits have been used.”. These being *often* the case in GNN research is not backed up by statistics and is not shown to be specific to GNN. Even if it’s assumed the 5 GNN methods , under this paper’s scrutiny, are representative of GNN classification, GNNs are used well beyond graph classification.\n- it should be clearly discussed what are the additional information that this work brings on top of Shchur et al. 2018. Is it only the shift of focus from node classification to graph classification?\n- the two notions of “fairness in comparison” and “ablation studies” are sometimes conflated. For instance, the argument against “fairness in comparison” of using one-hot encoding in GIN is not a matter of fairness (since it’s a novel proposal for node features of GNNs) but rather a matter of “ablation studies” (to show the effectiveness of individual components of a new GNN method). \n \n- many concerns regarding the nested cross validation:\na) it is important to note that the paper (as described in algorithm 2) does not do “nested cross validation” despite the claim in various parts of the work. Algorithm 2 conducts model selection based on a fixed train-val split. As such, there is no “cross” validation even with the minimum of 2 folds.\nb) the standard deviations reported in table 3 and 4 are unreasonably high. This important observation is not properly discussed. This high std is in contrast to the standard deviation reported in the baseline papers and makes most methods fall into one-std interval of each other. I strongly suspect that this is due to the non-nested cross validation that is done in this work. That is, the hyperparameters are found using a single val set consisting of 9% of all data. A proper nested cross validation would test each set of hyper parameters against k_in validation folds to make a robust inference of best hyperparameters.\nc) there are 3 runs used “to smooth the effect of unfavourable random weight initialization on test performances”, however the 10-fold outer CV should take care of this to some extent. I would argue it would have been much more helpful to use this budget to do a proper CV for model selection using 3 folds.\nd) The datasets are quite small (600-5000 samples). That might make the 10-fold cross validation problematic and overly optimistic in its performance report (and high variance). It might have been better to do say 5-fold cross validation but repeat the whole process twice. Of course, this is an old dilemma in the pratice regarding the bias and variance of the estimated CV error but as far as I am aware there is a general agreement that getting “closer” to leave-one-out setup is not a good idea, Furthermore, in my anecdotal experience 10-fold CV for a dataset with only 600 samples can really be problematic. \n \n- would the ranking of GNN methods change with different types of the node features being used?  Some GNN setups might be better at representation learning and thus should work better on raw features (as opposed to hand engineered ones).\n \n- how is the number of parameters (or capacity) across different GNN models and the baseline? Has this been taken into account for a fair comparison? For instance, could it be that higher/lower number of parameters explain the differences across GNNs and baseline due to over/under fitting?\n \n********** Disclaimer. I did not thoroughly check the paper’s report of the 5 GNN methods (summarized in table 1). \n \n\n********** Final Decision **********\n\nI believe the paper has merits as well as interesting findings. It is also true that this is an important concern in machine learning research. However, there are many issues that degrades the quality of the paper. Of all those critiques, I suggest “Weak Reject” mainly due to the ones raised regarding a proper nested cross validation; which is the main proposal of the paper.\n \n\n\n********** Minor Points **********\n- Page 1: “Our results put on a fair and unique reference scale many published results which, as we document, were obtained under unclear experimental settings.“: please rephrase; at least one preposition is missing.\n- Page 3: Nested Cross Validation: the brief textual explanation makes the simple idea more complicated than it is. Instead, I suggest to use citation as well as a small figure or short algorithm describing it since it’s the focus of the paper. It will also serve pedagogical purposes.\n- Table 1: precisely define the two mark “A” (ambiguous) and “-” (lack of information).\n- page 7: “higher or equal than” → “higher than or equal to”\n- in page 6, it’s mentioned that some experiments took more than 72 hours. This sounds excessive for such small datasets. My guess is that this is due to the fact that large number of epochs are allowed (e.g. 5000 in table A.3) in conjunction with extremely small learning rate (e.g. 1e-6). The question is if those extreme hyperparameter settings are actually important for this study?\n- the first paragraph of 6.1 argues that “GNNs are still unable to exploit the structure on such datasets [D&D, PROTEINS, ENZYMES]”. While this might be true, the conclusion is only based on 5 methods and limited by the design choices such as the architecture. It should be toned down.\n- page 5, “Features” paragraph is not very clearly written. For instance, from the sentence: “More in detail, in the former nodes ...”, it’s hard to understand what “former” refers to.\n- mean and standard deviation should have the same number of decimals (table 3,4)\n- page 4: “Moreover, the authors applied early stopping, which entails the use of a validation set“. Early stopping, in a less common setup, can be used by only looking at the training set and stopping with the same n-patience strategy as used in this work.\n- page 4: “we conform to the available code and do not use sampled neighborhood\nAggregation.“: needs more explanation.\n \n\n********** Points of improvements **********\n\nThe main point of improvement would be a proper nested cross validation setup. \nThe authors can also consider the ReScience journal: https://rescience.github.io/\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByliE0f49r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This type of benchmarking paper is long overdue for graph classification with deep neural networks.  The paper would've been strongly if it had the following:\n\n1. Considered more structural features than simple node degree and clustering coefficient.  Prior work [1] has looked at such features and answered questions like: How do structural features improve classification performance?And, which structural features are the most useful? \n\n2. Investigated which graph neural network performs better for which graph structures (preferential attachment, small world, regular, etc) and for how much homophily.  \n\n3. Investigated the robustness of graph neural networks on classification as the structure of graphs become more random (e.g., by rewiring edges while maintaining degree distribution).\n\n[1] B. Gallagher, T. Eliassi-Rad. Leveraging Label-Independent Features for Classification in Sparsely Labeled Networks: An Empirical Study. Lecture Notes in Computer Science: Advances in Social Network Mining and Analysis, Springer, 2009.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gg3FECKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an empirical comparison between several existing graph classification algorithms, aimed at providing a fair comparison among them, as well as proposing a simple baseline that does not take into account graph structural information.\n\nOverall, I found the experimental section very thorough and sound, with proper ways of performing parameter tuning and reporting test accuracies. On the other hand, I found this paper to miss some deeper insights into why some models perform better than others, and what are the challenges provided by these graph datasets. Also, there are other interesting dimensions of comparison that are not considered (see details below), as well as insights that could benefit future work in the area.\n\nAs a disclaimer, I would like to mention that I am more familiar with graph node classification methods, as opposed to whole-graph classification (which is the focus of this paper), so I cannot assess very well the authors’ choice of which models to compare, and which datasets they were tested on.\n\nPositive aspects of the paper:\na)  Extensive experiments, which try to find the best parameter configuration for each of these models.\nb) I appreciate the addition of the structure-agnostic baselines, although I was missing major details about the particular choice of baselines (see below).\nc) The writing is very clear and easy to follow.\n\nPoints where I found the paper lacking:\n1.  As mentioned above, I believe there are insufficient details about the baselines. For instance what is the global sum pooling over? If it is over neighbors, then this is not exactly “structure-agnostic”. If it is over features, then why was this architecture chosen as opposed to just a multilayer perceptron, without the Deep Sets component? I see the value in an order-invariant component when aggregating features over a node and its neighbors, but why is this necessary at node level? \n\n2.  What is the range of values considered for each parameter in the hyperparameter validation phase? For instance, I found it a bit surprising in table A.3. that all models need 500 patience. \n\n3. I found the discussion over the results to be quite limited. For instance:\n a) The authors point out that the performance on the NCI1 dataset is different than all the others (it is the only dataset where the baseline is not the best). How is this dataset different? \nb) How do these different models compare depending on certain dataset features? Perhaps a model is better than the rest if a dataset has some particular properties.\nc) Are the differences in performance shown in Tables 3-4 indeed significant, given the standard deviations, or these models practically perform the same?\nd) What makes these datasets challenging, such that the best performing models get up to 70-80%? This could provide interesting insights for future work.  \n\nIf the authors clarify some of the issues above, especially about the results discussion part, I believe this paper may indeed be of value to the graph classification community. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a careful, reproducible empirical comparison of 5 graph neural network models on 9 datasets for graph classification. The paper shows that baseline methods that use only node features (either counting node types, or summing node features) can be competitive. The authors also provide some guidelines for ways to improve reproducibility in empirical comparisons of graph classification.

The authors responded well to the issued raised during review, and updated the paper during the discussion period. The reviewers improved their score, and while there were reservations about the comprehensiveness of the set of experiments, they all agreed that the paper provides a solid empirical contribution to the literature.

As machine learning becomes increasingly popular, papers that perform a careful empirical survey of baselines provide an important sanity check that future work can be built upon. Therefore, this paper, while not covering all possible graph neural network questions, provides an excellent starting point for future work to extend.


Decision: Accept (Poster)