Data-driven Learning of Geometric Scattering Networks
Authors: Alexander Tong
Frederik Wenkel
Kincaid Macdonald
Smita Krishnaswamy
Guy Wolf
Venue:  (2021)
Keywords: Graph Neural Networks
GNNs
Geometric Scattering
Radial Basis Network
Graph Signal Processing
Wavelet

Abstract: Many popular graph neural network (GNN) architectures, which are often considered as the current state of the art, rely on encoding graph structure via smoothness or similarity between neighbors. While this approach performs well on a surprising number of standard benchmarks, the efficacy of such models does not translate consistently to more complex domains, such as graph data in the biochemistry domain. We argue that these more complex domains require priors that encourage learning of longer range features rather than oversmoothed signals of standard GNN architectures. Here, we propose an alternative GNN architecture, based on a relaxation of recently proposed geometric scattering transforms, which consists of a cascade of graph wavelet filters. Our learned geometric scattering (LEGS) architecture adaptively tunes these wavelets and their scales to encourage band-pass features to emerge in learned representations. This results in a simplified GNN with significantly fewer learned parameters compared to competing methods. We demonstrate the predictive performance of our method on several biochemistry graph classification benchmarks, as well as the descriptive quality of its learned features in biochemical graph data exploration tasks. Our results show that the proposed LEGS network matches or outperforms popular GNNs, as well as the original geometric scattering construction, while retaining certain mathematical properties of its handcrafted (nonlearned) design.

Under review as a conference paper at ICLR 2021
LEARNING OF GEOMETRIC SCATTERING NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Many popular graph neural network (GNN) architectures, which are often consid- ered as the current state of the art, rely on encoding graph structure via smoothness or similarity between neighbors. While this approach performs well on surpris- ing number of standard benchmarks, the efficacy of such models does not translate consistently to more complex domains, such as graph data the biochemistry do- main. We argue that these more complex domains require priors that encourage learning of longer range features rather than oversmoothed signals of standard GNN architectures. Here, propose alternative GNN architecture, based on a relaxation of recently proposed geometric scattering transforms, which consists of cascade of graph wavelet filters. Our learned geometric scattering (LEGS) ar- chitecture adaptively tunes these wavelets and their scales to encourage band-pass features emerge learned representations. This results in a simplified GNN with significantly fewer learned parameters compared to competing methods. We demonstrate the predictive performance of our method on several biochemistry graph classification benchmarks, as well as the descriptive quality of its learned features biochemical graph data exploration tasks. Our results show that the proposed LEGS network matches outperforms popular GNNs, as well as the original geometric scattering construction, while retaining certain mathematical properties of its handcrafted (nonlearned) design.
INTRODUCTION
Geometric deep learning has recently emerged as an increasingly prominent branch of machine learning in general, and deep learning in particular (Bronstein et al. 2017). It based on the obser- vation that many of the impressive achievements of neural networks come in applications where the data has an intrinsic geometric structure which can be used to inform network design and training procedures. For example, in computer vision, convolutional neural networks use the spatial orga- nization of pixels to define convolutional filters that hierarchically aggregate local information at multiple scales that in turn encode shape and texture information in data and task-driven represen- tations. Similarly, in time-series analysis, recurrent neural networks leverage memory mechanisms based the temporal organization of input data to collect multiresolution information from local subsequences, which can be interpreted geometrically via tools from dynamical systems and spec- tral analysis. While these examples only leverage Euclidean spatiotemporal structure in data, they exemplify the potential benefits of incorporating information about intrinsic data geometry in neural network design and processing. Indeed, recent advances have further generalized the utilization of geometric information in neural networks design to consider -ue-Eucideann structures, with partic- ular interest in graphs that represent data geometry, either directly given input or constructed as an approximation of a data manifold.
At the core of geometric deep learning the use of graph neural networks (GNNs) in general, and graph convolutional networks (GCNs) in particular, which ensure neuron activations follow the ge- ometric organization input data by propagating information across graph neighborhoods (Bruna)
et al. 2014; Defferrard et al. 2016 Kipf & Welling 2016 Hamilton et al. 2017 Xu et al. 2019 -Abu-El-Hajj et al. 2019). However, recent work has shown the difficulty in generalizing these methods more complex structures, identifying common problems and phrasing them in terms of oversmoothing (Li et al., (018), oversquashing (Alon Yahav (2020) or under-reate meaching (Barceló
et al. 2020). Using graph signal processing terminology from Kipf & Welling (016)) these issues

---

Under review as a conference paper at ICLR 2021
can be partly attributed to the limited construction of convolutional filters in many commonly used GCN architectures. Inspired by the filters learned in convolutional neural networks, GCNs con- sider node features as graph signals and aim to aggregate information from neighboring nodes. For example, Kipf & Welling (2016) presented a typical implementaioon of a GCN with a cascade of averaging (essentially low pass) filters. We note that more general variations of GCN architectures exist (Defferrard et al.) 2016 Hamilton et al.) 2017; Xu et al., 2019), which are capable of repre- senting other filters, but as investigated in Alon & Yahav (2020), they too often have difficulty in learning long range connections.
Recently, an altermative approach was presented to provide deep geometric representation learning by generalizing Mallat's 's scattering transform (Mallat 2012). originally proposed to provide math- ematical framework for understanding convolutional neural networks, to graphs (Gao et al.) 2019, Gama et al., 2019a Zou & Lerman 2019) and manifolds (Perlmutter et al. 2018). Similar to tra- ditional scattering, which can be seen as convolutional network with nonlearned wavelet filters, geometric scattering is defined as GNN with handcrafted graph filters, typically constructed as diffusion wavelets over the input graph (Coifman & Maggioni 2006), which are then cascaded with pointwise absolute-valle
nonlinearities. This wavelet cascade results in permutation equivariant node features that are typically aggregated via statistical moments over the graph nodes, as explained in detail in Sec. to provide a permutation invariant graph-level representation. The efficacy of geo- metric scattering features in graph processing tasks was demonstrated in Gao et al. (2019), with both supervised learning and data exploration applications. Moreover, their handcrafted design enables rigorous study of their properties, such as stability to deformations and perturbations, and provides clear understanding of the information extracted by them, which by design (e.g., the cascaded band-pass paas filters) goes beyond low frequencies to consider richer notions of regularity (Gama et al., 2019b Perlmutter et al. 2019).
However, while graph scattering transforms provide effective universal feature extractors, their rigid handcrafted design does not allow for the automatic task-driven representation learning that nat- urally arises in traditional GNNs. To address this deficiency, recent work has proposed a hybrid scattering-Ger (Min et al. 2020) model for obtaining node-level representations, which ensembles GCN model with fixed scattering feature extractor. In Min et al. (2020). integrating channels from both architectures alleviates the well-known oversmoothing problem and outperforms popular GNNs on node classification tasks. Here, we focus on improving the geometric scattering transform by learning, in particular its scales. We focus on whole- grape representations with an emphasis on biochemical molecular graphs, where relatively large diameters and non-planar structures usually limit the effectiveness of traditional GNNs. Instead of the ensemble approach of Min et al. 22020) we propose  native neural network architecture for learned geometric scattering (LEGS), which directly modifies the scattering architecture from Gao et al. (2019); Perlmutter et al. (2019), via relaxations described in Sec. [3] to allow task-driven adaptation of its wavelet configuration via methods approach the learning of long range connections by approximating the spectrum of the graph with the Lancoz algorithm Liao et al. (2019), or learning in block Krylov subspaces Luan et al. (2019). Such methods are complementary to the work presented here, in that their spectral approximation can also be applied in the computation of geometric scattering when considering very long range scales (c.g.. via spectral formulation of graph wavelet filters). However, we find that such approx- imations are not necessary in the datasets considered here and in other recent work focusing on whole-graph tasks, where direct computation of polynomials of the Laplacian is sufficient.
backpropagation implemented in Sec. 4 We note that other recent graph spectrum-
The resulting learnable geometric scattering network balances the mathematical properties inherited from the scattering transform (as shown in Sec. 3 with the flexibility enabled by adaptive repre- sentation learning. The benefits of our construction over standard GNNs, as well as pure geometric scattering, are discussed and demonstrated on graph classification and regression tasks in Sec. 5 In particular, we find that our network maintains the robustness to small training sets present in graph scattering while improving classification on biological graph classification and regression tasks, and we show that in tasks where the graphs have a large diameter relative to their size, learnable scatter- ing features improve performance over competing methods.
2

---

Under review as a conference paper at ICLR 2021
2 PRELIMINARIES: GEOMETRIC SCATTERING FEATURES
Let g = (V, E, w) be a weighted graph with V == fui,.... Un) the set of nodes, E C (fer uz} E V x V, 5} the set of (undirected) edges and E - (0, 0o) assigning (positive) edge weights to the graph edges, Note that w can equivalently be considered as a function of V x V, where we set the weights of non-adjacent node pairs zero. We define a graph signal as a function on the nodes of G and aggregate them in a signal vector I E IR" with the 7th entry being 2=U;].
I
We define the weighted adjacency matrix W R=xx of the graph 9 as
if (0) E otherwise
W my]
00
and the degree matrix D E R of G as D = diag(d1,...., da) with di = deg(u;)
=
Σj= W(e, 2y] being the degree of the node Vç.
The geometric scattering transform (Gao et al. from row stochastic diffusion matrix P probabilities of lazy random walk Markov process. The laziness of the process signifies that at each step it has equal probability of either staying at the current node or transitioning to a neighbor, where transition probabilities in the latter case are determined by (normalized) edge weights. Scattering filters are then defined via the graph-wavell matrices D R=x= of scale j E No. as
2019 relies cascade of graph filters constructed [I WD-1), which corresponds to transition
0 In P,
p2 P = P2 (I, - P p ) 221.
These diffusion wavelet operators partition the frequency spectrum into dyadic frequency bands, which are then organized into a full wavelet filter bank W, == j Lo)osjs. where 4 == p2 a pure low-pass filter, similar to the one used GCNs. is easy to verify that the resulting wavelet transform is invertible, since simple sum of filter matrices in W) yields the identity. Morcover, as discussed in Perlmutter et al. (2019), this filter bank forms a noncxpansive frame, which provides energy preservation guarantees as well stability to perturbations, and can be generalized to wider family of constructions that encompasses the variations of scattering transforms on graphs from Gama et al. (2019a] b) and Zou & Lerman (019)..
Given the wavelet filter bank W., node-level scattering features are computed by stacking cascades of bandpass filters and deeeettwwise absolute value nonlinearities to form
(2) indexed (or parametrized) by the scattering path p (Ji...., Im) E N that determines the filter scales captured by each scattering coefficient. Then, whole-graph scattering representation obtained by aggregating together node-level features via statistical moments over the nodes of the graph (Gao et al., 2019). This construction yields the geometric scattering features
U, = 10 Jm_ , L x ......|
Sp.g U,
(3)
indexed by the scattering path p and moment order q. Finally, we note that it can be shown that the graph-level scattering transform Sp.g guarantees node- sodde-eemmtttionn invariance, while Up is permu- tation equivariant (Perlmutter et al. 2019 Gao al. 2019).
3 RELAXED GEOMETRIC SCATTERING CONSTRUCTION TO ALLOW TRAINING
The geometric scattering construction, described in Sec. 2 can be seen particular GNN with handcrafted layers, rather than learned ones. This provides a solid mathematical framework for un- derstanding the encoding of geometric information in GNNs, shown in Perlmutter et al. (019))
while also providing effective unsupervised graph representation learning for data exploration, which also has some advantages even in supervised learning task, shown in Gao et al. 2019
While the handerafted design in iPrelmuuter et al. (2019); Gao et al. (2019) is not priori amenable to dak-drivee tuning provided by end-to-end GNN training, we note that the cascade in Eq. does
3

---

Under review as a conference paper at ICLR 2021
conform neural network architecture suitable for backpropagation. Therefore. in this section, we show how and under what conditions a relaxation of the laziness of the random walk and the se- lection of the scales preserves some of the useful mathematical properties established in Perlmutter et al. (2019). We then establish in section 5 the empirical benefits of learning the diffusion scales over purely handcrafted design.
We first note that the construction of the diffusion matrix P that forms the lowpass filter used in the fixed scattering construction can be relaxed to encode adaptive laziness by setting P. al, + (1 -aa a)) W D-1 Where a E [1/2, 1) controls the reluctance of the random walk to transition from one node to another. a = 1/2 gives an equal probability to stay in the same node as to transition to one of its neighbors. At this point, we note that one difference between the diffusion lowpass filter here and the one typically used in GCN and its variation is the symmetrization applied Kipt & Welling (016)). However, |Perlmutter et al. (2019) established that for the original construction, this only technical difference since P can be regarded as self-adjoint under an appropriate measure which encodes degree variations in the graph. This is then used to generate Hilbert space (g, D 1/3) of graph signals with inner product (ax, y) D-1/2 = (D-1/2x, D p-1/2y). The following lemma shows that similar property retained for our adaptive lowpass filter P..
Lemma 1. The matrix Pa is self-adjoint on the Hilbert space L?2 D D-1/2) from Perlmutter et al. 2079
We note that the self adjoindjossssss shown here is interesting, as it links models that use symmetric and asymmetric versions of the Laplacian adjacency matrix. Namely, Lemma I shows that the diffusion matrix (which is column normalized but not row normalized) is self-adjoint, as an oper- ator, and thus be considered as "symmetric" in suitable inner product space, thus establishing a theoretical link between these design choices.
As second relaxation, we propose to replace the handcrafted dyadic scales in Eq. [with an adaptive monotonic sequence of integer diffusion time scales 0 < 4 < <... < l,,, which can be selected or tuned via training. Then, an adaptive filter bank is constructed as W', = [T 5. $ }3) with
D', = Plu
V In ph
(4)
V/ P P I
The following theorem shows that for any selection of scales, the relaxed construction of W', con- structs a nonexpansive frame, similar the result from (errmmuttt et al. (2019) shown for the original handcrafted construction.
Theorem 1. There exist a constant C > 0 that only depends on t and t such that for all x E (G D 11/2),
C/|æe|/--12 & <1 D-1
Σ j=0
a & |
where the norm considered here is the one induced by the space (G,
Intuitively, the upper (i.e., nonexpansive) frame bound implies stability in the sense that small pertur- bations in the input graph signal will only result in small perturbations in the representation extracted by the constructed filter bank. Further, the lower frame bound ensures certain energy preservation by the constructed filter bank, thus indicating the nonexpansiveness is not implemented in a trivial fashion (e.g.. by constant features independent of input signal).
In the next section we leverage the two relaxations described here to design a neural network archi- tecture for learning the configuration a, ft,...,.k ,, this relaxed construction via backpropagation through the resulting scattering filter cascade. The following theorem establishes that for any such configuration, extracted from W", via Eqs. 2 is permutation equivariant at the node-level and per- mutation invariant at the graph level. This guarantees that the extracted (in this case learned) features indeed encode intrinsic graph geometry rather than priori indexation.
Theorem 2. Let U and Sy. be defined as in Eq.
2 aadd (correspondingly), with the filters from < L... Then, for any permutation II
W", with arbitrary configuration < a 1,

---

Under review as a conference paper at ICLR 2021
the nodes of and any graph signal (2
U'llæ 11 and Spgllæ= pp.
p Umen ,
where geometric scattering implicitly considers here the node ordering supporting its input signal.
We note that the results in Lemma and Theorems 12 as well their proofs, closely follow the theoretical framework proposed by Perlmutter al. (2019). We carefully account here for the relaxed learned configuration, which replaces the originally handcrafted configuration there. For completeness, the adjusted proofs appear in Sec. of the Appendix.
Geometric Scattering
Fixed
b) Learnable Scattering
P (D 2/
Trainable
W
X
X
yi prl
41
4P
Figure I: LEGSNet learns to select the appropriate scattering scales from the data.
4 LEARNABLE GEOMETRIC SCATTERING NETWORK ARCHITECTURE
In order to implement the relaxed geometric scattering construction (Sec. 3 via trainable neural network, throughout this section, we consider an input graph signal x R or. equivalently, collection of graph signals X E BR 1, The propagation of these signals can be divided into three major modules. First, a diffusion module implements the Markov process that forms the basis of the filter bank and transform, while allowing learning of the laziness parameter a. Then, scattering module implements the filters and the corresponding cascade, while allowing the learning Finally, the aggregation module collects the extracted features to provide
of the scales I ...... f.y. graph and produces the task-dependent output.
Building a diffusion process. We build set of m E N subsequent diffusion steps of the signal by iteratively multiplying the diffusion matrix Pa the left of the signal, resulting in
Pax, Pax,, P P....., P
Since Pa is often sparse, for efficiency reasons these filter responses implemented via an RNN structure consisting m RNN modules. Each module propagates the incoming hidden state h =1, ,,.... .m with P.. with the readout o, equal to the produced hidden state,
h P.h
h
Our architecture and theory enable the implementation of either trainable nontrainable a, which we believe will be useful for future work as indicated, for example, in Gao & Ji (201)). However, in the applications considered here (see Sec. we find that training a made training unstable and did not improve performance. Therefore, for sümplicity, we leave it fixed as a = 1/2 for the remainder of this work. In this case, the RNN portion of the network contains no trainable parameters, thus speeding up the computation, but still enables a convenient gradient flow back to the model input.
Learning diffusion filter bank. Next, we consider the selection of J < m diffusion scales for the relaxed filter bank construction with the wavelets defined according Eq. We found this was the most influential part of the architecture. We experimented with methods of increasing flexibility:
1. Selection of (= as dyadic scales (as in Sec. and Eq. fixed for all datasets (LEGS- FIXED),
2. Selection of each t, using softmax and sorting by j. learnable per model (LEGS-FFC and LEGS-RBE, depending output layer explained below).
5

---

Under review as a conference paper at ICLR 2021
For the softmax selection, use selection matrix F E R where each row FG.). I.....JJ is dedicated to identifying the diffusion scale the wavelet P via a one-hot encoding. This is achieved by setting
F softmax = (softmax(O1), sootmxa soot:m.ssssoaam))
where 0, !R' constitute the rows of the trainable weight matrix O While this construction may not strictly guarantee an exact one-hot encoding, we assume that the softmax activations yield sufficient approximation. Further, without loss of generality, we assume that the rows of F are ordered according to the position of the leading "one" activated in every row. In practice, this can be easily enforced by reordering the rows. We now construct the filter bank WF = ( $ with the filters
m v Σ 1=1
P. F(J.))
V v0 = In - m
(5)
a Em [Fo.)
1sjsJ-1
matching and implementing the construction of W3 from Eq.
Aggregating and classifying scattering features. While many approaches may be applied to
aggregate node-level features into graph-level features such as max, mean, sum pooling, and the more powerful TopK (Gao & Ji 2019) attention pooling (Veličković al. 2018), statistical moment aggregation explained in Secs. 2 (motivated by Gao et al. 2019 Perlmutter al. 2019) and leave exploration of other pooling methods to future work. As shown Gao al. (2019) graph classification, this aggregation works particularly well in conjunction with support vector machines (SVMs) based on the radial basis function (RBF) kernel.
follow the
Here, we consider two configurations for the task-dependent output layer of the network, either using a small neural network with two fully connected layers, which we denote LEGS-FCN, or using a modified RBF network (Broomhead & Lowe 1988), which we denote LEGS-RBE, to produce the final classification. The latter confi configuration more accurately processes scattering features as shown in Table 2 Our RBF network works by first initializing fixed number of movable anchor points. Then, for every point, new features are calculated based on the radial distances to these anchor points. In previous work on radial basis networks these anchor points were initialized independent of the data. We found that this led to training issues the range of the data was not similar to the initialization of the centers. Instead, first use a batch normalization layer to constrain the scale of the features and then pick anchors randomly from the initial features of the first pass through our data. This gives an RBF-kernel network with anchors that are always in the range of the data. Our RBF layer is then RBF(x)= = <{ ((BatchNorm())) cll) with <(x) =e ||=|"
5 EMPIRICAL RESULTS
Here we show results of LEGSNet on whole graph classification and graph re- gression tasks, that arise in a variety of contexts, with emphasis on the more com- plex biochemic datasets.
Table 1: Dataset statistics, diameter, nodes, edges, and clustering efficient averaged over graphs.
# Graphs
# Classes
Diameter
Nodes
Edges
Clust. Coeff
DD
1178
2
19.81
284.32
715.66
0.48
ENZYMES
600
6
10.92
32.63
62.14
0.45
MUTAG
188
2
8.22
17.93
19.79
0.00
We use biochemical graph datasets as they represent a new challenge in the field of graph learning. Unlike
NCII
4110
13.33
29.87
32.30
0.00
NC1109
4127
13.14
29.68
32.13
0.00
PROTEINS
1113
2
11.62
39.06
72.82
0.51
PTC
344
2
7.52
14.29
14.69
0.01
other types data, these datasets do not exhibit the small-world structure of social datasets and may have large graph diameters for their size. Further, the connectivity patterns of biomolecules very irregular due to 3D folding and long range connections, and thus ordinary local node aggrega- tion methods may miss such connectivity differences.

---

Under review as a conference paper at ICLR 2021
5.1
W WHOLE GRAPH CLASSIFICATION
We perform whole graph classification by using eccentricity and clustering coefficient as node fea- tures as is done in Gao et al. (2019). We compare against graph convolutional networks (GCN) (Kipf & Welling 2016), GraphSAGE (Hamilton et al., (017), graph attention network (GAT) (Veličković eet 2al.2011)) graph isomorphism network (GIN) (Xu al., 2019), Snowball network (Luan et a 2019 and fixed geometric scattering with support vector machine classifier (GS-SVM) in Gao ct al. 2019), and a baseline which is a 2-layer neural network on the features averaged across nodes (disregarding graph structure). These comparisons are meant to inform when including learnable graph scattering features are helpful in extracting whole graph features. Specifically, we are in- terested in the types of graph datasets where existing graph neural network performance can be improved upon with scattering features. We evaluate these methods across benchmark biochemi- cal datasets: DD, ENZYMES, MUTAG, NCT1, NC1109, PROTEINS, and PTC where the goal is to classify between two or more classes of compounds with hundreds to thousands of graphs and tens to hundreds of nodes (See Table []. For completeness we also show results on six social network datasets in Table S2 For more specific information on individual datasets see Appendix B We use 10-fold cross validation on all models which elaborated on in Appendix C
For an ensembling
comparison to Scattering-GCN (Min et al. 2020) see Appendix D
Table 2: Mean + standard deviation test set accuracy on biochemical datasets. Time limit expired (TLE) individual denotes models that did not finish in 10 hours.
DD
ENZYMES
MUTAG
NCII
NC1109
PROTEINS
PTC
LEGS-RBF
72.58 + 3,35
36.33 + 4,50
33,51
+ 4.34
74.26
1,53
72.47 + 2.11
70.89 + 3.91
57.26 + 5,54
LEGS- -SNN
72.07
+ 2.37
38.50
+ 8.18
82.98
9.85
70.83
2.65
70.17
+ 1.46
71.06
+ 3.17
56.92
9.36
LEGS-FIXED
69.09
4.82
32.33
5.04
81.4
11.24
71.24
1.63
69.25
1.75
67.30
2.94
54.31 + 6,92
GCN
67.82 + 3.81
31.33 +: 6.89
79.30 + 9.66
60.80 +: 4.26
61.30
+ 2,99
74.03
+: 3.20
56.34 +: 10.29
GraphSAGE
66.37 +: 4.45
15,83 +: 9.10
81.43
11.64
3.33
55.15
2.58
71.87
3.50
55,22 4: 9.13
GAT
3.62
25.83 4: 4.73
79.85 : 9,44
22.19
73.22
55,50 6,90
GIN
42.37 + 4.33
36.83 + 4,81
83.57 + 9.68
66.67 + 2.90
65.23 + 1.82
75.02
+ 4.55
55.82 + 8.07
Snowball
TLE
18.00 + 1.89
50.56
20.87
48.56
50.86
+ 2.65
39.36
1: 4.29
50.84 + 9.32
GS-SVM
72.66 4: 4.94
27.33
5.10
85.09
7.44
2..38
68.55 2,06
70.98
2.67
56.96 + 7.09
Baseline
75.98 =: 2.81
20.50 +: 5.99
79,80 1 9.92
3.009
$7.38 + 2.20
73.22
1: 3.76
56.71 + 5.54
LEGS outperforms on biological datasets. A somewhat less explored domain for GNNs is in biochemical graphs that represent molecules and tend to be overall smaller and less connected (see Tables [ and SSI than social networks. In particular ods by a significant margin on biochemical datasets with relatively small but high diameter graphs (NC11, NC1109, ENZYMES, PTC), shown in Table 2 On extremely small graphs we find that GS-SVM performs best, which is expected as other methods with more parameters can easily overfit the data. We reason that the performance increases exhibited by LEGSNet, and lesser extent GS-SVM, on these chemical and biological benchmarks is due the ability of geometric scattering to compute complex connectivity features via its multiscale diffusion wavelets, Thus, methods that rely on a scattering construction would in general perform better, with the flexibility and trainability LEGSNet giving it an edge on most tasks.
find that LEGSNet outperforms other meth-
LEGS performs consistently on social network datasets. the social network datasets LEGSNet performs consistently well, although its benefits here are not as clear as in the biochemi- cal datasets. Ignoring the fixed scattering transform GS-SVM, which was tuned in Gao et al.) (2019) with focus these particular social network datasets, version of LEGSNet is best three out of the six social datasets and second best on the other three. Since the advantages are clearer in the bio- chemical domain, we focus this in the remainder of this section. However, for completeness, provide results on social network datasets in Table S2 and leave further discussion to Appendix(B.]
LEGS preserves enzyme exchange preferences while increasing performance. One ad- vantage of geometric scattering over other graph embedding techniques lies in the rich in- formation present within the scattering feature space. This was demonstrated in Gao et al. (2019) where it was shown that the embeddings created through fixed geometric scatter- ing can be used to accurately infer inter anter-grahh relationships. Scattering features of en- zyme graphs within the ENZYMES dataset (Borgwardt et al. 2005) possessed sufficient
7

---

Under review as a conference paper at ICLR 2021
global information recreate the enzyme class exchange preferences observed empirically by Cuesta et al. (2013), using only linear methods of analysis, and despite working with a much smaller and artificially balanced dataset. We demonstrate here that LEGSNet retains similar descriptive capabilities, as shown in Figure 2 via chord diagrams where each exchange pref- erence between enzyme classes (estimated as suggested in Gao et al.) 2019) is represented as ribbon of the corresponding size. Our results here (and in Table [S5] which provides complementary quantitative comparison) show that, with relaxations on the scattering parame- ters, LEGS-FCI -CN achieves better classification accuracy than both LEGS-FIXED and GCN (see Table T while also retaining a more de- scriptive embedding that maintains the global structure of relations between enzyme classes. We ran two varieties of LEGSNet on the EN- ZYMES dataset: LEGS-FIXED and LEGS- FCN, which allows the diffusion scales be learned. For comparison, we also ran a stan- dard GCN whose graph embeddings were ob- tained via mean pooling. To infer enzyme ex- change preferences from their embeddings, we followed Gao et al. (2019) in defining the dis- projc, (ve)ll. where
Affinities between
Classes 1.
2. B. 4. 5.
aa) Observed
(b) LEGS-FCN
(c) LEGS-FIXED
(d) GCN
Figure 2: Enzyme class exchange preferences pirically observed in Cuesta et al. (2015), and es- timated from LEGS and GCN embeddings.
tance from enzyme e to the enzyme class EC; dist(e,EC,) ea is the embedding of e, and Cj is the PCA subspace of the enzyme feature vectors within ECj- The distance between the enzyme classes EC; and HC; is the average the individual distances, mean dist(e, EC2) E EC,). From here, the affinity between
Ue
enzyme classes is computed as (EC, EC,) = w min( Pa. Dyd)) where w; is the percentage of enzymes in class which are closer to another class than their own, and Di.j is the distance between EC; and ECj.
D.,,
Robustness to reduced training set size. We remark that similar to the robustness shown in (Gao ctt al. 2019) for handcrafted scattering, LEGSNet is able to maintain accuracy even when the training set size is shrunk to low 20% of the dataset, with median decrease of 4.7% accuracy when 80% the data is used for training, as discussed in the supplement (see Table S3
5.2 GRAPH REGRESSION
We next evaluate learnable scattering two graph regression tasks, the QM9 (Gilmer et al.) 2017 Wu et al. (018)) graph regression dataset, and a new task from the critical assessment of structure prediction (CASP) challenge (Moult eet al.] 2018). On the CASP task, the main objective to score protein structure predic- tion/simulation models in terms of the discrep- ancy between their predicted structure and the actual structure of the protein (which is known a priori). The accuracy of such 3D structure predictions are evaluated using variety of met- rics, but we focus on the global distance test (GDT) score Modi et al. measures the similarity between tertiary structures of two proteins with amino-aci correspondence. A higher score means two structures are more similar. For a set of predicted 3D structures for protein, would like score their quality quantified by the GDT score.
Table 3: Train and test set mean squared error on CASP GDT regression task over three seeds.
(μ O)
Train MSE
Test MSE
LEGS-FCN
134.34 +: 8.62
144.14 15.48
LEGS-RE RBG-
140.46 9.76
152.59 14.56
LEGS-FIXED
136.84 15.57
160.03 = 1.81
GCN
289.33 15.75
303.52 = 18.90
GraphSAGE
221.14
219.44 = 34.84
GIN
221.14 42.56
219.44 34.84
Baseline
393.78 + 4.02
402.21 + 21.45
2016).
The GDT score
8

---

Under review as a conference paper at ICLR 2021
For this task use the CASP12 dataset (Moult et al. 2018 and preprocess the data similarly to Ingraham et al. 2019), creating a KNN graph between proteins based on the 3D coordinates of each amioo acid. From this KNN graph regress against the GDT score. We evaluate on 12 proteins from the CASP12 dataset and choose random (but consistent) splits with 80% train, 10% validation, and 10% test. data out of 4000 total structures. We only concerned with structure similarity so use no non-structural node features.
LEGSNet outperforms on all CASP targets Across all CASP targets we find that LEGSNet significantly outperforms GNN and baseline methods (See [able This performance im- provement is particularly stark on the casiest
0679
19920
NSt decresse
Ke65
e 19942
0872
structures (measured by average GDT) but is
10944
consistent across all structures. In Figure 3 show the relationship between percent im-
1900
provement of LEGSNet over the GCN model and the average GDT score across the target structures. We draw attention to target t0879, where LEGSNet shows the greatest improve- ment over other methods. This target has long range dependencies (Ovchinnikov et al. 2018) as it exhibits metal coupling (Li et al., 2015)
Corr. Coef.: 0.55
55
Avg gdt
Figure 3: CASP dataset LEGS-FCN % improve- ment over GCN in MSE of GDT prediction vs. Average GDT score.
creating long range connections over the sequence. Since other methods are unable to model these long range connections LEGSNet is particularly important on these more difficult to model targets.
LEGSNet outperforms on the QM9 dataset We evalu- ate the performance of LEGSNet dataset QM9 (Gilmer et al. 2017 Wu et al. 2018), which con- sists of 130,000 molecules with 18 nodes per molecule. We use the node features from Gilmer et al. (2017), with the addi- tion of eccentricity and clustering coefficient features, and ig- nore the edge features. We whiten all targets to have zero mean and unit standard deviation. We train each network against all 19 targets and evaluate the mean squared eггоr on the test set with mean and std. over four runs. We find that learning the scales improves the overall MSE, and particularly improves the results over difficult targets (see Table for overall results and Table S7 for results by target). Indeed, on more difficult targets (i.e., those with large test error) LEGS-FCN able to perform better, where easy targets GIN is the best. Overall, scattering features offer robust sig- nal over many targets, and while perhaps less flexible (by construction), they achieve good average performance with significantly fewer parameters.
Table 4: Mean + std. over four runs of mean squared error over 19 targets for the QM9 dataset, lower is better.
the quantum chemistry
(n (
Test MSE
LEGS- FCN
0.216 + 0.009
LEGS-FIXED
0.228 + 0.019
GraphSAGE
0.524 + 0.224
GCN GIN
0.417 + 0.061
0,247 + 0.037
Baseline
0.533 + 0.041
6 CONCLUSION
In this work we have established relaxation from fixed geometric scattering with strong guarantees to a more flexible network with better performance by learning data dependent scales. Allowing the network to choose data-driven diffusion scales leads to improved performance particularly on biochemical datasets, while keeping strong guarantees on extracted features. This parameterization has advantages in representing long range connections with small number of weights, which are necessary in complex biochemical data. This also opens the possibility to provide additional relax- ation to enable node-specific graph-specific tuning via attention mechanisms, which we regard as an exciting future direction, but out of scope for the current work.
REFERENCES
Sami Abu-El-Haiie Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. MixHop: Higher-Order Graph Convolutional

---

Under review as a conference paper at ICLR 2021
Architectures via Sparsified Neighborhood Mixing. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Uri Alon and Eran Yahav, On the bottleneck of graph neural networks and its practical implications. arXiv preprini arXiv:2006.0522 05205, 2020.
Pablo Barceló, Egor V Kostylev, Mikael Monet, Jorge Pérez, Juan Reutter, and Juan Pablo Silva. The logical expressiveness of graph neural networks. In International Conference on Learning Representations, 2020.
K. M. Borgwardt, C. S. Ong, S. Schonauer, S. V. N. Vishwanathan, A. J. Smola, and H.-P. Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21(upp 1):147-i56, June 2005. ISSN 1367-4803, 1460-2059. doi: 10. 0001000fiionformaticssbil0001
Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geomet- ric deep learning: Going beyond Euclidean data. JEEE Signal Process. Mag., 34(4):18 42, 2017. ISSN 1053-5888. doi: 10.1 109/MS/..00...299999 2612344418.....
D. S. Broomhead and D Lowe. Radial Basis Functions, Multi- Variable Functional Interpolation and Adaptive Networks. R. Signals Raar Establ., Memorandum 4148, 1988.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral Networks and Locally Connected Networks on Graphs. In ICLR, 2014.
Ronald R. Coifman and Mauro Maggioni. Diffusion wavelets. Applied and Computational Har- monic Analysis, 21(1):53 July 2006. ISSN 10635203. doi: 10.1016/j.acha.20000 20tpha..000.....0 04.04.
Sergio Martinez Cuesta, Syed Asad Rahman, Nicholas Furnham, and Janet M. Thornton. The
Classification and Evolution of Enzyme Function. Biophysical Journal, 108 2015.
ISSN 00063495. doi: 10. 10.1016/j.bpj...... 2015..042211000000
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. Adv. Neural Inf. Process. Syst. 30, 2016.
Paul D. Dobson and Andrew J. Doig. Distinguishing Enzyme Structures from Non-enzymes Without Alignments. Journal of Molecular Biology, 330(4):771-783, July 2003. ISSN 0022-2836. doi: (20000000222223(00000006663
Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Diffusion Scattering Transforms Graphs. Int. Conf. Mach. Learn., 2019a.
Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Stability of Graph Scattering Transforms. Adv. Neurul Inf. Process. Syst. 33, 2019b.
Feng Gao, Guy Wolf, and Matthew Hirn. Geometric scattering for graph data analysis. Proceed- ings of the 36th International Conference Machine Learning, pp. 2122-112222222 12222131, 2019.
Hongyang Gao and Shuiwang Ji. Graph U-Nets. Proceedings of the 36th International Confer- ence on Machine Learning, 2019.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals. and George E. Dahl. Neural Message Passing for Quantum Chemistry. Proceedings of the 34th International Conference on Machine Learning. 2017.
William I.. Hamilton, Rex Ying, and Jure Leskovec. Inductive Representation Learning Large Graphs. Adv. Neural Inf. Process. Syst. 31, 2017.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative Models for Graph- Based Protein Design. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 33, pp. 15820-15831, 2019.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. 3rd Interna- lional Conference on Learning Representations, 2015.
10

---

Under review as a conference paper at ICLR 2021
Thomas N. Kipf and Max Welling. Semi- Supe-Supee Classification with Graph Convolutional Net- works. 4th Int. Conf. Mach. Learn., September 2016.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for sumerispere learning. In Proceedings of the Thirty-Seconi AAAI Conference on Artificial In- relligence (AAAI-18), pp. 1538-3545, Association for the Advancement of Artificial Intelligence, 2018.
Wenfei Li, Jun Wang, Jian Zhang, and Wei Wang. Molecular simulations of metal-coupled protein folding. Current Opinion in Structural Biology, 30:25-31, February 2015. ISSN 0959-440X. doi: 10.1016/j.sbi.200 2014, .006.
Renjie Liao, Zhizhen Zhao, Raquel Urtasun, and Richard S Zemel. LanczosNet: Multi-Scale Deep Graph Convolutional Networks. In ICLR, 2019.
Sitao Luan, Mingde Zhao, Xiao-Wen Chang, and Doina Precup. Break the Ceiling: Stronger Multi- scale Deep Graph Convolutional Networks. In Advances in Neural Information Processing Sys- tems 33, 2019.
Stéphane Mallat. Group Invariant Scattering. Commun, Pure Appl. Math., LXV:1331-13388 2012.
Yimeng Min, Frederik Wenkel, and Guy Wolf. Scattering gen: Overcoming oversmoothness in
graph convolutional networks. arXiv preprint arXiv:2003.08414. 2020.
Vivek Modi, Qifang Xu, Sam Adhikari, and Roland I.. Dunbrack. Assessment of Template- Modeling of Protein Structure in CASPIL. Proteins, 84(Suppl 1):200-220. September 2016.
ISSN 0887-3585. doi: 10. 10.1002//rot.25049
John Moult, Krzysztof Fidelis, Andriy Kryshtafovych, Torsten Schwede, and Anna Tramontano. Critical assessment of methods of protein structure prediction (CASP) XII. Proteins Struct. Funct. Bioinforma, 86(S1):77115 2018. ISSN 1097-0134. doi: 10. 10.1002/pr0..25415.
Sergey Ovehinnikov. Hahnbeom Park, David E. Kim, Frank DiMaio, and David Baker. Protein structure prediction using Rosetta in CASP12. Proteins Struct. Funct. Bioinforma. 86(S1):113- 121, 2018. ISSN 1097-0134.
doi: 10. 20.1002/rot..5390.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High- Perforrerfoe Deep Learning Library. In Advances in Neural Information Processing Systems 33, pp. 8026- 8037, 2019.
Michael Perlmutter, Guy Wolf, and Matthew Hirn Geometric scattering on manifolds. In NeurIPS 2018 Workshop on Integration of Deep Learning Theories, 2018
Michael Perlmutter, Feng Gao, Guy Wolf, and Matthew Hirn. Understanding graph neural networks with asymmetric geometric scattering transforms. arXiv preprint arXiv:1911.0623 2019.
H. Toivonen, Srinivasan, R. D. King, S. Kramer, and C. Helma. Statistical evaluation of the Pre- dictive Toxicology Challenge 2000-2001. Bioinformatics, 19(10):1183-1193 July 2003. ISSN 1367-4803, 1460-2059, doi: 10. 1...09//iiinnoommttcc//gg33333
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. 6th Int. Conf. Learn. Represent., 2018.
Nikil Wale, Ian A Watson, and George Karypis. Comparison of Descriptor Spaces for Chemical Compound Retrieval and Classification. Knowl. Inf. Syst., 2008
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: A benchmark for molecular machine learn- ing. Chem. Sei., 9(2)5133-330 2018. ISSN 2041-6520, 2041-6538 doi: 10. 10.30//675S

---

Under review as a conference paper at ICLR 2021
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful Are Graph Neural Networks? ICLR, 2019.
Pinar Yanardag and S. V.N. Vishwanathan. Deep Graph Kernels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD '15, pp. 1365-1374, Sydney, NSW, Australia, 2015. ACM Press. ISBN 978-1-4503-3664-2 doi: 10. 1145/2783258.278341
Dongmian Zou and Gilad Lerman. Graph convolutional neural networks via scattering. Applied and Computational Harmonic Analysis, 2019. ISSN 10635203, doi: 10.1016/j.achaaa act.1....cca.22.00000000..
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A PROOFS FOR SECTION 3
A.I PROOF OF LEMMA II
Let M. = D-11/2P, D1/2 then can be verified that M. is a symmetric conjugate of P.. and by construction is self-adjoint with respect to the standard inner product of (G). Let a; (g, D-1/2) then we have
Pa U)D 1/2 = = (D-1/2 1/2 Pa., D 1/2y)
(D- D 1/2 M. D 1-1/222 D 0-1/2y))
(MaD 1/2g, x, D 0-112y)
=(D (D- 1/2x, M D 1/2y)
M. (D 1--1/22 n. D D 1//
(D 1/2 x; D 1/2
(x, Pay) D 1/2,
which gives the result of the lemma.
A.2 PROOF OF THEOREM 1
As shown in the previous proof (Sec. A. p. P. has symmetric conjugate M.. Given the eigende- composition M, QAQ". can write P D'/2QN'Q" D 1/2 giving the eigendecomposi- tion of the propagated diffusion matrices. Furthermore, it can be verified that the eigenvalues on the diagonal of A are nonnegative. Briefly, this results from graph Laplacian eigenvalues being within the range [0, 1]. which means those of W D are in [-1, 1]. which combined with 1/2 S a S result A; = [A] [0, 1] for every j. Next, given this decomposition we can write:
D Q"
V D D A D D-1/2, OSJ
where we set to = to simplify notations. Then, we have:
b 10/ ( D'ja)
(D- 112 D -"""
=x ==' QA 1/2 1-2 (x D 12 D x-1/2x)
= DIA"Q""-'// 1/2 1/
Further, since Q is orthogonal (as it is constructed from an eigenbasis of symmetric matrix), we consider a change of variable to y = Q' D-122x, we have ||0||| = 1-12222) | 1 while | 1 1/2 = | y1l.. Similarly, we can also reformulate the operation of other filters in terms of diagonal matrices applied to y as Wy, -/ - = |l A ll1...
Given the reformulation in terms of y and standard LC(
now write
| 3 2112 + IIA - A511 Jal1 Σ : (Λ**) + Ej-0 (2 -2211122
i=1
Then, since 0 5 >j 5 1 and 0 to <4 <
را have
J-1
J-1
X 2t +
Σ
(2 - X v
x2 +
Σ x
-(1 + - -x01² =1,
1=0
==
13

---

Under review as a conference paper at ICLR 2021
On the other hand, which yields the upper bound y + Ej=o 11A 18++)a11 l1vl.. since 1 > 0 lo. then we also have
J-1 Σas - XQtt12 2 2 + +(1 )2
2
1=0
and therefore, by setting C = minosesi (52t) + )) 0. whose positivity is not difficult to verify, we get the lower bound ||A y Ej=0 || + >yll 2 C Finally, applying the reverse change of variable to æ and (g, D 0-1/2) yields the result of the theorem.
A.3 PROOF OF THROREM 2
Denote the permutation group on n elements as Sn, then for a permutation II E S. we l[) be the graph obtained by permuting the vertices of G with II. The corresponding permutation opera- tion a graph signal considered in the statement of the theorem, without specifying these notations for simplicity. Rewrit- ing the statement of the theorem more rigorously with the introduced notations, aim to show that U on the permuted graph G is denoted here by U, and likewise for Sp we have S Sp.q.
(G, D D-122) gives signal Hl E L', D-1/2,, which we implicitly
Ta = S, under suitable conditions, where the operation U from y TT IT", and 5 Thx
p.a
We start by showing U is permutation equivariant. First, notice that for any v, 0 j J have that V 1l1lx 11 a as for S j S J-11
1II = (IIP* IP
[I[(',, -Pb++)
Similar reasoning also holds for j E {0, J.. Further, notice that for the element-wiss wise nature of the absolute value nonlinearity yields [I II|=| for any permutation matrix II. Using these two observations, it follows inductively that
U II = L
L D
Iæ
V j.
D
II
x
V Jm Jm-1
1 x [....
-10
To show Sy is permutation invariant, first notice that for any statistical moment > O, we have [ = -I and further as sums are commutative, E, (IIx); = E, Ty.. We then have
n Σ
3 pp Tl =
Uor Σ a
U UN S,
i=1
i=1
which, together with the previous result, completes the proof of the theorem.
B DATASETS
In this section we further analyze individual datasets. Relating composition of the dataset as shown Table to to the relative performance of our models as shown in Table
DD Dobson & Doig (2003): Is dataset extracted from the protein data bank (PDB) of 1178 high resolution proteins. The task is to distinguish between enzymes and non-enzymes. Since these high resolution structures, these graphs significantly larger than those found in other biochemical datasets with a mean graph size of 284 nodes with the next largest biochemical dataset with mean size of 39 nodes.
14

---

Under review as a conference paper at ICLR 2021
ENZYMES Borgwardt et al. (2005): Is dataset of 600 enzymes divided into 6 balanced classes of 100 enzymes each. As we analyzes in the main text, scattering features are better able to preserve the structure between classes. LEGS-FCN slightly relaxes this structure but improves accuracy from 32 to 39% over LEGS-FIXED.
NCI1, NCT109 Wale al. (2008): Contains slight variants of 4100 chemical compounds encoded as graphs. Each compound is separated into one of two classes based on its activity against non- small cell lung cancer and ovarian cancer cell lines. Graphs this dataset are 30 nodes with similar number of edges. This makes for long graphs with high diameter.
Borgwardt et al. (2005): Contains 178 protein structures with the goal classifying enzymes vs. enzymes. GCN outperforms all other models this dataset, however the Baseline model, where no structure is used also performs very similarly. This suggests that the graph structure within this dataset does not add much information over the structure encoded in the eccentricity and clustering coefficient.
PROTEINS
PTC Toivonen et al. (2003): Contains 344 chemical compound graphs divided into two classes based whether not they cause cancer in rats. This dataset is very difficult to classify without features however LEGS-RBE and LEGS-FCN are able to capture the long range connections slightly better than other methods.
5000 ego-networks of different researchers from high energy physics, condensed matter physies or astrophysics. The goal to determine which field the research belongs to. The GraphS model performs best on this dataset although the LEGS-RBF network performs nearly well. Ego graphs have very small average diameter. Thus shallow networks can perform quite well on them as is the case here.
COLLAB Yanardag & Vishwanathan 2015
IMDB Yanardag & Vishwanathan (2015): For each graph nodes represent actresses/actors and there is an edge between them if they in the same move. These graphs also ego graphs around specific actors. IMDB-BIN -IDDARY classifies between action and romance genres. IMDB-MULTI classifies between classes. Somewhat surprisingly GS-SVM performs the best with other LEGS networks close behind. This could be due to oversmoothing on the part of GCN and GraphSAGE when the graphs are small.
REDDIT
Yanardag & Vishwanathan (0155: Graphs in
12K datasets each graph represents a discussion thread where nodes correspond users and there is an edge between two nodes if one replied to the other's comment. The task is to identify which subreddit a given graph came from. On these datasets GCN outperforms other models.
QM9 Gilmer al. (2017) Wu al. 2018): Graphs in the QM9 dataset each represent chemicals with 18 atoms. Regression targets represent chemical properties of the molecules.
B. PERFORMANCE OF LEGSNET ON SOCIAL NETWORK DATASETS
Table S2 shows that our model outperforms other GNNs on some biomedical benchmarks and that it pertorms comparably on social network datasets. Out of the six social network datasets, ignoring the fixed scattering model GS-SVM, which has been hand tuned with these datasets in mind, our model outperforms both GNN models on three them, and is second best on the other three. This is at least comparable if not slightly superior performance. GraphSAGE does bit better Collab, but much worse on -iBaarinary and Reddit-Binarg GCN does a bit better on Reddit-Mulll but worse Collab, IMDB-Bin -Bnary, and MaddittBBiary...
LEGSNet has significantly fewer parameters and achieves comparable or superior accuracy on com- mon benchmarks. Even when our method shows comparable results, and definitely when it outper- forms other GNNs, we believe that its smaller number of parameters could be useful in applications with limited compute or limited training examples.
15

---

Under review as a conference paper at ICLR 2021
Table S1: Dataset statisties, diameter, nodes, edges, clustering coefficient averaged over all graphs.
Split into bio-chemical and social network types.
# Graphs
Classes
Diameter
Nodes
Edges
Clust. Coeff
DD
1178
2
19.81
284.32
715.66
0.48
ENZYMES
600
6
10.92
32.63
62.14
0.45
MUTAG
188
2
8.22
17.93
19.79
0.00
NCII
4110
2
13.33
29.87
32.30
0.00
NCI109
4127
2
13.14
29.68
32.13
0.00
PROTEINS
1113
2
11.62
39.06
72.82
0.51
PTC
344
2
7.52
14.29
14.69
0.01
COLLAB
5000
3
1.86
74.49
2457.22
0.89
MMDB-BINARY
1000
2
1.86
19.77
96.53
0.95
IMDB-MULTY
1500
3
1.47
13.00
65.94
0.97
REDDIT-BINARY
2000
2
8.59
429.63
497.75
0.05
REDDIT-MULTI- 12K
11929
11
9.53
391.41
456.89
0.03
REDDIT-MULTI-SK
4999
5
10.57
508.52
594.87
0.03
Table S2: Mean + std. over 10 test sets bio-chemical and social datasets.
LEGSS RDF
LDGS FCN
LDGS FIXED
GCN
GraphS.AGE
GAT
GIN
GS SVM
Bascline
DD
72.58 + 1.35
72.07
2.37
69.09
4.82
82.83-2
66.37 4,45
68.50
3.62
42.37
4.32
72.66 + 4.04
75.98 2.81
ENZYMES
36.33
4.50
38,50
8.18
32.33
5.04
31.33 1 6.89
1583 9.10
25.83 1 4.73
36.33
4,81
27.33 1 5.10
20.50
5.99
MUTAG
33.51
9.85
81.54
11.24
79.30
9.66
81.43
11.64
9.44
83.57
85.09
7.44
79300
4.26
72.47
1.46
2.05
PROTEINS
70.89 1.91
71.06
3.17
67330
7400
1.550
73.23
75.02
4.55
2.67
73.23
3.76
PIC
$7.36 5:5.4
56.92
= 9.36
54.31
6.92
56.34
10.29
58.22 9.13
55.30
55.32
8.0
56.96
7.09
56.71
5,54
COLLAB
1.995
75.40
72.4
1:941:10 1.70
73.80
1.73
76.12 1.58
72.88
1: 2.06
02.98
1
74.54
64.76
1.66::63 283
IMDE-BINARY
64:00+3.45
64.50
3.50
64
(4.30--3.66
3.88
4740 6.24
46
4.03
45.50
3.14
64.20
5.77
66.70 + 1.3
47.30
5.57
IMDB-MULTI
1.901.33.1
40.13
2.77
41 3.19
39.33 1 3.13
39.73 1.45
39.73
361
38.67
3.9
42.13
2,53
39.53
3.63
REDDIITBINARY
REDDIEMULTI-SK
3.06
C TRAINING DETAILS
We train all models for a maximum 1000 epochs with an initial learning rate of 1e-4 using the ADAM optimizer (Kingma & Ba 2015). We terminate training if validation loss does not improve for 100 epochs testing every 10 epochs. Our models are implemented with Pytorch Paszke et al.) 2019) and Pytorch geometric. Models
variety of hardware resources. For all models we use = normalized statistical moments for the node to graph level feature extraction and m 16 diffusion scales line with choices in Gao et 2019
C.1 CROSS VALIDATION PROCEDURE
For all datasets we use 10-fold cross validation with 80% training data 10% validation data and 10% test data for each model. We first split data into (roughly) equal partitions. each model we take exactly one of the partitions to be the test set and one of the remaining nine to be the validation set. We then train the model on the remaining eight partitions using the cross-entropy loss on the validation for early stopping checking every ten epochs. For each test set, of the nine models trained with that test set. We then take the mean and standard deviation across these test set scores to average out any variability in the particular split chosen. This results in 900 models trained on every dataset. With mean and standard deviation over 10 ensembled models each with separate test set.
majority voting
D ENSEMBLING EVALUATION
Recent work by Min et al. 2020 combines the features from fixed scattering transform with GCN network, showing that this has empirical advantages in semi-supervised node classification, and theoretical representation advantages a standard Kipf & Welling (2016) style GCN. We
16

---

Under review as a conference paper at ICLR 2021
Table S3: Mean + std. over test selection on cross-validated LEGS-RBF Net with reduced training set size.
Train, Val, Test %
80%, 10%, 10%
70%, 10%. 20%
40%, 10%, 50%
20%, 10%, 70%
COLLAB
75.78 + 1.95
75.00 + 1.83
74.00 + 0.51
72.73 + 0.59
DD
72.58 3.35
70.88 2.83
69.95 + 1.85
69.43 + 1.24
ENZYMES
36.33 4.50
34.17 3.77
29.83
23.98 3.32
IMDB-BINARY
14.90:3..44
63.00 2.03
63.30 1.27
57.67 + 6.04
IMDB-MULE
41.93 3.01
40.80 1.79
41.80
36.83
MUTAG
33.51 4.34
33.51 1.14
33.52 1.26
33.51
NCI1
74.26 1.53
74.38 1.38
72.07 0.28
70.30 +.72
NCT109
72.47 + 2.11
72.21 + 0.92
+
+
PROTIENS
70.89 3.91
69.27 1.95
69.72 0.27
68.96 1.63
PTC
57.26 + 5.54
57.83 + 4.39
54.62 3.21
55,45 2.35
REDDIT-BINARY
86.10 2.92
86.05 2.51
85.15 1.77
83.71 + 0.97
REDDIT-MULTI-12 12K
38.47 1.07
38.60 + 0.52
37.55 + 0.05
36.65 + 0.50
REDDIT-MULTI-SK
47.83 2.61
47.81 1.32
46.73 + 1.46
44.59 + 1.02
Table S4: Test set mean squared error on CASP GDT regression task across targets over non- overlapping test sets.
LEGS-RBF
LEGS-FCN
LEGS-FIXED
GCN
GraphSAGE
GIN
Baseline
10800
197.68 1 34.29
164.22 :: 10.28
206.20
28.46
314.90 29.66
230.45 + 79.72
262.35 : 66.88
414.41 1: 26.96
10868
8.12
127.71
14.26
178.45
272.14
21.96
27.26
411.98
57.39
IU669
106.69
1: 9.97
132.12
31.37
14.16 134,48
104.47
317.22 1 12.75
244.38
= 40.58
217.02 - 57.01
393.12
1: 48.70
10872
144.11
23.63 148.20
8.25
293.96
221.13
28.74
374.48
33.70
10879
89.00
44.94
80.14
16.21
64.63
15.92
309.23 + 69.40
172.1 + 73.07
147.77 + 15.72
364.79 + 144.32
10900
10.78
25.41
158.56
9.87 150.70
254.11
79.99
10912
113.00 + 22.31
27.35 169.55
8.53
227.17
22.11
192.28
39.45
271.30 + 28.
406.25
31.42
10920
80.46
14.98
136.94
36.43
84.83
19.70
361.19
71,25
261.2
59.67
191.86
37.85
398.22
4: 25.60
1091
187 89 + 46.15
165.97 + 42.39
142.97
27.09
382.69
20.27
260
+ 16.09
207.19
+ 24.84
363.92
+ 35.79
10922
254.83
91.28
110.54
43.99
227.73
26.41
366.72
8.10
290.71
7.22
130.46
11.64
419.14 45.49
J092
188.55
1. 11.10
167.53
22.01
137.21
7.43
371.31
1 9.90
233.78
84.95
254.38
11 47.21
393.03
1: 24.93
10944
146.59 8.41
138.67
50.36
245.79
58.16
263.03
9.43
199.40
51.11
157.90
2.57
404.12 40.82
Table S5: Quantified distance between empirically observed enzyme class exchange preferences Cuesta al. (2015) and the class exchange preferences inferred from LEGS-FIXEE LEGS-FCN, and GCN. We measure the cosine distance between the graphs represented by the chord diagrams in Figure? As before, the self-affinities were discarded. LEGS-Fixed reproduces the exchange preferences the best, but LEGS-FCK FCN still reproduces well and has significantly better classification accuracy.
LEGS-FIXED LEGS-FCN
GCN
0.132
0.146
0.155
ensemble the learned features from a learnable scattering network LEGGSFCNN with those of GCN and compare this ensembling fixed scattering features with GCN as Min et al. (2020), as well as the solo features. Our setting is slightly different in that we use the GCN features from pretrained networks, only training small 2-layer ensembling network on the combined graph level features. This network consists of batch norm layer, 128 width fully connected layer, leakyReLU ac- tivation, and a final classification layer down to the number classes. In Table S6 we see that combining GCN features with fixed scattering features in LEGS-FIXED learned scattering fea- tures in LEGS-FCN always helps classification. Learnable scattering features help more than fixed scattering features overall and particularly in the biochemical domain.
17

---

Under review as a conference paper at ICLR 2021
Table S6: Mean + standard deviation test set accuracy on biochemical and social network datasets.
GCN
GCN-LEGS-FIXED
GCN-LEGS-FCN
DD
67.82 3.81
74.02 2.79
73.34 + 3.57
ENZYMES
31.33 6.89
31.83 + 6.78
35.83 5.57
MUTAG
79.30 + 9.66
82.46 + 7.88
83.54 + 9.39
NCII
60.80
70.80 2.27
72.21 + 2.32
NCI109
61.30
68.82 = 1,80
69.52 + 1.99
PROTEINS
74.03
73.94 = 3.88
74.30 3.41
PTC
56.34 10.29
58.11 + 6.06
56.64 + 7.34
COLLAB
7
76.60 1.75
75.76 + 1.83
IMDB-BINARY
47,40 1:40+:620 6.24
65.10 3.75
65.90 + 4.33
IMDB-MULTI
39.33 3.13
39.93 2.69
39.87 2.24
REDDIT-BINARY
81.60 + 2.32
86.90 1.90
87.00 2.36
REDDIT-MULTI- 12K
42.57 + 0.90
45.41 1.24
45.55 + 1.00
REDDIT-MULLT-SK
52.79 2.11
53.87 + 2.75
53.41 4 3.07
Table S7: Mean + std. over four runs mean squared error over 19 targets for the QM9 dataset, lower is better.
LEGS-FCN
LEGS-FIXED
GCN
GraphSAGE
GIN
Baselinc
Target
0.749 0.025
0.761 + 0.026
0.776 + 0.021
0.876 + 0.083
0.786 + 0.032
0.985 + 0.020
Target
0.158
0.014
0.024
0.295
0.013
Target
0.830 0.016
0.856 + 0.026
0.051
0.961
0.057
0.033
0.027
Target
0.511 + 0.012
0.508 + 0.005
0.549 + 0.010
0.688 + 0.216
0.555 + 0.006
0.805 + 0.025
Target
0.587 t: 0.007
0.587 + 0.006
0.609 t 0.009
0.755 + 0.177
0.613 + 0.013
0.792 + 0.010
Target 5
0.646 + 0.013
0.674
0.889 + 0.014
0.882 = 0.118
0.699 = 0.033
0.833 = 0.026
Target 6
0.018 + 0.012
-000++00011
0.099 0.011
0.321 + 0.454
0.012 0.006
0.468 + 0.005
Target
0.017 = 0.005
0.024
0.015
0.532 - 0.405
= 0.005
0.379 = 0.013
Target
0.017 0.005
0.024 0.008
Target
0.017 + 0.005
0.024 0.008
0.368
+ 0.015
0.532 + 0.404
0.015 0.005
0.378 = 0.013
Target 10
0.017 +:: 0.005
0.024 1.000 0.008
0.368
1 0.015
0.533 1 0.404
0.015 = 0.005
0.380 = 0.014
Target II
0.254 = 0.013
0.279 0.023
0.548
+ 0.023
0.617 = 0.282
0.294 = 0.003
0.631 = 0.013
Target
+ 0.014
0.033 + 0.010
0.215 0.009
0.356 - 0.437
0.002
0.478
Target
0.033 0
0.214 0.010
0.356 - 0.438
- 0.002
0.478 1 0.014
Target 0.033
0.010
0.213 0.009
0.355 0.438
Target 15
0.036 0.014
0.036
0.219 + 0.009
0.359 = 0.436
0.023 = 0.002
0.479 = 0.014
Target 16
0.002 1 0.002
0.001
1 0.001
0.017 1 0.034
0.012 1 0.022
0.000 - 0.000
0.033 1 0.013
Target 17
0.083 + 0.047
0.079 - 0.033
0.280 + 0.354
0.264 - 0.347
0.169 = 0.206
0.205 = 0.200
Target 0.062 - 0.005
0.176 - 0.231
0.482 + 0.753
0.470 - 0.740
0.321 = 0.507
0.368 - 0.525
18


Reviews:
[{"review_id": "U0hn8n_iCnA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a parameterization of the scatter transform on graphs and builds graph neural networks based on this parameterization. Authors also demonstrate that this scatter transform could theoretically lead to stable hidden representations of GNNs. Experimental results on biochemical datasets are conducted to support the arguments of the paper.\n\nPros:\n\n1, The connection between the recent advance of graph signal processing (i.e., the generalization of the scatter transform on graphs) and graph neural networks are inspiring and interesting.\n\n2, I agree with the motivation of designing band-pass or high-pass graph convolutional filters since the original ones used by GCNs indeed act like low-pass filters and tend to over-smooth signals on graphs. Although I am not so sure whether the scatter transform is the way to go to design data-driven/learnable graph convolutional filters which satisfy this purpose, the exploration along this line is still valuable and could inspire others in the community.\n\n3, Theorem 1 is interesting since it reveals the stability of hidden representations of the hand-crafted GNN. \n\n4, The overall paper is written in a clear manner.\n\n5, Authors provide experimental results on a wide range of datasets.\n\nCons & Questions & Suggestions:\n\n1, If I understood correctly, Theorem 1 holds for the construction shown in Eq. (1). However, it is unclear whether Theorem 1 still holds for the relaxed construction of filter banks. This part is not discussed in the paper. If the theoretical guarantees on the stability do not hold anymore, then the contribution of the proposed method will be degraded significantly.\n\n2, I’m concerned about the emphasis on significantly fewer learned parameters of the proposed model. If the performances of your proposed model are on par with or superior to other GNNs on many datasets, then having fewer parameters would be a merit. However, on common social networks, as shown in the appendix, the performance of the proposed model is worse than other basic GNNs, not mentioning the recent ones. This also links to the somewhat less convincing experimental comparison on biomedical datasets as discussed in point 6. On the other hand, it seems that there are a few options to increase the number of parameters (as discussed in point 4) which arguably could improve the model capacity.\n\n3, Some important references on the intersection of the graph signal processing and graph neural networks are missing, e.g., [1,2,3]. In particular, it would be great to discuss the relationship between your work and [2,3] which also go beyond low-pass filter on graphs by exploiting learnable spectral filters and capturing the multi-scale diffusion. Comparing the scatter transform with their approaches would help better position your contribution. \n\n4, Regarding the design choice of learning the selection index F, it is less expressive to share the selection index among different graphs. In other words, the current design of using the same set of \\theta for all graphs is less satisfying as one can imagine that some graphs may require capturing long-range dependencies (corresponding to the higher power of diffusion matrix P) whereas some may just require short-range dependencies. A better option would be designing a model that takes graph data as input and predicts the selection index. Therefore, the selection index would be graph-dependent. Moreover, compared to [2,3], if one wants to explore larger diffusion steps, the computational cost would be much higher than [2,3] since they avoid the direct computation of matrix powers by using the approximated top eigenpairs of the diffusion matrix P.\n\n5, For the aggregation part, there are quite a few simple options like max/mean/attention. It would be more convincing to include these options and conduct an ablation study to justify the design choices like the RBF aggregation.\n\n6, In the experiment section, it would be more convincing to add comparisons with some recent competitive methods, especially those motivated from the graph signal processing like [2,3]. Could you shed some light on why GraphSAGE has similar std as your method and other baselines on graph classification tasks while significantly larger std than yours and others on graph regression tasks? Also, why does the proposed LEGS-RBF have a huge performance drop on the MUTAG dataset compared to any other baselines? It seems that the proposed methods perform comparable or better than other baselines on biochemical datasets but worse on social science datasets (as shown in the appendix). Could you explain why this is the case?\n\n[1] Defferrard, M., Bresson, X. and Vandergheynst, P., 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In NeurIPS.\n\n[2] Liao, R., Zhao, Z., Urtasun, R. and Zemel, R.S., 2019. Lanczosnet: Multi-scale deep graph convolutional networks. In ICLR.\n\n[3] Luan, S., Zhao, M., Chang, X.W. and Precup, D., 2019. Break the ceiling: Stronger multi-scale deep graph convolutional networks. In NeurIPS.\n\nConclusion: Based on the above merits and issues, I am currently on the borderline and would like to hear feedback from the authors.\n\n===================================================================================================\n\nThe response and the updated version clarify and address many of my concerns regarding contributions and empirical conclusions. Overall, I lean towards acceptance.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KjHZ8ijc6w0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends geometric scattering network by relaxing its scattering construction to enable training / data-driven learning. There are three major modules in the proposed network architecture: diffusion module, scatter module, and aggregation module. They conduct experiments on two tasks: whole graph classification and graph regression.\n\nThe idea to relax geometric scattering network is novel to the best of my knowledge. However, I have the following concerns:\n\n* Why is it that LEGS only outperforms other GCN methods on biological datasets? The major advantage of LEGS compared with other low-pass filter based GCNs is that it goes beyond low frequencies and consider richer notions of regularity. Why doesn't this advantage manifest on performances on other types of graph data (e.g. social networks)?  \n\n* The result in Table 2 does not seem promising. If LEGS only performs well on graphs that exhibit certain properties, showing results on synthetic datasets would help.   \n\n* I suggest that authors should report results on larger datasets like QM9. All experiments are conducted on datasets with no more than 5000 instances. Or is that due to computational complexity and scalability issues? \n\n* What are the advantages of the proposed method when compared with scattering-GCN [1]?\nThey also address the problem of oversmoothing and scattering-GCN is also learned in a data-driven fashion. How does scattering-GCN  perform if we obtain whole-graph features by aggregating node features obtained by their method? Why is it not included in the baseline?\n\n[1] Yimeng Min, Frederik Wenkel, and Guy Wolf. Scattering gcn: Overcoming oversmoothness in graph convolutional networks. arXiv preprint arXiv:2003.08414, 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JHpZO3uubL3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post discussion period:\n-------------------------------\nI read the other reviews and the author's response. Thank you for performing these extra experiments - they make the experimental section more complete. I still feel that the technical contribution of this paper is marginal and stick to my original rating.\n\n\n-------------------------------\nSummary: \n\nThe paper suggests a new graph neural network architecture based on a recently suggested wavelet-like transform for graph data called geometric scattering transform (Gao et al., 2019; Gama et al., 2019a; Zou & Lerman, 2019).  The Geometric Scattering Transform is a fixed feature extractor with a GNN structure and fixed wavelet filters that can extract both high and low-frequency features. The resulting features were shown to be effective in various graph analysis and learning tasks. Importantly, this transform relies heavily on two fixed choices: (1) A row -stochastic diffusion matrix P= ½ (I_n +WD^{-1}) that is used to extract features (2) A specific choice of dyadic frequency bands to partition the spectrum and governs the wavelet filter bank. \n\nThe main contribution of the paper is to relax these two hard-coded choices (1-2) and suggest a mechanism that will learn them from data. Concretely, for (1) the authors suggest adding a learnable parameter \\alpha  that controls the laziness of the operator P, i.e. P=\\alpha* I_n + (1-\\alpha)WD^{-1}. For (2), the authors suggest learning (soft) frequency bands using a softmax mechanism. The authors then show that two useful properties of the original transform (stability, equivariance) are still valid The authors present results of their method on graph classification and regression tasks. The results indicate that the method performs well.\n\n \n\nStrong points:\n\nThe idea of taking a fixed architecture that works well and relaxing it to have components that are learned from data is a good idea that has worked well before.\nThe approach is principled: (1) Take a good fixed extractor (2) Relax it to better represent the data (3) show that the good properties of the original transform still hold (4) compare to the original transform. \n\nWeak points:\n\nThe main contribution of this paper, namely relaxing the laziness parameter and learning the frequency bands, is minor. I don’t think that it is sufficient for an ICLR paper.\nThe relaxation of the laziness parameter is not actually used throughout the paper since it did not work well.\nThe evaluation is rather limited, especially when considering the relatively minor novelty of the model. For example,  the authors do not compare to state of the art models such as GIN.\n\nRecommendation:\n\n Although the idea of the paper makes sense, and the method seems to work, I believe its contribution is limited.\n\n\nMinor comments:\n\n- Equation 2 is difficult to parse, please explain all notation.\n- Section 2 - it might be useful to exemplify/illustrate the transform on simple graphs. \n- Lemma 1 - why self-adjointness is important?\n- What is a nonexpansive frame?\n- When learning the F matrix - how do you make sure there are no repeating rows?\n-  “A somewhat less explored domain for GNNs is in biochemical graphs that represent molecules and tend to be overall smaller and less connected than social networks.” - I think that the datasets used in the paper are quite standard (DD, ENZYMES,...)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2zyR5NHSBEX", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a novel graph neural network-based architecture. Building upon the theoretical success of graph scattering transforms, the authors propose to learn some aspects of it providing them with more flexibility to adapt to data (recall that graph scattering transforms are built on pre-designed graph wavelet filter banks and do not learn from data). By dropping the dyadic distribution of frequencies within the wavelet bank, the proposed architecture actually learns a more suitable frequency separation among the different wavelets.\n\nStrong points:\n\nThe paper is well written and technically solid.\n\nThe idea of flexibilizing scattering transforms while retaining key theoretical contributions is a notable contribution. In particular, the idea of learning the dyadic nature of the frequency bands in the wavelet bank is a key aspect, since the frequencies in graphs are rarely evenly spaced, and thus the ability to partition the space of frequencies in other form than a dyadic one is bound to bring substantial improvements.\n\nThe numerical experiments in the paper deal with new biochemical datasets which are much more interesting and practically useful than the de-facto benchmarks of semi-supervised learning in citation networks. This is a welcome change to see actually useful applications of graph neural networks.\n\nWeak points:\n\nNo real weak points except for a common misunderstanding of the low-pass nature of filters in a graph convolutional neural network. Please, see below for details. I seriously encourage the authors to re-write parts of the abstract and introduction to avoid perpetuating the misconception that graph convolutional neural networks can only learn low-pass filters.\n\nRecommendation:\n\nThis is a very interesting contribution and, provided the clarifications that I mention below, I firmly support the acceptance of this paper into ICLR.\n\nMajor comment:\n\nThe claim that GCNs rely on low-pass graph filters is, at the very least, misleading. As a matter of fact, even a graph convolution with an order-one polynomial can be a high-pass. To see this, assume we adopt the graph Laplacian as the matrix description of the graph. If this is the case, we know that low-eigenvalues correspond to low-frequencies and high-eigenvalues to high-frequencies (according to the notion of total variation). If this is the case, then a filter like H(L) = 0*I+1*L gives a frequency response of h(lambda) = lambda which is 0 for the zero eigenvalue, and grows for larger eigenvalues. This is an example of a high-pass filter. Likewise, assume that we use the adjacency matrix. If this is the case, then the highest real eigenvalue is the one with the lowest frequency, and depending on how far we are from that eigenvalue (measured as modulus operation in the complex plane), the higher the frequency is. So, for the sake of argument, let us assume that the adjacency matrix has real eigenvalues and is normalized by the largest eigenvalue, so that all eigenvalues are contained in [-1,1]. For this situation, eigenvalues closest to 1 will be low frequencies, and eigenvalues closest to -1 will be high frequencies. Then, a filter of the form H(A) = 0.5*I - 0.5*A gives a frequency response of h(lambda) = 0.5 * (1-lambda) which has value 0 for the lowest frequency, and value 1 for the highest possible frequency. This makes such a filter a high-pass filter. As we can see, we have just constructed two filters (one for the Laplacian and one for the adjacency) where both filters are of order one, and still high-pass filters. Of course, it becomes easier to build high-pass filters if the orders of the polynomials are higher (i.e. ChebNets). Therefore, since the coefficients of the graph convolutions are learned from data, it is very hard to argue that the resulting filters will be low pass (i.e. they need not be, there is nothing preventing an order one graph filter to be a high-pass and there is no enforcement of such a constraint in the original formulation of graph convolutional neural networks).\n\nThis claim is first found in the abstract, and later repeated in the introduction. With the clarifications from the introduction, I may understand where the misinterpretation arises. Kipf's GCNs use as graph matrix a normalization of (I+A) and consider an order one polynomial with the zeroth coefficient set to zero: h*(I+A). This forces h_0 = h_1 in an order-one graph filter on the adjacency matrix, which would otherwise be written as h_0 * I + h_1 * A. By forcing h_0 = h_1, we are indeed forcing the filters to be low-pass, i.e., we always learn filters of the form h(lambda) = h*(1+lambda) where we only learn the coefficient h. However, this is a design choice of Kipf's GCNs. The general formulation of a graph convolutional neural network (see Defferrard's ChebNets, for instance), does not require h_0 = h_1 and thus can also learn high-pass filters as discussed above. Interestingly enough, even if we would consider the normalization of (I+A) as the graph matrix, but we don't force the zeroth order coefficient to be zero, we would be able to have filters of the form h_0 * I + h_1 * (I+A) which would actually allow the learned filters to be high pass (i.e. h_0 = 1, h_1 = -0.5, see above). In any case, what I mean here, is that learning low-pass filters is a self-inflicted problem by Kipf's GCNs that can be very easily avoided by just using a more general formulation of graph convolutional neural networks (for instance, Defferrard's ChebNets).\n\nIn summary, I would suggest the authors to avoid any mention that graph convolutional neural networks only learn low-pass filters. Otherwise, this would perpetuate an important misunderstanding that has been around for a while now and is misleading research efforts. In other words, yes, Kipf's 'GCN' implementation of graph convolutional neural networks can only learn low-pass filters. However, the general definition of graph convolutional neural networks (suggested in Defferrard's ChebNets, and formalized in the signal processing literature regarding GNNs) by no means implies that the learned filters are low-pass. Thus, 'graph convolutional networks' are not oversmoothers, or only learn low-pass filters. Kipf's GCN does (and it can be avoided by just using a different implementation).\n\nIn any case, this does not alter the contribution of the paper. I would just request the authors to correct this aspect to avoid perpetuating the misunderstanding surrounding graph convolutional neural networks.\n\nMinor comments:\n\n1) The authors suggest two learnable adaptations of graph scattering transforms. The parameter alpha in the graph matrix, and the scales. However, the authors find out that the parameter alpha does not contribute anything in training and is thus arbitrarily set to 1/2. I would suggest, then, that the authors focus on the scales as the learnable adaptation, and only mention in passing that alpha can be potentially trained, but that the numerical experiments in this paper show no improvement by doing so. I believe this would put the focus and draw attention to, probably, the main contribution of the paper.\n\n2) When referring to GCNs, the authors cite Kipf's paper, Veličković's paper and Abu-El-Haija. The GAT architecture in Veličkovic's paper is not a graph convolutional neural network, so it shouldn't be cited here (GATs are, probably, the first case of a popular graph neural network architecture that is not convolutional). Also, I believe it is unfair not to cite the two main contributions to GCNs: Bruna's 'Deep Spectral Networks' paper from ICLR 2014, and Defferrard's 'ChebNets' from NeurIPS 2016. (Omission of ChebNets would explain the misunderstanding with respect to the graph convolutional neural networks being low-pass filters).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple yet powerful generalisation of graph scattering transforms that allows a flexible scale dilation structure, retaining the stability guarantees of dyadic transforms. Experiments with strong empirical performance are reported on a variety of biochemical tasks.

Reviewers acknowledged the soundness of the approach as well as the quality of the empirical evaluation, but also raised some concerns about lack of novelty. Ultimately this AC believes that, although this work solidifies Graph Scattering Transforms as a good alternative to GNNs on certain structured physical domains, it provides little advancements on the theory front. Unfortunately not all good papers can be accepted, and therefore the AC recommends rejection at this time, encouraging a resubmission.
Decision: Reject