Title: Rule Based Learning with Dynamic (Graph) Neural Networks
Venue: NeurIPS 2024
Accepted: no
Keywords: graph neural networks

dynamic neural networks

rule based learning
Abstract: A common problem of classical neural network architectures is that additional information or expert knowledge cannot be naturally integrated into the learning process.
  To overcome this limitation, we propose a two-step approach consisting of (1) generating formal rules from knowledge and (2) using these rules to define  rule based layers -- a
  new type of dynamic neural network layer.
  The focus of this work is on the second step, i.e., rule based layers that are designed to dynamically arrange learnable parameters in the weight matrices and bias vectors for each input sample following a formal rule.
  Indeed, we prove that our approach generalizes classical feed-forward layers such as fully connected and convolutional layers by choosing appropriate rules.
  As a concrete application we present rule based graph neural networks (RuleGNNs) that are by definition permutation equivariant and able to handle graphs of arbitrary sizes.
  Our experiments show that RuleGNNs are comparable to state-of-the-art graph classifiers using simple rules based on the Weisfeiler-Leman labeling and pattern counting.
  Moreover, we introduce new synthetic benchmark graph datasets to show how to integrate expert knowledge into RuleGNNs making them more powerful than ordinary graph neural networks.

Abstract

A common problem of classical neural network architectures is that additional information or expert knowledge cannot be naturally integrated into the learning process. To overcome this limitation, we propose a two-step approach consisting of (1) generating formal rules from knowledge and (2) using these rules to define rule based layers - a new type of dynamic neural network layer. The focus of this work is on the second step, i.e., rule based layers that are designed to dynamically arrange learnable parameters in the weight matrices and bias vectors for each input sample following a formal rule. Indeed, we prove that our approach generalizes classical feed-forward layers such as fully connected and convolutional layers by choosing appropriate rules. As a concrete application we present rule based graph neural networks (RuleGNNs) that are by definition permutation equivariant and able to handle graphs of arbitrary sizes. Our experiments show that RuleGNNs are comparable to state-of-the-art graph classifiers using simple rules based on the Weisfeiler-Leman labeling and pattern counting. Moreover, we introduce new synthetic benchmark graph datasets to show how to integrate expert knowledge into RuleGNNs making them more powerful than ordinary graph neural networks.

1 Introduction

Using expert knowledge to increase the efficiency, interpretability or predictive performance of 18 a neural network is an evolving research direction in machine learning [21, 23]. Many ordinary 19 neural network architectures are not capable of using external and structural information such as 20 expert knowledge or meta-data, e.g., graph structures in a dynamic way. We would like to motivate 21 the importance of 'expert knowledge' by considering the following example. Maybe one of the 22 best studied examples based on knowledge integration are convolutional neural networks [12]. 23 Convolutional neural networks for images use at least two extra pieces of 'expert knowledge' that is: 24 neighbored pixels correlate , and the structure of images is homogeneous . The consequence of this 25 knowledge is the use of receptive fields and weight sharing. It is a common fact that the usage of 26 this information about images has highly improved the predictive performance over fully connected 27 neural networks. But what if expert knowledge suggests that rectangular convolutional kernels are 28 not suitable to solve the task? In this case the ordinary convolutional neural network architecture 29 is too static to adapt to the new information. Dynamic neural networks are not only applicable to 30 images but also to other data types such as video [25], text [10], or graphs [19]. The limitation 31 of such approaches is that expert knowledge is somehow implicit and not directly encoded in the 32

network structure, i.e., for each new information a new architecture has to be designed. Thus, our 33

goal is to extract the essence of dynamic neural networks by defining a new type of neural network 34

layer that is on the one side able to use expert knowledge in a dynamic way and on the other side 35

easily configurable. Our solution to this problem are rule based layers that are able to encode expert 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

(a) Learned weights and bias for the best model of the DHFR dataset.

(b) Learned weights for the best model of the IMDBBINARY dataset.

knowledge directly in the network structure. As far as we know, this is the first work that defines a dynamic neural network layer in this generality.

Main Idea We simplify and unify the integration of expert knowledge and additional information into neural networks by proposing a two-step approach and show how to encode given extra information directly into the structure of a neural network in a dynamic way. In the first step the extra information or expert knowledge is formalized using appropriate rules (e.g., certain pixels in images are important , only nodes in a graph of type A and B interact , some patterns, e.g., cycles or cliques, in a graph are important , etc.). In the second step the rules are used to manipulate the structure of the neural network. More precisely, the rules determine the positions of the weights in the weight matrix and the bias terms. We note that the focus of this work is on the second step as we show how to use given rules to dynamically adapt the layers. In fact, we do not provide a general instruction for deriving formal rules from given expert knowledge. In difference to ordinary network layers we consider a set W of learnable parameters instead of fixed weight matrices. The weight matrices and bias terms are then constructed for each input sample independently using the learnable parameters from W . Indeed, each learnable parameter in W is associated with a specific relation between an input and output feature of a layer. As an example consider Figure 1 where each input and output feature corresponds to a specific node in the graph. The input samples are (a) molecule graphs respectively (b) snippets of social networks and the task is to predict the graph class. Each colored arrow in the figure corresponds to a learned parameter from W , i.e., a specific relation between two atoms in the molecules or two nodes in the social network. Considering only the weights with the largest absolute values, see the second image of (a) respectively (b), our approach has learned how to propagate information from outer atoms to the rings respectively from the nodes to the 'important' nodes of the social network. This example shows several advantages of our approach: (1) rule based layer type has a much more flexible structure than layers in classical architectures and allow to deal with arbitrary input dimensions , (2) the layers are easily integrable into existing architectures, and (3) the learned parameters, hence the model, is interpretable and can possibly be used to extract new knowledge from the data or to improve the existing rules.

Main Contributions We define a new type of neural network layer called rule based layer. This 64 new layer can be integrated into arbitrary architectures making them dynamic, i.e., the structure 65 of the network changes based on the input data and predefined rules. We prove that rule based 66 layers generalize classical feed-forward layers such as fully connected and convolutional layers. 67 Additionally, we show that rule based layers can be applied to graph classification tasks, by introducing 68 RuleGNNs, a new type of graph neural networks. In this way we are able to extend the concept of 69 dynamic neural networks to graph neural networks together with all the advantages of dynamic neural 70 networks, e.g., that RuleGNNs are by definition permutation equivariant and able to handle graphs 71

of arbitrary sizes. Considering various real-world graph datasets, we demonstrate that RuleGNNs 72 are competitive with state-of-the-art graph neural networks and other graph classification methods. 73 Using synthetic graph datasets we show that 'expert knowledge' is easily integrable into our neural 74 network and also necessary for classification 1 75

76

77

78

79

80

The rest of the paper is organized as follows: We introduce the concept of rule based layers in Section 2 and prove in Section 3 that rule based layers generalize fully connected and convolutional layers. In Section 4 we present RuleGNNs and apply them in Section 5 to different benchmark datasets and compare the results with state-of the art graph neural networks. Finally, we discuss limitations, related work and conclude the paper in Section 6.

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

2 Rule Based Learning

Introducing the concept of rule based learning we first present some basic definitions followed by the formal definition of rule based layers.

Preliminaries For some n ∈ N we denote by [ n ] the set { 1 , . . . , n } . A neural network is denoted by a function f ( -, Θ) : R n -→ R m with the learnable parameters Θ . We extend this notation introducing an additional parameter R , that is the set of formal rules R = { R 1 , . . . , R k } . The exact definition of these rules is given in the next paragraph. Informally, a rule R is a function that determines the distribution of the weights in the weight matrix or the bias vector of a layer. A rule R is called dynamic if it is a function in the input samples x ∈ R n otherwise it is called static . An example of a static rule is the one used to define convolutional layers, see Proposition 2. An example of a dynamic rule can be found in Section 4. In our setting, a neural network is a function f ( -, Θ , R ) : R ∗ -→ R ∗ that depends on a set of learnable parameters denoted by Θ and some rule set R derived from expert knowledge or additional information. The notation ∗ in the domain and codomain of f indicates that the input and output can be of arbitrary or variable dimension. As usual f is a concatenation of sub-functions f 1 , . . . , f l called the layers of the neural network. More precisely, the i -th layer is a function f i ( -, Θ i , R i ) : R ∗ -→ R ∗ where Θ i is a subset of the learnable parameters Θ and R i is an element of the ruleset R . We call a layer f i static if R i is a static rule and dynamic if R i is a dynamic rule. The input data is a triple ( D , L , I ) , where D = { x 1 . . . , x k } with x i ∈ R ∗ is the set of examples drawn from some unknown distribution. The labels are denoted by L = ( y 1 . . . , y k ) with y i ∈ R ∗ and I is some additional information known about the input data D . This can be for example knowledge about the graph structure, node or edge labels, importance of neighborhoods and many more. One main assumption of this paper is that I can be used to derive a set of static or dynamic rules R . Again we would like to mention that we concentrate on the analysis of the effects of applying different rules R and not on the very interesting but also wide field of deriving the best rules R from I , see some discussion in Section 6. Nonetheless, we always motivate the choice of the rules derived by I .

Rule Based Layers We now give a formal definition of rule based layers. Given some dataset ( D , L , I ) defined as before and the rule set R derived from I , the task is to learn the weights Θ of the neural network f to predict the labels of unseen examples drawn from an unknown distribution. Our contribution concentrates on single layers and is fully compatible with other layers such as linear layers, convolutional layers Hence, in the following we restrict to the i -th layer f i ( -, Θ i , R i ) : R ∗ -→ R ∗ of a network f . For simplicity, we assume i = 1 and omit the indices, i.e., we write f := f i , Θ := Θ i and R := R i . The forward propagation step of the rule based layer f which will be a generalization of certain known layers as shown in Section 3 is as follows. Fix some input sample x ∈ D with x ∈ R n . Then f ( -, Θ , R ) : R n -→ R m for n, m ∈ N is given by

Here σ denotes an arbitrary activation function and W R W ( x ) ∈ R m × n rsp. b R b ( x ) ∈ R m is some weight matrix rsp. weight vector depending on the input vector x and the rule R . The set Θ := { w 1 , . . . , w N , b 1 , . . . , b M } consists of all possible learnable parameters of the layer. The parameters { w 1 , . . . , w N } are possible entries of the weight matrix while { b 1 , . . . , b M } are possible entries of the bias vector. The key point here is that the rule R determines the choices and the positions of the weights from Θ in the weight matrix W R W ( x ) and the bias vector b R b ( x ) depending on the input

134

̸

sample x . More precisely, not all learnable parameters must be used in the weight matrix and the 122 bias vector for some input sample x . Note that for two samples x, y ∈ D of different dimensionality, 123 e.g., x ∈ R n and y ∈ R k with n = k the weight matrices W R W ( x ) and W R W ( y ) also have different 124 dimensions and the learnable parameters can be in totally different positions in the weight matrix. 125 This is where the rules R and their associated rule functions, see (2) below, come into play. 126

127

128

Given the set of learnable parameters Θ := { w 1 , . . . , w N , b 1 , . . . , b M } , for each input x ∈ R n the rule R induces the following two rule functions

where m ∈ N is the output dimension of the layer that can also depend on x . In the following we 129 abbreviate R W ( x )( i, j ) by R W ( x, i, j ) and R b ( x )( i ) by R b ( x, i ) . We note that for simplicity we 130 assume that the matrix and vector indices start at 1 and not at 0 . Using the associated rule functions (2) 131 we can construct the weight matrix resp. bias vector by defining the entry ( i, j ) ∈ R m × n in the i -th 132 row and the j -th column of the weight matrix W R ( x ) ∈ R m × n via 133

and the entry at position k in the bias vector b R b ( x ) ∈ R m by

Summarizing, the rule based layer defined in (1) is a standard feed-forward layer with the difference 135 that the weights in the weight matrix and the bias vector are determined by a predifined rule R . 136 In fact, weight matrix and bias vector depend on the input and can contain shared weights. More 137 precisely, the rule controls the connection between the i -th input and the j -th output feature in the 138 weight matrix. A rule R is called static if it is independent of the input x ∈ D , i.e., R ( x ) ≡ R ( y ) 139 for all inputs x, y ∈ R ∈ D otherwise it is called dynamic . We call a rule based layer as defined in (1) 140 static if it is based on a static rule R and dynamic otherwise. We will show in Section 3 that rule 141 based layers generalize known concepts of neural network layers for specific rules R . In fact, we 142 show that fully connected layers and convolution layers are static rule based layers. Examples of 143 dynamic rule based layers are given later on in Section 4. The back-propagation of such a layer can 144 be done as usual enrolling the computation graph of the forward step and applying iteratively the 145 chain rule to all the computation steps. We will not go into the details of this computation as it is 146 similar to many other computations using backpropagation with shared weights. For the experiments 147 we use the automatic backpropagation tool of PyTorch [16] which fully meets our requirements. 148

149

150

151

152

153

154

155

156

157

158

159

160

161

Assumptions and Examples Rule based learning relies on the following two main assumptions: A 1) There is a connection between the additional information or expert knowledge I and the used rule R and A 2) The distribution of weights given by the rule R in the weight matrix W R ( x ) improves the predictive performance or increases the interpretability of the neural network. As stated before we concentrate on the second assumption and consider different distribution of weights in the weight matrix given by different rules. In fact, we assume without further consideration that it is possible to derive a meaningful ruleset R from the additional information or expert knowledge I . For example if the dataset consists of images we can derive the 'informal' rule that neighboured pixels are more important than pixels far away and in case of chemical data there exists, e.g., the ortho-para rule for benzene rings that makes assumptions about the influence of atoms for specific positions regarding the ring. This rule was already learned by a neural network in [28]. It is another very interesting task which is beyond the scope of this work how to formalize these 'informal' rules or to learn the 'best' formal rules from the additional information I .

162

163

164

In the following sections we focus on the concept of rule based layers and therefore for simplicity and space reasons only consider the rule function of weight matrices. The rule function associated with the bias vector can be constructed similarly. For simplicity, we write R instead of R W .

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

3 Theoretical Aspects of Rule Based Layers

In this section we provide a theoretical analysis of rule based layers and show that they generalize fully connected and convolutional layers. More precisely, we define two static rules R FC and R CNN and show that the rule based layer as defined in (1) based on R FC is a fully connected layer and the rule based layer based on R CNN is a convolutional layer. All the proofs can be found in the Appendix A.

be a rule based layer of a neural network as defined in (1) (without bias term) with learnable parameters Θ = { w 1 , . . . , w n · m } and y = f i ( x ) is the result of the first i -1 layers. Then for the rule function R FC ( x ) : [ m ] × [ n ] → [ m · n ] defined for all inputs x as follows

the rule based layer f is equivalent to a fully connected layer with activation function σ .

Proposition 1 shows that rule based layers generalize fully connected layers of arbitrary size without bias vector and can be easily adapted to include the bias vector. Hence, this shows that rule based layers generalize arbitrary fully connected layers. Moreover, fully connected layers are static rule based layers as the rule R FC is static because it does not depend on the particular input x .

be a rule based layer of a neural network as defined in (1) (without bias term) and W i = { w 1 , . . . , w N 2 } be the set of learnable parameters. Then for the rule function R CNN : [( n -N +1) · ( m -N +1)] × [ n · m ] → [ N 2 ] defined by

the rule based layer f is equivalent to a convolution layer with quadratic kernel of size N ( N < n , N < m ) and a stride of one over a two-dimensional image of size n × m (without padding and bias vector) with activation function σ . The notation a//b denotes the integer division of two integers a and b .

Proposition 2 shows that rule based layers generalize 2D-image convolution without padding and bias term. By adaption of the rule function it is possible to include the bias vector and padding. Moreover, the result can be generalized to higher dimensions kernels, non-quadratic kernels and arbitrary input and output channels. Hence, rule based layers also generalize arbitrary convolutional layers. Convolutional layers are static rule based layers as the rule R CNN is static because it is independent of the input. The following result is a direct implication from Propositions 1 and 2.

Theorem 1 Rule based layers generalize fully connected and convolutional feed-forward layers. Moreover, both layers are static rule based layers.

We claim that also other types of feed-forward layers can be generalized by rule based layers using appropriate rule functions. Because of space limitations we would rather present a specific application of dynamic rule based layers on graphs.

4 Rule Based Learning on Graphs

One of the main advantages of rule based layers as introduced in this work is that they give rise to a dynamic neural network architecture that is freely configurable using different rules. In fact, the network is independent of the dimension and structure of the input samples. Hence, a natural application of our approach is graph classification. We would like to emphasize that graph classification is only one of many possible applications of rule based layers. Other possible applications are node classification, regression tasks, graph embeddings or completely different data-structures.

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

Graph Preliminaries By a graph we mean a pair G = ( V, E ) with V denoting the set of nodes of G and E ⊆ {{ i, j } | i, j ∈ V } the set of edges. We assume that the graph is undirected and does not contain self-loops or parallel edges. In case that it is clear from the context we omit G and only use V and E . The distance between two nodes i, j ∈ V in a graph, i.e., the length of the shortest path between i and j , is denoted by d ( i, j ) . A labeled graph is a graph G = ( V, E ) equipped with a function l : V →L that assigns to each node a label from the set L ⊆ N . In this paper the input samples corresponding to a graph ( V, E ) are always vectors of length equal to | V | . In particular, the input vectors can be interpreted as signals over the graph and each dimension of the input vector corresponds to the one-dimensional input signal of a graph node.

4.1 Graph Rules

The example on molecule graphs in Figure 2 and Appendix A.4 motivates the intuition behind different graph specific rules that can be used to define a graph neural network based on rule layers. The underlying general scheme to define a rule based layer on graphs is as follows: Let G = ( V, E ) be a graph and l : V →L a permutation equivariant labeling function of the nodes, i.e., for some permutation π of V it holds l ( π ( V )) = π ( l ( V )) . Assuming that input and output dimension of the layer is equal to | V | the rule functions R as defined in (2) map each pair of nodes ( i, j ) ∈ V × V to an integer which is the index of the learnable parameter in the set of all learnable parameters. The mapping is injective based on the labels l ( i ) , l ( j ) and an additionally defined shared property between the nodes i and j . Examples for such shared properties can be the distance between i and j , the type of the edge connecting i and j or the information, that i and j are in one circle. As an example R Mol as defined in Appendix A.4 is induced by the permutation equivariant function l that maps each node to its atom label and the shared property between two nodes is the type of the edge connecting the nodes or the absence of an edge. Besides R Mol the simple rule that is based on the given node labels in this paper we focus on three different rule based layers for graphs.

Proposition 3 Let π be some permutation of the nodes of G = ( V, E ) and x its corresponding input vector. If R permutation equivariant, i.e., R ( π ( x ))( i, j ) = R ( x )( π ( i ) , π ( j )) then the rule based layer is also equivariant under node permutations, i.e., f ( π ( x ) , Θ , R Mol ) = π ( f ( x, Θ , R Mol )) .

Weisfeiler-Leman Rule Recent research has shown that the Weisfeiler-Leman labeling is a powerful tool for graph classification [18, 14, 2, 22]. Thus, we propose to use Weisfeiler-Leman labels as one option to define the rule based layer for graph classification. The Weisfeiler-Leman algorithm assigns in the k -th iteration to each node of a graph a label based on the structure of its local k -hop neighborhood, see [18]. Let l ( v ) be the result of the k -th iteration of the Weisfeiler-Leman algorithm for some node v ∈ V . Then the Weisfeiler-Leman Rule R WL k,d assigns to each node pair ( i, j ) an integer or zero based on the Weisfeiler-Leman labels l ( i ) , l ( j ) and the distance between the nodes i and j . The result is zero if the distance between i and j is not between 1 and d . Note that we are not restricted to look at consecutive distances from 1 to d . It is also possible to look at certain distances only if the expert knowledge suggests it. In fact, ( i, j ) and ( k, l ) are mapped to the same integer if and only if l ( i ) = l ( k ) , l ( j ) = l ( l ) and the distance between i and j is equal to the distance between k and l . The layer defined by this rule is related to ordinary message passing but messages can pass between nodes of arbitrary distances. For computational reasons in the experiments we restrict the maximum number of different Weisfeiler-Leman labels considered by some bound L . We relabel the most frequent l -1 labels to 1 , · · · , l -1 and set all other labels to l . The corresponding layer is denoted by f WL k,d,L .

Pattern Counting Rule Beyond labeling nodes via the Weisfeiler-Leman algorithm, it is a common approach to use subgraph isomorphism counting to distinguish graphs [3]. This is in fact necessary as the 1-Weisfeiler-Leman algorithm is not able to distinguish some types of graphs, for example circular skip link graphs [4] and strongly regular graphs [2, 3]. Thus, we propose the pattern counting rule and show in Section 5 that RuleGNNs based on this rule are able to perform well on synthetic benchmark datasets while message passing models based on the Weisfeiler-Leman algorithm fail. In general, subgraph isomorphis counting is a hard problem [5], but for the real-world and synthetic benchmark graph datasets that are usually considered, subgraphs of size k ∈ { 3 , 4 , 5 , 6 } can be enumerated in a preprocessing step in a reasonable time, see Table 5. Given a set of patterns, say P , we compute all possible embeddings of these patterns in the graph dataset in a preprocessing step. Then for each pattern P ∈ P and each node i ∈ V we count how often the node i is part of an

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

embedding of P . Using those counts we define a labeling function l : V →L . Two nodes i, j ∈ V are mapped to the same label if and only if their counts are equal for all patterns in P . Patterns that are often used in practice are small cycles, cliques, stars, paths, etc. The Pattern Counting Rule R P d assigns each node pair ( i, j ) an integer or zero based on the values of l ( i ) , l ( j ) and the distance between i and j . As for the Weisfeiler-Leman Rule we restrict the maximum number of different labels to some number L . The corresponding layer is denoted by f P d,L .

Summary Rule The summary rule R N Out can be used as the output layer as its output is a fixed dimensional vector of size N ∈ N independent of the size of the input data and the output is invariant under node permutations. Again, let l : V →L be a function that maps each node of a graph to some integer. Then the summary rule R N Out assigns each pair ( n, i ) with i ∈ V and n ∈ [ N ] an integer or zero based on n and l ( i ) . In fact, for each element of L the rule defines n different learnable parameters. The corresponding layer is denoted by f R N Out .

All the above rules define dynamic rule based neural network layers because the weight matrix and bias terms defined by the rules depend on the input vectors x corresponding to different graphs. Note that the layers defined by the above rules are permutation equivariant as the node labeling function l used to define the rule is equivariant under node permutations. Thus, using the layers corresponding to the above defined rules we can build a graph classification architecture that by definition does not depend on the order of the nodes in the input graphs. Moreover, a layer is able to pass information between nodes of arbitrary distances in the graph. Thus, as shown in the experiments below, it is not necessary to use deep networks to achieve good performance on the real-world benchmark datasets.

4.2 Rule Graph Neural Networks (RuleGNNs)

The layers derived from the above rules are the building blocks of the RuleGNNs. Each RuleGNN is a concatenation of different rule based layers from Weisfeiler-Leman rules and pattern counting rules followed by a summary rule using arbitrary activation functions. To define the learnable parameters of the bias term we also use the summary rule. The input of the network is a signal x ∈ R | V | corresponding to a graph G = ( V, E ) . We note that for simplicity we focus on one-dimensional signals but also multidimensional signals, i.e., x ∈ R | V |× d are possible. The output of the network is a vector of fixed size N ∈ N determined by the summary rule where N is usually the number of classes of the graph classification task. The output can be also used as an intermediate vectorial representation of the graph or for regression tasks.

5 Experiments

We evaluate the performance of RuleGNNs on different real-world and synthetic benchmark graph dataset and compare the results to the state-of-the-art graph classification algorithms. For comparability and reproducibility of the results, also with future algorithms, we make use of the experimental setup from [7]. That means, for each graph dataset we perform a 10 -fold cross validation, i.e., we use fixed splits of the dataset into 10 equally sized parts (the splits can be found in our repository), and use 9 of them for training, parameter tuning and validation. We then use the model that performs best on the validation set and report the performance on the previously unseen test set. We train the best model 3 times and average the results on each fold to decrease random effects. The standard deviation reported in the tables is computed over the results on the 10 folds.

Data and Algorithm Selection A problem of several heavily used graph benchmark datasets like MUTAG or PTC [13] is that node and edge labels seems to be more important than the graph structure itself, i.e., there is no significant improvement over simple baselines [17]. Moreover, in case of MUTAG the performance of the model is highly dependent on the data split because of the small number of samples. Thus, in this work for benchmarking we choose DHFR, Mutagenicity, NCI1, NCI109, IMDB-BINARY and IMDB-MULTI from the TU Dortmund Benchmark Graphs repository [13] because the structure of the graphs seems to play an important role, i.e., the simple baselines presented in [17, 7] are significantly worse than the state-of-the-art graph classification algorithms. Additionally, we consider circular skip link graphs CSL [4] and constructed some new synthetic benchmark graph datasets called LongRings, EvenOddRings and Snowflakes [15] to show the advantages of RuleGNNs on more complex graph structures with given expert knowledge. For

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

more details on the datasets see Appendix A.5. For NCI1, IMDB-BINARY and IMDB-MULTI we use the same splits as in [7] and for CSL we use the splits as in [6] and a 5 -fold cross validation. We evaluate the performance of the RuleGNNs on these datasets and compare the results to the baselines from [7] and [17] and the Weisfeiler-Leman subtree kernel (WL-Kernel) [18] which is one of the best performing graph classification algorithm besides the graph neural networks. For comparison with state-of-the-art graph classification algorithms we follow [7] and compare to DGCNN [27], GIN [26] and GraphSAGE [8]. Additionally, we compare to the results of some newer state-of-the-art graph classification algorithms [3, 1, 2, 22]. For the latter we use the results from the respective papers that might be obtained with different splits of the datasets.

Experimental Settings and Resources All experiments were conducted on a AMD Ryzen 9 7950X 16-Core Processor with 128 GB of RAM. For the competitors we use the implementations from [7]. For the real-world datasets we were not aware of expert-knowledge, hence we tested different rules and combinations of the layers defined in Section 4.1. More details on the tested hyperparameters can be found in Appendix A.7. We always use tanh for activation and the Adam optimizer [11] with a learning rate of 0 . 05 (real-world datasets) resp. 0 . 1 (synthetic datasets). For the real-world datasets the learning rate was decreased by a factor of 0 . 5 after each 10 epochs. For the loss function we use the cross entropy loss. All models are trained for 50 (real-world) resp. 200 (synthetic) epochs and the batch size was set to 128 . We stopped if the validation accuracy did not improve for 25 epochs.

Real-World Datasets The results on the real-world datasets (Table 1) show that RuleGNNs are able to outperform the state-of-the-art graph classification algorithms in the setting of [7] even if we add all the additional label information that RuleGNNs use to the input features of the graph neural networks (see the (features) results in Table 1). This shows that the structural encoding of the additional label information is crucial for the performance of the graph neural networks and not replacable by using more input features. Moreover, the results show that the Weisfeiler-Leman subtree kernel is the best performing graph classification algorithm on NC1, NCI109 and Mutagenicity. For IMDB-BINARY and IMDB-MULTI our approach performs worse than the state-of-the-art graph classification algorithms that are not evaluated within the same experimental setup.

Synthetic Datasets The results on the synthetic benchmark graph dataset show that RuleGNNs outperform the state-of-the-art graph classification algorithms if expert knowledge is available even in the case that mesage passing is enough to solve the task. In fact, CLS and Snowflakes are not solvable by the message passing model because they are not distinguishable by the 1-WL test. The results on LongRings show that long range dependencies can be easily captured by RuleGNNs and also dependencies between nodes of different distances as in case of the EvenOddRings dataset can be encoded by appropriate rules.

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

Interpretability of the Rule Based Layers Each learnable parameter of RuleGNNs can be interpreted in terms of the importance of a connection between two nodes in a graph with respect to their labels and their shared property (in our case the distance). In Figures 1 and6 we see how the network has learned the importance of different connections between nodes for different distances and labels.

6 Related Work, Limitations and Concluding Remarks

Dynamic neural networks have been proven to be more efficient, have more representation power and better interpretability than static neural networks [9]. Our approach can be seen as a sample dependent dynamic neural network as for each input sample the network structure is adapted. In contrast to other sample dependent dynamic neural networks [20, 24], our approach changes the layer structure based on a predefined rule instead of the whole architecture. The rule based layers of RuleGNNs use the Weissfeiler-Leman labeling algorithm and subgraph isomorphism counting which are both recently used concepts in graph classification algorithms [18, 3, 2, 1]. The challenge for graph neural networks is the heterogenicity of the input data and the lack of a fixed order of the input data. [19] proposes a dynamic neural network for graph classification that uses node and edge labels and is similar to our approach. In fact, they also show that their approach generalizes CNNs. In contrast, they do not provide a general scheme to encode expert knowledge into the network. Moreover, their approach is not able to encode long range dependencies in the graph using only one layer. There exist graph neural networks that have learned the ortho-para rule for molecules [28]. While the additional information used in these algorithms is mostly hard-coded, we are able to integrate arbitrary rules.

Limitations Input Features: So far we have only considered 1-dimensional input signals and node labels, i.e., our experimental results are restricted to graphs that have no multi-dimensional node features. Additionally, we have not considered edge features in our rules. In principle, multidimensional node features and edge labels can be handled by our approach with the cost of increased complexity. Space: For each graph we need to precompute the pairwise distances and store the positions of the weights in the weight-matrix. This is a disadvantage for large and dense graphs as we need to store a large number of positions. For dense graphs the number of positions can be quadratic in the number of nodes. Structure: To define a meaningful rule for a layer the input and output features need to be logically connected. Fortunately, this is the case for graphs but this fact can be a limitation for other structures. Combinatorics: If it is not possible to define a formal rule given some informal expert knowledge the number of possible rules that have to be tested can be very large. Thus, it is an interesting question if it is possible to automatically learn a rule that captures the expert knowledge in the best way. Implementation: As stated in [9] there is a 'gap between theoretical & practical efficiency' regarding dynamic neural networks, i.e., common libraries such as PyTorch or TensorFlow are not optimized for dynamic neural networks.

Concluding Remarks We have introduced a new type of neural network layer that dynamically arranges the learnable parameters in the weight matrices and bias vectors according to a formal rule. On the one hand our approach generalizes classical neural network components such as fully connected layers and convolutional layers. On the other hand we are able to apply rule based layers to the task of graph classification showing that expert knowledge can be integrated into the learning process. Moreover, our approach gives rise to a more interpretable neural network architecture as every learnable parameter is related to a specific connection between input and output features.

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

References

Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Li` o, Guido F. Mont´ ufar, and Michael M. Bronstein. Weisfeiler and lehman go cellular: CW networks. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 2625-2640, 2021.

Cristian Bodnar, Fabrizio Frasca, Yuguang Wang, Nina Otter, Guido F. Mont´ ufar, Pietro Li´ o, and Michael M. Bronstein. Weisfeiler and lehman go topological: Message passing simplicial networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 1026-1037. PMLR, 2021.

Giorgos Bouritsas, Fabrizio Frasca, Stefanos Zafeiriou, and Michael M. Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Trans. Pattern Anal. Mach. Intell. , 45(1):657-668, 2023.

Jin-yi Cai, Martin F¨ urer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Comb. , 12(4):389-410, 1992.

Stephen A. Cook. The complexity of theorem-proving procedures. In Michael A. Harrison, Ranan B. Banerji, and Jeffrey D. Ullman, editors, Proceedings of the 3rd Annual ACM Symposium on Theory of Computing, May 3-5, 1971, Shaker Heights, Ohio, USA , pages 151-158. ACM, 1971.

Vijay Prakash Dwivedi, Chaitanya K. Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. J. Mach. Learn. Res. , 24:43:1-43:48, 2023.

Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. ArXiv , abs/1912.09893, 2019.

William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Neural Information Processing Systems , 2017.

Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 44(11):7436-7456, 2022.

Yacine Jernite, Edouard Grave, Armand Joulin, and Tom´ as Mikolov. Variable computation in recurrent neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. Handwritten digit recognition with a backpropagation network. In David S. Touretzky, editor, Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989] , pages 396-404. Morgan Kaufmann, 1989.

Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) , 2020.

Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019 , pages 4602-4609. AAAI Press, 2019.

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

Harish G. Naik, Jan Polster, Raj Shekhar, Tam´ as Horv´ ath, and Gy¨ orgy Tur´ an. Iterative graph neural network enhancement via frequent subgraph mining of explanations, 2024.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

Till Hendrik Schulz and Pascal Welke. On the necessity of graph kernel baselines. 2019.

Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. Weisfeiler-lehman graph kernels. J. Mach. Learn. Res. , 12:2539-2561, 2011.

Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 29-38. IEEE Computer Society, 2017.

Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR 2016, Canc´ un, Mexico, December 4-8, 2016 , pages 2464-2469. IEEE, 2016.

Geoffrey G. Towell and Jude W. Shavlik. Knowledge-based artificial neural networks. Artif. Intell. , 70(1-2):119-165, 1994.

Quang Truong and Peter Chin. Weisfeiler and lehman go paths: Learning topological features via path complexes. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada , pages 15382-15391. AAAI Press, 2024.

Laura von R¨ uden, Sebastian Mayer, Katharina Beckh, Bogdan Georgiev, Sven Giesselbach, Raoul Heese, Birgit Kirsch, Julius Pfrommer, Annika Pick, Rajkumar Ramamurthy, Michal Walczak, Jochen Garcke, Christian Bauckhage, and Jannis Schuecker. Informed machine learning - A taxonomy and survey of integrating prior knowledge into learning systems. IEEE Trans. Knowl. Data Eng. , 35(1):614-633, 2023.

Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII , volume 11217 of Lecture Notes in Computer Science , pages 420-436. Springer, 2018.

Yulin Wang, Zhaoxi Chen, Haojun Jiang, Shiji Song, Yizeng Han, and Gao Huang. Adaptive focus for efficient video recognition. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 16229-16238. IEEE, 2021.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , pages 4438-4445. AAAI Press, 2018.

Zhenpeng Zhou and Xiaocheng Li. Graph convolution: A high-order and adaptive approach. arXiv: Learning , 2017.

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

H

H

H

H

C

C

H

H

H

H

C

C

H

H

C

C

C

H

H

C

C

C

A Appendix / supplemental material

A.1 Proof of Proposition 1

To show the equivalence between the two layers it suffices to show that their weight matrices coincide. In case of fully connected layers we have to show that the weight matrix W R FC ( x ) ∈ R m × n is filled with n · m distinct weights. This can be easily checked by computing W R FC ( x ) using the definition of the weight distribution based on the rule function in (3).

A.2 Proof of Proposition 2

Instead of the original two-dimensional image of size n × m we consider a reshaped vector x ∈ R n · m as our definition of rule based layers is restricted to simple vector matrix multiplication. The output vector of dimension ( n -N +1) · ( m -N +1) can then again be reshaped into a two-dimensional image of size ( n -N +1) × ( m -N +1) . Unfortunately, the reshaping makes the rule function complicated as the indices of the reshaped vector have to be mapped to the indices of the two-dimensional image.

First note that convolution with a N × N kernel corresponds to matrix-vector multiplication of a doubly block circulant matrix that is a special case of a block Toeplitz matrix. Hence, to show the equivalence between the layers we have to compare the weight matrices and show that the entries in W R CNN ( x ) ∈ R ( n -N +1) · ( m -N +1) × n · m exactly matches the entries in the block Toeplitz matrix of the same dimension that corresponds to the convolution kernel. Comparing the definition of block Toeplitz matrices with the above given rule shows that the rule exactly returns the entries of the block Toeplitz matrix. Hence, the multiplication of x with W R CNN ( x ) is equivalent to multiplication of x with the block Toeplitz matrix that is equivalent to the convolution of x with a kernel of size N × N .

A.3 Proof of Proposition 3

The proof of Proposition 3 follows directly from the definitions of the rule based layers, see (1), and the rule functions, see (2). If the order of the nodes in the graph is permuted and rule function is permutation equivariant, then the node labels are permuted accordingly. Hence, the positions of the weights in the weight matrix and the bias term are permuted in the same way as the node labels. Thus, the result of f , i.e., the multiplication of permuted weight matrix with the permuted input signal, is the same as the permutation of the result of the multiplication of the original weight matrix with the original input signal.

A.4 Example: RuleGNN for Molecule Graphs

Assume the task is to learn a property of a molecule based on its graph structure. In this example we present a RuleGNN that is a concatenation of two very simple rule based layers. The advantage of rule based layers and hence also RuleGNNs is that they encode the graph structure (in this example the structure of two molecules) directly into the neural network. Moreover, the input samples can be arbitrary molecule graphs and the output is a vector of fixed size k that encodes the property of the molecule or some intermediate vectorial representation. In this example we consider the molecule graphs of ethylene and cyclopropenylidene given in Figure 3 together with their corresponding input

526

527

528

529

530

531

532

533

534

C

6

C

5

H

1

H

2

H

3

H

4

C

3

H

1

C

5

C

4

H

2

signals x ∈ R 6 and y ∈ R 5 . The atoms of the molecules (hydrogen H and carbon C ) correspond to the nodes of a graph and the bond types ( single and double ) correspond to the edges. The atom labels and the atom bond types can be seen as additional information I that is known about the input samples. The graph nodes are indexed via integers in some arbitrary but fixed order and the atom corresponding to a graph node are given by the labeling function l : V →{ H,C } .

The RuleGNN consists of two rule based layers f 1 ( , Θ 1 , R Mol ) and f ( , Θ 2 , R Out ) with learnable parameters Θ 1 = { w 1 , . . . , w 6 } and Θ 2 = { w ′ 1 , . . . , w ′ 2 · k } and the following rule functions R Mol and R Out . For some graph G = ( V, E ) and its corresponding input signal z we define R Mol as follows:

For some graph G = ( V, E ) and its corresponding input signal z we define R Out as follows: 535

Note that R Mol and R Out are not restricted to the two molecules from above but can be applied to arbitrary molecule graphs. Indeed, applying it to molecules with atom labels different from H or C makes the rules less powerful, i.e., it should be adapted to the type of molecules. Using the definition (3) of weight distribution defined by the rule function we can construct the weight matrices W R Mol ( x ) , W R Out ( x ) for the ethylene graph and W R Mol ( y ) , W R Out ( y ) for the cyclopropenylidene graph as follows:

536

537

538

539

540

541

Combining the two rule based layers we obtain the RuleGNN and the forward propagation is given 542 by σ ( W R Out ( x ) · σ ( W R Mol ( x ) · x )) for the ethylene graph and σ ( W R Out ( y ) · σ ( W R Mol ( y ) · y )) for the 543 cyclopropenylidene graph. 544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

Note that the forward propagation of the layer corresponding to the rule R Mol is kind of a multiplication with a weighted adjacency matrix of the graph where the weights of the adjacency matrix are given by the learnable parameters, see also Figure 2. In contrast to adjacency matrices the weight matrix is not necessary symmetric. The computation graph induced by the weight matrix exactly represent the graph structure while the edge weights are shared across the network using the rule, see Figure 2. Note that the above defined rule is very flexible as also edge labels (e.g., atomic bonds) can be taken into account by increasing the size of the weight set. Moreover, it is possible to include bigger neighbourhoods, i.e., all nodes reachable by k -hops. Of course using other information of the graph (e.g., substructures (such as circles or cliques), node degrees, connections not depicted by edges) more complicated rules can be defined.

A.5 Dataset Details

In this section we provide additional details on the datasets used in the experiments. Table 3 shows an overview of the real-world datasets and Table 4 provides an overview of the synthetic datasets.

We consider the following synthetic datasets. The CSL dataset is from []. We constructed the other datasets to demonstrate the strength of our approach to encode expert knowledge into the neural network.

LongRings LongRings consists of 1200 cycles of 100 nodes each. Four nodes are labeled by 1 , 2 , 3 , 4 and all other nodes are labeled by 0 . The distance between each pair of the four nodes is exactly 25 or 50 . The label of the graph is 0 if 1 and 2 have distance 50 , 1 if 1 and 3 have distance 50 and 2 if 1 and 4 have distance 50 . There are 400 graphs for each class. The difficulty of the classification task is that information has to be propagated over a long distance. Regarding RuleGNNs this is very easy because if the expert knows that distance 50 is relevant we can define an appropriate rule.

EvenOddRings EvenOddRings consists of 1200 cycles of 16 nodes each. The nodes in each graph are labeled from 0 to 15 . The graph label is based on the labels of the nodes that have distance 8 respectively 4 to the node with label 0 . We denote them by x resp. y, z . We distinct four cases: x is even and y + z is even, x is even and y + z is odd, x is odd and y + z is even, x is odd and y + z is odd. There are 300 graphs for each class, i.e., each of the four cases. The expert knowledge we use is that the information has to be collected from nodes of distance 8 and 4 .

EvenOddRingsCount EvenOddRingsCount consists of the same graphs as EvenOddRings but the graph labels are different. For all nodes and their opposite node in the circle the sum of the labels is computed. If there are more even sums than odd sums the graph is labeled by 0 and by 1 otherwise. There are 600 graphs for each class. The expert knowledge we use is the information that only distance 8 is relevant.

Snowflakes Snowflakes is a dataset consisting of graphs proposed by [15] that are not distinguishable by the 1-WL test, see Figure 4 for an example. The dataset consists of circles of length 3 to 12 and at each circle node a graph from M 0 , M 1 , M 2 or M 3 is attached, see Figure 5 and [15] for the details. M 0 , M 1 , M 2 and M 3 are non-isomorphic graphs that are not distinguishable by the 1-WL test. One label in the circle is labeled by 1 and all other nodes are labeled by 0 . The label of the graph is determined by the graph M 0 , M 1 , M 2 or M 3 that is attached to the circle node with label 1 .

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

A.6 RuleGNNs: Runtimes

Table 5 shows more details of the RuleGNN model. In particular, we see that except for the DHFR dataset we need less than 12 epochs on average to reach the best result. This shows that our approach is very efficient and converges quickly. At the first glance the average time per epoch seems to be very high. This has two reasons. One is also mentioned in [9] that there is a gap between the theoretical and practical runtime of dynamic neural networks because the implementation in PyTorch is not optimized for dynamic neural networks. The other reason is that we parallelized the computation, i.e., we are able to run all the three runs and 10 folds in parallel on the same machine. Of course, this produces some overhead. As stated above the preprocessing times are not relevant for the experiments as they are only needed once. The third column shows the time needed to compute all the pairwise distances between the nodes of the graph. The fourth column shows the time needed to compute the node labels used for the best model. The most preprocessing time is needed for IMDB-BINARY and IMDB-MULTI because the graphs are much denser than the other datasets. For the synthetic datasets except for CSL we do not need any label preprocessing time as the original node labels are used.

A.7 RuleGNNs: Architectures and Hyperparameters

Table 6 provides an overview of the different architectures used in the experiments that achieved the best results. One advantage of our approach is that messages can be passed over long distances. Hence, except for the EvenOddRings dataset we used only one layer and the output layer. In case of NCI1, NCI109, Mutagenicity it turns out that the best model uses the Weisfeiler-Leman rule with k = 2 iterations. We restricted the number of maximum labels considered to 500 which results in 250000 learnable parameters for the weight matrix and 500 for the bias vector. For the output layer we used the bound of 50000 learnable parameters which was larger than the number of different Weisfeiler-Leman labels in the second iteration. Interestingly, for NCI1 and NCI109 the best validation accuracy was achieved if considering node pairs with maximum distance 10 . In case

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

of Mutagenicity the best model uses only node pairs with distance 3 although we also considered the hyperparameter d = 10 . We also tested different small patterns, e.g., simple cycles, but they did not improve the results. For DHFR this was different as the best model uses the pattern (simple cycles with length at most 10 ) for the output layer. We also tested the Weisfeiler-Leman rule in this case but the validation accuracy was lower. For IMDB-BINARY and IMDB-MULTI the best model uses the pattern (simple cycles with length at most 10 , triangle, edge). Note that the embedding of one edge as a pattern is equivalent to the degree of the node. We also tested the Weisfeiler-Leman rule but the validation accuracy was lower. All in all we considered many different rules from type Weisfeiler-Leman and patterns but of course we did not test all possible rules. A full list of tested hyperparameters can be found here. As a next step it would be interesting to consider more rules, rules that come from expert knowledge or also deeper architectures with more rule based layers concatenated. Regarding the number of learnable parameters we would like to mention that the number is relatively high but lots of parameters are not used in the weight matrix. Hence, it might be possible to prune the set of learnable parameters by removing those that are not used or those that have a small absolute value.

For the synthetic datasets we use 'expert knowledge' to define the rules. Hence we did not tested other rules than those in Table 6. For LongRings, EvenOddRings and EvenOddRingsCount we used the original node labels for the rule based layers. Moreover, instead considering learnable parameters for all node pairs of certain labels with distance smaller or equal to d we considered only the node pairs with distance d (denoted by 'only: d '). In case of EvenOddRings we used two layers. The first layer that considers only node pairs with distance 8 collects all the necessary information of opposite nodes. The second layer that considers only node pairs with distance 4 collects the information of the nodes that are 4 hops away from the nodes with label 0 , see also Figure 6. For CSL we used as patterns all simple cycles with length at most 10 . For the Snowflakes dataset we used the patterns cycle of length 4 and 5 and collect the information of the nodes that have pairwise distance 3 . In this way the RuleGNN is able to distinguish the graphs M 0 , M 1 , M 2 and M 3 that are not distinguishable by the 1-WL test. In the output layer we used the Weisfeiler-Leman rule with k = 2 iterations to collect the relevant information from nodes with different Weisfeiler-Leman labels.

A.8 RuleGNNs: Interpretability

One advantage of our approach is that each weight can be interpreted, i.e., we can see the relevance 638 of two nodes i, j in a graph with labels l ( i ) , l ( j ) and distance d ( i, j ) . Figure 6 shows an example of 639 the learned parameters for some synthetic dataset. Figure 1 shows an example of the relevance of the 640 weights for a graph from the DHFR dataset using the weights of the best model. Considering Figure 6b 641 we can see that in the first layer the RuleGNN passes the messages between opposite nodes as given 642 by the rule. In the second layer it has learned to collect the information from the nodes that have 643 distance 4 to the node with label 0 (dark blue node) all other connections of distance 4 have a smaller 644 weight. 645

(a) EvenOddCount

(b) EvenOddRings

(c) Snowflakes

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The theoretical and experimentally claims made in the abstract are consistent with the results presented in the paper and reflect the contributions made.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the conclusion and also in the experiments section we discuss the limitations of the approach.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each of the theoretical results we provide a complete proof (in the appendix) and a full set of assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We give all information that is needed to reproduce the experimental results and also provide the code and data which is not online.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to the code, datasplits and synthetic datasets used in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all data splits and hyperparameter choices for our algorithm in the paper. Moreover, in the code we have understandable config files that contain all the hyperparameters used in the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We use standard deviation to show the variability of the results. We do not use statistical significance tests because our main claim is not that our method is significantly better than state-of-the-art methods, but that it is an interesting new approach that is applicable to a wide range of tasks.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify the computer resources and the time needed to run the experiments in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper does not address societal impact because we present a very basic approach that is not directly applicable to any specific societal problem.

Guidelines: 855

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not use scraped datasets or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We mention all creators and original owners of code and data we use in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

958

959

960

961

962

963

964

965

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for neural network-based learning to incorporate expert knowledge in the neural network architecture by building rules and utilizing them in \"rule-based\" layers of the learned neural networks. It introduces RuleGNNs as a concrete application of the proposed method and evaluates its performance against a few other SOTA methods. Empirical studies show competitive performance of RuleGNNs compared with other alternative methods.\n\nSTRENGTHS:\n- The idea of having dynamic rule-based layers in a neural network and especially for graph neural network learning is interesting. Although some existing methods in the literature including WL labeling could be considered doing the same, the proposed method builds on top of these building blocks and extends their ideas.\n- Theoretical discussions in the paper and the assumptions behind them are clear.\n- Experimental results cover an adequate set of alternative methods.\n\nWEAKNESSES:\n- The performance of RuleGNNs is expected to heavily rely on the quality of the rules generated from additional information or domain knowledge, however, the paper solely focuses on application of such rules without adequately discussing the challenges of building quality rules and feasibility of this fundamental step in the proposed method.\n- Lack of clarity around how rules in RuleGNNs look like and how they can influence learning model parameters. \n- Experimental results are not fully discussed. For example, WL-Kernel shows superior performance in three data sets and it would have been useful to provide more insights about what data set characteristics contribute to this.\n\nQUESTIONS:\nAs stated under the weaknesses section, the main question from the reviewer is about where (i.e. for what type of data sets) one can expect RuleGNNs to outperform other alternatives methods, especially WL-Kernel which shows promising results and superior performance in a few tested data sets.\n\nLIMITATIONS:\nAuthors have adequately addressed the limitations of their work by listing the following limitations:\n- They have only considered 1 dimensional input signals and labels.\n- They have not considered graphs with multi-dimensional node features.\n- Edge features are not considered.\n- Computation and storage limitation for large/dense graphs.\nIn addition, authors have clearly discussed structure, Combinatorics, and Implementation limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel model architecture rule-based layer, which induces different parameters given different inputs. Theoretical analysis demonstrates how the proposed architecture reduces back to classical feed-forward layers, and empirical results on both synthetic and real-world data sets demonstrate that the proposed method can improve upon existing works.\n\nSTRENGTHS:\nThe idea of rule-based layers and rule-based GNN is novel and interesting.\n\nWEAKNESSES:\n- The implementation in this work may need further elaboration to make the proposed method easier to understand. \n- Empirical results may need further improvements to better support the proposed method.\n\nQUESTIONS:\n- The notations may require further explanation. Regarding the “additional information” $I$ introduced in Section 2, how do we assume the additional information can be used to derive a set of static or dynamic rules $R$? While the authors provide some rule examples in Appendix A.4, it would be better if the authors can provide some examples on the additional information as well. \n- Also, how can we actually derive rules from the “additional information”? Do we need to manually design some rules or we need to apply some learning algorithms? While the authors claim it may not be the main focus of this paper, some clarification may still be needed to better understand the proposed method. \n- I am also confused on how we obtain the parameters for the proposed rule-based GNN. Basically, compared with standard GNN models, I suppose the rules in $R$ enable us to use different parameters for different input $x$. Then do we simply learn these parameters by standard stochastic optimizers? The authors may need to explain more on that. \n- While the proposed method seems to yield significant improvements in Table 2, the improvement in Table 1 for real-world data sets on the contrary seems quite marginal. As such, are there any other advantages of the proposed method compared with baseline methods in these tables? The authors may need to add more explanations on that. \n- It seems also confusing why the authors do not explicitly mention the rules used for experiments in Table 1 and 2, which should be essential to understand how the proposed method performs well on these data sets.  \n- Moreover, no ablation studies or hyper-parameter sensitivity analysis is present in current submission. How do different rules (if they are manually set) or different learning algorithms affect the performance of proposed method? Such results and discussion are certainly needed to better understand the proposed method.\n\nLIMITATIONS:\nThe authors discuss about possible limitations in the conclusion part, and no direct negative societal impact exists for this work from my perspective.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces rule-based (dynamic) neural network layers. The basic idea is to have a common set of parameters, i.e., weights and biases, where, depending on a certain rule, only a subset of these parameters are used in the forward pass. They show that certain fully connected and convolutional layers can be regarded as a type of static rule-based neural network layer. In the remainder of the paper, the authors introduce three dynamic rules for graph classification tasks and perform experiments on synthetic and real-world datasets.\n\nSTRENGTHS:\nOverall, the concept of using rules based on expert knowledge to select different subsets of weights for various data samples or tasks seems useful and promising. This approach could offer significant benefits, such as training the same model on different tasks or on different datasets. Moreover, an approach which is able to learn on variably sized input data could be valuable on its own. The proposed rule-based layers for graph classification tasks outperform standard message-passing graph neural networks on synthetic and real-word datasets.\n\nWEAKNESSES:\nOne of my primary concerns is that the main theoretical result of the paper, Theorem 1, is not proven. Specifically, while the authors show in Prop. 1 and Prop. 2 that fully connected layers _without bias_ and convolutional layers _without bias, padding, stride of one, and quadratic kernels_ can be expressed using their proposed (static) rule-based layer, the following paragraph leading to Theorem 1 claims this can be generalized to arbitrary convolutional layers. Although this might be straightforward to prove (and could be included in the appendix), the lack of a complete and formal proof severely undermines the soundness of the submission. If Theorem 1 is intended as a summary of Proposition 1 and Proposition 2, I suggest making this explicit by clearly stating the specific types of FC and CNN layers and renaming Theorem 1 to Corollary 1, or merging Prop. 1 and Prop. 2 into Theorem 1. Moreover, while the paper introduces some mathematical framework and formalizes existing concepts within this framework, it lacks proofs demonstrating what this framework can achieve and fails to establish connections to existing work. Given the lack of substantial theory, I think a more thorough empirical investigation could strengthen the submission. Comparisons with more expressive architectures are missing (e.g., in Table 1 there are no results reported for more expressive architectures for almost half of the datasets; for the synthetic datasets no comparison is done with more expressive architectures), making it difficult to appreciate the practical advantages of using the rule-based layers in practice. The practical relevance is limited further by the fact that the rule-based layer can only process one-dimensional features, and the higher space complexity for dense graphs.\n\nRegarding clarity, there is considerable room for improvement. The concept of how a rule-based layer works was not fully clear to me until page 4. If my understanding is correct, we have a matrix $\\mathcal{W}$ that contains all possible weights (and similarly a bias vector $\\mathcal{B}$ with all possible biases). A rule restricts $\\mathcal{W}$ to a subset of weights; applying rule *R* means setting some entries in $\\mathcal{W}$ to zero. If my understanding is incorrect, this indicates that the writing lacks some clarity. I suggest shortening the introduction and preliminaries, which are at times verbose, and including a briefer example from Appendix A.4 earlier in the paper, or providing a clearer definition sooner. Additionally, the notation for the rule-based layer presentation is somewhat convoluted. The readability of the paper is also hindered by the inconsistent use of formal definitions and natural language. While both approaches can be fine (as long as they are precise), there is a noticeable mismatch between the rigor in the preliminaries and, for example, Section 4. Many aspects of the paper are thus unclear; please refer to the *Questions* and *Minor Remarks* for specific examples.\n\nOverall, I think this paper presents promising ideas in a preliminary manner. As also stated by the authors, the dynamic rule-based layer seems to be reasonable for graphs, but is more difficult to devise for other structures. One approach could be to revise the paper from a graph learning perspective, and, if the authors have novel results which hold for general structures, present these results in a follow-up paper. Another exciting direction could be to use rules to create flexible machine learning models for different tasks and input data.\n\n*Minor remarks*:\n\n* line 33: each new information -> each new piece of information\n* line 34: the essence -> a bit vague, what is the essence of dynamic NNs?\n* Fig. 1 is too small and difficult to parse in general; there is also and typo in the last sentence\n* line 75: dot missing after end of sentence\n* line 95: concatentation -> should this be \"composition\"?\n* line 111: dot missing after end of sentence\n* Somewhat inflationary use of \"respectively\"\n* line 123, 140: I would strongly advise to not use $y$ here for $x, y \\in D$, as $y$ is already used to denote labels earlier\n* It would be helpful to refer to equations as eq. (1) (instead of just (1))\n* Could it simplify presentation if you define $\\Theta$ as tuple $(\\mathcal{W}, \\mathcal{B})$?\n* Last sentence of Prop. 1 is difficult to read\n* Why do we call the learnable parameters $\\Theta$ in Prop. 1 and $W^i$ in Prop. 2?\n* line 190: higher dimensions -> higher dimensional\n* line 202: network -> network architecture\n* lines 206-214: I suggest to consider moving this to the preliminaries\n* line 221: either rule function (singular) or rule functions R_W, R_B\n* line 225: circle -> cycle\n* Prop. 3: \"its\" -> not clear what it refers to\n* line 231: If R permutation-equivariant -> language sounds off, maybe \"For permutation-equivariant R\" or \"If R is permutation-equivariant\"\n* line 255: typo in isomorphism\n* Pattern counting rule: $d$ is never defined\n* line 347: missing space\n\nQUESTIONS:\n1. Is it possible to encode multiple rules into one layer? E.g., if a rule consists of a conjunction, are there beneftis/downsides to encoding the conjuncts in different layers or all in one?\n2. Could you give more examples of how to exploit the fact that dynamic layers allow for arbitrarily sized input (beyond graphs)?\n3. Could you elaborate on the precise meaning of \"injective based on the labels and an additionally defined shared property\"? \n4. Does the presented approach allow to combine different data types/modalities and train them in the same network (towards, e.g., in-context learning)?\n5. Did you experiment on strongly regular graphs? The ability to count cycles is helpful for that graph class and it could give some information about the expressivity of the rule-based graph layers.\n6. Line 258: \"We compute all possible embeddings\". What does embedding refer to here?\n7. Could you put your work into context with other modular (dynamic) neural network approaches (e.g., https://arxiv.org/pdf/1910.04915 or https://arxiv.org/pdf/2010.02066).\n8. What happens if we want to classify a new sample, but we do not know what rule to use. How can we use the proposed approach in this scenario?\n\nLIMITATIONS:\nOne of the main limitations, as the authors point out themselves, is that their proposed rule-based layer can only process one-dimensional node features, and no edge features, which impacts the practical value of their method. For more limitations, please refer to *Weaknesses* and *Questions*.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors develop a broad framework for adding expert knowledge to Neural Networks. They formalize this by extending the learnable parameterized functions with an additional parameter consisting of the set of formal rules. In general, these rules maybe learnable as well. However, the authors focuses on these rules being given in the form of expert-knowledge. The authors then introduce the set of rule based layers. And shows that fully connected NN layers and CNN layers are special cases of the rule based layers. They introduce three rule based layers for graphs: Weisfeiler-Leman Layer, Pattern-Counting layer and Aggregation layer. The author shows that there exists a GNN with rule based layers that can distinguish any two isomorphic graphs. Finally the author introduces some examples of rule based layers for specific molecule graphs. And presents experimental results on synthetic and real-world data.\n\nSTRENGTHS:\n-- The idea of adding expert knowledge to NNs and GNNs specifically is quite interesting and widely investigated.\n\n-- The presented theory is very general and simple\n\nWEAKNESSES:\n-- The author has used the notion of rules rather broadly. There is no formal language (logic or matrix language) for the rules. They are just arbitrary functions. This basically means that any existing NN model, in one way or another, can be seen as a special case of Rule based NN. In my understanding, this makes the introduced framework a rather simple formalization of how expert knowledge maybe added to NNs. However, this formalization is so loose, that it does not really admit any meaningful analysis or provide any meaningful guidance to the user for adding knowledge.\n\n-- None of the examples presented by the author are beyond what would be anyway possible by adding some simple graph features to the node features. This could be an interesting direction to investigate. But just formally stating that this is possible is not very interesting.\n\nQUESTIONS:\nHow does your framework help a practitioner, in a meaningful way, more than just augmenting feature vectors in GNNs with expert knowledge?\nHow does your framework enable any new theoretical analysis of GNNs?\n\nLIMITATIONS:\nThe authors have indeed touched upon most of the points I mention as weaknesses.\nHowever, as mentioned earlier, the proposed framework is very broad and does not provide a meaningful way to proceed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}