Title: Complete Neural Networks for Complete Euclidean Graphs
Venue: NeurIPS 2023
Accepted: no
Keywords: Graph Neural Networks

Weisfeiler Leman Test

graph isomorphism

universal approximation

equivariant neural network

Euclidean graphs
Abstract: Neural networks for point clouds, which respect their natural invariance to permutation and rigid motion,  have enjoyed recent success in modeling geometric phenomena, from molecular dynamics to recommender systems.  Yet, to date, no architecture with polynomial complexity is known to be complete, that is, able to distinguish between any pair of non-isomorphic point clouds. We fill this theoretical gap by showing that point clouds can be completely determined, up to permutation and rigid motion, by applying the 3-WL graph isomorphism test to the point cloud's centralized Gram matrix.  Moreover, we formulate a Euclidean variant of the 2-WL test and show that it is also sufficient to achieve completeness. We then show how our complete Euclidean WL tests can be simulated by a Euclidean graph neural network of moderate size and demonstrate their separation capability on highly-symmetrical point clouds.


Abstract

Neural networks for point clouds, which respect their natural invariance to permutation and rigid motion, have enjoyed recent success in modeling geometric phenomena, from molecular dynamics Reiser et al. [2022] to recommender systems Yi et al. [2023]. Yet, to date, no architecture with polynomial complexity is known to be complete , that is, able to distinguish between any pair of non-isomorphic point clouds. We fill this theoretical gap by showing that point clouds can be completely determined, up to permutation and rigid motion, by applying the 3-WL graph isomorphism test to the point cloud's centralized Gram matrix. Moreover, we formulate a Euclidean variant of the 2-WL test and show that it is also sufficient to achieve completeness. We then show how our complete Euclidean WL tests can be simulated by a Euclidean graph neural network of moderate size and demonstrate their separation capability on highly-symmetrical point clouds.

1 Introduction

A point cloud is a collection of n points in R d , where typically in applications d = 3 . Machine learning on point clouds is an important task with applications in chemistry Gilmer et al. [2017], Wang et al. [2022], physical systems Finzi et al. [2021] and image processing Ma et al. [2023]. Many successful architectures for point clouds are invariant by construction to the natural symmetries of point clouds: permutations and rigid motions.

The rapidly increasing literature on point-cloud networks with permutation and rigid-motion symmetries has motivated research aimed at theoretically understanding the expressive power of the various architectures. This analysis typically focuses on two closely related concepts: Separation and Universality . We say an invariant architecture is separating , or complete , if it can assign distinct values to any pair of point clouds that are not related by symmetry. An invariant architecture is universal if it can approximate all continuous invariant functions on compact sets. Generally speaking, these two concepts are essentially equivalent, as discussed in Villar et al. [2021], Joshi et al. [2022], Chen et al. [2019], and in our context, in Appendix A.

Dym and Maron [2020] proved that the well-known Tensor Field Network Thomas et al. [2018] invariant architecture is universal, but the construction in their proof requires arbitrarily high-order representations of the rotation group. Similarly, universality can be obtained using high-order representations of the permutation group Lim et al. [2022]. However, prior to this work, it was not known whether the same theoretical guarantees can be achieved by realistic point-cloud architectures that use low-dimensional representations, and whose complexity has a mild polynomial dependency on the data dimension. In the words of Pozdnyakov and Ceriotti [2022]: "...provably universal equivariant frameworks are such in the limit in which they generate high-order correlations. . . It is an interesting, and open, question whether a given order suffices to guarantee complete resolving power." (p. 6). We note that it is known that separation of point clouds in polynomial time in n

is possible, assuming that d is fixed (e.g., d = 3 ) Arvind and Rattan [2014], Dym and Kovalsky 37 [2019], Kurlin [2022]. What still remains to be established is whether separation is achievable for 38 common invariant machine learning models, and more generally, whether separation can be achieved 39 by computing a continuous invariant feature that is piecewise differentiable. 40

In this paper, we give what seems to be the first positive answer to this question. We focus on analyzing 41 a popular method for the construction of invariant point-cloud networks via Graph Neural Networks 42 (GNNs) . This is done in two steps: first, point clouds are represented as a Euclidean graph - which we 43 define to be a complete weighted graph whose edge features are simple, rotation-invariant features: 44 the inner products between pairs of (centralized) points. We then apply permutation-invariant Graph 45 Neural Networks (GNNs) to the Euclidean graphs to obtain a rotation- and permutation-invariant 46 global point-cloud feature. This leads to a rich family of invariant point-cloud architectures, which is 47 determined by the type of GNN chosen. 48

49

50

51

52

53

54

55

56

57

The most straightforward implementation of this idea would be to apply the popular message passing GNNs to the Euclidean graphs. One could also consider applying more expressive GNNs. For combinatorial graphs, it is known that message-passing GNNs are only as expressive as the 1-WL graph isomorphism test. There exists a hierarchy of k -WL graph isomorphism tests, where larger values of k correspond to more expressive, and more expensive, graph isomorphism tests. There are also corresponding GNNs that simulate the k -WL tests and have an equivalent separation power Morris et al. [2018], Maron et al. [2019]. One could then consider applying these more expressive architectures to Euclidean graphs, as suggested in Lim et al. [2022]. Accordingly, we aim to answer the following questions:

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

Question 1 For which k is the k -WL test, when applied to Euclidean graphs, complete?

Question 2 Can this test be implemented in polynomial time by a continuous, piecewise-differentiable architecture?

We begin by addressing Question 1. First, we consider a variation of the WL-test adapted for point clouds, which we refer to as 1 -EWL ('E' for Euclidean). This test was first proposed by Pozdnyakov and Ceriotti [2022], where it was shown that it cannot distinguish between all 3 -dimensional point clouds, and consequently, neither can GNNs like Victor Garcia Satorras [2021], Schütt et al. [2017], which can be shown to simulate it. Our first result, described in Section 2.1, balances this by showing that two iterations of 1 -EWL are enough to separate almost any pair of point clouds.

To achieve complete separationfor all point clouds, we consider higher-order k -EWL tests. We first consider a natural adaptation of k -WL for Euclidean graphs, which we name the Vanillak -EWL test. In this test, the standard k -WL is applied to the Euclidean graph induced by the point clouds. We show that when k = 3 , this test is complete for 3 -dimensional point clouds. Additionally, we propose a variant of the Vanilla 2 -EWL, which incorporates additional geometric information while having the same complexity. We call this test the 2 -EWL test, and show that it is complete on 3D point clouds. We also propose a natural variation of 2 -EWL called 2 -SEWL , which can distinguish between point clouds that are related by a reflection. This ability is important for chemical applications, as most biological molecules that are related by a reflection are not chemically identical Kapon et al. [2021] (this molecular property is called chirality ).

We next address the second question of how to construct a GNN for Euclidean data with the same separation power as that of the various k -EWL tests we describe. For combinatorial graphs, such equivalence results rely on injective functions defined on multisets of discrete features Xu et al. [2018]. For Euclidean graphs, one can similarly rely on injective functions for multisets with continuous features, such as those proposed in Dym and Gortler [2023]. However, a naive application of this approach leads to a very large number of hidden features, which grows exponentially with the number of message-passing iterations (see Figure 2). We show how this problem can be remedied, so that the number of features needed depends only linearly on the number of message-passing iterations.

To summarize, our main results in this paper are:

We show that two iterations of 1 -EWL can separate almost all point clouds in any dimension.

We prove the completeness of a single iteration of the vanilla 3 -EWL for point clouds in R 3 .

We formulate the 2 -SEWL and 2 -EWL tests, and prove their completeness for point clouds in R 3 .

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

Weexplain how to build differentiable architectures for point clouds with the same separation power as Euclidean k -WL tests, with reasonable complexity.

Experiments In Section 5 we present synthetic experiments that demonstrate that 2 -SEWL can separate challenging point-cloud pairs that cannot be separated by several popular architectures.

Disambiguation: Euclidean Graphs In this paper we use a simple definition of a Euclidean graph as the centralized Gram matrix of a point cloud, and focus on a fundamental theoretical question related to this representation. In the learning literature, terms like 'geometric graphs' (not used here) could refer to graphs that have both geometric and non-geometric edge and vertex features, or graphs where pairwise distances are only available for specific point pairs (edges in an incomplete graph).

1.1 Related Work

Euclidean WL Pozdnyakov and Ceriotti [2022] showed that 1 -EWL is incomplete for 3dimensional point clouds. Joshi et al. [2022] defines separation for a more general definition of geometric graph, which combines geometric and combinatorial features. This work holds various interesting insights for this more general problem but they do not prove completeness as we do here.

Other complete constructions As mentioned earlier, Dym and Maron [2020] proved universality with respect to permutations and rigid motions for architectures using high-dimensional representations of the rotation group. Similar results were obtained inFinkelshtein et al. [2022], Gasteiger et al. [2021]. In Lim et al. [2022] universality was proven for Euclidean GNNs with very high-order permutation representations. In the planar case d = 2 , universality using low-dimensional features was achieved in Bökman et al. [2022]. For d ≥ 3 our construction seems to be the first to achieve universality using low dimensional representations.

For general fixed d , there do exist algorithms that can separate point clouds up to equivalence in polynomial time, but they do not seem to lend themselves directly to neural architectures. In Kurlin [2022], Widdowson and Kurlin [2023] complete tests are described, but they represent each point cloud as a 'multiset of multisets' rather than as a vector as we do, and so are not suitable for gradient descent based learning. Efficient tests for equivalence of Euclidean graphs were described in Brass and Knauer [2000], Arvind and Rattan [2014], but they compute features that do not depend continuously on the point cloud.

Weaker notions of universality In Widdowson and Kurlin [2022] the authors suggest a method for distinguishing almost every point clouds up to equivalence, similar to our result here on 1 -EWL. Similarly, efficient separation/universality can also be obtained for point clouds with distinct principal axes Puny et al. [2021], Kurlin [2022]. Another setting in which universality is easier to obtain is when only rigid symmetries are considered and permutation symmetries are ignored Wang et al. [2022], Villar et al. [2021], Victor Garcia Satorras [2021]. All these results do not provide universality for all point clouds, with respect to the joint action of permutations and rigid motions.

Mathematical notation

A(finite) multiset { { y 1 , . . . , y N } } is an unordered collection of elements where repetitions are allowed. Let G be a group acting on a set X . For X,Y ∈ X , we say that X = G Y if Y = gX for some g ∈ G .

We say that a function f : X → Y is invariant if f ( gx ) = f ( x ) for all x ∈ X,g ∈ G . We say that f is equivariant if Y is also endowed with some action of G and f ( gx ) = gf ( x ) for all x ∈ X , g ∈ G . A separating invariant mapping is an invariant mapping that is injective, up to group equivalence:

Definition 1.1 (Separating Invariant) . Let G be a group acting on a set X . Wesay F : X → R K is a G -separating invariant with embedding dimension K if for all X,Y ∈ X , F ( X ) = F ( Y ) ⇔ X = G Y .

We focus on the case where X is some Euclidean domain. To enable gradient-based learning, we shall need separating mappings that are continuous everywhere and differentiable almost everywhere.

Generic

Hard

Harder

GeometricDegreeHistogram

Hard

a

d

e

Harder

y

Z

GeometricDegree

1

2

3

4

5

a

b

C

d

e

X

X

y

y

3

3

X

6

6

10

a

b

C

d

e

The symmetry group we consider for point clouds ( x 1 , . . . , x n ) ∈ R d × n is generated by a rotation matrix R ∈ SO ( d ) , and a permutation σ ∈ S n . These act on a point cloud by

We denote this group by SO [ d, n ] . In some instances, reflections R ∈ O ( d ) are also permitted, 135 leading to a slightly larger symmetry group, which we denote by O [ d, n ] . Our goal shall be to 136 construct separating invariants for these groups. For the sake of brevity, we do not discuss translation 137 invariance and separation, as these can easily be achieved by centering the input point clouds, once 138 SO [ d, n ] (or O [ d, n ] ) separating invariants are constructed, see Dym and Gortler [2023]. 139

For simplicity of notation, throughout this paper, we focus on the case d = 3 . In Appendix C we 140 explain how our constructions and theorems can be generalized to d > 3 . 141

142

143

144

145

146

147

148

149

150

2 Euclidean Graph Isomorphism Tests

The k -WL Graph Isomorphism Test Weisfeiler and Leman [1968] is a classical paradigm for testing the isomorphism of combinatorial graphs, which we shall now briefly describe. Let G be a graph with vertices indexed by [ n ] = { 1 , 2 , . . . , n } . We denote each ordered k -tuple of vertices by a multi-index i = ( i 1 , . . . , i k ) ∈ [ n ] k . Essentially, for each such k -tuple i , the test maintains a coloring C ( i ) that belongs to a discrete set, and updates it iteratively. First, the coloring of each k -tuple is assigned an initial value that encodes the isomorphism type of the corresponding k -dimensional subgraph:

Then the color of each k -tuple i is iteratively refined according to the colors of its 'neighboring' k -tuples. The update rule is given by

where i [ j \ t ] is the multi-index i with its t -th coordinate replaced by j ; e.g. for j = 1 , i [ j \ 1] = ( j, i 2 , . . . , i k ) . Embed is a function that maps its input injectively to some discrete set. This process is repeated T times to obtain a final coloring { { C ( T ) ( i ) } } i ∈ [ n ] k . A global label is then calculated by

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

where Embed ( T +1) is a function that maps label-multisets injectively to some discrete set.

̸

To test whether two graphs G and G ′ are isomorphic, the k -WL test computes the corresponding colorings C G and C G ′ for some chosen T . If C G = C G ′ then G and G ′ are guaranteed not to be isomorphic, whereas if C G = C G ′ , then G and G ′ may either be isomorphic or not, and the test does not, in general, provide a decisive answer for combinatorial graphs. It is known that this test is able to distinguish a strictly larger class of combinatorial graphs for every strict increase in the value of k, i.e. it is a strict hierarchy of tests in terms of distinguishing power Cai et al. [1992], Grohe [2017].

Vanillak -WL tests As a first step from a combinatorial to a Euclidean setting, we identify each point cloud X = ( x 1 , . . . , x n ) ∈ R d × n with a complete graph on n vertices, wherein each edge ( i, j ) is endowed with the weight w ij ( X ) = ⟨ x i , x j ⟩ . We name such a graph a Euclidean graph . Similarly to k -WL for combinatorial graphs, k -WL for Euclidean graphs maintains a coloring of the k -tuples of vertices. However, the initial color of each k -tuple i is not a discrete label as in the combinatorial case, but rather a k × k matrix of continuous features, which represent all edge weights w ij corresponding to pairs of indices from i . We will call the k -WL test defined by this initial coloring the vanilla k -WL test. This test is invariant by construction to reflections, rotations, and permutations. We note that our definition of the vanilla k -EWL test via inner products follows that of Lim et al. [2022]. Another popular, and essentially equivalent, formulation, uses distances instead.

k -EWL tests An inherent limitation of the Vanilla-1-EWL test is that no pairwise Euclidean information is passed, yielding it rather uninformative. Indeed, Pozdnyakov and Ceriotti [2022] proposed a Euclidean analog of the 1 -WL test, where the update rule (2) is replaced with

̸

We call this test the 1 -EWL test. This formulation is motivated by the fact that many symmetrypreserving networks for point clouds are in fact a realization of it, though they use Embed functions that are continuous and, in general, may assign the same value to different multisets. Consequently, the separation power of these architectures is at most that of 1 -EWL with discrete injective hash functions. Moreover, the separation power will be equivalent if continuous injective multiset functions are used for embedding, as we discuss in Section 4.

The 1 -EWL test strengthens the Vanilla1 -EWL test by allowing the messages passed to a node in each step to contain not only previous colorings but also geometric information in the form of pairwise distances. More generally, we shall use the term k -EWL to refer to tests that follow the Euclidean k -WL paradigm, but incorporate geometric invariants into the message-passing procedure. In particular, for point clouds with dimension 3 , we define the 2 -SEWL test ('SE' for Special Euclidean ) by replacing the update step (2) with

Note that ⟨ x i × x j , x k ⟩ is equal to the determinant of the 3 × 3 matrix whose rows are the three vectors x i , x j , x k , which makes this a natural choice as all polynomial invariants of SO (3) are generated by these determinants and the inner products we use for the initial coloring Kraft and Procesi [1996].

We note that, Using the fact that O (3) is just two copies of SO (3) , it is not difficult to generalize 2 -SEWL to a complete O [3 , n ] test, which we name 2-EWL. for general d , similar complete ( d -1) -SEWL and ( d -1) -EWL tests can be formulated for point clouds in R d via the Hodge-star operator; see Appendix C for more details.

In the rest of this section, we shall prove that the 2 -SEWL, 2 -EWL and vanilla 3 -EWL tests are complete when applied to R 3 × n , even when using a single iteration ( T = 1) . We shall also show that two iterations of the 1 -EWL test is complete, except on a set of measure zero.

2.1 Generic completeness of 1 -EWL

The separation power of 1-EWL is closely linked to the notion of geometric degree : For a point cloud X = ( x 1 , . . . , x n ) , we define the geometric degree d ( i, X ) of the i th point, and the induced geometric degree histogram d H ( X ) , to be the multisets

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

̸

It is not difficult to see that if d H ( X ) = d H ( Y ) then X and Y can be separated by a single 1 -EWL iteration . An example of such a pair is shown in the left of Figure 1. With two 1 -EWL iterations, we show that can separate X and Y even if d H ( X ) = d H ( Y ) , provided that they both belong to the set of point clouds defined by

̸

̸

Such an example, taken from Pozdnyakov et al. [2020], is visualized in the middle column of Figure 1

Theorem 2.1. Two iterations of the 1 -EWL test assign two point clouds X , Y ∈ R 3 × n distinct the same value, if and only if X = O [3 ,n ] Y .

In the appendix we show that the complement of R 3 × n distinct has measure zero. Thus this result complements long-standing results for combinatorial graphs, stating that 1-WL can classify almost all such graphs as the number of nodes tends to infinity Babai et al. [1980].

The right-most pair of point clouds ('Harder') in Figure 1 is taken from Pozdnyakov and Ceriotti [2022]. The degree histograms of these point clouds are identical, and they are not in R 3 × n distinct . Pozdnyakov and Ceriotti [2022] show that this pair cannot be separated by any number of 1 -EWL iterations.

2.2 Is 1-EWL All You Need?

Theorem 2.1 shows that the probability of a failure of the 1 -EWL is zero. A natural question to ask is whether more powerful tests are needed. We believe the answer to this question is yes. Typical hypothesis classes used for machine learning, such as neural networks, are Lipschitz continuous Gama et al. [2020]. In this setting, failure to separate on a measure zero set could have implications for non-trivial positive measure. This phenomenon is depicted in the figure in the inset. On the right, a plot of a Gaussian

distribution centered at x ∈ R , depicting a target function is shown in blue. In red, a schematic plot of how a Lipschitz continuous function that does not distinguish x from y would model the target function.

3 2 -SEWL and Vanilla 3 -EWL are complete

We now prove that the vanilla 3 -EWL test is complete.

Theorem 3.1. For every X,Y ∈ R 3 × n , a single iteration of the vanilla 3 -EWL test assigns X and Y the same value if and only if X = O [3 ,n ] Y .

Proof. First, it is clear that if X = O [3 ,n ] Y then C G ( X ) = C G ( Y ) since the vanilla 3 -EWL test is invariant by construction. The challenge is proving the other direction. To this end, let us assume that C G ( X ) = C G ( Y ) , and assume without loss of generality that r := rank( X ) ≥ rank( Y ) . Note that X has rank r ≤ 3 , and so it must contain some three points whose rank is also r . By applying a permutation to X we can assume without loss of generality that these three points are the first three points. The initial coloring C 0 (1 , 2 , 3)( X ) of this triplet is their Gram matrix ( ⟨ x i , x j ⟩ ) 1 ≤ i,j ≤ 3 , which has the same rank r as the space spanned by the three points. Next, since C G ( X ) = C G ( Y ) are the same, there exists a triplet of points i, j, k such that C (1) (1 , 2 , 3)( X ) = C (1) ( i, j, k )( Y ) which implies that the initial colorings are also the same. By applying a permutation to Y we can assume without loss of generality that i = 1 , j = 2 , k = 3 . Next, since the Gram matrix of x 1 , x 2 , x 3 and y 1 , y 2 , y 3 are identical, there is an orthogonal transformation that takes x i to y i for i = 1 , 2 , 3 , and by applying this transformation to all points in X we can assume without loss of generality that x i = y i for i = 1 , 2 , 3 . It remains to show that the rest of the points of X and Y are equal, up to permutation. To see this, first note that X and Y have the same rank since

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

Thus the space spanned by x 1 = y 1 , x 2 = y 2 , x 3 = y 3 contains all points in X and Y . Next, we can deduce from the aggregation rule defining C 1 (1 , 2 , 3)( X ) in (2), that

Since all points in X and Y belong to the span of x 1 = y 1 , x 2 = y 2 , x 3 = y 3 , X and Y are the same up to permutation of the last n -3 coordinates. This concludes the proof of the theorem.

We next outline the completeness proof of the more efficient 2 -SEWL.

Theorem 3.2. For every X,Y ∈ R 3 × n , a single iteration of the 2 -SEWL test assigns X and Y the same value if and only if X = SO [3 ,n ] Y .

Proof idea. The completeness of Vanilla3 -EWLwasbased on the fact that its initial coloring captures the Gram matrix of triplets of vectors that span the space spanned by X , and on the availability of projections onto this basis in the aggregation step defined in (2). Our proof for 2 -EWL completeness relies on the fact that a pair of non-degenerate vectors x i , x j induces a basis x i , x j , x i × x j of R 3 . The Gram matrix of this basis can be recovered from the Gram matrix of the first two points x i , x j , and the projection onto this basis can be obtained from the extra geometric information we added in (18). A full proof is given in the appendix.

To conclude this section, we note that the above theorem can be readily used to also show that the 2 -EWL test us also complete with respect to O [3 , n ] . For details see Appendix A.

4 WL-equivalent GNNs with continuous features

In the previous section we discussed the generic completeness of 1-EWL and the completeness of 2 -SEWL and vanilla 3 -EWL. The Embed functions in these tests are hash functions, which can be redefined independently for each pair of point clouds X,Y . In this section, our goal is to explain how to construct GNNs with equivalent separation power to that of these tests, while choosing continuous, piecewise differentiable Embed functions that are injective. While this question is well studied for combinatorial graphs with discrete features Xu et al. [2018], Morris et al. [2018], Maron et al. [2019], Aamand et al. [2022], here we focus on addressing it for Euclidean graphs with continuous features.

4.1 Multiset injective functions

Let us first review some known results on injective multiset functions. Recall that a function defined on multisets with n elements coming from some alphabet Ω ⊆ R D can be identified with a permutation invariant function defined on Ω n . A multiset function is injective if and only if its corresponding function on Ω n is separating with respect to the action of the permutation group (see Definition 1.1).

In Corso et al. [2020], Wagstaff et al. [2022] it was shown that for any separating, permutation invariant mappings from R n to R K , the embedding dimension K will be at least n . Two famous examples of continuous functions that achieve this bound are

When the multiset elements are in R D , the picture is similar: if there exists a continuous, permutation invariant and separating mapping from R D × n to R K , then necessarily K ≥ n · D Joshi et al. [2022]. In Dym and Gortler [2023] it is shown that continuous separating invariants for D > 1 , with nearoptimal dimension, can be derived from the D = 1 separating invariants Ψ = Ψ pow or Ψ = Ψ sort , by considering random invariants of the form

where each a j and b j are d and n dimensional random vectors, and we denote θ = ( a 1 , . . . , a K , b 1 , . . . , b K ) ∈ R K ( D + n ) . When K = 2 nD + 1 , for almost any choice of θ , the function Embed θ will be separating on R D × n . Thus the embedding dimension in this construction is optimal up to a multiplicative constant of two.

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

An important property of this results of Dym and Gortler [2023] for our discussion, is that the embedding dimension K can be reduced if the domain of interest is a non-linear subset of R D × n of low dimension. For example, if the domain of interest is a finite union of lines in R D × n , then the instrinsic dimension of the domain is 1 , and so we will only need an embedding dimension of K = 2 · 1 + 1 = 3 . Thus, the required embedding dimension depends on the intrinsic dimension of the domain rather than on its ambient dimension , which in our case is n · D .

To formulate these results precisely we will need to introduce some real algebraic geometry terminology (see Basu et al. [2006] for more details): A semi-algebraic subset of a real finite-dimensional vector space is a finite union of subsets that are defined by polynomial equality and inequality constraints. For example, polygons, hyperplanes, spheres, and finite unions of these sets, are all semi-algebraic sets. A semi-algebraic set is always a finite union of manifolds, and its dimension is the maximal dimension of the manifolds in this union. Using these notions, we can now state the 'intrinsic version' of the results in Dym and Gortler [2023]:

Theorem 4.1 (Dym and Gortler [2023]) . Let X be an S n -invariant semi-algebraic subset of R D × n of dimension D X . Denote K = 2 D X +1 . Then for Lebesgue almost every θ ∈ R K ( D + n ) the mapping Embed θ : X → R K is S n invariant and separating.

4.2 Multiset injective functions for GNNs

We now return to discuss GNNs and explain the importance of the distinction between the intrinsic and ambient dimensions in our context. Suppose we are given n initial features ( h (0) 1 , . . . , h (0) n ) in R d , and for simplicity let us assume they are recursively refined via the simple aggregation rule:

̸

Let us assume that each Embed ( t ) is injective on the space of all multisets with n -1 elements in the ambient space of h ( t ) j . Then the injectivity of Embed (1) implies that h (1) i is of dimension at least ( n -1) · d . The requirement that Embed (2) is injective on a mult-set of n -1 features in R ( n -1) · d implies that h (2) i will be of dimension at least ( n -1) 2 · d . Continuing recursively with this argument we obtain an estimate of ∼ ( n -1) T d for the dimensions of each h ( T ) i after T iterations of (7).

Fortunately the analysis presented above is overly pessimistic, because it focused only on the ambient dimension . Let us denote the matrix containing all n features at time t by H ( t ) . Then H ( t ) = F t ( H (0) ) , where F t is the concatenation of all Embed ( t ′ ) functions from all previous time-steps. Thus H ( t ) resides in the set F t ( R d × n ) . Here we again rely on results from algebraic geometry: if F t is a composition of piecewise linear and polynomial mappings, then it is a semi-algebraic mapping, which means that F t ( H (0) ) will be a semi-algebraic set of dimension dim( R n × d ) = n · d . This point will be explained in more detail in the proof of Theo-

௜

଴

௜

ଵ

௜

ଶ

௜

ଷ

௜

்

output

…

…

dim

dim

𝟐

𝟑

ambient

intrinsic

rem 4.2. By Theorem 4.1 we can then use Embed θ as a multiset injective function on X t with a fixed embedding dimension of 2 n · d +1 which does not depend on T . This is visualized in Figure 2.

2-SEWLnet Based on the discussion above, we can devise architectures that simulate the various tests discussed in this paper and have reasonable feature dimensions throughout the construction, In particular, we can simulate T iterations of the 2 -SEWL test by replacing all Embed ( t ) functions 1 with Embed ( t ) θ , where in our implementation we choose Ψ = Ψ sort in (6). The embedding dimension for all t is taken to be 6 n +1 , since the input is in R 3 × n . We denote the obtained parametric function by F ϕ . Based on a formalization of the discussion above, we prove in the appendix that F ϕ has the separation power of the complete 2 -SEWL test, and therefore F ϕ is separating.

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

Theorem 4.2. Let F ϕ denote the parametric function simulating the 2 -SEWL test. Then for Lebesgue almost every ϕ the function F ϕ : R 3 × n → R 6 n +1 is separating with respect to the action of SO [3 , n ] .

To conclude this subsection, we note that while sort-based permutation invariants are used as aggregators in GNNs Zhang et al. [2020, 2018], Blondel et al. [2020], the polynomial-based aggregators Ψ pow are not as common. To a certain extent, one can use the approach in Xu et al. [2018], Maron et al. [2019], replace the polynomials in Ψ pow by MLPs, and justify this by the universal approximation power of MLPs. A limitation of this approach is that it only guarantees separation at the limit.

5 Synthetic Experiments

In this section we implement 2-SEWLnet, described in Section 4, and empirically evaluate its separation power, and the separation power of alternative SO [3 , n ] invariant point cloud architectures. We trained the architectures on permuted and rotated variations of highly-challenging point-cloud pairs, and measured separation by the test classification accuracy. We considered three pairs of point clouds (Hard1-Hard3) from Pozdnyakov et al. [2020]. These pairs were designed to be challenging for distance-based invariant methods. However, our analysis reveals that they are in fact separable by two iterations of the 1-EWL test. We then consider a pair of point clouds from Pozdnyakov and Ceriotti [2022] which was proven to be indstinguishable by the 1-EWL tests. The results of this experiment are given in Table 1. Further details on the experimental setup appear in Appendix B.

As expected, we find that 2 -SEWLnet, which has complete separation power, succeeded in perfectly separating all examples. We also found that EGNN Victor Garcia Satorras [2021], which is essentially an implementation of 1-EWL, does not separate the Harder example, but does separate the Hard example after two iterations, as predicted by Theorem 2.1. We also considered three additional invariant point cloud models whose separation power is not as well understood. We find that MACE Batatia et al. [2022] and TFN Thomas et al. [2018] achieve perfect separation, (when applying them with at least 3-order correlations and three-order SO (3) representations). The third GVPGNN Jing et al. [2021] architecture attains mixed results. We note that we cannot necessarily deduce from our empirical results that MACE and TFN are complete. While it is true that TFN is complete when considering arbitrarily high order representations Dym and Maron [2020], it is not clear whether order three representation suffices for complete separation. We conjecture that this is not the case. However, finding counterexamples is a challenging problem we leave for future work.

Future Work In this work, we presented several invariant tests for point clouds that are provably complete, and have presented and implemented 2 -SEWL-net which simulates the complete 2 -SEWL test. Currently, this is a basic implementation that only serves to corroborate our theoretical results. A practically useful implementation requires addressing several challenges, including dealing with point clouds of different sizes, the non-trivial ∼ n 4 complexity of computing even the relatively efficient 2 -SEWL-net, and finding learning tasks where complete separation leads to gains in performance. We are actively researching these directions and hope this paper will inspire others to do the same.

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

References

Patrick Reiser, Marlen Neubert, Andr'e Eberhard, Luca Torresi, Chen Zhou, Chen Shao, Houssam Metni, Clint van Hoesel, Henrik Schopmans, Timo Sommer, and Pascal Friederich. Graph neural networks for materials science and chemistry. Communications Materials , 3(1):93, 2022. doi: 10.1038/s43246-022-00315-6.

Zixuan Yi, Iadh Ounis, and Craig Macdonald. Graph contrastive learning with positional representation for recommendation. In Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, and Annalina Caputo, editors, Advances in Information Retrieval , pages 288-303, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-28238-6.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. CoRR , 2017.

Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. Comenet: Towards complete and efficient message passing for 3d molecular graphs, 2022.

Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. arXiv preprint arXiv:2104.09459 , 2021.

Xu Ma, Yuqian Zhou, Huan Wang, Can Qin, Bin Sun, Chang Liu, and Yun Fu. Image as set of points, 2023.

Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 28848-28863. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ f1b0775946bc0329b35b823b86eeb5f5-Paper.pdf .

Chaitanya K. Joshi, Cristian Bodnar, Simon V. Mathis, Taco Cohen, and Pietro Liò. On the expressive power of geometric graph neural networks. NeurIPS Workshop on Symmetry and Geometry in Neural Representations , 2022.

Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. Advances in neural information processing systems , 32, 2019.

Nadav Dym and Haggai Maron. On the universality of rotation equivariant point cloud networks. ArXiv , abs/2010.02449, 2020.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219 , 2018.

Derek Lim, Joshua Robinson, Lingxiao Zhao, Tess Smidt, Suvrit Sra, Haggai Maron, and Stefanie Jegelka. Sign and basis invariant networks for spectral graph representation learning. arXiv preprint arXiv:2202.13013 , 2022.

Sergey N. Pozdnyakov and Michele Ceriotti. Incompleteness of graph neural networks for points clouds in three dimensions, 2022.

Vikraman Arvind and Gaurav Rattan. The complexity of geometric graph isomorphism. In Electron. Colloquium Comput. Complex. , volume 21, page 70, 2014.

Nadav Dym and Shahar Ziv Kovalsky. Linearly converging quasi branch and bound algorithms for global rigid registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1628-1636, 2019.

Kurlin. Computable complete invariants for finite clouds of unlabeled points under euclidean isometry. ArXiv , abs/2207.08502, 2022.

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gurkirt Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. arXiv preprint arXiv:1810.02244 , 2018.

Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks, 2019. URL https://arxiv.org/abs/1905.11136 .

Emiel Hoogeboom. Max Welling Victor Garcia Satorras. E(n) equivariant graph neural networks. Proceedings of the 38-th International Conference on Machine Learning , PMLR(139), 2021.

Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-filter convolutional neural network for modeling quantum interactions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_ files/paper/2017/file/303ed4c69846ab36c2904d3ba8573050-Paper.pdf .

Yael Kapon, Abhijit Saha, Tal Duanis-Assaf, Thijs Stuyver, Amir Ziv, Tzuriel Metzger, Shira Yochelis, Sason Shaik, Ron Naaman, Meital Reches, et al. Evidence for new enantiospecific interaction force in chiral biomolecules. Chem , 7(10):2787-2799, 2021.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks?, 2018.

Nadav Dym and Steven J. Gortler. Low dimensional invariant embeddings for universal geometric learning, 2023.

Ben Finkelshtein, Chaim Baskin, Haggai Maron, and Nadav Dym. A simple and universal rotation equivariant point-cloud network. arXiv preprint arXiv:2203.01216 , 2022.

Johannes Gasteiger, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules, 2021. URL https://arxiv.org/abs/2106.08903 .

Georg Bökman, Fredrik Kahl, and Axel Flinth. Zz-net: A universal rotation equivariant architecture for 2d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10976-10985, 2022.

Daniel Widdowson and Vitaliy Kurlin. Recognizing rigid patterns of unlabeled point clouds by complete and continuous isometry invariants with no false negatives and no false positives. arXiv preprint arXiv:2303.15385 , 2023.

Peter Brass and Christian Knauer. Testing the congruence of d-dimensional point sets. In Proceedings of the sixteenth annual symposium on Computational geometry , pages 310-314, 2000.

Daniel Widdowson and Vitaliy Kurlin. Resolving the data ambiguity for periodic crystals. In Advances in Neural Information Processing Systems , 2022.

Omri Puny, Matan Atzmon, Edward J Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations , 2021.

Sergey N Pozdnyakov, Michael J Willatt, Albert P Bartók, Christoph Ortner, Gábor Csányi, and Michele Ceriotti. Incompleteness of atomic structure representations. Physical Review Letters , 125(16):166001, 2020.

Boris Weisfeiler and A. A. Leman. The reduction of a graph to canonical form and the algebra which appears therein. Nauchno-Technicheskaya Informatsia , 2:12-16, 1968.

Jin-Yi Cai, Martin Fürer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica , 12(4):389-410, 1992. doi: 10.1007/BF01305232.

Martin Grohe. Descriptive complexity, canonisation, and definable graph structure theory , volume 47. Cambridge University Press, 2017.

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

Hanspeter Kraft and Claudio Procesi. Classical invariant theory, a primer. Lecture Notes. Preliminary version , 1996.

László Babai, Paul Erd˝ os, and Stanley M Selkow. Random graph isomorphism. SIAM Journal on Computing , 9(3):628-635, 1980.

Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Stability properties of graph neural networks. IEEE Transactions on Signal Processing , 68:5680-5695, 2020. doi: 10.1109/tsp.2020.3026980. URL https://doi.org/10.1109%2Ftsp.2020.3026980 .

Anders Aamand, Justin Chen, Piotr Indyk, Shyam Narayanan, Ronitt Rubinfeld, Nicholas Schiefer, Sandeep Silwal, and Tal Wagner. Exponentially improving the complexity of simulating the weisfeiler-lehman test with graph neural networks. Advances in Neural Information Processing Systems , 35:27333-27346, 2022.

Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veliˇ ckovi´ c. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems , 33:13260-13271, 2020.

Edward Wagstaff, Fabian B Fuchs, Martin Engelcke, Michael A Osborne, and Ingmar Posner. Universal approximation of functions on sets. Journal of Machine Learning Research , 23(151): 1-56, 2022.

Saugata Basu, Richard Pollack, and Marie-Françoise Roy. Algorithms in real algebraic geometry , volume 10. Springer, 2006.

Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Fspool: Learning set representations with featurewise sort pooling, 2020.

Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning , pages 950-959. PMLR, 2020.

Ilyes Batatia, David P Kovacs, Gregor Simm, Christoph Ortner, and Gabor Csanyi. Mace: Higher order equivariant message passing neural networks for fast and accurate force fields. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 11423-11436. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 4a36c3c51af11ed9f34615b81edb5bbc-Paper-Conference.pdf .

Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=1YLJDvSx6J4 .

J.R. Munkres. Topology . Featured Titles for Topology. Prentice Hall, Incorporated, 2000. ISBN 9780131816299. URL https://books.google.co.il/books?id=XjoZAQAAIAAJ .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provided theoretical analyses and proof that the 3-WL algorithm and the Euclidean version of the 2-WL algorithm can distinguish any complete Euclidean graph pairs. The authors then demonstrated that the algorithm can be approximated with GNNs and ran the proposed model on synthetic data to show that it was indeed able to \"separate\" the hard graph pairs.\n\nSTRENGTHS:\n1. The authors provided rigid mathematical formulation and proof. The problem was well-defined and formulated in the paper.\n2. The organization of the paper is clear. The authors first discussed the problem, followed by theoretical analyses and proof. Model architectures and experiments are then provided to support the theoretical claims.\n3. The theoretical results are general and can be potentially applied to a wide range of models.\n\nWEAKNESSES:\n1. The experiments are inadequate. The authors only tested their model and the baselines on small-scale synthetic datasets. The dataset used in the paper only contains small molecular graph pairs whereas, in practical 3D point cloud scenarios, there can be easily thousands of points. Furthermore, at such a scale, many traditional GNN-based networks including MACE and TFN are also \"separating\" according to the results. The authors may test their model and the baselines on larger practical datasets.\n\n2. The potential benefit of a model being \"separating\" was also not experimented. Eventually, we want models to output some meaningful values in the classification or regression task. The author may experiment with such tasks to demonstrate the capability of graph isomorphism tests indeed help with representation learning.\n\n3. The potential application of the algorithm is greatly limited by the assumption that the graph is complete. The authors also mentioned in the future work section that the proposed model scales as $O(n^4)$ with respect to the number of nodes, which is prohibitively large even for small point clouds.\n\n4. The writing of the manuscript needs to be improved in some places. All references were not in parenthesis, making it hard to read the manuscript (e.g. line 143 when quoting the WL test). The quotation marks are not paired. The sections are also a bit strange. I would suggest making the related work and future work a separate section and the current Sec 3 into a subsection (as it is also a theoretical analysis).\n\nQUESTIONS:\n1. Regarding the experiment part, how will the proposed model behave when scaling up to larger practical datasets? (See Weakness 1)\n\n2. Regarding the experiment part, will the capability of graph isomorphism tests help with representation learning and other downstream tasks? (See Weakness 2)\n\n3. The proposed model scales as $O(n^4)$ with respect to the number of nodes. How can it scale up to practical point clouds with thousands of points? Can we drop or relax the *complete graph* assumption? This problem is crucial as the locality assumption is crucial for GNNs. \n\n4. Will the theoretical results hold if locality is assumed? That is, each vertex only aggregates information from a subset of the vertices (its nearby neighbors). I doubt they would also hold under the locality assumption. Consider the simple case of two regular triangles and one regular hexagon, any rounds of 1-EWL will not be able to distinguish from them.\n\nLIMITATIONS:\nThe authors have mentioned the limitation of this work in the manuscript and no further negative societal impact is expected.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes neural networks for point clouds toward modeling of geometric phenomena. It considers the application of message passing networks/GNNs to Euclidean graphs, whereby a variation of the well-studied k-WL test is adapted to point clouds by using a complete graph on the point cloud and making use of Euclidean pairwise distances. To this effect, the authors propose the k-EWL test and show that: (1) For k=1, two iterations of message passing are sufficient to separate most point clouds in any dimension, (2) A single iteration is sufficient for all point 3D point clouds when k=3. Furthermore, additional differential architectures are proposed and demonstrated to have similar separation power as k-EWL tests.\n\nI think the paper has some promise and is generally well-written. But it can be strengthened by better motivating the problem and providing more detailed experimentation, ideally on chemistry/molecular datasets (given that this was cited as a motivation/application in the intro). I think the paper needs some more work, but addressing some of these points would make me open to raising my score.\n\nSTRENGTHS:\n(1.) The paper is generally easy to follow and well-written. I understood the definitions and theorem statements without any problem.\n(2.) GNNs + point clouds seems like an underexplored area, and the authors make progress in this area (motivation nonwithstanding).\n\nWEAKNESSES:\n(1.) I think there is some motivation lacking for why one wishes to separate point clouds via GNNs, or why it is desirable to construct variants of k-WL. The paper hints at the importance of this for chemical applications, but there isn't much discussion about this beyond the intro.\n(2.) The formulation of EWL doesn't seem novel; it is simply a standard MPNN on a complete graph with use of distance as an edge feature in the update rule (in the framework of Gilmer et. al 2017). SEWL seems more interesting, but the motivation is a bit lacking.\n(3.) The experiments could be stronger. While the authors provide experiments on synthetic datasets of point clouds demonstrating effectiveness of the proposed architectures, the paper could benefit from some experiments on real-world chemistry tasks, as chemical applications, biological molecular datasets, etc. were cited as a motivation in the intro.\n\nQUESTIONS:\nSuggestions:\n\n(1.) Make the motivations in the intro stronger (see weakness above).\n(2.) Including further experiments, e.g., on real world chemistry datasets (in line with the motivation in the intro) would highly strengthen the paper.\n\nQuestions:\n\n(1.) Could the authors comment on non-neural net based approaches for the task? How does the proposed approach compare to these? The methods in Table 1 seem to be all NN-based.\n\nLIMITATIONS:\nSome limitations are highlighted in the Future Work section at the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the theoretical completeness of neural networks for Euclidean/3D point clouds, from the perspective of whether they can distinguish all non-isomorphic point clouds. \n\nKey theoretical contributions include showing that variations of the k-WL graph isomorphism test are complete for 3D point clouds, and that distance-based 1-WL tests are complete for *almost all* point clouds (measure theoretic perspective). \n\nThe work also demonstrates that a GNN can be designed with the proposed completeness guarantees, and sanity checks the theoretical results on synthetic counterexamples from previous studies.\n\nSTRENGTHS:\n- This work shows that adaptations of the k-WL hierarchy of graph isomorphism tests can be 'complete' on 3D point clouds. I believe this is a **novel** theoretical contributions for neural networks on point clouds in Euclidean space.\n\n- I believe the findings are **significant**, as neural networks on Euclidean graphs and point clouds are an emerging area of interest from both theoretical and applied perspectives.\n\n- The paper is **well written** and **clear** in terms of presentation:\n    - The Introduction does a good job highlighting the research gap.\n    - The coverage of related work in Section 1.1 is useful.\n    - Section 2 makes a good bridge from WL to the Euclidean setting.\n\n- I went through the proofs, which are correct to the best of my understanding.\n\nWEAKNESSES:\n- It seems challenging to translate this paper's ideas into practice as the model's parameters depend on the number of points $n$ taken as input. This probably makes it very difficult to build a trainable model **while retaining** theoretical guarantees.\n    - The authors are upfront about this when discussing limitations.\n\n- Beyond sanity-checking the theoretical ideas on the counterexample from Pozdnyakov-Ceriotti, 2022, the synthetic experiment does not seem to provide any further insights into practical instantiations of the ideas in this paper, or about this class of models more broadly.\n\nQUESTIONS:\nQuestions and clarifications:\n\n- Regarding E(n) Equivariant GNN (Satorras et al.) being less expressive than 1-EWL: won't more than one iteration of E(n) Equivariant GNN be able to distinguish between the counterexample of Pozdnyakov-Ceriotti, 2022?\n    - The appendix states that the experiments use the QM9 variant of E(n) Equivariant GNN without the position updates. If you meant that this version is less expressive than 1-EWL: yes, in that case I agree, but that version of the model is not what I as a reader would usually consider E(n) Equivariant GNN. That model is invariant.\n\n- Regarding the relationship between Theorem 3.2, Figure 2, and 2-SEWLnet: technically, if a layer is injective or complete (as you prove for one iteration of 2-SEWL), **why do we even need to stack multiple of them?**\n\n- Regarding Theorem 2.1 and Theorem A.2:\n    - The main takeaway here is that 1-EWL is sufficient to separate almost all point clouds. The reason is that, for counterexamples which cannot be separated by 1-EWL, the size of the manifold that those counterexamples belong to is very small w.r.t. that of all possible point clouds. Is my understanding correct?\n    - On line 533-535, I tried to follow the argumentation but: (1) Could you expand on why the dimensionality is <= 3n-1? (2) Basu et al. is a textbook; is there a better reference? Is this a very simple result?\n\n- Regarding experimental setup: Why were different # of layers used for different models?\n\nSuggestions:\n\n- In Figure 1, it may be useful to draw the actual graphs and also state the exact figures from the Pozdnyakov papers that each graph is taken from, to be least ambiguous. \n- Fix the citation for the E(n)-GNN paper (it should appear as Satorras et al., 2021 and include the rest of the authors).\n- Consider adding the equation for 2-EWL after eq.4.\n- Consider discussing the ACE framework and MACE in Related Work, as this is a framework for building a complete basis for equivariant functions on a set of points up to some interaction body order.\n- Typo in line 235.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations but not discussed any potential negative social impact.\n\nBeyond what the authors mention regarding practical instantiation of their models, one major theoretical limitation is that the framework is restricted to complete geometric graphs, and the construction of complete/universal models for the general sparse graph setting remains an open question. This my be worth reiterating.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper seeks to theoretically demonstrate the complete determination of point clouds, up to permutation and rigid motion. The authors formulate a Euclidean variant of the 2-WL test, effectively illustrating the separation capacity of the Euclidean Graph Neural Network on highly symmetrical point clouds.

STRENGTHS:
1. The paper delivers a theoretical exploration of point cloud completeness.
2. It discusses the separation capability of the Euclidean Graph Neural Network in high-dimensional representations.

WEAKNESSES:
1. In appendix Line 564, what does $(\star)$ stand for? 
2. Does the proposed method risk confounding reflection equivariance?

QUESTIONS:
See above

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10