Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work extends the concept of steerable kernel to nonlinear transformations in MLP. The steerable MLPs are then incorporated into recently introduced equivariant message passing layers thus forming Steerable E(3) Equivariant GNNs (SEGNNs). At the outset SEGNNs are generalisation of EGNNs (Satorras et al., 2021) with non-isotropic message functions utilising relative pose information instead of mere distances. When applied to molecular property prediction tasks, it achieves mixed results on QM9 data and state-of-the-art results on IS2RE task of OC20 dataset.
main_review: The paper is well written. The main strengths of this paper are the strong empirical results on recently introduced benchmark (OC20) for learning on small molecules. Authors put down a good effort to jot connection between various convolutional and message passing algorithms in the main paper. Although I enjoyed reading the manuscript I think it would benefit from a clearer presentation of the novel technical contributions. For example, some form of comparison using pseudo code will help readers.

The overall contribution and novelty is minimal. As pointed earlier, SEGNNs is an extension of EGNNs (Satorras et al., 2021) wherein the linear layer and nonlinear activation within MLPs are replaced by their steerable version. The steerable linear transformations was earlier proposed in (Anderson, Fuchs, Thomas) while steerable activation is extended from (Wieler).

Other drawback and feedback: 

1. For comparison on QM9, SEGNN performs better than EGNN on first six properties. However, SEGNNs performance on last six properties (non-energy ones) is much worse than previous state-of-the-art. Author points that more involved architecture might perform better. Given the level of complexity in SEGNN, I feel SEGNN is quite involved. How about increasing order or cutoff radius ?
2. From an ablation study on cutoff radius and order provided in Table C.2, I noticed that result improves considerably on increasing the cutoff radius. Can you also provide ablation results for higher cutoff radius (say at 7, 10, 15 A) ? Will lower order and increased cutoff radii give similar result ?
3. I suggest that for Table 2 and elsewhere, please present all baselines and results with fixed number of decimal points. This makes it easier to compare across models. Also, for each of the properties, please highlight the best model in bold. 
4. The model claims scalability to large molecules due to the use of fewer MP steps. But the application of same on large molecule datasets is missing. OC20 contains maximum of 70 atoms which is not large enough. Some datasets such as GEOM-Drugs contains max. of 180 atoms. Moreover, given that the model is applicable to any physical system, I would like to see the performance on fluid simulation (A).
5. What is the actual difference between SE(non-linear) and SEGNN_G ?
6. The results on N-body system in Table 1 suggest that in comparison to EGNN, SEGNN (order 1) significantly reduce error but at the cost of ~10x slower runtime.
7. What is odd parity ?

A. Learning to Simulate Complex Physics with Graph Networks, ICML 2020.
summary_of_the_review: Drawing connection with steerable network, this work attempts to rectify a missing link in prior art for equivariant network. Although, the technical contribution is minimal, the paper would have stand out if the empirical results were strong. I suggest that authors improve upon their results and further demonstrate wide applicability.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This study presents a improved version of EGNN (E(n) Equivariant GNN, Victor et al. 2021), where the node features and edge attributes are replaced by steerable vectors. In addition, the MLPs in message passing should also be replaced by steerable MLPs. The advantage of steerable vectors on the node and edge is that attributes are not restricted to scalars but also can be covariant information (e.g., vectors, tensors). The experimental results on N-body, QM9 and OC20 demonstrate the effectiveness of proposed method.
main_review: This paper is well-motivated and well-written. I have few concerns below for the authors to address.

Minor concerns:
1. How to understand that using the spherical harmonic embedding of relative positions is including physical information? Is that a pure geometric information?
2. How to include relative force or relative momentum in the molecular tasks, e.g., the QM9 and OC20 experiments used in the paper.
3. I assume that EGNN should be an important baseline of this paper, but seems that EGNN has not been reported on OC20.
4. I'm very curious that why equivarience is important (or say, how to prove it) for the molecular tasks, e.g., QM9 and OC20, which induces a series of equivarient model architecture development. As far as I know, some methods which aren't equivarient perform well on similar tasks, e.g., Very Deep Graph Neural Networks Via Noise Regularisation, Godwin et al.
5. The performance of the proposed method is not very competitive. For example, UNiTE largely outperform baseline methods (SphereNet, PaiNN, etc) on QM9, but the performance gain of proposed SEGNN is marginal. Similarily, GemNet (Klicpera et al., NeurIPS 2021) and NoisyNode (Godwin et al.) perform well on OC20 IS2RE.
6. Considering that the recent GemNet is also a powerful equivarient GNN, would the authors make a more comprehensive discussion to explain the pros/cons between these two methods?
7. Equivarient model is a hot topic today in the community, and they could be developed by different techniques, e.g., integral (group equivarient CNN, spherical CNN,...), tensor product (TFN, SE(3)-Tr, ...), or equivarient coordinates (EGNN, AF2, ...). Would the authors explain more about how should we think about these equivarient models, how to justify whether an equivarient model is better than others, and what's the future directions of equivarient models?
8. I'm not very familiar with the mathmatics in this paper, so I could not guarantee all the theories are correct in the paper. I'm open to other reviewers opinions.

summary_of_the_review: Overall, I enjoy reading this paper, but I have minor concerns. I'm definitely willing to raise my score if my concerns are well addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes SEGNNs, a message passing neural network which is equivariant to 3D rotations.  The message passing neural network is a type of graph neural network consisting of edge networks and node networks.  Equivariance is achieved by enforcing SO(3) equivariance on the node and edge networks using what the authors call Steerable MLPs.  Steerable MLPs use spherical harmonics and Clebsch-Gordon coefficients similar to Tensor Field Networks but without aggregation.  SEGNNs may be considered non-linear convolutional networks with equivariant anisotropic messages.  The base space for the convolution may be taken to be not just spatial, but a state space consisting of, e.g. position and velocity.  The method achieves competitive results with several baselines on an N-body problem and two molecular properties tasks.

main_review: ### Strengths
- There are several existing equivariant graph neural networks including TFN, SE(3)-transformers, Schnet, and E(n)-Equivariant GNNs (EGNN).  I was thus a bit surprised to see there does not exist a full-power equivariant GNN in the popular message passing flavor (MPNNs).  By “full power” I mean able to handle a range of equivariant feature types and learn over a large space of interwiners between them.  Schnet and EGNNs are MPNNs which are technically equivariant but primarily handle scalar features (invariants) and to a limited extent vector features (standard rep) but without representing the full space of equivariant linear mappings.  The current work is thus novel and extends to a very reasonable combination of successful aspects since both MPNNs and equivariant networks with higher order features types have both proved successful in the past.
- In particular the current method uses anisotropic convolutions which is one of the primary advantages of uses non-invariant equivariant feature types. 
- The method achieves competitive results with several baselines on an N-body problem and two molecular properties tasks.
- One of the more interesting aspects of the method is that convolution is taken not necessarily over the usual spatial base space.  The graphs here are embedded in R^3, however, the features may be considered as living over a higher dimensional state space consisting of position and velocity.  In physics, it is often more reasonable to compute over this higher dimensional space, and the author find that doing so improves their model.  

### Weaknesses
- The paper suffers from some lack of precision and formalism.  The authors describe their method as “non-linear” convolution.  With respect to which definition of convolution?  Section 3, in particular is very difficult to read.  The first part appears to try to compare regular group convolutions and steerable group convolutions but without defining either or stating a precise theorem. 
- Section 3 is also claimed as a contribution in the form of “unifying view.”  As it stands, I would not consider that to be the case.    
- The authors claim, for example, that their method provides a new class of activation functions for equivariant NNs, but so far as I can tell, the non-linearity is comparable to TFN, this claim merely stems from considering a NN as a non-linearity. This does not seem like a useful extension of the terminology and obfuscates the contribution. 

### Questions
- Why are TFN and EGNN not used as baselines for OC20? 
- DimeNet, SphereNet, PaiNN are all shown with better numbers for QM9, but an asterisk notes they use a different set.  It is hard for me to know whether it is reasonable for me to use these numbers to compare.  Is the code or data not available to make a fair comparison? 
- On page 7, it is noted that l_a = l_f = 1 usually worked best.  Isn’t this case already considered by SchNet and EGNN? 
summary_of_the_review: Although parts of the paper were difficult to understand and some of the contributions are questionable, the core contribution, making equivariant GNNs in the message passing flavor is a useful step and together with considering higher-dimensional state spaces seems to result in improved performance.  Thus I tend towards accept, but I would prefer revisions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper extends the recently proposed E(n) GNN (Satorras et al.) such that node and edge attributes can contain covariant information. This allows the easy incorporation of geometric and physical information in the MPNN framework in a steerable manner. Excellent results are reported across a set of benchmarks.
main_review: The starting point for this paper is the observation that regular MPNNs (of which CNNs can be seen to be a special case) get their power from the fact that node features (which are usually invariant scalars) are transformed and propagated in a non-linear manner. Work on equivariant GNNs  is more general, but they still linearly transform the graph and the non-linearity is only obtained via point-wise activations. The authors propose to generalize this formalism and propose E(3) equivariant graph networks, viewing the message passing phase itself as performing non-linear convolution. Further, covariant information is incorporated into the nodes and edges, and they also transform equivariantly. A major inspiration is a recent work by Satorras et al. Indeed, the paper can be seen building on it directly while adding in what they call "non-linear convolutions" and augmenting the nodes and edges to include covariant information. 

To go into more detail about some of the above points: the usual message passing formalism is defined through equations (1) and (2). phi_m is the message function and phi_f is the aggregation/update function. phi_m and phi_f are required to be equivariant, specifically to the group E(3). Furthermore, physical and geometric information is incorporated in the node and edge attributes as steerable vectors which transform covariantly as the message passing propagates. For this, the authors define steerable MLPs on page 3 (same functional form as MLPs, but the weight matrices are conditioned on geometric information using the CG products, and thus support covariance). The exact form is encapsulated in equation (6). Section 2.1 is essentially the same scheme as in Satorras et al. but with the steerable MLP added in. Section 3 goes over the related work on viewing message passing as convolution. The usual convolution layers are discussed first (eq. 6), followed by their usual treatment in point clouds aka "point convolution." The important point is that the matrices W (the convolution kernel) are conditioned on relative positions. The exact form depends on the type of input data. The setup used in the paper corresponds to a steerable convolution given in equations (12) and (13) (the relative distances are embedded in the spherical harmonic space). Because of the CG products that W crucially relies on, the authors interpret their method as performing a non-linear convolution (the approach of Satorras can be seen as corresponding to the special case when the kernel has an isotropic form). 

Quite simply, I see the paper as proposing an extension of the approach of Satorras. The message passing scheme is the same. The relative distances are embedded into the spherical harmonic space, the node and edge attributes are steerable, and the message passing ensures these are transformed covariantly. In this sense, I feel like the paper has limited novelty, or at least overstates its contributions (this happens frequently throughout the paper).  

In the beginning one gets the impression that something new is going on with "non-linear convolutions", but it just amounts to taking CG products, an idea that has been used in other contexts quite successfully 2-3 years ago. It is also said that incorporating steerability into GNNs is a new contribution -- this is only true with respect to the work of Satorras which just passes invariant messages, which was indeed pleasantly surprising at the time in terms of performance and simplicity. For instance the paper "Covariant Compositional Networks for Learning Graphs.", Risi Kondor, Hy Truong Son, Horace Pan, Brandon Anderson, Shubhendu Trivedi, has considered this problem in detail. The case for steerability is mentioned in the abstract, and the intro. Besides, the "type 0" and "type k" message passing types are also discussed in detail later. The difference is that node and edge attributes are not required to be covariant with respect to some Euclidean group. However, the main idea there, although it just deals with the permutation group, is about steerability. One can replace the defining representation there with a representation for a group of interest, along with attaching steerable vectors on nodes and edges, and their approach will also work with the same import as reported in this paper. Papers such as "Vector Neurons: A General Framework for SO(3)-Equivariant Networks" Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, Leonidas Guibas, are not cited and should be discussed properly, along with some related papers such as Jing et al. (2020). These are more related to this paper than the author let on. 
Taking tensor products can also become prohibitive quite quickly, unless I missed it (and I am sorry if I did), I don't see a discussion about this in the main paper (nor about what L is considered and what effect it has on the runtime versus accuracy curve). 

The experimental results reported in the paper are quite encouraging, and for me the hallmark of the paper. 

Minor comment:
Page 6, below equation (15): "There are, however, two import differences" -> "There are, however, two important differences"
summary_of_the_review: I agonized over the rating to give this paper. On one hand, it reports a clear and unambiguous benefit of incorporating covariant information in the node and edge attributes. It is also well-motivated and well written and is definitely valuable. However, the technical contribution of the paper at this point (in how the equivariant neural network literature is progressing) is relatively minor. The paper also seems to overclaim its contributions, one case being in the sense of how it relates to steerable graph networks (although this happens because they are positioning themselves in contrast to Satorras et al., Thomas et al., etc). While I am a big fan of simple methods that just work well, I feel conflicted about this paper. I am currently rating the paper a 6 (which is mostly due to their experimental results), I would be willing to adjust my rating if the authors can present arguments showing that my concerns are unfounded or not rational.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a kind of Steerable E(3) Equivariant Graph Neural Networks (SEGNNs), exhibiting these merits: 1) Compared to EGNN (Satorras et al., 2021) and other concurrent works, SEGNN does not restrict the message passing to be scalars but to be steerable features, and it also permits the injection of geometric and physical quantities into node updates. 2) Compared to previous steerable architectures (such as the works by Thomas et al., 2018 and Fuchs et al., 2020), SEGNN employs more general steerable convolutions, built upon the message passing network (Gilmer et al., 2017). The experiments are conducted on several benchmarks: N-body, QM9 and OC20.
main_review: I have listed the contributions of this submission in the summary above. I will mainly focus on the questions below:

1.	This paper is not easy to follow. I understand that this is not owing the writing (the presentation is good anyway) but due to the abundant preliminaries to understand the geometry of steerable functions. The authors are suggested to separate the introductions of the preliminaries, related work, and the proposed methodology, instead of introducing them as a whole (section 2 and 3). 

2.	The authors claim that the proposed SEGNN (Eq. (16)) is a general form of various previous steerable models (Eq. (14-15)). In my understanding, the main difference of SEGNN from previous steerable models (Thomas et al., 2018 and Fuchs et al., 2020) is that the hidden feature h in Eq. (16) is now a concatenation of the node steerable features and the squared relative distance. The idea of making the MLP (i.e. W) conditional on the attribute a_{ij} (e.g. be a spherical harmonic embedding of xj-xi) has been explored before. So, it seems the contribution is minor. Do I miss something? Perhaps, the authors should specify the form of each previous model under the same denotations and pipeline, from which we can easily check the difference between different models.

3.	 In terms of the nonlinearity, the authors claim that “Although powerful, such layers only (pseudo1) linearly transform the graphs and non-linearity is only obtained via point-wise activations.” However, in my opinion, Eq. (16) first computes the linear steerable convolution and then performs the nonlinear activation \sigma which is also point-wise. So, why we have the claim that the convolution is nonlinear? By the way, what is the form of \sigma? Does it directly follow the previous papers?

4.	I do not check the very details of the experimental part. Just justified from the performance, it seems on QM9, the performance between SEGNN and EGNN is close. Given that more complexity is involved in SEGNN for the computation of Spherical harmonic and Clebsch-Gordan products, the benefit of involving steerable features is not well verified experimentally. Besides, on OC20, there is no comparison with EGNN, why? Regarding the ablation studies, the difference between SEGNN-linear and -nonlinear is unclear. And it is suggested that the authors should compare their SEGNN with previous models under a more complete and fair setting. Since they are claimed as a specific case of SEGNN, the authors can keep other parts unchanged and check which component works actually. 

summary_of_the_review: I generally think the proposed general steerable GNN is meaningful. Yet, given certain unclarities related to the novelty and experimental significance, I initially suggest that it is just marginally abve the acceptance bar.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: Motivated by a wide array of applications across the physical sciences, there has recently been a flurry of interest in building neural networks that are equivariant to the symmetries of the Euclidean group in three-dimensions. In my opinion, this paper presents a particularly elegant and general framework for building equivariant networks that generalizes a number of recent results. There are two essential ingredients: 1) steerable linear transformations and 2) steerable nonlinearities. Both pieces are defined over direct sums of steerable vector spaces and, with these two ingredients, it is easy to promote any message-passing neural network to an equivariant one. By describing the problem in terms of direct sums of steerable vector spaces, there is sort of a nice “type checking” nature when wanting to include new geometric quantities.
main_review: In my opinion, this is an outstanding paper that synthesizes an exciting area of recent work into a relatively authoritative general framework. I found the exposition to be extremely clear, especially given the technical nature of the subject matter. I also really liked the related works section that did a great job of positioning this paper in the context of the literature. Finally, the experimental results and ablation studies are thorough and seem to show a significant advantage for equivariant networks.

To summarize what I thought were the strengths of the paper:

1. The exposition was very clear throughout, but especially in the description of the model, the introduction of steerable vector spaces, the affine transformations, and related work. The supplementary material was also very thorough and contained a lot of really interesting and useful information. I would almost encourage the authors to publish a longer form version of this paper with everything in the main text.

2. I really liked the identification of the direct sum of steerable vector spaces (including multiplicity) with a type system that makes it straight-forward to add new physical quantities.

3. The formulation seems very general; while the authors apply it to MPNNs as defined in Gilmer et al. it seems like you could plug in the equivariant MLPs into any graph-network architecture.

4. The included code looks relatively clean; do the authors plan to open source it?

5. The empirical results seem very strong and three pretty different datasets. The Open Catalyst results are particularly nice to see, since the dataset seems much harder / larger than QM9 or the N-body system.

6. The ablations seem pretty thorough and it’s nice to see that equivariance seems to improve results of the l=0 (invariant) baseline.

While I did really like this paper, I do think there are a few places where it could be a bit more clear.

1. Although most of the design decisions seemed quite reasonable, I thought the paper could do a better job describing how the conditioning vectors, a, were chosen. Are they always the node or edge features? For the N-body system, were the conditioning vectors the velocity vector or the displacement vector to the center of the system? Did you try any other choices?

2. I would include a description of the gated nonlinearities in the SI of the paper (if not also in the main text). I would also be a little bit more straightforward about the fact that you use gated non-linearities. In particular, in the text you currently write [1] but in the text you only ever use gated non-linearities. I would lead with the choice you made and then mention that one could use any equivariant MLP.

3. One of the things I liked about this paper was how straightforward the proposed equivariant MLPs are. The submitted code uses e3nn, which (while very nice) is a pretty complex codebase. Can the authors discuss this choice vs writing things from scratch using Clebsch-Gordan coefficients as described in the paper?

4. Do the authors ever use the fact that the weights can be made dependent on a parameter? I couldn't find any place in the paper where this was used (perhaps I missed it), but if it's not used then I'm not sure it adds much to the discussion.

[1] _The common recipe for deep neural networks is to alternate linear layers with element-wise non-linear activation functions. In the steerable setting, careful consideration is required to ensure that the activation functions are equivariant; currently available classes of activations include Fourier-based (Cohen et al., 2018), norm-altering (Thomas et al., 2018), or gated non-linearities (Weiler et al., 2018). All these can be used in alternation with (6). The resulting steerable MLPs themselves in turn provide a new class of steerable activation functions that is able to directly leverage local geometric cues. Namely, through steerable node attributes a, either derived from the physical setup (forces, velocities) or from predictions (similar to gating), the MLPs can be applied node-wise and generally used in steerable feature fields as non-linear activations._

summary_of_the_review: A very strong paper that provides an elegant framework for building E(3) equivariant graph networks. The proposal is very general and well-motivated; it is also well-situated within the literature. The exposition is generally high quality, and the experiments are rigorous and compelling. The supplementary material gives thorough background and the code seems pretty clean. A few comments / questions about some details of the paper remain outstanding.
review_title: null

Meta-review: This work combines steerable MLPs with equivariant message passing layers to form Steerable E(3) Equivariant GNNs (SEGNNs). It extends previous work such as Schnet and EGNNs, by allowing equivariant tensor messages (in contrast to scalar or vector messages). The paper also provides a unifying view of related work which is a nice overview for the ML community. It is overall well written, but would benefit from further revision to improve readability in some parts (in particular section 3, cf. reviews).
It shows strong empirical results on the IS2RE task of the OC20 dataset and mixed results on the QM9 dataset.
Decision: Accept (Spotlight)