Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose to enforce soft equivariance constraints in ANN layers through a Gaussian prior on layer weights. Experimental results on a variety of problems support the benefit of the approach perhaps unsurprisingly. The overall contribution feels quite straightforward and incremental.
main_review: Overall the contribution of the paper is to use MAP optimization to propose and intermediate between stricly equivariant layers and non-equivariant ones (MLPs typically). This is enforced by using a Gaussian prior whose convariance is a sum of unstructured (proportional to identity) and structured (contrained to a basis of equivariant matrices) matrices.
Perhaps unsurprisingly, the resulting model achieves a trade off between unstructured approches and stricly equivariant approaches that is beneficial in several examples.

The contribution feels thus quite iterative, while there would be many ways to address the problem more in depth: e.g. can we use sparser priors in ordre to relax equivariance only where/when necessary? Can we implement this soft constraint such that computational benefits of structued models are preserved as much as possible? 


limitations_and_societal_impact: I could not find a discussion on the limitations (e.g. computational complexity).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The idea here is to use a combination of fully connected layers with equivariant layers and use a prior to bias the feedforward layer towards simpler (equivariant) linear maps. Standard residual connections are given as an extreme example where the simpler linear map is the identity operator. L2 penalty on the weights is used to implement this prior. Experiments study this type of model in settings where 1) the exact symmetry holds, 2)  it is partially violated, and 3) the data has no such symmetry. Example application domains used for experiments include physical models and applications in RL, where the correct (approximate) symmetries are known.

main_review: The proposed idea is quite simple yet it seems quite powerful and flexible, in the sense that the model performs well even with misspecified or “approximate” symmetries. The experiments are chosen very well, and the results are supportive. The presentation is quite clear, the paper is an enjoyable read. I have a few (minor) questions:

The material in appendix D about the effect of prior variance seems quite central to your claim. I suggest expanding the experiments to see if similar patter (of robustness to the hyperparameters) appears across different problems and potentially add this to the main body. 

Could you please elaborate on the claim of lines  165-168? Are these only based on empirical observations?

How do you explain the RL experiments in which RPP fails (Hopper-v2, Ant-v2)?

I wonder if presenting this idea as a generalization of residual connection is valid, given that a vanilla residual connection does not have an associated parameter.

Is it necessary to package this idea with the EMLP’s method for finding equivariant linear maps, given that one could use any other method for designing equivariant kernels or tying the parameters in the feedforward layer?

limitations_and_societal_impact: The appendix includes some discussion on the limitations.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method to introduce inductive biases into neural networks in a *soft* way. In a nutshell, it achieves this by combining strict equivariant layers with flexible non-structured layers, akin to how residual layers combine a identity skip connection with a normal linear layer. This induces soft equivariance priors that, in principle, can accomodate both exact and approximate symmetries. 

In a first set of experiments, the paper demonstrates that the proposed RPPs provide a trade-off between unstructured MLPs and hard-equivariant EMLP's in datasets with exact, approximate, and mis-specified symmetries. Then, it showcases the promise of RPPs in the context of reinforcement learning, where recent deep learning-based models have typically been fully unstructured, and for which the results show RPPs provide significant improvements. 
main_review: This paper contains one simple main idea (soft equivariance via residual-type additive layers), but which is motivated very compellingly, executed elegantly, and validated convincingly. It is also written well, and makes for an entertaining read. Overall, I appreciate this type of simple-but-well-executed papers, and I think it makes for a valuale contribution to the conference. Most of my comments and concerns relate to clarity, minor issues, and some requests for additional results. 

Meta-Comments:
* Originality. As far as I can tell, this approach to softly enforce equivariance constraints through residual pathway priors is new, although admittedly I'm not on top of all the recent equivariance literature
* Significance. While the experimental results are somewhat limited to make a definitive conclusion on the applicability, robustness, and performance of RPP-based models, they are promising, making this paper potentially quite significant
* Quality. In terms of quality, albeit simple and limited, the theoretical derivations are correct, and the experimental results seem reliable and replicable
* Clarity. The clarity of the paper can be improved in some aspects, e.g., see comments below. 

Specific Comments:
* I found it counterintuitive that the solutions for the modified inertia setting in Fig 3 are less spread than those of the pure Inertia ones. Given the approximate notion of symmetry encoded in this dataset, I would have expected it to lead to solutions with more variance in their equivariance error. I would appreciate if the authors could comment on this.
* In Sec 5.3, it would be nice if the main text could specifiy whether the exact same architecture is used for all these datasets. If so, how much padding is there on the UCI datasets?
* Table 2 needs some more explanation. E.g., it is my understanding that the Hooper dataset does have exact symmetries, so it's not clear why it's crossed out for 'Exact' in this table. 
* The results in Table 3 and Figure 5 consider only a fully-unstructured MLP baseline. In order to get a full picture here, it would have been nice to have results for some hard-equivariance baseline.
* While I appreciate the discussion on limitations in the Appendix, deferring it to the Supplement instead of the main text dilutes its perceived importance, and goes against the spirit of the 9-th page policy of this year's CFP

Questions:
* In L135, is the prior on the *size*, or on the *magnitude*? 
* In L168, do you mean *less* flexible layer? (or replace A by B), otherwise this seems to contradict the intuition for A/B provided above
* Why does the method fail for the Humanoid-v2 setting in Figure 5? The paper says it 'destabilitizes SAC'. How so? Could this be prevented? How prevalent is this phenomenon? Answering such questions would bring better understanding of the limitations of this class of models.


Typos:
* word in missing in L67
* coma missing in L67
* repeated 'the' L129
* The definition of the space of solutions in L150.5 would be clearer wrapping it around {$ W \in \mathbb{R}^{n_{\text{out}} \times n_{\text{in}}} : \quad \dots $}.
* break<->down L162
* A couple of typos in L194
* The figure reference in L197 is incorrect
limitations_and_societal_impact: Yes, albeit tangentially, in the appendix
review_title: null

Meta-review: This paper presents a simple solution to an important problem - deciding what kind of inductive biases to use when we don’t have full confidence about the type of symmetries that exist in the domain.  The solution is to combine a flexible model with one that has strong equivariance inductive biases.

All reviewers recognize the importance of the problem, clarity of the presentation and the simplicity of the approach.  The proposed method being simple does not diminish its value, and simple methods are typically more applicable to a wide range of problems and more easily adopted by many.

Reviewer wXcB who gave a 5 rating does have a point about the baselines though: this paper mostly contrasts the proposed approach with the two extremes - the MLP without any explicitly symmetry built in and models like convnets that have strong equivariance built in.  It would make the paper stronger to have a stronger baseline, that is for example trained with data-augmentation as a soft way to build some inductive bias into the model.  The advantage of the proposed approach, as the authors explained in the discussion, is that it will be much more data-efficient and generalize much better in data-sparse regions.

After discussion among the reviewers and calibrating across a range of borderline papers we decided to accept this paper based on its clarity and elegance, and believe it can be a useful contribution to the community.

Decision: Accept (Poster)