Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a parameter-efficient neural layer for relaxed equivariance. The main idea is to use a non-stationary integral operator whose kernel depends on absolute group elements as well as relative group elements.  This allows to interpolate between non-equivariant, strictly-equivariant, and strictly-invarint mappings in learning. The proposed method is validated on CIPAR-10 and CIFAR-100 image classification benchmarks, outperforming other strictly and partially equivariant methods. 

strengths_and_weaknesses: Strengths
- Clear motivation for relaxed equivariance
- Simple yet effective idea of non-stationary filter
- Enabling targeted relaxation of a specific subgroup
- Convincing results on MNIST and CIFAR image classification

Weaknesses
- Missing comparison to related work. The proposed method is compared only with G-CNN and Romero and Lohit [18] while there exist other recent related work, e.g., E(2)-CNN, Implicit Equivariance in Convolutional Networks. 
- Limited experimental setup. The experiments are done using a small network on small datasets. It is unclear whether the performance gain is retained when bigger networks (ResNet18, 50) and larger datasets (ImageNet) are used. Furthermore, the benefit of equivariance relaxation is not demonstrated beyond image classification. 
- Missing analyses. Comparison of FLOPs and # of parameters and ablation studies on design choices (activation function..) are not provided. 
- How to set non-frequency parameter w? While the advantage of the proposed method is the controllability of equivarience levels, the performance of a learned model looks significantly affected by the control parameter w as shown in Fig. 4. The sensitivity of performance to w and the determination of w in practice need to be discussed. 

questions: I think this is a good paper, but addressing the weaknesses I described above would strengthen the contribution. 
Please respond to my points of the weaknesses. 

limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Strict equivariance constraints in deep learning may sometimes be overly restrictive for some tasks. This paper proposes a relaxation of equivariance by allowing the convolutional kernel to be non-stationary (thus depending on the absolute input location). They achieve so by introducing frequency parameters, which can interpolate between strict equivariance and element-wise linear product, thus adjusting the "softness" of equivariance. They empirically demonstrate that their proposed soft equivariance leads to better performance on CIFAR10 and CIFAR100 in terms of classification accuracy.
strengths_and_weaknesses: **Strength**
1. This paper studies an important problem: strict equivariance constraints can be overly restrictive and is misspecified for some problems. It would be nice to have soft equivariance and automatically adjust symmetries on data. This paper provides a simple, practical and parameter-effective solution.
2. This paper shows clear performance improvements using the proposed soft-equivariance on CIFAR10 and CIFAR100. It would have a great impact if this improvement is widely applicable to lots of tasks.
3. The proposed solution is new and it seems to solve the over-strict equivariance problems. (However, I think more careful ablation study and potentially more experiments on new datasets are needed in the future)

**Weaknesses**
1. Although this paper shows clear performance improvements for the proposed soft equivariance compared to strict equivariance on CIFAR10 and CIFAR100, it is not convincing enough that the performance gain actually comes from soft equivariance. It would be ideal to have an ablation study showing that, the performance gain does not come from using the random Fourier features. It would be more convincing if the authors could show performance improvements on more tasks and datasets. But I understand that this is not possible during the rebuttal period. So I am not asking the authors to run experiments on extra datasets, just a suggestion for future versions.
2. This paper is not careful enough with mathematical terminology: For example, 
    * (line 166) "cross-product of two vector spaces". I believe they mean the product space of these two spaces, rather than the vector product here (https://en.wikipedia.org/wiki/Cross_product). 
    * (line 175) "let $\hat{k}$ be a Gaussian Processes (GPs)". Firstly, $\hat{k}$ is a kernel function while GP is a stochastic process with a mean function and kernel function. It does not make sense to me by saying a kernel function is a stochastic process. Secondly, the kernel function here corresponds to the convolutional filter, which is different from the kernel functions in GP literature. In GPs, the kernel function needs to be positive semi-definite. Thirdly, I think the authors only use the kernel function here and it has nothing to do with GPs (the probabilistic model)
    * (eq 3) I believe it is only an element-wise product. For me, it is a bit contrived to say this corresponds to the fully-connected layers. Even if you apply the averaging pooling layers, you still have only one output unit? 

(Please correct me if I misunderstand)

questions: 1. How much performance gain comes from the usage of random Fourier features rather than the soft equivariance? Can the authors consider running an ablation study for this?
2. I am very confused by the concept of "kernel" in this paper. On the one hand, it is only the convolutional filters, so it does not need to satisfy kernel properties such as positive semi-definite (See https://people.eecs.berkeley.edu/~jordan/kernels/0521813972c03_p47-84.pdf). However, on the other hand, the authors mention GPs and lengthscales, and use the random Fourier features. All of these are for the "kernels" in kernel methods literature. Furthermore, even though they are using random Fourier features, the neural networks take in the concatenation of features and directly produce a scalar as output. In this case, the kernel function is merely an arbitrary function on a pair of inputs.

I am ready to raise my score if the authors could clarify my confusion and give a convincing justification.
limitations: I have no concern over negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a deep-neural network architecture for relaxing equivariance constraints. The main idea is to use a non-stationary kernel in a group convolution. They propose to model the kernels via continuous kernels parameterized via a neural network. Experiments on CIFAR10 and CIFAR100 demonstrate improvements over baseline models which are strictly equivariant.
strengths_and_weaknesses: # Strengths
The topic of modeling parietal equivariance or learning symmetries from data is relevant and of interest to the NeurIPS community. Overall the writing of this paper is clear and the approach is clearly described. While similar to prior works in continuous kernels, the proposed non-stationary kernel is sufficiently novel.
# Weaknesses
1. The main weaknesses of this work is the empirical validation. While the proposed method is capable of modeling soft equivariances, it is unclear whether the proposed method is better than alternative approaches. For example, one could choose a model that has very few built in equivariance assumptions, e.g. transformer, and perform data augmentation. The model will learn to exhibit soft equiviarance behavior. For example, a Vision Transformer with positional encoding can be trained to be partially shift-equivariant. The current experiment lacks comparison and only conducts experiments on CIFAR10 and CIFAR 100 dataset, which is not very convincing. 
2. The paper could be strengthened if additional comparisons with other baselines were made. For example, a comparison with parameter sharing method [33] or augmentation based method [A] (Augerino also has an equivariant setting (Sec. 3.1)). 
[A] Benton, Gregory, et al. "Learning invariances in neural networks from training data." Advances in neural information processing systems 33 (2020): 17605-17616.
3. The authors propose to use Fourier features when parametrizing the kernel (Line. 193), it would be great to conduct an ablation study on this.
4. Some experimental and training details are missing. It might be difficult to reproduce the results. Is data augmentation being used for the baseline?
5. The authors should report the number of trainable parameters introduced and the model’s inference speed. It seems like the proposed approach would be slower than a standard ResNet. 

# Miscellaneous
* The checklist should have been attached with the main paper and not in the supplemental materials.
* Some references still cites the arXiv version instead of the conference version. For example, Deep-Sets [32] is published at NeursIPS.
* Line 18. "euclidean" --> Euclidean
* Line 75. "continuosly" --> continuously
* Line 186. "fourier" --> "Fourier"  
questions: Please address the concerns in the weaknesses section above. Particularly, I am most interested in a comparison with data augmentation method, Augerino, and knowing whether the baselines use data augmentation in table 1. 
limitations: The paper does not have a limitations section. It would be great to discuss some shortcomings of the approach.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes soft equivariance parameterization of neural networks that can interpolate between non-equivariant linear product thru strict-equivariant convolution to strict-invariant mapping. The contributions of the paper are twofold: 1) improving applications where strict equivariance is suboptimal; 2) providing a way to learn equivariances from data. The main idea is to use a non-stationary kernel for the group convolution. The contribution of this paper is how to parameterize that kernel. While the paper proposes a general approach, which might work for any group, in practice it only shows the idea for T(2) and SO(2). The paper obtains marginal improvements over [18]. 

While I like the idea, I am afraid that the paper is missing experimental and implementation details. Furthermore, can the method be extended to larger datasets and tasks?



strengths_and_weaknesses: Pros: 

1) Natural idea to interpolate between invariance, equivariance and non-equivariance.
2) Effective implementation, improving reasonable baselines.
3) Mostly well-written, with a few question (see below).

Cons: 

1) Computational metrics, such as training and inference time, memory constraints and implementation details are missing.
2) The method provides only marginal gains over small datasets.

questions: Related work: please, consider citing works on data augmentation with self-supervised training objectives as an effective way of baking in approximate equivariance into neural networks, e.g.: https://arxiv.org/abs/2111.00899.

Line 84: please, specify that $A_i$ are the basis vectors of the Lie algebra.

Line 97: please, specify that *the layers of the neural network being a group convolution* (Eq. 1) is actually the necessary and sufficient condition (following Theorem 1 in [11]). Currently, it's not clear what exactly that condition is.

Line 128: "see ??"; there is a broken link; please, fix it.

Line 161: you mean Lie group elements instead of *Lie algebra*? The exponential map maps from the group to the algebra. Actually, isn't this supposed to be a "log" map instead of "exp"?


limitations: I am not sure if the authors have addressed the limitations of the approach properly. In particular, I would like to see implementation challenges and the limitations stemming from the implementation constraints, discussed in Section 5. Ablation studies would be helpful.
review_title: null

Meta-review: metareview: This paper received initially quite mixed reviews, but after a strong rebuttal from the author's, a number of reviewers increased their scores, leaving it with an overall borderline positive rating. The work was praised for the interesting and novel core idea, the potential significance of the work's contribution, generally clear writing, and providing good empirical results for small-scale image datasets. Concerns that remained after the author's rebuttal included the lack of experiments on larger scale datasets or architectures, issues in the presentation/care of some mathematical formulations and notation, and a lack of comparison to certain potential benchmark approaches.

Taking these into account, my own view of the work is that the strengths outweigh the negatives. In particular, I feel like most of the remaining concerns raised are either not reasonable or not appropriate grounds for rejection. In particular, I do not think that direct comparisons to Augerino are warranted beyond the new augmentation experiments in Appendix C (which I would consider promoting to the main paper for the final version) or that the issues with the mathematical formulations are severe enough to be reasonable grounds for rejection. Given that the paper has some clear strengths that all reviewers agree on, my recommendation is therefore that the paper should be accepted.
confidence: Less certain
Decision: Accept