Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the problem of measuring to what degree the the function is invariant to certain transformations. The authors suggest to use the Lie Derivative to measure such a quantity. Then Local equivariance error (LEE) is introduced. The paper demonstrates how different parts of various models contribute to the equivariant properties of the model in terms of LEE and how different models demonstrate a correlation between the global LEE and the resulting accuracy.
strength_and_weaknesses: The paper is well-written. I found it easy to read. The storytelling of hte paper is of high quality. The derivations are accurate and the math notation is well-understood. 

**Strengths**
1. The authors pay attention to the problem which is less studied in the community - the problem of measuring the equivariant properties of various models.
2. The theory of the paper is solid. And demonstrates a very good understanding of the problem.
3. The experimental results of the paper demonstrate that such a method is useful for the analysis

**No Strengths**
I don't see any significant issues, which may affect my rating drastically. However, I would like to address several questions and suggestions, which may improve the quality of the paper.
1. The related work misses a section on why we are so motivated to measure the equivariance error. There were several papers which demonstrated that the model's accuracy and its equivariance error are highly correlated. In [1] the authors take a network and train it by minimizing the equivariance error which leads to a better accuracy than any other counterparts. In [2] the authors compare different scale-equivariant models, conclude that the equivariance error is not 0, and demonstrate that the lower it is the more accurate the model becomes.
2. [Does not affect my rating. Just a pure interest] Although it was not possible to include this paper beforehand, there is a recent paper which considers a similar problem. In [3] the authors introduce a similar metric for measuring the symmetric properties of the model. What is the main difference between their approach and yours? 
3. Let us consider two neural networks, which perform binary classification. The very last layer outputs a positive or a negative value for either class 1 or class 2. We initialize them with the same weights. However, for network 2 we multiply the output by $10^6$. Do I understand it correctly, that for network 2 LEE will be $10^6$ larger than LEE for network 1? If so, can we really compare two neural networks based on Eq 4? Do we need an extra step of output normalization?
4. In supplementary materials, in A3, paragraph 2, you mention that there is an inequality between the $\mathcal{L}$ of the network and the sum of the $\mathcal{L}$  of its layers. Let us consider a network, for which the very last layer just multiplies everything by $0$. Such a network is absolutely invariant, thus $\mathcal{L} = 0$. However, the $\mathcal{L}$ of the layers of the network can be arbitrary large. Thus, Figure 4 does not really tell us anything about the network as a whole. Could you make a comparison where you plot side-by-side the cumulative loss as a sum per-layer losses. And a series of losses calculated for all truncated subnetworks of the original networks $\text{subnet}_k = f_k \circ \dots \circ f_1$


- [1] Khetan N. et al. Implicit Equivariance in Convolutional Networks, preprint 2021
- [2] Sosnovik I., Moskalev A., Smeulders A. Disco: accurate discrete scale convolutions, BMVC 2021
- [3] Moskalev A. et al. LieGG: Studying Learned Lie Group Generators, NeurIPS 2022
clarity,_quality,_novelty_and_reproducibility: The provided code and the text of the paper allow one to reproduce the approach.
summary_of_the_review: The paper is worth sharing with the community because of its potential significant contribution to the field. Minor changes may improve its quality.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work compares the ability of models with built-in equivariance versus those without in how well they ultimately learn equivariance.  The majority of experiments and analysis focuses on the case of translation equivariance in image classification comparing CNNs to ViT and Mixer, although rotation and scale are also considered.  
- In order to measure equivariance error, the authors define local equivariance error, LEE, based on infinitesimal action of the symmetry group on the function space defined by each layer.  This definition allows the authors to decompose the final LEE of the model in terms of per-layer LEE.  
- The paper puts forth the hypothesis that the interaction of aliasing, the aberration which result from discretizing a continuous signal, and continuous symmetry is the main driver of equivariance error in all models.   This hypothesis is supported by Theorem 1, which directly computes the equivariance error due to aliasing under translations, and by empirical analysis of layer-wise LEE showing that activation layers contribute most towards LEE.  
- Experiments show model performance is correlated with degree of learned equivariance and that there is not a significant difference between CNNs and ViTs in terms of equivariance learning (if anything ViTs have lower LEE.)  
- Several architechure-agnostic solutions to better symmetry learning are tested including BlurPool, larger models, more data, and SSL pretraining.  All such interventions (to various degrees) do improve equivariance learning. 
strength_and_weaknesses: ## Strengths
- The paper considers an important question in the field.  Instead of proposing a novel architecture, this work provides a useful comparison of the strengths and weaknesses of existing state-of-the-art methods from the viewpoint of their ability to learn equivariance.  It could provide insights for improving equivariance in both models with built-in and no built-in inductive bias or help to make model selection easier. 
-  The experiments are really quite extensive.  I agree with the authors' claim that the scale and scope of their analysis are helpful in making the trends clear against the noise and other factors contributing to model performance and equivariance error.  Given the goal here is to say something broad about CNNs or ViT in general, it is essential a wide variety of architectures in both classes are considered. 
- Previously works on equivariant neural networks have not been overly focused on aliasing.  The equivariance is usually analyzed from the point of view of continuous signals and equivariance is proved in the continuous case.  Empirical measurements are used to assure that discretization errors are acceptable.  However, it seems clear that discretization is important and limiting the potential advantages.  For example, many applications of E(2)-CNN use small discrete rotation groups instead of continuous groups.  This work addresses aliasing head-on and provides evidence of the extent to which it contributes to equivariance error and specifically where (non-linearity). 
- The definition of LEE and the observation that LEE can be broken down on a layer-by-layer basis seems like an important contribution. Previously, there was not a precise way to quantify the contribution of each layer to EE.  Practically, one could measure the EE over a single layer (as is done here for alternate EE metrics, I believe), but this is not guaranteed to combine in any reasonable way.
- The result shown in Figure 4 and subsequent insights are very interesting. I appreciate that several different comparisons are done: CNNs vs. ViT, different symmetries, different EE metrics.  The finding that non-linearities contribute the most to LEE raises interesting questions.  What about the fact early layers seem to contribute more?  The observation the contributions to EE for each layer are similar for different symmetry groups suggesting aliasing the root cause is interesting. 
- Section 6 discusses several ways to improve equivariance learning given that baking equivariance seems to come up short. 


## Weaknesses / Questions
- Since the task is invariant (equivariant with output having trivial rep), the representation must go from having non-trivial group action to trivial group action.  In practice, this invariance can be baked in either by mapping to invariants and having later layers be unconstrained or having equivariant layers and then a final invariant layer.  Thus for a classification task, it is not necessary for the intermediate layers to have low EE for the final NN to be invariant.  I feel this could somewhat undermine the assumptions of the layer-by-layer LEE analysis.  Could the authors comment?  I would feel more confident if the experiments were performed for a task where the output representation was equal to the input representation, for example, image segmentation.  
- Another potential aspect overlooked in the setup is the potential for a non-trivial action on the channels of the hidden features.  This absolutely key to making equivariant neural networks work. My understanding is that LEE is assuming the channel representation is trivial (though I could be mistaken).  Thus we could be measuring high LEE when in fact it would be low if we knew the way the channels should transform.       
- As noted in the paper, CNNs are often not very equivariant (compared to many other equivariant networks) due to discretization, pooling, and stride. I'd like to see results showing the equivariance error for the CNNs vs. ViTs at initialization and throughout training.  If the compared CNNs do not have low EE at initialization, then the argument that they have built-in equivariance is not very strong.  I'd feel more comfortable with a comparison between E(2)-CNN and unconstrained networks for rotational equivariance error.  In this case, I think the E(2)-CNN architecture would have relatively low EE.
- Although it was posted on the arXiv only in October and has now been accepted to NeurIPS (per the schedule), I believe that LieGG (https://arxiv.org/abs/2210.04345) covers some similar territory in terms of using lie algebra action on functions space to analyze the learned symmetry of the model.  That work also includes analysis of "symmetry bias" (analogous to equivariance error) and per-layer analysis (see Fig.5).  I don't consider it to pre-empt this work, both because they are concurrent and because the focus is very different, however, I would like to see a comparison added.  
- I take some issue with the claim that it is surprising that non-equivariance models attain lower equivariance error after training.  Equivariant networks certainly have lower equivariance error at initialization that unconstrained networks, but learning equivariance is necessary in order to get good performance, so it follows that if an unconstrained network achieves better performance than an equivariant one, it may well have lower trained equivariance error (EE).  Consider, for example, a regression task.  Assuming the ground truth is perfectly equivariant, then the equivariance error will be bounded by 2*(approximation error) for any sample by the triangle inequality.   Since you consider a classification task, this relationship is a bit more obscured, but still well represented by your results.

## Questions 
- I'm unclear on the aliasing operation $A$ versus the map $\mathrm{Alias}(n)$.  Can you clarify?  In particular, on page 3, it seems $A \colon \mathbb{N} \to \mathbb{N}$, but on page 6, it seems $A \colon (\mathbb{R}^2)^{\mathbb{R}^c} \to (\mathbb{R}^2)^{\mathbb{R}^c}$, i.e. images to images. 
- Could section 4 avoid referring to flows and be written in terms of the exponential map $\mathfrak{g} \to G$? 
- In section 5, you prove that Alias results in translation LEE.  I didn't fully understand why this points to the non-linear layers.

## Minor Points 
- The abstract says "introduce the Lie derivative," I would re-word to make clear Lie derivatives are pre-existing work, they are merely being introduced as a method here. 
- I'd replace "it's" with "it is" since this is formal writing. 
- In two places $\mapsto$ appears when it should be $\to$.  $\mapsto$ is for elements and $\to$ is for sets.  As in $f \colon X \to Y$ mapping $x \mapsto y$. 
- Page 4 "the operator $\rho_{21}(g)[\Phi^t_Y]$ "  Should it be  $\rho_{21}([\Phi^t_Y])$ ?  
- In the notation for LEE in eqn. 4, you might want to include the dependence on $X$. 
- Section 5, first sentence "equivariance" -> "equivariant" 
clarity,_quality,_novelty_and_reproducibility: Very clearly written.    High quality analysis and  rigorous experiments.  Code included for reproducibility.  The work is novel, but I'd like an added to comparison to a very recent work (LieGG).  
summary_of_the_review: I very much liked reading this paper.  A study comparing learned symmetry to built-in symmetry is very useful to the field today. Equivariant architectures are increasing popular and at the same time, non-equivariant methods such as transformers are overtaking them in some domains.  The experiments are extensive and the analysis is clear and rigorous.  I think the conclusions will be useful to practitioners who are choosing between different architectures. I hope the tool of layer-wise LEE analysis could help to improve equivariant neural networks.    I have some disagreement with how some parts of the study are set up, things I would like to do differently, and ultimately with the implication that built-in equivariance is often unnecessary.  However, that is really to say I think this paper will inspire much future research both supporting, extending, and arguing with the hypothesis studied here.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes Lie derivative, which is a simple method for measuring equivariance. The authors use the proposed method to analyze the equivalence properties of hundreds of pre-trained models including most popular deep learning architectures for classification. They also demonstrate a layerwise breakdown of the equivariance error for each model, which only becomes possible under their proposed metric. From analysis, they separate the impact of architectures from factors such as model size and training method and identify the contributions of different modules in each model. They also show the interesting finding that large models such as ViT and Mixer are actually "more equivariant" than CNNs which use convolutions with build-in exact equivariance.
strength_and_weaknesses: **Strength:**
1. A simple yet useful technical contribution. Interesting empirical findings that can inspire future research.
2. Solid work with theoretical and empirical support.
3. Nice presentation of the proposed method. Clear motivation.

**Weaknesses:**
1. The explanation for aliasing and equivariance could be more readable to people who are not familiar with signal processing. One solution is that the authors can provide more background knowledge in the appendix, and only keep the high-level essential conclusions and intuition about aliasing in the main text.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
The paper is well-written and it is a pleasure to read. All parts are well-motivated and well-paced, except for the aliasing part in the background section, which is still a bit difficult to grasp without extra reference (But I understand that this is a hard job with limited space).

**Quality:**
This paper studies how to measure equivariance and compare a wide range of existing model architectures in terms of equivariance. It is very important for us to better understand the role of equivariance in deep learning and the relationship between equivariance and generalization. The proposed method is very easy to use (can therefore inspire further investigation) and has a solid theoretical foundation. The empirical results are solid and support their claims well. The main mathematical results seem to be all correct to me. 

**Novelty:**
Although there are other metrics to measure equivariance, the proposed local equivariance error based on the Lie derivative has notable advantages (e.g. can analyze the contribution of each layer, local but is connected to global equivariance, very few hyperparameters) and has never been explored in previous literature. Furthermore, there are several very interesting empirical observations in the paper which I think are very inspiring for the community. Although similar findings do exist in other papers, the investigation has never been conducted at such a scale with a wide range of models and training procedures.

**Reproducibility:**
The authors provided the code. Although I did not run the code, lots of experimental details are available from the repository. They also provided some important information about experiments in the text including a simple implementation using automatic differentiation and additional information about empirical study in the appendix.

summary_of_the_review: I recommend acceptance of this paper mainly based on the following:
1. This paper studies an important question that can inspire further investigations. I think it can benefit both the equivariance community and the broader deep-learning community.
2. The findings in this paper are interesting. The proposed method is simple yet effective. The empirical work is solid.
3. The paper is polished and well-presented.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a method for empirically measuring the equivariance error based on the Lie derivative, and performs an extensive empirical study. Comparing the equivariance error of different architectures such as CNNs, transformers and mixers at different scales and training methods, it is found that lower equivariance error correlates with performance, and that much of the equivariance error can be traced to spatial aliassing due to nonlinearities and other common layers. The paper also presents a very clear theoretical analysis of aliasing, equivariance error and their relation.

The reviewers unanimously agreed that this is an excellent paper, and found few significant weaknesses. I personally enjoyed reading the paper, and believe that it has significant implications for future research. So I wholeheartedly recommend the paper for acceptance.
justification_for_why_not_higher_score: n/a
justification_for_why_not_lower_score: This is a well rounded paper, with interesting and important theoretical and empirical contributions and significant potential for impact on the direction of the field of equivariant networks.
Decision: Accept: notable-top-5%