The Lie Derivative for Measuring Learned Equivariance
Authors: Nate Gruver
Marc Anton Finzi
Micah Goldblum
Andrew Gordon Wilson
Venue: ICLR 2023 notable top 5% (2023)

Abstract: Equivariance guarantees that a model's predictions capture key symmetries in data. When an image is translated or rotated, an equivariant model's representation of that image will translate or rotate accordingly. The success of convolutional neural networks has historically been tied to translation equivariance directly encoded in their architecture. The rising success of vision transformers, which have no explicit architectural bias towards equivariance, challenges this narrative and suggests that augmentations and training data might also play a significant role in their performance. In order to better understand the role of equivariance in recent vision models, we apply the Lie derivative, a method for measuring equivariance with strong mathematical foundations and minimal hyperparameters. Using the Lie derivative, we study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. The scale of our analysis allows us to separate the impact of architecture from other factors like model size or training method. Surprisingly, we find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities, and that as models get larger and more accurate they tend to display more equivariance, regardless of architecture. For example, transformers can be more equivariant than convolutional neural networks after training.

THE LIE DERIVATIVE FOR MEASURING LEARNED EQUIVARIANCE
Nate Gruver", Marc Finzi", Micah Goldblum, Andrew Gordon Wilson New York University
ABSTRACT
Equivariance guarantees that a model's predictions capture key symmetries in data. When an image is translated or rotated, an equivariant model's representation of that image will translate or rotate accordingly. The success of convolutional neural networks has historically been tied to translation equivariance directly encoded in their architecture. The rising success vision transformers, which have no explicit architectural bias towards equivariance, challenges this narrative and suggests that augmentations and training data might also play significant role their performance. In order to better understand the role of equivariance in recent vision models, we apply the Lie derivative, a method for measuring equivariance with strong mathematical foundations and minimal hyperparameters. Using the Lie derivative, we study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. The scale of analysis allows to separate the impact of architecture from other factors like model size or training method. Surprisingly, we find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities, and that as models get larger and more accurate they tend to display more equivariance, regardless of architecture. For example, transformers can be more equivariant than convolutional neural networks after training.
Cumulative Equivariance Error
80
Activation
Translation Equivari qquivarianc riance Error
0.15
0.333
CNN VIT Mixer
Attention Pool
60
0.10
C lim furofor
40
0.05
20
70%
75%
80%
85%
90%
ResNet-50 ViT-Small
Imagenet Test Accuracy
Figure (Left)) The Lie derivative measures the equivariance of function under continuous transformations, here rotation. (Center): Using the Lie derivative, we quantify how much each layer contributes to the equivariance error of model. Our analysis highlights surprisingly large contri- butions from non-linearities, which affects both CNNs and VIT architectures. (Right): Translation cquivariance as measured by the Lie derivative correlates with generalization classification models, across convolutional and non-convolutional architectures. Although CNNs are often noted for their intrinsic translation equivariance, ViT and Mixer models are often more translation equivariant than CNN models after training.
INTRODUCTION
Symmetries allow machine learning models to generalize properties of one data point to the properties of an entire class of data points. A model that captures translational symmetry, for example, will have
Egual contribution.

---

the same output for an image and a version of the same image shifted a half pixel to the left or right. If a classification model produces dramatically different predictions as result of translation by half a pixel or rotation by few degrees it is likely misaligned with physical reality. Equivariance provides a formal notion of consistency under transformation. A function is equivariant if symmetries in the input space are preserved in the output space.
Baking equivariance into models through architecture design has led to breakthrough performance across many data modalities, including images (Cohen Welling, 2016; Veeling et al., 2018), proteins (Jumper et al., 2021) and atom force fields (Batzner et al., 2022; Frey et al., 2022). In computer vision, translation equivariance has historically been regarded as a particularly compelling property of convolutional neural networks (CNNs) (LeCun et al., 1995). Imposing equivariance restricts the size of the hypothesis space, reducing the complexity of the learning problem and improving generalization (Goodfellow et al., 2016).
In most neural networks classifiers, however, true equivariance has been challenging to achieve, and many works have shown that model outputs change dramatically for small changes in the input space (Azulay & Weiss, 2018; Engstrom et al., 20;8; Vasconcelos et al., 2021; Ribeiro & Schön, 2021). Several authors have significantly improved the equivariance properties of CNNs with architectural changes inspired by careful signal processing (Zhang, 2019; Karras et al., 2021), but non-architectural mechanisms for encouraging equivariance, such as data augmentations, continue to be necessary for good generalization performance (Wightman et al., 2021).
The increased prominence of concconvolutional architectures, such as vision transformers (ViTs) and mixer models, simultaneously demonstrates that explicitly encoding architectural biases for equivari- ance is not necessary for good generalization in image classification, as ViT models perform on-par with or better than their convolutional counterparts with sufficient data and well-chosen augmentations (Dosovitskiy et al., 2020; Tolstikhin et al., 2021). Given the success of large flexible architectures and data augmentations, it is unclear what clear practical advantages are provided by explicit architectural constraints over learning equivariances from the data and augmentations. Resolving these questions systemically requires unified equivariance metric and large-scal scale evaluation.
In what follows, we introduce the Lic derivative as a tool for measuring the equivariance of neural networks under continuous transformations. The local equivariance error (LEE), constructed with the Lie derivative, makes it possible to compare equivariance across models and to analyze the contribution of each layer of model to its overall equivariance. Using LEE, we conduct large-scale analysis of hundreds of image classification models. The breadth of this study allows us to uncover a novel connection between equivariance and model generalization, and the surprising result that ViTs are often more equivariant than their convolutional counterparts after training. To explain this result, we use the layer-wise decomposition of LEE to demonstrate how common building block layers shared across ViTs and CNNs, such as pointwise non- -ineaincasss..s. frequently give rise to aliasing and violations of equivariance.
We make our code publicly available at http://gtthbbocommgggvrrrilleeet
2 BACKGROUND
Groups and equivariance Equivariance provides formal notion of consistency under transforma- tion. A function f V12 + V22 is equivariant to transformations from a symmetry group applying the symmetry to the input of f is the same as applying it to the output
E E - = (2)))
where p(g) is the representation of the group element, which is linear map V V.
The most common example of equivariance in deep learning is the translation equivariance of convolutional layers: if we translate the input image by integer number of pixels in and y. the output is also translated by the same amount, ignoring the regions close to the boundary of the image. Here E Vi V/2 is image and the representation # P2 expresses translations of the image. The translation invariance of certain neural networks is also an expression of the equivariance property, but where the output vector space 1/2 has the trivial p2(g) model outputs are unaffected by translations of the inputs. Equivariance is therefore a much richer framework, in which we can reason about representations the input and the output of a function.
representation, such that
2

---

Continuous signals The inputs to classification models are discrete images sampled from contin- uous reality. Although discrete representations are necessary for computers, the goal of classification models should be learning functions that generalize in the real world. It is therefore useful to consider an image as function h I + R rather than discrete set of pixel values and broaden the symmetry groups we might consider, such as translations of an image by vector E R2, rather than an integer number of pixels.
Fourier analysis a powerful tool for understanding the relationship between continuous signals and discrete samples by way of frequency decompositions. Any M x M image, h(æ), can be constructed from its frequency components, Hmm. using 2d Fourier series, h((x) Σ En.m H e
2 (n,
where æ E [0, 1] and n, m [-M/22 -M/22 /2 + 1.... M/2], the bandlimit defined by the image size.
Aliasing Aliasing occurs when sampling at limited frequency [s. for example the size of an image in pixels, causes high frequency content (above the Nyquist frequency fs/2) to be converted into spurious low frequency content. Content with frequency n is observed as lower frequency contribution at frequency
Alias(n) {
n mod fa (n mod fa) f fa if (n mod f.) fa/2 }
if (n mod f.) fa/2 }
If a discretely sampled signal such as an image is assumed to have no frequency content higher than fa. then the continuous signal can be uniquely reconstructed using the Fourier series and have consistent continuous representation. But the signal contains higher frequency content which gets aliased down by the sampling, then there is an ambiguity and exact reconstruction is not possible.
Aliasing and equivariance Aliasing is critically important to our study because breaks equiv- ariance to continuous transformations like translation and rotation. When a continuous image is translated its Fourier components pick up a phase:
h(x) + h(x b) Ham Hame (rrbb(n..))
However, when an aliased signal translated, the aliasing operation A introduces a scaling factor:
Hmm Hnme 22ri(bu
-2ai Alias(n) (-b+)) Alias(m))
In other words, aliasing causes translation by the wrong amount: the frequency component H will effectively be translated by [(Alias(n)/a n),,, (alias(m)/m)b) which may point in a different direction than b, and potentially even the opposite direction. Applying shifts an aliased image will yield the correct shifts for true frequencies less than the Nyquist but incorrect shifts for frequencies higher than the Nyquist. Other continuous transformations, like rotation, create similar asymmetries.
Many common operations in CNNs can lead to aliasing in subtle ways, breaking equivariance in turn. Zhang (2019), for example, demonstrates that downsampling layers causes CNNs to have inconsistent outputs for translated images. The underlying cause of the invariance is aliasing, which occurs when downsampling alters the high frequency content of the network activations. The M M activations at a given layer of a convolutional network have spatial Nyquist frequencies f. M/2. Downsampling halves the size of the activations and corresponding Nyquist frequencies. The result is aliasing of all nonzero content with [M/4, M/2). To prevent this aliasing, Zhang (2019) uses a local low pass filter (Blur-Pool) to directly remove the problematic frequency regions from the spectrum.
While studying generative image models, Karras et al. (2021) unearth a similar phenomenon in the pointwise nonlinearities of CNNs. Imagine an image at a single frequency h(2) sin [nt, m)). Applying nonlin- car transformation to l creates new high frequencies in the Fourier series, as illustrated in Figure 2. These high frequencies may fall outside of the bandlimit, leading to aliasing. To counteract this effect, Karras et al. (2021) opt for smoother non-linearities and perform upsampling before calculating the activations.
MMM
Ma0a
Fouter Spectrur
10
10
Fauter
30
10
30
500
505
500
Frequency
Figure 2: Non-linearities generate new high-frequency harmonics.
3

---

3 RELATED WORK
While many papers propose architectural changes to improve the equivariance of CNNs (Zhang, 2019; Karras et al., 2021; Weiler & Cesa, 2019), others focus purely on measuring and understanding how equivariance can emerge through learning from the training data (Lenc & Vedaldi, 2015). Olah et al. (2020), for example, studies learned equivariance in CNNs using model inversions techniques. While they uncover several fascinating properties, such as rotation equivariance that emerges naturally without architectural priors, their method limited by requiring manual inspection of neuron activations. Most relevant to our work, Bouchacourt et al. (2021) measure equivariance in CNNs and ViTs by sampling group transformations. Parallel to our findings, they conclude that data and data augmentation play larger role than architecture in the ultimate equivariance of model. Because their study is limited to just a single ResNet and ViT architecture, however, they do not uncover the broader relationship between equívariance and generalization that show here.
Many papers introduce consistency metrics based on sampling group transformations (Zhang, 2019; Karras al., 2021; Bouchacourt et al., 2021), but most come with significant drawbacks. When translations are sampled with an integer number of pixels (Zhang, 2019; Bouchacourt et al., 2021), aliasing effects will be completely overlooked. As remedy, (Karras al., 2021) propose a subpixel translation equivariance metric (EQ-T trae) that appropriately captures aliasing effects. While this metric is major improvement, requires many design decisions not required by LEE, which has relatively few hyperparameters and seamlessly breaks down equivariance across layers. Relative to these other approaches, LEE offers a unifying perspective, with significant theoretical and practical benefits.
4 MEASURING LOCAL EQUIVARIANCE ERROR WITH LIE DERIVATIVES
Lie Derivatives The Lie derivative gives a general way of evaluating the degree to which a function violates symmetry. To define the Lie derivative, first consider how symmetry group element can transform a function by rearranging Equation I:
(a)())f(((>>>>>>))))))))>))))))) = ((p(())))
The resulting linear operator, P2i [+] acts on the vector space of functions, and P21 (a)( = f if the function is equivariant. Every continuous symmetry group (Lie group), G, has corresponding vector space (Lie algebra) fields R² > R". For images, these vector fields encode infinitesimal transformations IR2 + RR² over the domain of continuous image signals ʃ R² + & One can represent group elements g E G (which lie in the connected component of the identity) as the flow along a particular vector field Ly, where Y = =E a,X, is a linear combination of basis elements. The flow (20) of a point zo along& vector field Y by value t is defined the solution the ODE dæ initial value zip. The flow b6 smoothly parameterizes the group elements by I that the operator P21 (+b)){ [+] connects changes in the space of group elements to changes in symmetry behavior of a function.
with basis elements X, that can be interpreted as vector
de Y(x) at time t with
The Lie derivative of the function f is the derivative of the operator P21 at = Identity = to along a particular vector field Y.
d
Cr()) lim I
(p21 (++) - -) dt 1=
p21 (F})
Intuitively, the Lie derivative measures the sensitivity a function to infinitesimal symmetry trans- formations. This local definition of equivariance error is related to the typical global notion of equivariance error. As we derive in Appendix B.1, if = 1...., d: (f) = 0 (and the exponential map is surjective) then Vg G f(pi (g)x) (z)) and for all æ in the domain, and vice versa. In practice, the Lie derivative is only a proxy for strict global equivariance. We note global cquivariance includes radical transformations like translation by 75% of an image, which is not necessarily desirable. In section 6 we show that our local formulation of the Lie derivative can capture the effects of many practically relevant transformations.
The Lie derivative of a function with multiple outputs will also have multiple outputs, so if we want to summarize the equivariance error with a single number, we can compute the norm of the Lie
4

---

import torch. nn funcciiona
2 from torch. autograd. functional import jvp
def rotate (imgs, theta):
*** Rotate images by angle theta and interpolate"
[[torch, cos (theta), torch.sin (theta), 0],
7
[-torch. sin (theta), torch. cos (theta), 0]]
I torch.tenson (m) [None] expand (imgs. shape [0], -1, -11
return F. grid__ampplee (ings, M.affine_grid imgs. 81 size True))
def rotation 1ie deriv (model, imgs):
н** Lie deriv. model w.r.t. rotation, can be scalar/image
13
def motated_modell (theta);
14.
2 model (rotate (imgs,
mike = (len (z. shape) == 4) # more complex for ViT/Mixer return rotate tthtt)) if img_like else
return jvp (rotated__mddal) torch. zeros (1, requires_graad-eee [-1]
18
19 def e_lee (model, ings]:
Expected equiv. error 22// W.r.E. rotation"*
return ietation_lie_des (model, imgs) poo (2) mean
Figure Lie derivatives can be computed using automatic differentiation. We show how a Lie derivative for continuous rotations can be implemented in PyTorch (Paszke et al., 2019). The implementation our experiments differs slightly, for computational efficiency and pass second- order gradients through
derivative scaled by the size of the output. Taking the average of the Lic derivative over the data distribution, we define Local Equivariance Error (LEE),
LEE(S) = Ex-pllEx
/imm
(4)
We provide a Python implementation of the Lie derivative calculation for rotations in Figure 3 as an example. Mathematically, LEE also has an appealing connection to consistency regularization (Athiwaratkun et al., 2018), which we discuss in Appendix C.I.
Layerwise Decomposition of Lie Derivative Unlike alternative equivariance metrics, the Lie derivative decomposes naturally over the layers of neural network, since it satisfies the chain rule. As we show Appendix B.2, the Lie derivative of the composition of two functions h V + V/ and f V/2 V3 satisfies
Cx o h)(x)) = (Cxf)(((2)) + of
(5)
where of (k(r)) is the Jacobian of at h(x)) which we will abbreviate as df. Note that this decomposition captures the fact that intermediate layers of the network may transform in their own way:
f(h(z)) P (f)lf (z()) = pa(g) - ((2))))))00)))))
= (s)( (2)(g)(4((
and the Lie derivatives split accordingly.
Applying this property to an entire model composition of layers NN(x) (N)
(N (x)))) we can identify the contribution that each layer fi makes to the equivariance
error of the whole. Unrolling Equation 5, we have
Cx Σ Cx i=1
(6)
Intuitively, the equivariance eπто of a sequence of layers is determined by the sum of the equivariance enror for each layer multiplied by the degree to which that emror is attenuated or magnified by the other layers (as measured by the Jacobian). We evaluate the norm of each of the contributions df Lx f to the (vector) equivariance error Lx (NN) which we compute using autograd and stochastic trace estimation,
describe Appendix B.3. Importantly, the sum of norms may differ from the norm of the sum, but this analysis allows us to identify patterns across layers and pinpoint operations that contribute most equivariance error.
5

---

120
50
Altetion
Actio
CorvZd
Foal
Artraatan
100
Cerbb
6PSX
Loyehor
Normi and Eauvernree oe Emoor
1,0
80
0.
69
0.8
0.6
Canulebee
40
04
komallod
0.4
20
0.2
Incoplinnaa
10033
Rrstee.0 5g
Widefestonn
VST
ConViT
Res MLD
0
Tranorain Rotattan Scalin
Hypothotic Rox.
IMAAA frocond Fractend
LDS
Figure 4: Contributions equivariance shown cumulatively by layer, in the order the layers oceur in the network. Left: Convolutional architectures. In all the CNNs, much of the equivariance error comes from downsampling and non-linearities. Middle-Leeft Non-convolutional architectures. The initial patch embedding, a strided convolution, is the largest contributor for the ViTs and Mixers. The rest of the error is distributed uniformly across other nonlinear operations. Middle-Rigg Rightt ResNet-50 across different transformations as percentage. Despite being designed for translation equivariance, the fraction of equivariance error produced by each layer is almost identical for other affine transformations, suggesting that aliasing is the primary source of equivariance error. Right: Comparing LEE with alternative metrics for translation equivariance. Using integer translations misses key contributors to equivariance errors, such as activations, while using fractional translations can lead to radically different outcomes depending on choice of normalization (N or vN). LEE captures aliasing effects and has minimal design decisions.
5 LAYERWISE EQUIVARIANCH ERROR
As described in section 3, subtle architectural details often prevent models from being perfectly equivariant. Aliasing can result from careless downsampling or from activation function with wide spectrum. In this section, we explore how the Lie derivative uncovers these types of effects automatically, across several popular architectures. We evaluate the equivariance of pretrained models on 100 images from the ImageNet (Deng et al., 2009) test set.
Using the layerwise analysis, we can dissect the sources of translation equívariance error in con- volutional and non-convolutional networks as shown in Figure 4 (left) and (middle-left). For the Vision Transformer and Mixer models, produces a significant portion of the error, and the remaining error is split uniformly between the other nonlinear layers: LayerNorm, tokenwise MLP, and self-attention. The contribution from these nonlinear layers is seldom recognized and potentially counterintuitive, until we fully grasp the deep connection between equivariance and aliasing. In Figure (middle-right), we show that this breakdown is strikingly similar for other image transformations like rotation, scaling, and hyperbolic rotations, providing evidence that the cause of equivariance error is not specific to translations but is instead a general culprit across whole host of continuous transformations that can lead to aliasing.
see that the initial conversion from image to patches
We can make the relationship between aliasing and equivariance error precise by considering the aliasing operation Alias defined in Equation 2.
Theorem 1. For translations along the vector v = [U,,,y], the aliasing operation A introduces translation equivariance error of
(()(()) = (2 Σ H2, Hiim (1alisis(()) n)² + (Alliss(() - -m)²).
where h(x) = 2T Σn.m Hnm e (n. is the Fourier series for the input image h.
We provide the proof in Appendix C.2. The connection between aliasing and LEE is important because aliasing is often challenging to identify despite being ubiquitous (Zhang, 2019; Karras et al., 2021). Aliasing in non-linear layers impacts all vision models and is thus a key factor in any fair comparison of equivariance.
As alternative equivariance metrics exist, it is natural to wonder whether they can also be used for lay- erwise analysis. In Figure 4 (right), we show how two equivariance metrics from Karras al. (2021)

---

Consistency
Group Sample
Sheer
0.08
0.588
R² = 0,205
0.360
R² = 0.367
Equiveriance Error
0.04
0.06
36-4
0.08
26
0.02
70%
807
70% 80%
90%
75%
85%
70%
80N
90%
Imagenet Test Accuracy
Equivariance metrics evaluated on the ImageNet test set. Left: Non-LEE equivariance metrics display similar trends to Figure I, despite using larger, multi-pixel transformations. Right: Norm of rotation and shear Lie derivatives. Across all architectures, models with strong generalization become more equivariant to many common affine transformations. Marker size indicates model size. Error bars show one standard error over test set images used in the equivariance calculation.
Figure 5:
compare with LEE, highlighting notable drawbacks. (1) Integer translation equivariance completely ignores aliasing effects, which are captured by both LEE and fractional translations. (2) Though fractional translation metrics (EQ-T free) correctly capture aliasing, comparing the equivariance of layers with different resolutions (C x Hl x W) requires an arbitrary choice normalization. This choice can lead to radically different outcomes in the perceived contribution of each layer and is not required when using LEE, which decomposes across layers as described in section 4. We provide a detailed description of the baselines in Appendix D.I.
TRENDS IN LEARNED EQUIVARIANCE
Methodology We evaluate the Lie derivative of many popular classification models under transfor- mations including 2d translation, rotation, and shearing. We define continuous transformations on images using bilinear interpolation with reflection padding. In total, evaluate 410 classification models, a collection comprising popular CNNs, vision transformers, and MLP-based architectures (Wightman, 2019). Beyond diversity in architectures, there is also substantial diversity in model size, training recipe, and the amount of data used for training or pretraining. This collection of models therefore covers many of the relevant axes of variance one is likely consider in designing system for classification. We include an exhaustive list of models in the Appendix D.3.
Equivariance across architectures As shown in Figure (right), the translation equivariance error (Lie derivative norm) is strongly correlated with the ultimate test accuracy that the model achieves. Surprisingly, despite convolutional architectures being motivated and designed for their translation equivariance, find significant difference in the equivariance achieved by convolutional architec- tures and the equivariance their more flexible ViT and Mixer counterparts when conditioning on test accuracy. This trend also extends to rotation and shearing transformations, which are common in data augmentation pipelines (Cubuk et al., 2020) (in Figure 5 (right)). Additional transformation results included in Appendix D.5.
For comparison, we also evaluate the same set of models using two alternative equivariance metrics: prediction consistency under discrete translation (Zhang, 2019) and expected equivariance under group samples (Finzi et al., 2020; Hutchinson et al., 2021), which is similar in spirit to EQ-T frac (Karras et al., 2021) (exact calculations Appendix D.4). Crucially, these metrics are slightly less local than LEE, as they evaluate equivariance under transformations of up to 10 pixels at time. The fact that we obtain similar trends highlights LEE's relevance beyond subpixel transformations.
Effects of Training and Scale In section we described many architectural design choices that have been used to improve the equivariance of vision models, for example Zhang (2019)'s Blur-Pool low-pass filter. We now investigate how equivariance error can be reduced with archttcchttttr design decisions, such as increasing model size, dataset size, or training method. Surprisingly, we show that equivariance error can often be significantly reduced without any changes in architecture.
7

---

BlurFeol (ResNer 50)
Model Scale (VITs)
Model Scale (EMiclentNists)
Size (RosMLP Sigi
Mothod (Restiett S0)
0.20
Imagenet 1t Imagenet 22k
Equiarancc
0.08
Parumsterrs de-ide
Parameter Se de
Tordiwision Improved Recipe SSL Pretraining
Standard Burthol
0.04
Zeréo
26-d7
80%
02%
80%
15.
90%
80%
85%
85% 75%
10%
80%
Miver
Imaganet Test Accuracy
Figure 6: Case studies in decreasing translational equivariance error, numbered left-to-right. 1: Blur-Pool (Zhang, 2019), architectural change improve equivariance, decreases the equivariance error but by less than can be accomplished by improving the training recipe or increasing the scale of model or dataset. 2-3: Increasing the number of parameters for fixed model family (here ViTs (El-Nouby et al., 2021) and EfficientNets (Tan & Le. 2019a)). 4: Increasing the training dataset size for a ResMLP Big (Touvron al., 2021a) model. 5: Changing the training recipe for ResNeXt-50 (Xie et al., 2017) with improved augmentations (Wightman et al., 2021) or SSL pretraining (Yalniz al., 2019). Error bars show one standard error over images in the Lie derivative calculation.
In Figure 6, we show slices of the data from Figure 1 along shared axis for equivariance error. As a point of comparison, in Figure 6 (left), we show the impact of the Blur-Pool operation discussed above on ResNet-50 (Zhang, 2019). In the accompanying four plots, we show the effects of increasing model scale (for both ViTs and CNNs), increasing dataset size, and finally different training procedures. Although Zhang (2019)'s architectural adjustment does have noticeable effect, factors such as dataset size, model scale, and use of modern training methods, have a much greater impact learned equivariance.
As a prime example, in Figure 6 (right), we show a comparison of three training strategies for ResNeXt-50 architecture almost identical to ResNet-50.. We use Wightman et al. (2021)'s pretrained model to illustrate the role of an improved training recipe and Mahajan et al. (201 220)8b)'s semi-supervised model and model size, these changes lead to decreases in equivariance error on par with architectural interventions (BlurPool). This result is surprising when consider that Wightman al. (2021)'s improved training recipe benefits significantly from Mixup (Zhang et al., 2017) and CutMix (Yun al., 2019), which have no obvious connection to equivariance. Similarly. Mahajan al. (2018b)'s semi-supervised method has no explicit incentive for equivariance.
example of scaling training data. Notably, for fixed architecture
Equivariance out of distribution From our analysis above, large models appear to learn equivari- ances that rival architecture engineering in the classification setting. When learning equivariances through data augmentation, however, there is no guarantee that the equivariance will generalize to data that is far from the training distribution. Indeed, Engstrom et al. (2019) shows that carefully chosen translations or rotations can be as devastating to model performance as adversarial examples. less equivariant on
We find that vision models do indeed have an equivariance gap: models test data than train, and this gap grows for OOD inputs as shown in Figure 7. Notably, however, architectural biases do have strong effect on the equivariance gap. as both CNN and ViT models have comparable gaps for OOD inputs.
Why aren't 't CNNs more equivariant than ViTs? Given the deep historical connection between CNNs and equivariance, the results in Figure and Figure 7 might appear counterintuitive. ViTs, CNNs, and Mixer have quite different inductive biases and therefore often learn very different repre- sentations of data (Raghu et al., 2021). Despite their differences, however, all of these architectures are fundamentally constructed from similar building blocks-such as convolutions, normalization layers, and non-linear activations which can all contribute to aliasing and equivariance error. Given this shared foundation, vision models with high capacity and effective training recipes are more capable of fitting equivariances already present in augmented training data.

---

Imagenet Train
Imagenet Test
CIFAR-100
Histology
Retinopathy
Equivartanes Errar
0.2
0.0
70% 75% 80%
85% 90%
70% 75% 80% 85% 90%
70% 75% 80% 85% 90%
70% 75% 80% 85% 90%
70% 75% 80% 85% 90%
Imagenet Test Accuracy
Mixer
Figure 7: Models are less equivariant on test data and becoming decreasingly equivariant as the data moves away from the training manifold. As examples of data with similar distributions, show equivariance error on the ImageNet train and test sets as well as CIFAR-100. As examples of out-of-distribution data, use medical datasets (which often use Imagenet pretraining), one for Histology (Kather et al., 2016) and one for Retinopathy (Kaggle & EyePacs, 2015).
Learning rotation equivariance We finally consider the extent to which large-scale pretraining can match strong ar- chitectural priors in case where equivariance is obviously desirable. We fine-tune a state-of-the-ar vision transformer model pretrained with masked autoencoding (He et al., 2021) for 100 epochs on rotated MNIST (Weiler & Cesa, 2019) (details Appendix D.6). This dataset, which con- tains MNIST digits rotated uniformly between -880 and 180 degrees, is common benchmark for papers that design equivariant architectures. In Table we show the test er- rors for many popular architectures with strict equivariance constrainets alongside the error for finetuned model. Surprisingly, the finetuned model achieves competitive test accuracy, in this Despite having relatively weak architectural biases, trans- formers capable of learning and generalizing on well on symmetric data.
Model
Test Error (%)
G-CNN [13]
2.28
H-NET [76]
1.69
ORN [86]
1.54
TI-Pooling [43]
1.2
Finetuned MAE
1.14
RotEqNet [51]
1.09
E(2)-CNN [73]
0.68
Table Our finetuned MAE is compet- itive with several architectures explic- itly engineered to encode rotation in- variance RoMMNIST, where rotation invariance is clearly crucial to general- ization.
strong proxy for rotation invariance.
7 CONCLUSION
We introduced a new metric for measuring equivariance which enables a nuanced investigation of how architecture design and training procedures affect representations discovered by neural networks. Using this metric we are able to pinpoint equivariance violation to individual layers, finding that pointwise nonlinearities contribute substantially even in networks that have been designed for equivariance. We argue that aliasing is the primary mechanism for how equivariance to continuous transformations are broken, which we support theoretically and empirically. We use our measure to study equivariance learned from data and augmentations, showing model scale, data scale, training recipe can have a greater effect on the ability to learn equivariances than architecture.
Many of these results contrary to the conventional wisdom. For example, transformers can be more equivariant than convolutional neural networks after training, and can learn equivariances needed to match the performance of specially designed architectures on benchmarks like rotated MNIST, despite a lack of explicit architectural constraints. These results suggest we can be more judicious in deciding when explicit interventions for equivariance real world problems where we only desire approximate and local equivariances.c On the other hand, explicit constraints will continue have immense value when exact equivariances and extrapolation are required- - such as rotation invariance for molecules. Moreover, despite the ability to learn cquivariances on training data, we find that there is an equivariance gap on test and OOD data which persists regardless of the model class. Thus other ways of combating aliasing outside of architectural interventions may be the path forward for improving the equivariance and invariance properties of models.
required, especially in many


Reviews:
[{"review_id": "kblG4q8Lpf", "reviewer": "Reviewer_rDjb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of measuring to what degree the the function is invariant to certain transformations. The authors suggest to use the Lie Derivative to measure such a quantity. Then Local equivariance error (LEE) is introduced. The paper demonstrates how different parts of various models contribute to the equivariant properties of the model in terms of LEE and how different models demonstrate a correlation between the global LEE and the resulting accuracy.", "summary_of_the_review": "The paper is worth sharing with the community because of its potential significant contribution to the field. Minor changes may improve its quality.", "strength_and_weaknesses": "The paper is well-written. I found it easy to read. The storytelling of hte paper is of high quality. The derivations are accurate and the math notation is well-understood. \n\n**Strengths**\n1. The authors pay attention to the problem which is less studied in the community - the problem of measuring the equivariant properties of various models.\n2. The theory of the paper is solid. And demonstrates a very good understanding of the problem.\n3. The experimental results of the paper demonstrate that such a method is useful for the analysis\n\n**No Strengths**\nI don't see any significant issues, which may affect my rating drastically. However, I would like to address several questions and suggestions, which may improve the quality of the paper.\n1. The related work misses a section on why we are so motivated to measure the equivariance error. There were several papers which demonstrated that the model's accuracy and its equivariance error are highly correlated. In [1] the authors take a network and train it by minimizing the equivariance error which leads to a better accuracy than any other counterparts. In [2] the authors compare different scale-equivariant models, conclude that the equivariance error is not 0, and demonstrate that the lower it is the more accurate the model becomes.\n2. [Does not affect my rating. Just a pure interest] Although it was not possible to include this paper beforehand, there is a recent paper which considers a similar problem. In [3] the authors introduce a similar metric for measuring the symmetric properties of the model. What is the main difference between their approach and yours? \n3. Let us consider two neural networks, which perform binary classification. The very last layer outputs a positive or a negative value for either class 1 or class 2. We initialize them with the same weights. However, for network 2 we multiply the output by $10^6$. Do I understand it correctly, that for network 2 LEE will be $10^6$ larger than LEE for network 1? If so, can we really compare two neural networks based on Eq 4? Do we need an extra step of output normalization?\n4. In supplementary materials, in A3, paragraph 2, you mention that there is an inequality between the $\\mathcal{L}$ of the network and the sum of the $\\mathcal{L}$  of its layers. Let us consider a network, for which the very last layer just multiplies everything by $0$. Such a network is absolutely invariant, thus $\\mathcal{L} = 0$. However, the $\\mathcal{L}$ of the layers of the network can be arbitrary large. Thus, Figure 4 does not really tell us anything about the network as a whole. Could you make a comparison where you plot side-by-side the cumulative loss as a sum per-layer losses. And a series of losses calculated for all truncated subnetworks of the original networks $\\text{subnet}_k = f_k \\circ \\dots \\circ f_1$\n\n\n- [1] Khetan N. et al. Implicit Equivariance in Convolutional Networks, preprint 2021\n- [2] Sosnovik I., Moskalev A., Smeulders A. Disco: accurate discrete scale convolutions, BMVC 2021\n- [3] Moskalev A. et al. LieGG: Studying Learned Lie Group Generators, NeurIPS 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MbIFXQT6rn", "reviewer": "Reviewer_TkYs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work compares the ability of models with built-in equivariance versus those without in how well they ultimately learn equivariance.  The majority of experiments and analysis focuses on the case of translation equivariance in image classification comparing CNNs to ViT and Mixer, although rotation and scale are also considered.  \n- In order to measure equivariance error, the authors define local equivariance error, LEE, based on infinitesimal action of the symmetry group on the function space defined by each layer.  This definition allows the authors to decompose the final LEE of the model in terms of per-layer LEE.  \n- The paper puts forth the hypothesis that the interaction of aliasing, the aberration which result from discretizing a continuous signal, and continuous symmetry is the main driver of equivariance error in all models.   This hypothesis is supported by Theorem 1, which directly computes the equivariance error due to aliasing under translations, and by empirical analysis of layer-wise LEE showing that activation layers contribute most towards LEE.  \n- Experiments show model performance is correlated with degree of learned equivariance and that there is not a significant difference between CNNs and ViTs in terms of equivariance learning (if anything ViTs have lower LEE.)  \n- Several architechure-agnostic solutions to better symmetry learning are tested including BlurPool, larger models, more data, and SSL pretraining.  All such interventions (to various degrees) do improve equivariance learning. ", "summary_of_the_review": "I very much liked reading this paper.  A study comparing learned symmetry to built-in symmetry is very useful to the field today. Equivariant architectures are increasing popular and at the same time, non-equivariant methods such as transformers are overtaking them in some domains.  The experiments are extensive and the analysis is clear and rigorous.  I think the conclusions will be useful to practitioners who are choosing between different architectures. I hope the tool of layer-wise LEE analysis could help to improve equivariant neural networks.    I have some disagreement with how some parts of the study are set up, things I would like to do differently, and ultimately with the implication that built-in equivariance is often unnecessary.  However, that is really to say I think this paper will inspire much future research both supporting, extending, and arguing with the hypothesis studied here. ", "strength_and_weaknesses": "## Strengths\n- The paper considers an important question in the field.  Instead of proposing a novel architecture, this work provides a useful comparison of the strengths and weaknesses of existing state-of-the-art methods from the viewpoint of their ability to learn equivariance.  It could provide insights for improving equivariance in both models with built-in and no built-in inductive bias or help to make model selection easier. \n-  The experiments are really quite extensive.  I agree with the authors' claim that the scale and scope of their analysis are helpful in making the trends clear against the noise and other factors contributing to model performance and equivariance error.  Given the goal here is to say something broad about CNNs or ViT in general, it is essential a wide variety of architectures in both classes are considered. \n- Previously works on equivariant neural networks have not been overly focused on aliasing.  The equivariance is usually analyzed from the point of view of continuous signals and equivariance is proved in the continuous case.  Empirical measurements are used to assure that discretization errors are acceptable.  However, it seems clear that discretization is important and limiting the potential advantages.  For example, many applications of E(2)-CNN use small discrete rotation groups instead of continuous groups.  This work addresses aliasing head-on and provides evidence of the extent to which it contributes to equivariance error and specifically where (non-linearity). \n- The definition of LEE and the observation that LEE can be broken down on a layer-by-layer basis seems like an important contribution. Previously, there was not a precise way to quantify the contribution of each layer to EE.  Practically, one could measure the EE over a single layer (as is done here for alternate EE metrics, I believe), but this is not guaranteed to combine in any reasonable way.\n- The result shown in Figure 4 and subsequent insights are very interesting. I appreciate that several different comparisons are done: CNNs vs. ViT, different symmetries, different EE metrics.  The finding that non-linearities contribute the most to LEE raises interesting questions.  What about the fact early layers seem to contribute more?  The observation the contributions to EE for each layer are similar for different symmetry groups suggesting aliasing the root cause is interesting. \n- Section 6 discusses several ways to improve equivariance learning given that baking equivariance seems to come up short. \n\n\n## Weaknesses / Questions\n- Since the task is invariant (equivariant with output having trivial rep), the representation must go from having non-trivial group action to trivial group action.  In practice, this invariance can be baked in either by mapping to invariants and having later layers be unconstrained or having equivariant layers and then a final invariant layer.  Thus for a classification task, it is not necessary for the intermediate layers to have low EE for the final NN to be invariant.  I feel this could somewhat undermine the assumptions of the layer-by-layer LEE analysis.  Could the authors comment?  I would feel more confident if the experiments were performed for a task where the output representation was equal to the input representation, for example, image segmentation.  \n- Another potential aspect overlooked in the setup is the potential for a non-trivial action on the channels of the hidden features.  This absolutely key to making equivariant neural networks work. My understanding is that LEE is assuming the channel representation is trivial (though I could be mistaken).  Thus we could be measuring high LEE when in fact it would be low if we knew the way the channels should transform.       \n- As noted in the paper, CNNs are often not very equivariant (compared to many other equivariant networks) due to discretization, pooling, and stride. I'd like to see results showing the equivariance error for the CNNs vs. ViTs at initialization and throughout training.  If the compared CNNs do not have low EE at initialization, then the argument that they have built-in equivariance is not very strong.  I'd feel more comfortable with a comparison between E(2)-CNN and unconstrained networks for rotational equivariance error.  In this case, I think the E(2)-CNN architecture would have relatively low EE.\n- Although it was posted on the arXiv only in October and has now been accepted to NeurIPS (per the schedule), I believe that LieGG (https://arxiv.org/abs/2210.04345) covers some similar territory in terms of using lie algebra action on functions space to analyze the learned symmetry of the model.  That work also includes analysis of \"symmetry bias\" (analogous to equivariance error) and per-layer analysis (see Fig.5).  I don't consider it to pre-empt this work, both because they are concurrent and because the focus is very different, however, I would like to see a comparison added.  \n- I take some issue with the claim that it is surprising that non-equivariance models attain lower equivariance error after training.  Equivariant networks certainly have lower equivariance error at initialization that unconstrained networks, but learning equivariance is necessary in order to get good performance, so it follows that if an unconstrained network achieves better performance than an equivariant one, it may well have lower trained equivariance error (EE).  Consider, for example, a regression task.  Assuming the ground truth is perfectly equivariant, then the equivariance error will be bounded by 2*(approximation error) for any sample by the triangle inequality.   Since you consider a classification task, this relationship is a bit more obscured, but still well represented by your results.\n\n## Questions \n- I'm unclear on the aliasing operation $A$ versus the map $\\mathrm{Alias}(n)$.  Can you clarify?  In particular, on page 3, it seems $A \\colon \\mathbb{N} \\to \\mathbb{N}$, but on page 6, it seems $A \\colon (\\mathbb{R}^2)^{\\mathbb{R}^c} \\to (\\mathbb{R}^2)^{\\mathbb{R}^c}$, i.e. images to images. \n- Could section 4 avoid referring to flows and be written in terms of the exponential map $\\mathfrak{g} \\to G$? \n- In section 5, you prove that Alias results in translation LEE.  I didn't fully understand why this points to the non-linear layers.\n\n## Minor Points \n- The abstract says \"introduce the Lie derivative,\" I would re-word to make clear Lie derivatives are pre-existing work, they are merely being introduced as a method here. \n- I'd replace \"it's\" with \"it is\" since this is formal writing. \n- In two places $\\mapsto$ appears when it should be $\\to$.  $\\mapsto$ is for elements and $\\to$ is for sets.  As in $f \\colon X \\to Y$ mapping $x \\mapsto y$. \n- Page 4 \"the operator $\\rho_{21}(g)[\\Phi^t_Y]$ \"  Should it be  $\\rho_{21}([\\Phi^t_Y])$ ?  \n- In the notation for LEE in eqn. 4, you might want to include the dependence on $X$. \n- Section 5, first sentence \"equivariance\" -> \"equivariant\" ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EeVVPlWTHg", "reviewer": "Reviewer_46iy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Lie derivative, which is a simple method for measuring equivariance. The authors use the proposed method to analyze the equivalence properties of hundreds of pre-trained models including most popular deep learning architectures for classification. They also demonstrate a layerwise breakdown of the equivariance error for each model, which only becomes possible under their proposed metric. From analysis, they separate the impact of architectures from factors such as model size and training method and identify the contributions of different modules in each model. They also show the interesting finding that large models such as ViT and Mixer are actually \"more equivariant\" than CNNs which use convolutions with build-in exact equivariance.", "summary_of_the_review": "I recommend acceptance of this paper mainly based on the following:\n1. This paper studies an important question that can inspire further investigations. I think it can benefit both the equivariance community and the broader deep-learning community.\n2. The findings in this paper are interesting. The proposed method is simple yet effective. The empirical work is solid.\n3. The paper is polished and well-presented.", "strength_and_weaknesses": "**Strength:**\n1. A simple yet useful technical contribution. Interesting empirical findings that can inspire future research.\n2. Solid work with theoretical and empirical support.\n3. Nice presentation of the proposed method. Clear motivation.\n\n**Weaknesses:**\n1. The explanation for aliasing and equivariance could be more readable to people who are not familiar with signal processing. One solution is that the authors can provide more background knowledge in the appendix, and only keep the high-level essential conclusions and intuition about aliasing in the main text.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%