Flopping for FLOPs: Leveraging Equivariance for Computational Efficiency
Authors: Georg Bökman
David Nordström
Fredrik Kahl
Venue: ICML 2025 spotlightposter (2025)
Keywords: Equivariance
efficiency
scalable
image classification
vision
Primary area: applications->computer_vision
TL;DR: Equivariant neural networks for image input are scalable.

Abstract: Incorporating geometric invariance into neural networks enhances parameter efficiency but typically increases computational costs.
This paper introduces new equivariant neural networks
that preserve symmetry while maintaining a comparable number of floating-point operations (FLOPs) per parameter to standard non-equivariant networks. 
We focus on horizontal mirroring (flopping) invariance, common in many computer vision tasks.
The main idea is to parametrize the feature spaces in terms of mirror-symmetric and mirror-antisymmetric features, i.e., irreps of the flopping group.
This decomposes the linear layers to be block-diagonal, requiring half the number of FLOPs.
Our approach reduces both FLOPs and wall-clock time,
providing a practical solution for efficient, scalable symmetry-aware architectures.

Flopping for FLOPs: Leveraging Equivariance for Computational Efficiency
Georg Bökman' David Nordström
Fredrik Kahl
Abstract
Incorporating geometric invariance into neural networks enhances parameter efficiency but typ- ically increases computational costs. This paper introduces new equivariant neural networks that preserve symmetry while maintaining compara- ble number of foating-point operations (FLOPs) per parameter to standard non-equivariant net- works. We focus horizontal mirroring (flop- ping) invariance, common in many computer vi- sion tasks. The main idea is to parametrize the feature spaces in terms of mirror-symmetric and mimmorr features, i.c., irreps of the flopping group. This decomposes the linear layers to be block-diagonal, requiring half the number of FLOPs. Our approach reduces both FLOPs and wall-clock time, providing practical solution for efficient, scalable symmetry-awe aware architectures.
FLOP
Figure 1: Common image classification tasks are invariant flopping (horizontal mirroring). In implementation, this invariance is enforced through equivariant network layers, halving the required floating-poine operations (FLOPs).
enable more efficient computations. In nutshell, our case is that equivariant neural networks can be simple models that scale well.
1. Introduction
Many vision problems are invariant (or more generally equivariant) to horizontal mirroring, also known flop- ping', and partially invariant other geometrie transforma- tions such rotations. For instance, both images in Figure I show a butterfly of the same species. Enforcing geometric invariance in a neural network architecture can be done by hard mord-coding the symmetry in each network layer, by enfore- ing specific weight sharing (Wood Shawe-Taylor, 1996; Cohen & Welling, 2016). It has been shown in the literature that such architectural symmetry constraints typically im- prove the parameter efficiency of neural networks (Cohen & Welling, 2016; Bekkers et al., 2018: Weiler & Cesa, 2019). However, this comes at the cost of requiring more compute per parameter. For instance, Klee et al. (2023) found that rotation invariant ResNets typically outperformed ordinary ResNets on ImageNet- IK the equal-parameter setting, but that the training time of the invariant ResNets far ceeded that of the ordinary ResNets. This is because the symmetry-inducee weight sharing means that each trainable parameter is used in more computations.
One of the main drivers of progress in deep learning is the scaling of compute. This idea is perhaps best summarized in Sutton's Bitter Lesson (Sutton, 2019). As Sutton states:
Seeking an improvement that makes a difference in the shorter term, researchers seek leverage their human knowledge of the domain, but the only thing that matters in the long run is the lever- aging of computation.
It follows from the Bitter Lesson that guiding principle for designing deep learning models should be to find simple models that scale well. In computer vision, this has proven fruitful and led to models such as the Vision Transformer (ViT) (Dosovitskiy et al., 2021). However, a straightfor- ward critique of Sutton's argument that leveraging domain knowledge can yield algorithms that better utilize computa- tion. This paper explores common image symmetries that
CChalmers University of Technology. Correspondence to: Georg Bökman <boomman chalmers.se>.
In this paper, we show that invariant neural networks can achieve comparable number of floating-poin operations (FLOPs) per parameter ordinary neural networks. Imple-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'See Wikipedia article or Oxford definition.

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
menting these networks improves both FLOPs and actual wall-clock time. Our FLOPPefficient design is enabled by parametrizing features using irreducible representations of the flopping group.
2.2. Equivariant Networks for Other Input Than Images
Equivariant networks constitute a broad research direc- tion. One early line of research is by Wood & Shawe- Taylor (1996) and going back earlier the topic connects to steerable filters (Knutsson & Granlund, 1983; Freeman & Adelson, 1991). Recent approaches include canonicaliza- tion (Kaba et al., 2023; Mondal et al., 2023), learned equiv- ariance (Gupta et al., 2024), structured matrices (Samudre et al., 2025) et cetera. We will use the standard approach of enforcing each layer of the network to be equivariant through constraints the weight matrices (Wood & Shawe- Taylor, 1996; Cohen & Welling. 2016).
Section reviews related work on equivariant networks and Section 3 provides an exposition of Nopping-equivarian networks for image classification, while Section A lays out the underlying mathematical background in detail. In Section 4, we design flopping-equivariaat versions of popu- lar modern vision architectures. Experiments in Section 6 on ImageNet-IK IK demonstrate that as network size scales up,
models achieve comparable or im- proved classification accuracy for ResMLPs (Touvron et al., 2023), ConvNeXts (Liu et al., 2022) and ViTs (Dosovitskiy et al., 2021), while requiring only half the FLOPs.
There has been recent interest in the scaling properties of equivariant networks. Brehmer et al. (2024) find that the equivariant transformer GATr (Brehmer et al., 2023) outper- forms non-equivariant transformers on task of 3D rigid- body interactions, both in terms of parameter-efffciency and i.e., the number of FLOPs required dur- ing training.
2. Related Work
Our work is part of the broad field of geometric deep learn- ing (Bronstein et al., 2021). In this section, we aim to discuss the most relevant work to the present paper, but we reference many other relevant works throughout.
Bekkers et al. (2024) propose a fast equivariant group con- volutional network on point cloud input by using separable group convolutions on position orientation space TR$ x 5 and parallelizing the message passing step. Their networks use NonyNeXt-stylee blocks, do some of our networks. Follow-up work extends the message passing to use uni- versal invariants (Bellaard et al., 2025), and experimentally analyzes the benefits of equivariance in different point cloud processing tasks (Vadgama et al., 2025).
2.1. Equivariant Networks for Image Input
Most work on equivariant networks on image input concerns convolutional networks (ConvNets) (Fukushima, 1975; Le- Cun et al., 1989). ConvNets are themselves equivariant to (cyclic) image translations. The ConvNets in this paper are built on ConvNeXt (Liu et al., 2022), which is modern variant with state-of-the-art classification performance.
In contrast to our networks, none of the mentioned works develop networks that are directly comparable with non- equivariant ones, with similar FLOPs FLOPs-per-paramme per-parameter ratio.
Making ConvNets equivariant to rotations and reflections was done by Cohen & Welling (2016) and Dieleman al. (2016), with many follow-up works. The unifying frame- work of steerable F(())equivariant ConvNets (Cohen & Welling, 2017; Weiler & Cesa, 2019), subsumes much of the prior and subsequent work and proposes decomposing the feature spaces into irreps (irreducible representations, see Section A) of the symmetry group. Our Nen variants are special cases of this general framework too.
3. Flopping Equivariance
We focus on horizontal mirroring invariance in image classi- fication as a simple prototype task, aiming to inspire further research on scaling equivariant neural networks. Readers interested in more abstract diseussion in terms of general mathematical groups are referred to Section A. This section provides an accessible introduction to equivariant neural networks with minimal prerequisites. By emphasizing com- putational aspects, we aim to offer an intuitive motivation for our approach, making the key ideas easier to grasp.
Apart from ConvNets, there are works proposing attention- and transformer-based equivariant vision architec- tures (Romero et al., 2020; Xu et al., 2023). Most similar to our ViT-based networks are the S0(2) -steerbtee transform- ers by Kundu& & Kondor (2024). The main differences are the group considered, the fact that they use complex-valued features, the nonlinearities used and the type of positional embeddings. However, the main idea of parametrizing the features in terms of irreps (following steerable ConvNets) and modifying the ViT architecture accordingly is the same in our VIT's and the one by Kundu& & Kondor (2024).
The core idea is to split all feature maps in our network into two types: flopping- invariant features, which remain unchanged when the image is flopped, and flopping (-1)- equivariant features, which flip sign under flopping. By explicitly keeping track of how the features transform, it is possible to feed only invariant features into the final classifi- cation layer and guarantee flopping invariant predictions.
2

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
3.1. Linear Layers
Equivariant PatchEmbed
Beyond ensuring aopping-invarian, predictions, splitting features into invariant and (-1)-equivariant components also provides significant computational advantage. Con- sider linear mapping W from R to Rd. Assume that the first c/2 dimensions of the input are flopping invariant while the last 2/2 dimensions are --ujeauuirai,, and likewise that the first d/2 dimensions of the output are invariant, while the last d/2 dimensions are (-1)-equivariant. We split the linear map W into four parts,
Input
Filters
Output feature maps
( y1 y1-1
W1,1
W1,-1 W-1,-1
)
31
(I)
W-1,1
x-1
Filters
where F, are the invariant input features and _ the (-1)- equivariant input features and likewise for y. Now, by in- spection, if any element of W1,-1 is non-zero, then y1 will change when --1 changes sign and if any element of W-1.1 is non-zero, then y-1 can not transform by multiplication with -1 jointly with x-1. We conclude that W1,-1 and 1,1 must be zero so that (1) actually reads
Output feature maps
(1))) (
W1.1
W W-1--)
0
I1
(2)
x-1
Figure 2: Patch embedding layer. The patch embed- ding (PatchEmbed) layer common in modern networks, proposed with the ViT-model (Dosovitskiy et al., 2021). PatchEmbed is convolutional layer with stride equal to the kernel size of the convolution filters. In our equivariant architectures, we enforce half of the filters to be symmet- ric and half to be antisymmetric. When the input image is flopped, the output feature map of a symmetric filter is flopped as well. The output feature map of an antisymmet- ric filter is flopped and changes sign. Half of the features output from our PatchEmbed layer are flopping invariant, while half are (-1) (-1)-equivariant, enabling efficient equiv- ariant processing in subsequent linear layers, as detailed in Section 3.1.
This result reveals a key computational benefit: instead of performing a full d by e matrix-vector multiplication, we only need two smaller ones of size (d/2) x (c/2) by c/2. This effectively reduces the FLOPS by half while preserv- ing equivariance. More generally, the block-diagonalization from (1) to (2) follows from Schur's ' lemma (Lemma A.2) which applics to all finite symmetry groups. is closely related to the fact that convolutions reduce into elementwise multiplications in the Fourier domain.
Linear layers of the form (2) are called equivariant. In gen- eral, any layer that preserves the transformation properties of the input is called equivariant. For instance, we can add a bias parameter to the equivariant linear layer we en- force the bias to be zero for the (-1)-equivariant features. We give a more formal definition of equivariance in Sec- tion A, cf. (11). The stacking of equivariant layers build symmetry-respeddiiee
Three important questions remain and will be discussed next. First, how do we obtain features that are invariant and (-1)- equivariant? Second, how do we design the other parts of the network? Third, do we reduce representation power by forcing our features be invariant and (- --)-equuvaarint??
network is sometimes called the ge- ometric deep learning blueprint (Bronstein et al.. 2021), which goes back at least to the 90's with the works by Wood & Shawe-Taylor (1996). Using flopping invariant and (-1 ----equivariant features a special case of the steerable features proposed by Cohen & Welling (2017). However, general convolutional layers can not be be straightfor- wardly digg-diigo as in (2), since the convolutional kernels have spatial extent. In particular, in the networks by Cohen & Welling (2017), the described computational boost is not realized, as the equivariant convolutions are implemented in terms of ordinary convolutions. We will discuss this in more depth in Section 4.4.
3.2. Patch Embedding Layer
Obtaining invariant and -equiiarri features from an image is easy. One straightforward manner is convolve the image with symmetric and antisymmetric filters illus- trated in Figure 2. Such convolutions can be incorporated as the first layer in convolutional neural networks and vision transformers alike (the so-called patch embedding layer, or "PatchEmbed") and correspond to the lifting layer in an equivariant ConvNet (Cohen & Welling, 2016).
3

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
3.3. Non-Linear Layers
particular layer in the network, there exists a "steering matrix" A, such that when the input is flopped, the output feature is (approximately) transformed by A. Since flopping an image twice returns the original image, we must have that A² = T, which means that A can be diagonalized as A = QDQ-1 with D diagonal containing only entries of 11. (The generalization of this diagonalization is called the isotypical decomposition and is possible for more general symmetry groups well, due to Maschke's 's theorem (The- orem A))).) In other words, by changing the basis that we parametrize the network features in by Q. we get features that are invariant and --i-vvaaiaaiiant as before. Therefore, restricting the network to be equivariant is not really lim- iting what features are learnt if the equivariance is learned even without this restriction.
Non-linear layers of the neural network also need be equivariant, i.e. respect the transformation rules of the fea- tures. For instance, common layers in modern networks include layer normalization (Ba et al., 2016), activation functions such as GELU (Hendrycks & Gimpel, 2016), as well as multi-head self-attention (Bahdanau et al., 2015; Vaswani et al., 2017). Layer normalization is equivariant without modification since the norm of neither 21 nor x-1 changes the input is flopped. For pointwise activation functions o, we can create equivariant versions by comput- ing the output V1, V-1 through
t t ( (011==11/v2) $ 8o (01 + x v)/v2)
(3)
111 + (s+ t)/V2,
H - t//V
The emergence of equivariance from data has sometimes been given as an argument against hard-coding equivariance in the network. We want to turn this argument around (or flop it) by saying that if the network learns equivariance in any case, we might as well hard-code it and take advantage of the computational benefits that come for free. It should be mentioned, however, that it is not obvious that the choice with an equal number of invariant and (-1)-equivariant features is optimal. In fact, Bökman & Kahl (2023) found that networks often learn more invariant features than equiv- ariant features for classification tasks, while Bökman et al. (2024) found that for keypoint description, networks often learn an equal amount of invariant and equivariant features..
The reader is encouraged to check that when x- is multi- plied by 11 3/1 remains constant and y- is multiplied by -1. Here (3) can be interpreted as transforming from the "Fourier" domain to the "spatial" domain (s and t permute when x=1 changes sign), applying there and transforming back. We include the factor 1/v2 to preserve the norm of the features. The computation in (3) is heavier than just applying a, but this overhead is negligible compared to, for instance, linear layers.
3.4. Attention
For self-attention, if the queries qi and keys ki are split into flopping invariant and (-1)-equivariant features, their dot products form an invariant quantity:
It is also not obvious that an equivariant layer has as good parametrization an ordinary layer for network training, particularly when using standard training recipes developed for ordinary layers. Recent work by Pertigkiozoglou et al. (2024) shows that it may be possible to improve the train- ing of equivariant networks by relaxing the equivariance constraint during training. In any case, the main aim of the present paper to demonstrate that equivariant net- works, when parametrized right, are as FLOP-efficient per parameter as ordinary networks. We leave the investigation of optimal design and training of equivariant networks as orthogonal research directions for future work.
a. = qr,1 1k,, -k,,1 .k, = 9.,1 + 1 ky
(4) The asj's 's are then normalized using softmax as in standard scaled dot-product attention and the same attention score is multiplied by both invariant and (-1)-equivarriat values #1 to obtain the output.
3.5. Limitations of Equivariant Networks
Finally, we need to discuss the limitations put on the model by enforcing each layer to be equivariant. Prior work has shown experimentally, and theoretically in some special cases, that ordinary networks learn to be approximately equivariant by training on symmetric data (Lenc& & Vedaldi, 2015; Olah et al., 2020; Gruver et al., 2023; Bruintjes et al., 2023; Bökman & Kahl, 2023; Marchetti al., 2024)2. What is meant by this is that for given output feature space of
3.6. Generalization to Other Groups
The argument given for computational savings in linear lay- ers can be generalized to other groups than the flopping group. In Section A, we go through the mathematical back- ground in detail. Importantly, whenever the representations
for motivating hard-coding equivariance--sying that it can be learned from data-thert is still a clear computational argument in favour of hard-coding.
"Some prior work also found evidence in the other direction, i.c., that weight symmetries do not always appear in networks trained on symmetric data (Moskaley et al., 2023). Thus, it is partially unsettled question to what extent and in which situations cquivariance can be learned from data. As we argue in this section, is however the case that even if we take the least favourable view
'Of course, nothing guarantees that training an ordinary net- work with gradient-basee optimization yields the optimal alloca- tion of invariant and equivariant features.
4

---

Flopping for FLOPs: Leveraging Equivariance for Computational Efficiency
acting on the feature spaces are completely reducible (i.e. direct sum of irreps), linear layers can always be block- diagonalized by parameterizing them in terms of the irreps. The computational savings depend on (a) the number of irreps, (b) the dimensionality of the irreps, and (c) whether the real irreps are real, complex or quaternion type. Com- putational savings are possible in general but the number of FLOPs per parameter will be higher for irreps with dimen- sion larger than 1.
Equivariant PatchEmbed
4. Modern Neural Networks for Image Data
Equivariant features
XN
Since the development of Highway nets (Srivastava et al., 2015) and ResNets (He et al., 2016a), modern neural net- works for image data consist of stacking residual blocks
Equivariant Residual Block
Invariant features only
B(x) = x + ((x).
(5)
Classifier
Often, ø is a composition of normalization layer, such as batch normalization or layer normalization, and a computa- tional sequence, such as two-layer MLP, a self-attention layer or two-layer ConvNet. The main reason for using residual blocks is to facilitate network training by enabling gradient propagation through many layers.
Figure 3: Schematic. Our architectures follow the basic geometric deep learning blueprint (Wood & Shawe-Taylor, 1996; Cohen & Welling, 2016; Bronstein et al., 2021) in combination with modern vision models (Dosovitskiy et al., 2021; Liu et al., 2022; Touvron et al., 2023). The equivariant PatchEmbed described in Section 3.2. The residual blocks differ per architecture and are explained in Section 4. The classifier head is single linear layer.
We cover three architectures based on (5), and outline the minimal changes to make them flopping-equivarinne We start with the simplest architecture, ResMLP (Touvron et al., 2023), working our way through ViT (Dosovitskiy et al., 2021) and finally ConvNeXt (Liu et al., 2022).
ant and efficient, we decompose the features not only into invariant and equivariant in the channel dimension, but also in the patch dimension. This detailed in Section
Figure 3 shows schematic of the network design. All three network families incorporate patch embedding layer ("PatchEmmeed") as the first layer, as outlined Section 3.2. We replace all linear layers (except the depthwise convolu- tions in ConyNeXt) by block diagonal linear layers as in (2) and use equivariant layer norm, attention and GELU- nonlinearities as discussed in Section 3. Further special layers will be discussed in the subsections below.
One unique aspect of ResMLP is that it doesn't 't normalize features, but instead uses an affine layer (over the channel dimension), in symbols
aff (x) Diag(a)= 3,
(6)
where a and @ are learnable weight vectors. To make (6) euuping-equivarriat we only need to set 33 to constant 0 for the --iieequivariant features.
We chose architectures and training recipes based on three criteria: the architectures should be well established, a Py TyTchh implementation should available, and train- ing recipes for ImageNet- IK should be available, as larger datasets are out of reach for our computational budget.
Touvron et al. (2021) proposed LayerScale, a learnable di- agonal matrix applied on the output of each residual block, initialized close to O. This is just an affine layer as in (6) without bias / and is also included in ResMLP.
4.1. ResMLP
Tolstikhin et al. (2021), Melas-Kyriazi (2021) and Touvron
The ResMLP- are the form B = Bg o B1 with
al. (2023) independently proposed a simplification of vision transformers, where the attention layers replaced by MLPs over the patch dimension. These model types are most often denoted as MI.P-Mixers in the literature, but as we use the ResMLP version and training recipe by Touvron et al. (2023), we will usually refer to the models as ResMLP. To make the linear layers over the patch dimension equivari-
(x) = x LS(FC patches (Aff(z))). B²(x) = x LS(MLP channels (Aff()))).
where LS denotes layer scale, FC a fully connected (block- diagonal in our case) layer and MLP two fully connected (block-diagonal) layers separated by GELUU
5

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
4.2. VIT
4.3. ConvNeXt
Vision transformers (ViTs) use transformer encoder blocks (Vaswani et al., 2017) residual blocks. The layer closely follows the original transformer implementation and only deviates by the layer normalization being applied be- fore the computation, as proposed by He et al. (2016b).
ConwNeXt, proposed by Liu al. (2022), builds on the ideas of the ViT architecture, outlined above, to create modern family of pure ConvNet models that compare fa- vorably with ViTs in terms of accuracy and scalability. The isotropic ConvNeXt architecture, denoted Convnext (iso.), even more closely resembles ViT as has no downsampling layers and keeps the same number of patches/pixels at all depths. We use the isotropic ConvNeXt architecture in this paper as it allows for more efficient implementation. The reason is that we do not have efficient implementations of symmetric and antisymmetric convolutions for the down- sampling layers; refer to Section 4.4.
In particular, the residual block can be expressed as B B2 o B1 with
Bi (zr = x + (())) B2(x) = x LS(MLP(LN(()))
(8)
where LS is LayerScale, LN layer normalization, MSA is multi-head self attention and MLP consists of two fully connected (block diagonal in our case) layers separated by GELU. LayerScale was part of the original ViTs but is included in the ViTs used as benchmark in this paper as we follow the training recipe by Touvron et al. (2022).
The main residual block is composed of a depthwise con- volution, using a 7 convolutional kernel, followed by two 1 1-convolutions, separated by a GELU non-linearity, The residual block can be expressed B B2 B1 with
To make VIT positional encoding to consist of half channels that are sym- metric about the middle of the image and half channels that are anti-symmetric, thus forming invariant and (-1)- equivariant features. Positional encodings added to the features after the initial PatchEmbed layer. We also enforce the classification token appended to the embedded image patches in ViTs to be constant zero for the (-1)-equivariant features. Multi-head self attention is made equivariant by using block- diigonna linear layers for the projections to queries, keys and values followed by attention described in Section 3.4.
we further modify the
(1(z) = x LNDDWConv777(()) B2(x) = x
(9)
where Dw stands for depthwise.
Out of the depthwise convolutions, half are set to be symmet- ric and half be antisymmetric. They are alternated so that out of the invariant input features, half are convolved with symmetric filters and half with aatisymmetric-ggnneeee invariant and (-1)-equivariant features respectively, and ( (-1) -------rquvaiiam inputs. The 1 x 1 convo-
vice versa for the lutions are implemented using the block diagonal form (2).
To experiment with varying the group representations in the intermediate layers, we implement ViT-versions that use only invariant features in half the layers. After the first half of the network, we map the d/2 invariant dimen- sions linearly invariant dimensions and throw away the (-1)- --u--euuiarri features. The subsequent layers are then ordinary ViT-layers, which preserve invariance. In these last layers, we do not get any computational saving compared to the baseline since the block-diagonalizan
4.4. On Efficient Steerable Convolutions
In prior work (Weiler & Cesa, 2019), steerable convolu- tions were implemented by inputting equivariant filters in ordinary convolution software routines. This did not pro- vide any computational benefit over -quivvarvar filters, which use the same routines. Our implementations efficient because the ConvNeXt architecture utilizes depth- wise convolutions followed by 1 x 1-convolutions, and the x 1-convolutions are pixel-wise linear layers that can be decomposed into block-diagonals as in (2). There are pre- vious works that use separable convolutions in equivariant networks, but they parametrize the features in the spatial domain rather than in terms of irreps and do not obtain a computational advantage over ordinary separable convolu- tions (Lengyel & van Gemert, 2021: Knigge et al., 2022).
in (2) will not have any W-1,11 or O-blocks. We denote these half-way invariantized networks by Z(ViT).
more
Following Weiler & Cesa (2019), we also implement a hybrid model where the first half of the residual blocks equivariant and the second half are standard blocks. The intuition is that inductive bias is most useful in early network layers for learning general features in all orientations, while the last layers use more specialized processing. As this is not the focus of the paper, we include only hybrid versions of the ViTs, denoted H(ViT), to test the limits of scaling equivariant layers.
It would actually be possible to implement more efficient steerable t x k-convolutions as well. For illustration, con- sider a one-dimensional correlation of a symmetric filter [a,a] by a signal [x. y, 2). We can reduce the number of
*These are called "restricted" by Weiler Cesa (2019).
6

---

Flopping for FLOPs: Leveraging Equivariance for Computational Efficiency
required FLOPs by writing the output as
10
[a,a] [x, 21 [ax + oy, + a=] (10)
=aa( + v). a(v + =)].
FLOPs (x11))
Even more efficiently, one can use so-called Winograd schemes specific for symmetric or anti-symmetric fil- ters (Winograd, 1980), We do not pursue this optimization in the present paper, it requires significant effort into writing GPU-code, but we believe that it is a promising direction for further improving the runtime of steerable Con- vNets. Thus, the FLOP and throughput values reported for (ConnNNeXt )iso.)) in Table do not include the possible further reduction obtainable from more efficient symmetric and anti-symmetric depthwise convolutions.
10'-
DeiT III
ResMLP ConvNeXI-IO Equivariant
102 Number Parameters (xl0f)
Figure 4: FLOPS scaling by model size. Comparing the number of FLOPs to the number of parameters in the model.
5. Equivariant vs. Non-Equivariant Networks
There are couple of different ways that equivariant net- works can be compared to ordinary networks. Cohen & Welling (2016) and most later works compare an equiv- ariant network with M trainable parameters with ordinary networks with M parameters. This is fair comparison from learning theory perspective since the parameters can then store the same amount of information. Subsequent works noted that this is unfair from computational perspective, (Weiler & Cesa, 2019; Klee et al., 2023; Roos & Kroon, 2024), because of the fact that the equivariant ConvNets had the same computational cost as ordinary network with many more parameters, since as mentioned they used the same convolution routines.
6. Experiments
In this section, we evaluate the effectiveness of flopping- equivariant networks. We first briefly discuss the setting of the experiments. Then we compare the efficiency and accuracy of equivariant versions of ResMLPs, ViTs and ConvNeXts from Section to their -quivvarva counter- parts. For a given architecture X, the equivariant version is E (X). {()) always has around half the number of trainable parameters and FLOPs X due to the block diaoonaagnnaiiztiioon of the weight matrices (2). We will release code and weights at github.com//eorg. bn/flopping for-flops.
The first measure for computational cost that we will use in this paper is the number of FLOPs required for a forward pass of the network. This represents a bound on how ef- ficient the network can be made. In the limit, as feature spaces grow and dense linear layers dominate the compute, the number FLOPs accurately predicts how fast the net- work will run on modern GPUs. FLOPs are therefore used for analyzing the compute scaling of large networks (Kaplan al., 2020). However, the most direct measure of computa- tional cost is the network's 's throughput in terms of images per second. This is a somewhat inconsistent measure as it depends heavily on the hardware used and can sometimes vary drastically with minor implementation changes. We believe that the best way to compare networks is to report all three measures: number of parameters, FLOPs and through- non-equivariant
Dataset. We benchmark model implementations the ImageNet- IK dataset (Deng al., 2009; Russakovsky et al., 2015; Recht et al., 2019), which includes 1 1.2M images evenly spread over 1,000 object categories.
Hyperparameters. We use the same training recipes as the baselines. The complete set of hyperparameters can be found Table 2 in the appendix.
Implementation. The major building blocks of modern deep learning, including linear layers, convolution layers and attention layers, are all implemented using general ma- trix multiplications (GEMMs). Modern GPUs enable ef- ficient GEMMs, typically implemented in a tiled fashion, processing blocks of the output matrix in parallel. Con- sequently, reducing the linear layers into block diagonals does in principle not hinder GPU utilization, the same tile-GEMMs can be used for the full product (1) or the block block-diagonal (2) (with the latter requiring half as many). We use the Py Torch (Paszke et al., 2019) compiler to obtain efficiently running networks, but writing custom GPU-code would likely further improve the throughput.
put, is usually done in the literature networks. One of the main points of this paper is to illus- trate that contrary to popular belief, flopping equivariant networks can be designed to have approximately the same number FLOPs per parameter as ordinary networks (see Figure 4).

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
85.5
83
81
Top-1 Accuracy (%)
$1.5
29-
84.0
83.5
75-
E(VIT)
ResMLP
82.5
H(VIT)
ConvNeXt-ISO Equivariant
B
I(VIT)
82.0
128
32
FLOPs (x10))
FLOPs (x10*)
(a) DeiT III.
ConvNeXt-ISO and ResMLP.
Figure 5: Validation accuracy on ImageNet- IK versus model complexity as measured by the number of FLOPs required per image for forward pass. Letters indicate the respective model sizes, see Tables 1 and 3 for details.
6.1. Results and Discussion
computational overhead from less optimized kernels becom- ing less significant. This is best illustrated by the superior speed of the equivariant ResMLP models for which linear layers dominate the computation. Our networks have no throughput improvement for attention or depthwise convo- lutions, which, despite not contributing the most FLOPs, are sometimes comparable in runtime to the dense layers. Future more efficient implementations of these layers, and further scaling of the embedding dimension, would pro- nounce the gain in throughput for the equivariant models.
Figure 4 shows that our networks maintain a comparable number of floating-pi point operations (FLOPs) per parameter to standard networks.
In prior work, image classification is often contextualized trade-off between accuracy and measures of model com- plexity, such as FLOPs and number of parameters. To that end, in Table I, we highlight both the theoretical complexity, iee model size, and FI.OPs, as well as the empirical speed as measured by the throughput (images processed per second), and the Top-1 classification accuracy on ImageNet- 1K. Our results demonstrate that increase the scale of the net- works, aupping-cquivariin networks achieve competitive classification accuracy while requiring half the FLOPs, as highlighted in Figure 5, and showcase higher throughput compared to baseline implementations. The hybrid model H(ViT-H) achieves the best accuracy all models, align- ing with the results for hybrid models by Weiler Cesa (2019). One reason the smaller equivariant models under- perform in terms of accuracy that they quite limited in the number of parameters-the regularization methods in the training recipes for the corresponding non-equivariant models (with more parameters) may be too heavy. We leave the compute-hes heavy task of optimizing the training recipes for future work, as we believe that our experiments clearly demonstrate the main point of this paper- that equivariant networks can be scalable.
Notably, for small models, such VIT-S and ConvNeXt- S, the equivariant versions have worse throughput. For making these efficient, custom GPU-code is likely required.
6.2. Experimental Limitations
We did not do any hyperparameter tuning (so the baselines we compare against are at an advantage). Particularly, we noted that some of the training runs were quite unstable: We were not able to get (VVT--S) to converge despite trying different random seeds, and the result of the non-equivariant ResMLP-L24 (the only model size that we introduced our- selves and thus trained the baseline for) was unexpectedly low. We only ited compute budget. This practice aligns with the prior work. We limited the experiments to supervised training on ImageNet- while the best results are usually obtained with pretraining on Imagenet-211 or distillation of larger
training per model, as we had a lim-
As shown in Figure 6, the relative throughput improvement of the equivariant networks becomes more pronounced as the size of the model increases, which can be attributed to a higher proportion of time spent on the linear layers, with the
'The reader will notice a flattening out of the rate of improve- ment between ViT-L and -H. ViT-H uses patch size 14 14 while the other ViTs use patch size 16 16. Smaller patch sizes means more patches and more compute the attention layers.
8

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
Throughhput Gain
Table Classification with Imagenet training. We contrast our Hopping-equivarinn networks to the originals using the same training recipes. The equivariant version of architecture is denoted (X)). H(X) is a hybrid model with equivariant layers for the first half the network. The baselines are not rerun, instead we show the results reported in their respective papers. The throughput and peak memory single A A100-40GB GPU with batch size fixed to 64 and compiled networks running mixed precision forward passes with no gradients.
20
Vitd
ReaMLP
ConWNeXt-1SO
10'
measured
FLOPs (x10')
Figure 6: Throughput gain based on model size. The throughput gain as measured by the percentage differ- ence between the equivariant and baseline implementations, placed according to the baseline's number of FLOPs.
Architecture
params throughput FLOPs/img Peak Mem (x10%)
Top-1
(im/s)
(x10")
(MB)
Acc.
MLP-Mixers (ResMLP (Touvron al., 2023))
ResML_P-1.44
318.1
1107
63.3
1778
80.4
(ResMLLP-224
159.2
1756
31.7
1056
81.5
models. The hardware and software used for our training runs differed from the baselines, see Section
ResMLP-B2
115.7
2482
23.2
779
81.0
((ResMLP-B24)
58.0
3459
11.7
519
80.8
ResMLP-S24
30.0
6445
6.0
320
79.4
(RRsMMLP-224)
15.1
7025
3.1
249
74.9
7. Conclusion
ResMLP-T12
15.4
12133
3.0
264
76.6
8 (ResMLP-TI2)
7.7
13154
1.6
221
72.0
In this paper, we have introduced aapping-equivarii neu- ral networks that maintain a comparable number of FLOPs per parameter to standard non-equivane non-equivariant networks. We showed that these have increased computational efficiency only in terms of FLOPs, but also actual throughput.
Vision Transformers (DeiT III (Touvron et al., 2022))
VIT-H
632.1
431
168.0
3366
84.6
H(VIT-H)
474.2
466
127.6
3231
85.0
I(VIT-H)
474.2
462
127.6
3231
84.7
S(VIT-H)
316.1
501
87.3
2598
84.4
VIT-L
304.4
1064
61.9
1726
84.2
The considered task of upright image classification is in sense the least interesting for equivariant networks the symmetry group is small and the output is invariant. We believe that future efforts should be directed towards equivariant networks as general visual feature extractors. Classic and recent works have already demonstrated the utility of having equivariant features for downstream vision tasks (Lowe, 2004; Loy & Eklundh, 2006; Lee al., 2023; Bökman et al., 2024; Garrido et al., 2024).
H(VIT-L)
228.3
1123
47.0
1743
84.5
I(VIT-L)
228.3
1123
47.0
1743
84.1
S(VIT-L)
152.2
1204
32,2
1459
83.4
VIT-B
86.6
3088
17.7
777
83.1
N(VIT-B
65.0
3162
13.5
936
82.9
I(VIT-B
65.0
3163
13.5
921
82.5
E(VT-B)
43.3
3266
9,3
828
82.2
VIT-S
22.1
7174
4.7
338
80.4
E(VI-S)
11.0
6565
2.6
417
Convolutional Networks (ConvNeXt (Liu et al., 2022))
ConvNeXt-L. (iso.)
305.9
1284
60.0
1420
82.6
E(ConvNeXC (iso.)
153.1
1527
30.3
93.5
82.6
We hope that efficient equivariant networks can be adopted into modern vision backbones and that this work prompts more researchers to view equivariant architectures as not only theoretically compelling but also practically useful, not only for parameter efficiency in the small data regime but also for compute efficiency in the large data regime.
ConvNeXt-B (iso.)
87.1
3890
17.0
540
82.0
E(ConwNeXt-B (iso.)
4094
8.6
450
ConvNeXt-S (isa.)
22.3
9649
4.3
226
79.7
E(ConvNeXts) (ixo.)
11.2
7851
2.3
220
76.8
P-L22 trained by bascline.
Failod to converge.

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
Impact Statement
In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 2024.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Brehmer, J., de Haan, P., Behrends, S., and Cohen, 'T. S. Geometric algebra transformer. In Advances in Neural Information Processing Systems, volume 36, pp. 35472- 35496. Curran Associates, Inc., 2023.
Acknowledgements
Brehmer, J., Behrends, S., de Haan, P., and Cohen, T. Does equivariance matter at scale? arXiv preprint arXiv:2410.2317, 2024.
This work was supported by the Wallenberg Artificial In- telligence, Autonomous Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg Foun- dation. The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE, partially funded by the Swedish Research Council through grant agreement no. 2022-06725, and by the Berzelius resource, provided by the Knut and Al- ice Wallenberg Foundation at the National Supercomputer Centre.
Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv:2104. 13478 [cs, stat], May 2021.
Bruintjes, R.-J,, Motyka, T., and van Gemert, J. What affects learned equivariance in deep image recognition models? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4838-484444 June 2023.
References
Ja, J. L., Kiros, J. R., and Hinton, E. Layer normalization. arXiv preprint arXiv: 10M7v:6607..644 06450, 2016.
Cohen, T. and Welling, M. Group equivariant convolutional networks. In Conf. Machine Learning. 2016.
Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning align and translate. ICLR, 2015.
Cohen, T. and Welling, M. Steerable CNNs. In Int. Conf. Learn. Represent. 2017.
Dao, T. Flashattention-2: Faster attention with better par- allelism and work partitioning, 2023. URL https: / //aaxiv..com org/ 20000//333.0000
Bekkers, E. J., Lafarge, M. W., Veta, M., Eppenhof, K. A., Pluim, J. P., and Duits, R. Roto-translation covari- ant convolutional networks for medical image analysis. In Medical Imuge Computing and Computer Assisted InterventionMMccC 2018: 21st International Confer- ence, Granada, Spain, September 16-20, 2018, Proceed- ings, Part I, pp. 440 448. Springer, 2018
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 (009///PR.2009.5200660
Bekkers, E. J., Vadgama. S., Hesselink, R., der Lin- den, P. A. V., and Romero, D. W. Fast, expressive S maathrm (n) equivariant networks through weight- sharing in position-orientation space. In The Twelfth International Conference on Learning Representations, 2024. URL https: http::::///ppeeeveeeee net/ forum? id=dPHLbUqGbr.
Dieleman, S., De Fauw, J., and Kavukcuoglu, Exploiting cyclic symmetry in convolutional neural networks. In International conference machine learning, pp. 1889 1898. PMLR, 2016
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
Bellaard, G., Smets, B., and Duits, R. Universal collec- tion of euclidean invariants between pairs of position- orientations. arXiv preprint arXiv:2504.03299) 2025.
Bökman, G. and Kahl, F. Investigating how relu-networks encode symmetries. In Advances in Neural Information Processing Systems, volume 36, pp. 13720-13744, Cur- ran Associates, Inc., 2023.
Freeman, W. T. and Adelson, F. H. The design and use of steerable filters. IEEE Trans. Pattern Anal. Mach. Intell., 13(9)891-9966, 1991.
Fukushima, K. Cognitron: self-organizing multilayered neural network. Biological cybernetics, 10(3):121-138 1975.
Bökman, G., Edstedt, J., Felsberg, M., and Kahl, F. Steerers: framework for rotation equivariant keypoint descriptors.
10

---

Flopping for FLOPs: Leveraging Equivariance for Computational Efficiency
Garrido, Q., Assran, M., Ballas, N., Bardes, A., Najman, L.,, and LeCun, Y. Learning and leveraging world mod- els in visual representation learning. arXiv preprint arXiv:2403.00504 2024.
Kondor, R., Lin, Z., and Trivedi, S. Clebsch-gordan nets:
fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Gerken, J. E., Aronsson, J., Carlsson, O., Linander, H., Ohisson, F., Petersson, C., and Persson, D. Geometric deep learning and equivariant neural networks. Artificial Intelligence Review, (6(12)::400--14660 2023.
Kundu, S. and Kondor, R. Steerable transformers, 2024. URL https: watpiriiiivv///..///..aa.aaaaav. org/ httpgabb/2445..5992
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. Backpropaga- tion applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989.
Gruver, N., Finzi, M. A., Goldblum, M., and Wilson, A. G. The lie derivative for measuring learned equivariance. In The Eleventh International Conference on Learning Representations, 2023, URL https: /ppp/eeeee..eeeeeeeeeeeeee.e.eee (ol.oruunmii--1117avlllllll
Lee, J., Kim, B., Kim, S., and Cho, M. Learning rotation- equivariant features for visual correspondence. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 21887-21897, 2023.
Gupta, S., Robinson, J., Lim, D., Villar, S., and Jegelka, S. Structuring representation geometry with rotationally equivariant contrastive learning. In The Twelfth Interna- lional Conference on Learning Representations, 2024.
Lenc, K. and Vedaldi, A. Understanding image representa- tions by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999, 2015.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 201
Lengyel, A. and van Gemert, J. Exploiting learned symme- tries in group equivariant convolutions. In 2021 IEEE International Conference on Image Processing (ICIP),
pp. 759-763, IEEE, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. CoRR, abs/1603.05027,, 2016b. URL http: http:://aaxvvvooo wttpx//ww.org/bb//...333333
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T. and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986, 2022.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
Lowe, D. G. Distinetive Image Features from Scale- Invariant Keypoints. Int. J. Comput. Vis., 60(2):91-110, November 2004. doi: 10/bqrmsp.
Kaba, S.-O., Mondal, A. K., Zhang, Y., Bengio, Y., and Ravanbakhsh, S. Equivariance with learned canonicaliza- tion functions. In International Conference on Machine Learning, pp. 15546 15546-15566. PMLR, 2023.
Loy, G. and Eklundh. J.-O. Detecting symmetry and sym- metric constellations of features. In Computer Vision- ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006. Proceedings, Part I1 9, pp. 508 0211521.55 Springrr, 2006.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. 2020.
Marchetti, G. L., Hillar, C. J., Kragic, D., and Sanborn, S. Harmonics of learning: Universal fourier features emerge in invariant networks. In The Thirty Seventh Annual Conference on Learning Theory, pp. 3775-3797. PMLR, 2024.
Klee, D., Park, J. Y., Platt, R., and Walters, R. com- parison of equivariant vision models with imagenet pre- training. In NeurIPS 2023 Workshop on Symmelry and Geometry in Neural Representations, 2023. URLhttpe //openreview net/ / forum? id= 2NHPOV..
Melas-Kyriaze L. Do you even need attention? stack of feed-forward layers does surprisingly well on imagenet. arXiv preprint
Knigge, D. M., Romero, D. W., and Bekkers, E. J. Exploit- ing redundancy: Separable group convolutional networks on lie groups. In International Conference on Machine Learning, pp. 11359-13388 PMLR, 2022.
2021.
Mondal, A. K., Panigrahi, S. S., Kaba, O., Mudumba, S. R., and Ravanbakhsh, S. Equivariant adaptation of large pretrained models. Advances in Neural Information Pro- cessing Systems, 06:50293-50309 2023.
Knutsson, H. and Granlund, G. Texture analysis using two- dimensional quadrature filters. 01 1983.

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
Moskalev, A., Sepliarskaia, A., Bekkers, E. J., and Smeul- ders, A. W. On genuine invariance learning with- out weight-tying. In Doster, T., Emerson, T., Kvinge, H., Miolane, N., Papillon, M.. Rieck, B., and San- born, S. (eds.), Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learn- ing (TAG-ML), volume 221 of Proceedings of Ma- chine Learning Research, pp. 218-227. PMLR, 28 Jul 2023. URL wwt.hhttss//pproeeee http:://oocceeddee www.eeeiiinsmm........ mllr press/ v221 //mosoalle23aaa html.
28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pp. 1171-1179, PMLR, 03-05 May 2025. UtLhttps: httpttts:t http:sss//pppceeeiegeeee mlr press/ v258 hts//smmudee2.. .ht.mmree.a...........
Serre. J.-P. Linear representations of finite groups. Springer, 1977.
Srivastava, R. K., Greff, K., and Schmidhuber, J. Training very deep networks. Advances in neural information processing systems, 28, 2015.
Olah, C., Cammarata, N., Voss, C., Schubert, L., and Goh, G. Naturally occurring equivariance in neural net- works. Distill, 2020. doi:
Sutton, R. The bitter lesson. Incomplete Ideas (blog), 2019.
http://fiillllbb22222//ciiivieer
Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung. J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all-mlp architec- ture for vision. Advances in neural information process- ing systems, 34:24261-242777 2021.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., Detoto, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang. L., Bai, J., and Chintala, S. Pytorch: An imperative style,
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and Jégou, H. Going deeper with image transformers, 2021. URL https: /ttp/rrxiiv.////////a/////aa/..a org/ /atp//bb//.1113333
deep learning library. In Wal- lach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural In- formation Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Touvron, H., Cord, M., and Jégou, H. Deit ii:: Revenge of the vit. In European conference on computer vision, pp. 516-533. Springer, 2022.
Pertigkiozoglou, S., Chatzipantazis, E., Trivedi, S., and Danillidis, K. Improving equivariant model training via constraint relaxation. In The Thirty-eighth Annual Con- ference on Neural Information Processing Systems, 2024.
Touvron, H., Bojanowski, P.. Caron, M., Cord, M., El- Nouby, A., Grave, E., Izacard, G., Joulin, A., Syn- naeve, G., Verbeek, J., and Jégou, H. Resmlp: Feed- forward networks for image classification with data- efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):5314 5321, 2023. doi: 10.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? CORR, 10s/1902.10811, 2019. URL abs/1902. 10811.
Romero, D., Bekkers, E., Tomezak, J., and Hoogendoorn, M. Attentive group equivariant convolutional networks. In International Conference on Machine Learning, pp. 8188 8199. PMLR, 2020.
Vadgama, S., Islam, M. M., Buracus, D., Shewmake, C., and Bekkers, E. On the utility of equivariance and symmetry breaking deep learning architectures on point clouds. arXiv preprint arXiv:2501.0100 2025.
Roos, L. and Kroon, R. S. On fairly comparing group equiv- ariant networks. In ICML 2024 Workshop on Geometry- grounded Representation Learning and Generative Mod- eling, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is All you Need. In Adv. Neural Inform. Process. Syst., 2017.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211 211-252, 2015.
Weiler, M. and Cesa, G. General Equiiiauiaaiiantt steerable CNNs. Adv. Neural Inform. Process. Syst., 2019. URL https: https://proceedingg neurips. heurrpp..cc/ppeer///////eee 4 4566377011880F7423377338888 pdf.
Samudre, A., Petrache, M., Nord, B., and Trivedi, S. Symmetry- based structured matrices for efficient approx- imately equivariant networks. In Li, Y., Mandt, S., Agrawal, S., and Khan, E. (eds.), Proceedings of The
Weiler, M., Forré, P., Verlinde, E.,, and Welling, M. Equivari- ant and Coordinate Independent Convolutional Networks. 2023.
12

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
Wightman, R. Pytorch image models. https://github: com/ rwi g 2019.
Winograd, S. Arithmetic complexity of computations, vol- ume 33, Siam, 1980.
Wood, J. and Shawe-Taylor, J. Representation theory and invariant neural networks. Discrete Applied Mathematics, 690,-2):33-00, August 1996. ISSN 0166218X. doi: 10/c3qmr6.
Xu, R., Yang, K., Liu, K., and He, F. e2))-qquiaarian vision transformer. In Uncertainty in Artificial Intelligence, pp- 2356-2366. PMLR, 2023.
13

---

Flopping for FLOPs: Leveraging Equivariance for Computational Efficiency
A. Groups, Representations and Schur's Lemma
choices of p; in terms of fundamental building blocks called irreducible representations.
An irreducible representation (irrep) 2 G - GL(n) is a representation of G such that the only subspaces of R" that are invariant under all ((g) are IR" itself and the O-vector space", We will use the notation p for representations in general and e for irreducible representations.
This section is not required reading to understand the main takeaway of the paper. It serves to describe how the con- tent of Section 3 can be formalized and generalized using group theory. The theory content of this section is com- mon knowledge in geometric deep learning (Bronstein et al., 2021; Gerken et al., 2023; Weiler et al., 2023) and part of standard representation theory courses (Serre, 1977). Our contribution is to highlight the computational aspect more than has perhaps been done in previous work.
Representations that only differ by a change of basis Q are not qualitatively different mathematically and are called isomorphic. Maschke's 's theorem implies that the feature spaces in equivariant neural networks are always isomorphic to concatenations of feature spaces that irreps act on.
We use the language of group representations to facilitate a precise discussion of symmetries. Groups are the math- ematical abstraction of sets of symmetry transformations, such as the flopping of images. A group (G, o) is set G together with an associative binary composition operation To be group it has to have (i) unit element u E G such that u g = g o E G an inverse g-1 E G' such that g gog For image flopping, the mathematical group is called D2 (the dihedral group with two ) and consists of the identity u and another element h, which are both their own inverse (flopping an image twice returns the original image).
Theorem A.1 (Maschke's ' theorem). Every representation P: G ~ GT((n) of a finite group G is a direct sum of irreps.
Maschke's 's theorem means that for every representation p. there exists a change of basis matrix Q E GL(n), non- isomorphic irreps @i (i 1i=1,.. l......, m) and multiplicities k; E Ezo such that
9 for all g E G and (ii) for each
g-' = u.
)
+ (ell)) )0:,
p(g) =Q
(12)
where ① denotes the direct sum, i.e. assembling the matri- ces e (g))@k into block-diagonal matrix and X means a block-diagond matrix with & copies of X along the diag- onal. (12) is called the isotypical decomposition of p.
A group representation concretises the group set of invertible matrices. We denote the set of invertible real n X n matrices by GL(n), and a representation of G is a map p : G + GL(n), that encodes the composition operation as matrix multiplication: p(g) og2) = p(g) P((2)) It follows from p(u) p(u o u) = p(u) that p(u) is the identity matrix. Group representations enable us to discuss the same group acting on vector spaces of different dimensions. Therefore, group representations are useful to describe what happens to internal neural network features when the input is transformed. Specifically, a function f is said to be equivariant with respect to representations #1 and P2 if
Having characterized all representations in terms of irreps, we next characterize all equivariant linear maps through Schur's lemma. Given two real representations P1 G + GL(n1) and p2 : G + GL(n2), we denote the set of linear maps A : Ih1 + IR that are equivariant under P1 and P2 by Hom(p1, p2). Hom(pi, P2) is vector space since we can add and multiply by scalars without changing the equivariance property.
fer = (()))
(11)
Lemma A.2 (Schur's lemma). Let @1 and 02 be two real irreps of a group G.
If f consists of the first couple of layers of an equivariant neural network, and P1 (h) is the permutation of pixels that flops an image x, then (11) says exactly what happens to the internal features of the neural network as the input image is flopped-the features are transformed by p2(h).
1. If @n and @2 are not isomorphic, then Hom(ei, (2) contains only the zero-map.
2. If @ı and Q2 are isomorphic, then Hom(e1, e2) is ei- ther 1-, 2- or 4-dimensional.
Following the geometric deep learning blueprint (Bronstein et al., 2021), to design flopping-ing hopping-invariant invariant neural networks fNx ..... f2 S1, we enforce each network layer Si to be -opping-quuvvriiat. The last layer is designed be flopping invariant, which is the special case of (11) where p2(g)= = I. Designing an invariant neural network hence involves a choice of intermediate group representation pi for the output of each layer. We can write down all possible
The dimension in case 2 depends on whether the irrep 01 is of so-called real, complex or quaternary type (Serre, 1977).
We can put Maschke's 's theorem and Schur's lemma to work in a very useful standard corollary.
This definition can also be stated for complex vector spaces and infinite vector spaces, but will be satisfied with finite- dimensional real vector spaces in this paper.
'Other names for D2 are C2. Z2, S2 and (confusingly) Dj.
14

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
Corollary A.3. If P1 G GL(n)) and P2 G + (T((2) are real representations of finite group G, then there are Q1 E GL(m1) and Q2 GL(n2) such that for any A E Hom(p1. P2), Q2 AQ1 is block- diagonaa with blocks only mapping between isomorphic irreps.
"Huge" (H) are specified in Table 3.
Accelerating throughput. We use three tools improve the throughput of the networks, both the baselines and the equivariant implementations.
1. Flash Attention (Dao, 2023), used by the ViTs.
For discussing image flopping. it is useful to know that D2 has only two irreps, both one-dimensional, defined by @i(h) == 1 (the "trivial" irrep) and e-1 == -1 (the "sign-flip" irrep). Therefore, by Corollary A.3, every equiv- ariant linear map decomposes into a block-diagonal map with two blocks. Working in this block-diagonal basis hence gives computational advantage. Features transforming by 21 are those called invariant and those transforming by e-1 are called (-1 1) -equivarrin in Section 3.
2. Mixeedprecision training using NVIDIA's Apex library, used by all models except the ResMLPss
3. PyTorchh's compiler and high precision matmul.
Calculating throughput. As is specified in Table 1, the throughput and peak memory are measured on single A100-40GB GPU with batch size fixed to 64 and compiled networks running mixed precision forward passes with no gradients. Moreover, we utilize 10 warm-up iterations and then average over 100 runs. To measure peak memory we make use of PyTorch's built in device memory allocation monitor.
For larger groups than D2, the number of irreps are more numerous and so the computational savings in linear layers can be even larger. However, for groups with irreps of di- mension 1 the PLOPs-per-paramene with more irreps, the hyperparameter-task of choosing rep- resentation in each layer is also more difficult. We believe that a promising direction for future research on scaling up efficient equivariant networks to work with larger dihe- dral groups such as Dg the symmetry group of the square pixel grid. Dg has four irreps of dimension 1 and only one irrep of dimension 2 > 1. Further, dihedral groups have the conceptual advantage that all of their irreps are of real type, so that the dimension of Hom e) is 1 according to Schur's lemma.
will be worse and
Counting FLOPs. For counting the number of FLOPs, we use fvcore. nn Fl Fw..Fllpppooc opCountAnal Aasalysissss yssi (fvcore). We further make sure to add support for the operations added by the flopping-cquivariant implementation as the default counter in Evcore does not count elementwise additions for instance. FL.OPs are normalized with respect to the batch size.
C. ResMLP Linear Layers Over Patches
To make the linear layers over the patch dimension flopping- equivariant, we need to keep track of what happens with the patches as the image is flopped. If we denote by x the N x N x d/2 tensor of invariant patch embeddings, then 31 f: k 1] changes place with 21(:, -k] when the image is flopped (where use NumPy-style tensor indexing). Thus,
B. Experimental Setting
We train on images of size 224 x 224 throughout. Hyper- parameters are listed in Table 2. While ConvNeXt uses exponential moving accuracy by default, we find for the equivariant nets that the non-averaged model gives better accuracy and therefore report that accuracy.
3 11 k 1] == x k 1] + æi -k]
Software versioning. Our experiments build upon PyTorch (Paszke et al., 2019) and the timm (Wightman. 2019) li- brary. We enable mixed-precision training using the depre- cated NVIDIA library Apex, this is to mirror the training recipes of the benchmarks as closely as possible. To en- able PyTorch's 's compiler, we use a modern version (2 2.0). Specifically, we use PyTorch 2.5.1 with CUDA 11.8.
is actually an invariant quantity while
2 z1,- I1, k - 1] : x. [t. 1] 2 2: -k]
is (-1)-equivvarint, Similarly, from the (-1) --uiivvrrra output from PatchEmbed, x- we can construct the (-1)- equivariant
Hardware. All experiments were run on NVIDIA A100- 40GB. The per GPU batch size ranged between 64 (for larger models) to 256 (for smaller models). The biggest model requires training on 32 A100 GPUs for c. 2 days. The baselines were trained on V100 GPUs by the respective authors.
x x-111,, 1] := x- 1ii 1] + +
and the invariant
x -- (E, kt == x 11: k
x- 1|: -k].
Model sizes. The model sizes, referred to in the paper as "Tiny" (T), "Small" (S), "Base" (B), "Large" (L.), and
By keeping track of the four tensors x+1,11, each of shape
(N x N/2) x d/2, we can make both the linear layers over
15

---

Flopping for FLOPS: Leveraging Equivariance for Computational Efficiency
Table 2: Training recipes for different model architectures. We try to, as closely as possible, replicate the training recipe of the baselines.
DelT III (louvron et al., 2022)
ResMLP (Torvron al., 2025)
ConvNeXI (1 et al,, 2022)
Model
VIT-S
ResMLP-T
ResMLP-L
ConvNeXi-S (iso.)
(iza.)
Con-NeXI-L (isp.)
Batch sicc
2018
2048
2048
2018
2018
2048
2018
2018
1096
4096
1096
Optimizer
Adam
LR
3.10-
3.10-
3.10
3.10-a
5.10-a
5.10-a
5.10-4
5.10
4.10-1
1.30-
4.10-1
decay
Weight decay
0.02
0.02
0.02
0.02
0.2
0.2
0.2
0.2
0.05
0,055
0.05
Training epochs
Warmup epochs
5
50
50
50
smonthing
Dropout
X
X
Stoch, Depth
0.0
0.1
0.4
0.05
0.5
Repeated Aug
V
Gradical Clip.
1.0
1.0
Mixed Procision
Rand Augment
X
3 Augment
/
Mixup alpha.
0.8
0.8
0.8
0.8
0.8
Cutmix alpha
1.0
1.0
1.0
Erasing prob.
X
0.25
0.25
0.25
0.15
Colorlitter
0.3
0.3
0.3
0.3
0.4
0.4
Test crop mtio
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
BCE
CB
Table Specification of model notation in terms of depth and width (embedding dimension).
Architecture
Depth
Width
MLP-Mixers (ResMLP (Touvron et al., 2023))
ResMLP-L
1280
ResMLP-B
768
ResMLP-S
384
ResMLP-T
384
Vision Transformers (DeiT III (Touvron al., 2022))
VIT-H
32
1280
VIT-L
24
1024
VIT-B
12
768
VIT-S
12
384
Convolutional Networks (ConvNeXt (Liu et al., 2022))
ConNNeXt-L (iso.)
36
1024
NeX-Ne (iso.)
18
768
ConvNeXi-S (iso.) 18
384
the patch dimension (N x N) and the channel dimension d efficient through the decomposition (2).
This calculation an example of Clebsch-Gordal product or tensor product of representations, which has been used in some prior designs of equivariant networks (Kondor al., 2018).
16


Reviews:
[{"review_id": "HbgQuLpxev", "reviewer": "Reviewer_3YAc", "summary": "The paper's main goal is to develop new equivariant vision models that scale effectively. It presents equivariant networks that preserve mirror symmetry while keeping FLOPs comparable to non-equivariant models. The focus is on simple image symmetries for more efficient computation. The proposed network divides all feature maps into two types: flopping invariant features and flopping equivariant features. The latter undergo a sign flip when mirroring occurs. This split architecture enables smaller matrix multiplication while maintaining equivariance, cutting the number of FLOPs in half. The paper discusses a patch embedding layer along with considerations for non-linearities and attention. It examines three successful vision architectures: ResMLP (Touvron et al., 2023), ViT (Dosovitskiy et al., 2021), and ConvNeXt (Liu et al., 2022). Each architecture is explained simply, with adaptations proposed using the flopping equivariant architecture. The experiments compare: 1) model size versus FLOPs, 2) network throughput, and 3) memory usage. Results for all architectures are presented on ImageNet, showing that the proposed models scale well compared to state-of-the-art non-equivariant vision models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the clear review, which accurately summarises our main contributions.\n\n> I think references are discussed adequately. However, I would like to suggest the authors also take a look at the following paper: \"Symmetry-Based Structured Matrices for Efficient Approximately Equivariant Networks\" by Samudre et al. This paper is about exploring a similar theme, even though the paper is mainly theoretical, and about approximate equivariance. Note that the experiments seem to have a typo (FLOPS instead of FLOPs).\n\nThis is a good suggestion for an additional reference, which we will add. Combining the structured matrix approach with the irrep-decomposition approach could be interesting for future work.\n\n> As such, the ideas used in the paper are technically straightforward. However, its main goal is to encourage designing equivariant models that have favourable scaling properties. In that goal, I believe the paper does a decent job.\n\nWe hope the technical straightforwardness can be regarded as a strength in this context.\n\n> Could the authors expand on why implementing the Winograd scheme might be involved on GPU?\n\nImplementing a naive version of the scheme is quite straightforward. What is challenging is making the implementation competitive with highly optimised GPU-kernels for ordinary convolutions, such as the ones in the widely used closed source cuDNN library. It requires careful handling of data shuffling within the GPU as well as proper utilization of hardware specifics such as tensor cores.\n\nFinally, we have included one extra experiment in the rebuttal to reviewer s6u9 (Q1), which may interest all reviewers.", "claims_and_evidence": "The main motivation of the paper is to encourage work on scaling equivariant neural networks. The main claims are 1) flopping equivariant networks can achieve accuracy comparable to the state of the art vision networks. 2) the number of FLOPs vs model size tradeoff is comparable. 3) The memory footprint is reduced. The results in figure 5 and table 1 do accurately support the main claims of the paper.", "methods_and_evaluation_criteria": "Yes, the evaluation criteria (comparison in terms of accuracy, throughput, memory, FLOPs, model size) are sensible. The three baseline architectures are well-optimized in the literature on ImageNet, so the comparison actually puts the proposed model at a slight disadvantage.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experimental design and the choice of metric is sensible. The appendix describes experimental recipes in more detail.", "relation_to_broader_scientific_literature": "Exploring the scaling behaviour of equivariant models is only a recent area of interest. This paper considers specific vision models and then proposes equivariant counterparts that scale well. I think the key contribution is well-motivated and of broader interest.", "essential_references_not_discussed": "I think references are discussed adequately. However, I would like to suggest the authors also take a look at the following paper: \"Symmetry-Based Structured Matrices for Efficient Approximately Equivariant Networks\" by Samudre et al. This paper is about exploring a similar theme, even though the paper is mainly theoretical, and about approximate equivariance. Note that the experiments seem to have a typo (FLOPS instead of FLOPs).", "other_strengths_and_weaknesses": "As such, the ideas used in the paper are technically straightforward. However, its main goal is to encourage designing equivariant models that have favourable scaling properties. In that goal, I believe the paper does a decent job.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Could the authors expand on why implementing the Winograd scheme might be involved on GPU?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a65Q8En4fs", "reviewer": "Reviewer_s6u9", "summary": "In this paper, the authors present a flopping equivariant variant of known vision models like ConvNext, ViTs, and ResMLP. The key idea is to parameterize feature space in terms of mirror symmetry and mirror anti-symmetry features. This approach reduces FLOPs and wall-clock time to give an efficient scalable and efficient symmetry architecture. The empirical experiments are done on the Imagenet 1K dataset Flops, throughput, and number of parameters are reported for three different architectures.\n\n## post rebuttal\nI will keep my score as the authors gave a satisfactory rebuttal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful review and relevant questions.\n\n> The claim in section 5: equivariant networks can be designed to have the same number of FLOPs... seems to lack evidence [...].\n\nWe thank the reviewer for the pointer and will refine it to “flopping equivariant networks”.\n\n> Essential References Not Discussed\n\nThe suggested references are relevant and will be included. Note that our idea for speeding up equivariant networks has not been applied in any of these papers.\n\n> In section 3.5, Limitations: It is unclear what the limitations described in the first paragraph.\n\nOne argument against hard-coding equivariance is that it limits what features can be learned. We argue that this limitation is weak in the first paragraph. We will make it clearer.\n\n> Q1. [...] How would the framework’s performance change if this ratio were adjusted based on task-specific learned symmetries and is there a way to learn the optimal division for?\n\nTo answer this question, we have implemented a ViT that has an equal split of invariant and (-1)-equivariant features for the first half of the network and only invariant features for the second half of the network (the (-1)-equivariant features from the first half of the network are invariarized by taking their absolute value, incidentally this is a “complete invariant”). For layers with only invariant features, the block-decomposition in equation (2) turns into a dense matrix $W_{1,1}$ since $W_{-1,-1}$ doesn’t exist. So, there are no FLOP savings compared to ordinary layers, and the FLOP-per-parameter ratio is also the same.\n\nThe accuracies on ImageNet are as follows:\n\nViT-S: no convergence, ViT-B: 82.5%, ViT-L: 84.1%, ViT-H: 84.7%.\n\nComparing these with the results in Table 1, we see that they consistently outperform the equivariant ViT with equal amounts of invariant and (-1)-equivariant features. This aligns with the prior work as discussed in Section 3.5. In terms of total FLOPs and parameters the new networks are between the previous equivariant and the baseline networks, for ViT-H the values are almost exactly the same as the previously considered hybrid network.\n\nWe are not aware of a method to guarantee learning the optimal division of irreps, but as outlined in Section 3.5, we regard this as promising and interesting future work.\n\n> Q2 [and earlier comments]. Is it possible to generalize this approach beyond D2 (flopping)? How does that affect the performance with similar FLOPs?\n\nYes, this is possible. We discuss this extension in Appendix A. The FLOPs-per-parameter ratio depends on the group and it is always possible to achieve computational savings. Please refer to the rebuttal to Reviewer RXrY (Q2) for a detailed discussion.\n\n> Q3. Methods like learned canonicalization and its adaptation to large models improve performance without huge computational costs [1, 2]. How does the proposed approach compare to these techniques?\n\nCanonicalization would mean training an extra network that maps an image to a canonical orientation and then feeding that image into a pretrained network. This is useful when a pretrained network exists which is not invariant. A common example would be a network trained on upright images not generalising to rotated images. If the data that the pretrained network was trained on already contains the symmetry (e.g. upright images are equally common in flopped format) then canonicalization can not improve performance because the pretrained network is not better on a particular orientation of the images. Also, highly symmetric images can pose a problem for canonicalization-based methods (see e.g. Appendix A of the mentioned [2]). Finally, canonicalization methods incur an extra overhead of running the canonicalization network, while we instead obtain networks that run faster than the baseline networks.\n\n> Q4. Does data augmentation like rotations or flips (commonly used in contrastive learning) help with performance in the proposed approach?\n\nYes, we use heavy data augmentation like the baseline methods, including Mixup, Cutmix, etc. Refer to Table 2 in the appendix.\n\n> Q5. With libraries like Cu-equivariance [3] which provides improved irreps computation with cuda implementation, how does this affect the impact of the proposed work?\n\nLibraries with faster equivariant layers further strengthen the broad case for equivariant networks made in Section 3. Cu-equivariance is specialised for processing 3D graphs with rotation ambiguity, such as molecular data, and thus it is orthogonal to our implementations for images.", "claims_and_evidence": "The claim in section 5: *equivariant networks can be designed to have the same number of FLOPs...* seems to lack evidence, as the paper only shows this for a flop equivariance, and this might not be true for higher dihedral groups.", "methods_and_evaluation_criteria": "Yes, ImageNet 1K makes sense as well and the evaluation criteria are reasonable for the problem.", "theoretical_claims": "Yes, they seem correct.", "experimental_designs_or_analyses": "Yes, all of the experiments were presented. \n\nThe experimental design seems reasonable, although there are a few questions regarding the broader use/ implications of the choice made in the paper. The relation to contrastive learning (with equivariance) as well as learned canonicalization especially post hoc way in models are some ways interesting comparisons to the proposed approach as they are more commonly used for images. [1,2] show that augmentation helps with performance even with equivariant models, and given the task of image classification, augmentations are not used in the pipeline. \n\nThe classification task which is an invariant task with respect to transformations is considered, and tasks like image editing which is equivariant to transformations are not considered without any justification. \n\n1. Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning, Gupta et al. \n2. On the Utility of Equivariance and Symmetry Breaking in Deep Learning Architectures on Point Clouds, Vadgama et al.", "relation_to_broader_scientific_literature": "The key contribution of the paper is to scale equivariant models using mirror symmetry and mirror anti-symmetry is interesting and is novel, although the idea of using invariants to design equivariant networks is not uncommon.\nImportant links to existing works in a slightly different domain (point clouds, contrastive learning approach) are missing and also effective use for image classification tasks seems superfluous as ViT with post hoc canonilication or conservative learning transformation pipeline can achieve similar results. \n\nThe important takeaway from the paper is that equivariant models can be scaled in the proposed manner, and there could be more use cases beyond the ones mentioned in the paper.", "essential_references_not_discussed": "Yes, most of the related works are covered in the paper. Although a few important works regarding are missing. \n\n\n1. Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning, Gupta et al. \n2. On the Utility of Equivariance and Symmetry Breaking in Deep Learning Architectures on Point Clouds, Vadgama et al. \n3. On genuine invariance learning without weight-tying, Moskalev et al. \n4. Equivariant Adaptation of Large Pretrained Models, Mondal et al.\n5. Equivariance with learned canonicalization functions, Kaba et al", "other_strengths_and_weaknesses": "## Strengths\n- This paper presents a way to scale equivariant models by considering flop equivariant neural networks. \n- The proposed approach is used for three different architectural choices: ViT, ConvNext, and ResMLPs which provides a good understanding of image classification using these architectures with their equivariant and hybrid counterparts. \n\n\n\n## Weaknesses\n- The paper does not include any suggestions for effective improvement in the pipeline like augmentation or generalizability to other group. \n- The low performance of smaller equivariant models is not justified.", "other_comments_or_suggestions": "- In section 3.5, Limitations: It is unclear what the limitations described in the first paragraph.\n\n- In section 3.5, important works on relaxing equivariance have not been mentioned. A lot of symmetry-breaking and relaxed equivariance works have been done for the domain of point clouds (molecules etc). See the Essential References section.", "questions_for_authors": "1. The equal split of invariant and (-1)-equivariant features are used in the paper. How would the framework’s performance change if this ratio were adjusted based on task-specific learned symmetries and is there a way to learn the optimal division for? \n\n2. Is it possible to generalize this approach beyond D2 (flopping)? How does that affect the performance with similar FLOPs? \n\n3. Methods like learned canonicalization and its adaptation to large models improve performance without huge computational costs [1, 2]. How does the proposed approach compare to these techniques?\n\n4. Does data augmentation like rotations or flips (commonly used in contrastive learning) help with performance in the proposed approach?\n\n5. With libraries like Cu-equivariance [3] which provides improved irreps computation with cuda implementation, how does this affect the impact of the proposed work? \n\n[1] Equivariant Adaptation of Large Pretrained Models, Mondal et al.\n\n[2] Equivariance with learned canonicalization functions, Kaba et al\n\n[3] [cu-equivariance](https://github.com/NVIDIA/cuEquivariance)\n\nand see weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jeZNfGaZel", "reviewer": "Reviewer_debe", "summary": "The paper suggests a clever implementation of flop-equivariant linear layers - together with some strategies to adapt other layers too - which allows to achieve flop-equivariance in most popular vision architecture while halving the computation cost with respect to their non-equivariant counterparts. \nThe paper is very well written and motivated and includes extensive empirical analysis.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and for highlighting Section 3.5. We are pleased to read that the reviewer finds the paper valuable.\n\n> The block diagonal decomposition of the linear layers in Sec 3.1 doesn't directly work for convolution layers, unless one only considers flop-invariant filters since anti-symmetric filters can map between the two types of features. This is briefly discussed later in Sec 4.3 but I think the authors could expand this point earlier in the paper for improved clarity.\n\nWe agree with this comment and will modify Section 3.1. We thank the reviewer for the suggestion.\n\nFinally, we have included one extra experiment in the rebuttal to reviewer s6u9 (Q1), which could interest all reviewers.", "claims_and_evidence": "All claims seem well supported by empirical evidence and theoretical arguments.\nMoreover, all the assumptions are backed up by extensively citing many previous works which provided relevant insights.\nOverall, this makes the story of the paper very convincing.", "methods_and_evaluation_criteria": "See above", "theoretical_claims": "See above", "experimental_designs_or_analyses": "See above", "relation_to_broader_scientific_literature": "The manuscript properly places itself in the literature, extensively citing the long line of works which provided the insights used to motivate this work.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is really well motivated, with a very convincing story and introduction.\nIn particular, I really enjoyed reading Sec 3.5, which includes some very elegant arguments and provides in a clear and concise way many practical insights about the design of equivariant architectures.\nI would even suggest the authors to mention some of these points in the introduction as part of the contribution.\n\nSec 3.1 also manages to give an intuitive yet precise description of equivariant layers and the potential computational gain, making the main ideas in the paper accessible for a wider audience.\n\nI do not see any significant weaknesses in this work.\nOf course, the empirical evaluation could be made stronger by scaling further and performing hyper-parameter tuning of the equivariant models as already argued by the authors, but I agree this is beyond the scope of this paper.\nI am looking forward to see the community picking up on these ideas and further extend the empirical validation.", "other_comments_or_suggestions": "The block diagonal decomposition of the linear layers in Sec 3.1 doesn't directly work for convolution layers, unless one only considers flop-invariant filters since anti-symmetric filters can map between the two types of features.\nThis is briefly discussed later in Sec 4.3 but I think the authors could expand this point earlier in the paper for improved clarity.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hoguHPfVlL", "reviewer": "Reviewer_RXrY", "summary": "The paper introduces three new equivariant neural networks (adapted from three non-equivariant models: ResMLP, ViT and ConvNeXt) for the purpose of showing that the equivariant models maintain a comparable number of floating-point operations (FLOPs) per parameter versus their non-equivariant counterparts. The main idea behind the authors' work is that they replace linear layers with block diagonal equivariant linear layers using irreps of the group C_2 (the flopping group), which ultimately parameterise feature spaces in terms of mirror-symmetric and mirror-antisymmetric features. They describe how to obtain invariant and flopping equivariant features using a patch embedding layer, how to implement the non-linear equivariant layers, and how to adapt self-attention to obtain equivariance. The authors evaluate their models on the ImageNet-1K dataset and compare them based on three measures: number of parameters, FLOPs and throughput.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review. We address their concerns here, which will sharpen the paper.\n\n> [...] the version of Schur's Lemma (A.2) provided by the authors was a bit overkill [...]\n\nWe are unaware of simpler versions of Schur’s Lemma for real irreps of finite groups than the one given. If the reviewer has a simpler form in mind, we welcome any suggestions.\n\n> Q1: Could the authors provide other examples where their C_2 equivariant construction might be useful [...]?\n\nSince $C_2$ is the simplest possible symmetry group, it appears in many situations. Examples include\n\n- 3D tasks: Many natural and man-made objects (and parts of objects) are equally likely to appear mirrored. Thus, tasks involving these are often mirror equivariant/invariant. This includes data such as meshes, point clouds, or 3D images.\n- Many games have mirror symmetry. E.g., for training neural networks to play Go, it is common to apply reflection augmentation.\n- Mirror equivariance/invariance is also common in 1D data. E.g., measurements such as soil samples along a path should be classified the same regardless of the path's direction.\n\nWe believe that experiments on images are sufficient to demonstrate the idea and that the new architectures are a valuable contribution to the literature on vision models.\n\n> Q2 [and earlier comments]: Do the authors think that the FLOPs results that they have presented here for C_2 equivariant models extend to groups with more difficult irreps?\n\nThe paper makes two different points. \n\nThe first point is that the FLOPs-per-parameter ratio is exactly the same for non-equivariant and equivariant linear layers with the same input and output feature dimensions. This only holds when the irreps are one-dimensional, as mentioned in Appendix A. We will go through the main paper and make it clearer there. For instance, as pointed out by Reviewer s6u9, there is a formulation in Section 5 where “equivariant networks can be designed to have the same number of FLOPs…” will be changed to “flopping equivariant networks…”. We do not think we have over-claimed the contribution since the paper's title already clarifies that we mainly consider the flopping symmetry in this work. \n\nThe second point is that hard-coding equivariance results in computational savings, due to block-diagonalization of the linear layers. This argument generalizes straightforwardly to any other groups due to Schur’s lemma. In particular, for groups with more than two irreps, layers mapping between feature spaces transforming according to (multiples of the isotypical decomposition of) the regular representation will decompose into more blocks, resulting in larger computational savings. To rephrase slightly, the FLOPs-per-feature is much reduced in linear layers with hard-coded equivariance, which generalises to all finite groups.\n\nWe think the Reviewer’s suggestion to include a discussion around the generalisation to other groups in the main paper is a good idea, and we will do so.\n\n> Q3: What happens to the analysis provided in the paper if the number of invariant and flopping equivariant features is not equal? \n\nThe block decomposition still works if we do not have an equal amount of invariant and (-1)-equivariant features. The 0-blocks in equation (2) will be smaller and so the computational savings per feature will be smaller. The number of FLOPs per parameter will still be the same.\n\nWe have included an extra experiment in the rebuttal to Reviewer s6u9 (Q1) to test whether having more invariant features is helpful for the classification task.\n\nHaving an equal amount corresponds to having features that transform according to the regular representation of $C_2$, but in a diagonal basis (the isotypical decomposition). Weiler & Cesa (2019) found that the regular representation is generally a good choice. \n\n> I have a big gripe about [...] \"In a nutshell, our case is that equivariant neural networks are simple models that scale well.\" \n\nWe thank the reviewer for pointing this out. Indeed, the intended meaning is not that all equivariant neural networks are simple and scale well. We will revise the sentence: \"In a nutshell, our case is that equivariant neural networks *can be* simple models that scale well.” \n\n> I think that the Patch embedding layer could be described better [...]\n\nConsider the case of 1D-signals: We have a signal [a, b, c, d] and its flopped version is [d, c, b, a]. If we use a symmetric filter [v, v] with stride 2, we get [v(a + b), v(c + d)] and [v(d + c), v(b + a)] respectively. Thus, when the input is flopped, the feature map is flopped too. If we use an antisymmetric filter [w, -w], we get [w(a - b), w(c - d)] and [w(d - c), w(b - a)] respectively. Thus, when the input is flopped, the feature map is flopped and changes sign.\n\nIn the PatchEmbed-layer we use equally many symmetric and antisymmetric filters to obtain invariant and (-1)-equivariant features. As mentioned above, this is a common choice.", "claims_and_evidence": "The claims appear to be well-founded, with well-explained supporting evidence. However, as stated below, I question the extent of the contribution made, given that the analysis that appears only considers equivariance to the group C_2.", "methods_and_evaluation_criteria": "Given the question that the authors posed throughout, I thought that the proposed methods and benchmarks made sense for the problem at hand.", "theoretical_claims": "I checked the correctness of the derivation of the linear layers in Section 3.1, and deemed it to be correct. I also checked the equivariant version of the pointwise activation in (3), and confirmed that it is correct.", "experimental_designs_or_analyses": "I read about the design of all of the experiments, and found them to be sound, with fair comparisons being made. In particular, I liked the explanation behind the authors' decision to consider the three measures that they did, and thought that their argument was well-considered.", "relation_to_broader_scientific_literature": "I found that the authors set their paper well in the context of the broader scientific literature, in particular, with a well-framed related work section. They set their work in the context of geometric deep learning (Bronstein et al. 2021), considered the numerous works behind steerable equivariant ConvNets, and highlighted the differences behind their ViT networks and those used by Kundu and Kondor (2024). I also enjoyed the discussion behind equivariant networks for other input than images, although I note that the group considered in these related works is different from the one under consideration in this paper (there is, however, no issue with this).", "essential_references_not_discussed": "No, I didn't pick up any important ones.", "other_strengths_and_weaknesses": "Strengths: I thought that the paper was well structured and easy to read, and I commend the authors for achieving this. The question of achieving scalable equivariant models that do not dramatically increase computational costs is an important one that needs further study in the equivariant community. The authors clearly demonstrate that the equivariant models that they have considered achieve a comparable number of floating-point operations per parameter to the baseline ordinary neural networks. I commend the authors for providing a clear example where equivariant models can be scaled whilst maintaining their usefulness.\n\nWeaknesses: Despite this, I question the depth of the originality behind these results, given that the paper targets only one type of equivariance whose symmetries, beyond perhaps the data studied (image data), are maybe not so important - this is my biggest concern behind this paper. However, I am open to the authors addressing this point of mine in their rebuttal if they can provide me with convincing evidence to the contrary. Complementary to my previous point, I think that the commentary given in lines 676-689, 1st column, needs to be given instead in the main paper and discussed in much more detail. In particular, I think that the authors need to discuss whether they think that the conclusion that they give in this paper (about the comparability of FLOPs per parameter) would extend to groups with the more \"difficult\" irreps. If what they have presented here is only a special case, then that would, in my view, reduce the significance of the present contribution. Finally, the way that the equivariant linear layers are constructed use an equal number of invariant and flopping equivariant features: as the authors themselves say in lines 181-184, 2nd column, it is not obvious that this is optimal. What happens if they are not equal? Does the entire analysis/construction of the models break down? It seems to me that it might do, but again, I am open to the authors clarifying this for me.", "other_comments_or_suggestions": "I have a big gripe about the sentence given in lines 46-48, 2nd column: \"In a nutshell, our case is that equivariant neural networks are simple models that scale well.\" For me, this claim as stated is far beyond what is presented in this paper: indeed, there are many more equivariant neural networks that exist for far many more groups beyond C_2 in the literature. I would strongly recommend that the authors adjust this sentence in a camera-ready version of this paper should it be accepted.\n\nI think that the Patch embedding layer could be described better - it is not entirely clear from Figure 3 how this works. Could the authors provide a clear example of this procedure, perhaps even in the appendix?\n\nThis is a small point but I would personally move Figures 1 and 2 to the second page - it looks a bit messy on the first page.", "questions_for_authors": "I have provided the questions that I would need answering to be able to adjust my score in the weaknesses section. To repeat them here, they focus on:\n\n1) Could the authors provide other examples where their C_2 equivariant construction might be useful beyond the image data they have experimented with?\n\n2) Do the authors think that the FLOPs results that they have presented here for C_2 equivariant models extend to groups with more difficult irreps? I am concerned, given the way that the authors have presented their work, that the results only work for this group (and this situation) as a special case; however, as stated above, I am open to being told otherwise.\n\n3) What happens to the analysis provided in the paper if the number of invariant and flopping equivariant features is not equal? Can their construction still be used, or not? If not, how likely would one use an equal number of these features?\n\nEDIT: Following the authors' rebuttal, I have decided to raise my score from 3 to 4.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper suggests to modify popular vision models by taking into account their equivariance to flopping (mirror symmetry). Importantly, the number of operations used within the network is reduced by two (in contrast with most equivariant models where typically compute time can be higher than non-equivariant models, though number of parameters is reduced). The reviewers were all happy with this paper and with the rebuttal. I ask the authors to remember to incorporate the improvements requested by the reviewers in the camera-ready version.
Decision: Accept (spotlight poster)