Efficient Model-Agnostic Multi-Group Equivariant Networks
Authors: Razan Baltaji
Sourya Basu
Lav R. Varshney
Venue: Accepted by TMLR ()

Abstract: Constructing model-agnostic group equivariant networks, such as equitune (Basu et al., 2023b) and its generalizations (Kim et al., 2023), can be computationally expensive for large product groups. We address this problem by providing efficient model-agnostic equivariant designs for two related problems: one where the network has multiple inputs each with potentially different groups acting on them, and another where there is a single input but the group acting on it is a large product group. For the first design, we initially consider a linear model and characterize the entire equivariant space that satisfies this constraint. This characterization gives rise to a novel fusion layer between different channels that satisfies an invariance-symmetry (IS) constraint, which we call an IS layer. We then extend this design beyond linear models, similar to equitune, consisting of equivariant and IS layers. We also show that the IS layer is a universal approximator of invariant-symmetric functions. Inspired by the first design, we use the notion of the IS property to design a second efficient model-agnostic equivariant design for large product groups acting on a single input. For the first design, we provide experiments on multi-image classification where each view is transformed independently with transformations such as rotations. We find equivariant models are robust to such transformations and perform competitively otherwise. For the second design, we consider three applications: language compositionality on the SCAN dataset to product groups; fairness in natural language generation from GPT-2 to address intersectionality; and robust zero-shot image classification with CLIP. Overall, our methods are simple and general, competitive with equitune and its variants, while also being computationally more efficient.

Published in Transactions on Machine Learning Research (10/2024)
Efficient Model-Agnostic Multi-Group Equivariant Networks
Razan Baltaji Sourya Basu ; & Lav R. Varshney
Deportment of Electrical and Computer Engineering, Coordinated Science Laboratory University of Miiois Urbona- Champaign (baltaji, sourya, varshney] @illinois.eds
Reviewed on OpenReview: https: // openreview. net/ ptt/formm? id- HCMDtc0ZZV
Abstract
Constructing model-agnostic group equivariant networks, such as equilune (Basu et al. 2023b) and its generalizations (Kim et al. 2023). can be computationally expensive for large product groups. We address this problem by providing efficient model-agnostic equivariant designs for two related problems: one where the network has multiple inputs each with potentially different groups acting them, and another where there is a single input but the group acting it is large product group. For the first design, initially consider linear model and characterize the entire equivariant space that satisfies this constraint. This characterization gives rise to a novel fusion layer between different channels that satisfies an invariance-symmetty (IS) constraint, which we call an IS layer. We then extend this design beyond linear models, similar to equitune, consisting equivariant and IS layers. We also show that the layer a universal approximator of livariant-symmetrii functions. Inspired by the first design, we use the notion of the IS property design a second efficient model- agnostic equivariant design for large product groups acting single input. For the first design, we provide experiments on multi-image classification where each view is transformed independently with transformations such as rotations. We find equivariant models are robust to such transformations and perform competitively otherwise. For the second design, we consider three applications: language compositionality on the SCAN dataset to product groups; fairness natural language generation from GPT-2 address intersectionality; and robust zero-shot image classification with CLIP. Overall, our methods are simple and general, competitive with equitune and its variants, while also being computationally more efficient.
1 Introduction
Equivariance to group transformations is crucial for data-efficient and robust training of large neural net- works. Traditional architectures such as convolutional neural networks (CNNs) [LeCun et al. 1998), Al- phafold (Jumper et al. 202T), and graph neural networks (Gilmer al. 2017 use group equivariance for efficient design. Several works have generalized the design of equivariant networks to general discrete (Cohen & WWeling 2016) and continuous groups (Fizzi al., 2021b). Recently, Puny al. (2021) introduced frame averaging, which makes non-equivariant model equivariant by averaging over appropriate frame or equivariant set. One advantage of this method is that it can be used to finetune pretrained models, leveraging
the benefits of pretrained models and equivariance simultaneously (Basu et al. 2023b Kim et al. 2023).
Methods based on frame-averaging have high computational complexity when the frames are large. And, in general, it is not trivial to find small frames. Hence, several frame everaging-based methods, such as equitune [Basu et. al. 2023b) and its generalizations (Basu et al. 2023a Kim al. 2023), simply use the entire group their frames. As such, these methods attain perfect equivariance at high computational cost for large groups. Similar computational issues arise when network has multiple inputs with each input
FEqual contribution. Code available
http::r-oo-iiii-r-------- NNtwwrks

---

Published in Transactions on Machine Learning Research (10/2024)
X1
M
1.G.
Y
XInv
M Gi
M4S 12.G..z
y-Sym 1.C.
+
Y
21.....
yohm 2.G.
X2
Eq
+
Y
MEg G2
2,G
(a)
(b)
Figure 1: (a) multi-inpul group equivariant nelwork defined in $3.3 where groups G1, G2 act on the inputs X1, X2. Here M.G. denotes layer equivariant. to G, and M Sf.G, G, denote layer invariant-symmetrig to groups G,, Gy- (b) a model equivariant G G2 defined in $3.4 but with only a computational complexity of O(G |G2)). Here X/ denotes that the input features are invariant G and Sym denotes that the output features are symmetric with respect to G.
having an independent group acting on Here, we design efficient methods that can work with large groups or multiple inputs with independent groups. Our methods are applicable for both training from scratch and for equivariant finetuning of protrained modcls.
We first characterize the entire space of linear equivariant. functions with multiple inputs, where all inputs are acted upon by independent groups. The resulting design has invariant-symmetii (IS) fusion layer between channels. We find that the obtained design for lincar models can be casily extended to non-lincau models. We also show this IS layer has ils own universality properties. The universality result not only shows that we are extracting the most out of the design but, as we will see, also helps us extend the formulation of the layer beyond the linear framework.
Inspired by the IS layer, we propose efficient method to construct group-equivariant network for large discrete groups. For product group of the form (Gr x (GN-1 x G.)...), the computational complexity of equituning is ([Gi] ... [Gvl), whereas our method provides the equivariance for the same group in N x (|G]/+..++/My) compute. Here, we measure computational complexity by the number of forward passes of pretrained models required. The advantage comes at a loss expressivity of the constructed network, but we show empirically that our network still leverages the benefits of equivariance and outperforms non- equivariant models, while gaining computational benefit. For applications on large product groups, we emphasize that our goal is not to scale the models to extremely large groups. Rather, we are interested in validating the computational benefits gains the size of the product groups grows while leveraging the performance gains from equivariance.
For our first equivariant design with multiple inputs and groups, we apply it on multi-input image classifi- cation task. Then for our second design with single input and large product group, we apply it on diverse applications, namely, compositional generalization ion in language, intersectional fairness natural language generation, and robust. image classification using CLIP. Our model designs, i.c., linear as well as their ex- tension model-agnostic designs, are given in $3 Details of applications they are used in given Finally, experiments provided in $
2 Background and Related Works
Basics groups and group actions are provided in Appendix A
Group equivariance and invarianc-symmeery A function f x + y is G-equivariant for a group G if f(g)) gx))) for all g E G, X, where the action of G on is written gx and that on f(x)
2

---

Published in Transactions on Machine Learning Research (10/2024)
is written as 9z) for all g E G, E X. We call function f: X H y (G1, Gaa.iimarrianttymmmtriiicc in that order of groups, if [((x) f() for all x E X and 91 E G1; and (x) = 92f(() for all E X, 92 E G2-
Model-agnostic group equivariant networks There has been a recent surge of interest in designing model-agnostic group equivariant network designs, such as equitune (Basu et al. 2023b), probabilistic sym- metrization (imm et al. 2023), X-equilune (Basu et al.) 2023a), and canoniealization (Kaba et al. 2023 These designs are based on the frame-averagin method (Puny et al. 2021). where a (potentially pretrained)
non-oquivariant model is averaged over an equivariant set, called frame. The computational costs of these methods grow proportionally with the size of the frame. Finding small frames for general groups and tasks is not trivial, hence, several previous works such as equitune, probabilistic symmetrizzttion, and X-equit simply use the entire group as the frame. These methods become inefficient for large groups. Canonical- ization uses frame of size exactly one but assumes small auxiliary cquivariant network is given, which might itself require frame-averaging, Canonicalization also assumes known map from the outputs of this auxiliary network to group elements, which is non-trivial for general groups. Moreover, canonicalization does not provide good zero-shot performance as do some special cases of X-equitune. Thus, it is crucial to design efficient avame-averaging techniques for large groups.
Given a pretrained model M X + Y and a group G, equitune produces the equivariant model MG as which makes |G] passes through the model M. Thus, the size of the group grows, so does the complexity of several of these frame-averaging-boee x G.)...) and provide efficient modcl-agnostic oquivariant
Ma = क S
methods. In this work, we consider
product groups of the form (G1 X ...... (GN-1 .....GGN-- network designs for two related problems, as described in $3.1 Our construction has complexity proportional to |G1|+ [G2] + +.... + |Gxl compared to |G1| |G2] .... x Gvl for equitune. We empirically confirm our methods are competitive with equitune and related methods while being computationally inexpensive.
The main contribution of our work is to provide an efficient model agnostic equivariant method that works with pretrained models. The efficiency arises by dividing large groups into small product groups and provid- ing a method symmetrize over the smaller groups that gives cquivariance with respect to the larger group. Since this is an emerging area of research in the equivariance literature, there are very few works in this area. Compared to Basu et al. (2023b) which uses a simple averaging over the entire group obtaín sym- metrization, we simply perform averaging over subgroups when the group can be decomposed as products. Kim et al. 2023);; Mondal et al. (2023): Basu et al. (2022) use weighted averaging over group elements to oblain symmetrization, which is complementary to our work and can be used on top of our work for future
work.
Additional related works Several techniques exist to design group-equivariant networks such as pa-
rameter sharing and convolutions (Cohen & Welling 2016 Ravanbakhsh et al.) 2017
Kondor & Trivedi
2018), computing the basis of equivariant space (Cohen & WWeling 2017 Weiler & Cesa
2019
Finzi et al.
2021b Yang et al. 2023 Fuchs et al. 2020
Thomas et 2018 De Haan et al. 2020
Basu et al. 2022)
ropresentation-based methods (Deng et al
2021 Satorras ct al. 2021). and regularization-base moth-
ods MMoskalev el al.
2023 Finzi et. al. 2021a
Patel & Dold 2022: Arjovsky et al.
2019
Choraria el al.
2023 These methods typically rely on training from scratch, whereas our method also works with pretrained models.
Atzmon et al.
2022 ; Duval et al. (0233 use frame-averaging for shape learning and materials modeling,
respectively. These works focus on designing frames for specific tasks, unlike ours which focuses on a general
efficient design. Maile et al.] (2023) provide mechanisms to construct approximate equivariant networks to
multiple groups, whereas our work focuses on perfect equivariance.
3
Method
3.1 Problem Formulation and Proof of Equivariance
Multiple inputs Let XI...., XN and Yı...., Yy be N inputs and outputs, respectively, to a neural network. Let X; R4 and Y, R Let GI...., Gv be N groups acting X1,...; Xx respectively. That is, Ge
3

---

Published in Transactions on Machine Learning Research (10/2024)
acts on X, independent of the other group actions. We want to construct model M(GI....Gww) such that Yt transforms equivariantly when G; acts on X, A naive construction to attain such an equivariant model would be to construct N separate equivariant models equivariant to the groups GI...., Gv. But such a model would not be very expressive since information does not flow between the ith and jth input channels. We construct efficient and expressive equivariant networks for this problem.
Large product groups Now, we consider a single input X and a large product group G that can be written G (G, x (GN-1 X GN)..... where x denotes the semi-direct product. We assume that G transforms X 9192.... .N-X for 9, E G4, ie., the subgroups 9: act in the same order. Further, note that are assuming left group action of G on X. For constructing G-oquivariant models, we assume the groups act commutatively on the output, whereas for constructing G-invariant models we do not need commutativity. As we will see in $ most experiments covered in previous works such as Basu et al. (2023b a) are covered by these basic assumptions. Naively using equituning on a pretrained model M using G can be expensive. Hence, we aim to design efficient group equivariant models for large product groups.
3.2 Characterization of the Linear Equivariant Space
We first start with the problem of group oquivariance for multiple inputs XI...., XN being acted upon by groups Gi,..., Gn, respectively. Using equituning for this problem would require group averaging over the product group G1 x ... GN, which is expensive because of the size of the product group. Thus, instead of naively applying equitaning, we want first to understand how equivariance to this large group be obtained from a combination of layers equivariant/ intivarranntivvaiannnt 'inaarinnt-symnnnnnn we first characterize the entire space of linear equivariant layers for the product group using linear layers equivariant/ ieriiianniiiiiiiintttt /tt.aniaaatttymmmeeeeee
to smaller groups. To this end,
to the smaller groups.
This simple linear layer characterization can help build equivariant deep neural networks by stacking a number of these layers along with pointwise nonlinearities (with discrete groups) as done by Cohen & Welling 2016). Further, this characterization will give us intuition on how to construct model agnostic equivariant layers similar to equiluning (Basu el al. 2023b) for the concerned group action. We take N = for simplicity, but the obtained results can be easily extended to general N as discussed in Appendix C.1
Let L be a G-oquivariant linear matrix, i.c. Eg (aX) = aLE'( for all a E G. And let G (x) be (G1; Gg)-invariant-symmett linear matrix, i.e., T GGG,G, (ax) = Is G,, (x) = =bL. ,,, for all E G1.
b E G2. Then, we define multi-group equivariant lincar layer as
G, ([X1 X2)) (x,)())) + L a, (X2), (X2) + L G. (X1)).
(1)
where [.] denotes concatenation. In Thm. ! we prove that L,,, X2)) is equivariant to (G1, G2) applied X1 and X2, respectively. More preciscly, for any a a &GG,b b E G2 we show that
LG, (1aXi, bX2)) = ([(()))(X)) + ((s,c,(Xx,)) ((xx)),b(TTE bLL Eq (X2) + L (aXi)))
(2)
Theorem 1. The malti-group equivariant layer La, (xX,,, defined in equation i is equivariant to (G1, G2) applied to X2), respectively.
All proofs are provided in Appendix B Now we show that Lci.o, ([,, characterizes the entire lincar equivariant space under the given equivariant constraint. First, recall from Maron et al. (2020) that the dimension of linear equivariant space for a discrete group G is given by
E(G)
Σ Tr(P(t)) gEG
(3)
G
where G is subgroup of a permutation group and let P(g) is the permutation group element corresponding to E G and Tr(-) denotes the trace of the P(g) matrix. Here, for simplicity, it is assumnd that the linear space is represented by a matrix of same input and output dimensions. Hence P(g) has the same dimensions as the matrix. Now we compute the dimension of the lincar invariant-symmetric space for groups G,, G2 in Lem. D where G2 acts on the input and G2 acts on the output. The proof closely follows the method for computing the dimension of the equivariant space in Maron al. (202
4

---

Published in Transactions on Machine Learning Research (10/2024)
Lemma 1. The dimension of linear invariant-symmecrii space corresponding to groups (G1, G2) is given
by
1 G2) Gl||22
Σ 91E01 92EG2
E Ir(P(g))
(4)
Now, in Thm. 2 we show that LG, GGG,G.(la ([X1, in equation characterizes the entire space of lincar weight matrices that satisfies the equivariant constraint in equation
Theorem 2. The linear equivariant matric L. ([x,, in equation characterizes the entire space of linear weight matrices that satisfies the equivariant constraint in equation
Thus, in Thm. D we first show that the construction in equation is equivariant to the product group (G1, G2). Then, in Thm. 2 we show that the equation in equation characterizes the entire linear space of equivariant networks for the given imput and output dimensions.
It easy to construct the oquivariant and layer, equivariant to G can be obtained as LG [ ল E the same as equiluning (Basu ct al.
layers given some weight matrix L. A lincar
2023b). Similarly, a lincar invariant-symman layer with respect to (G1, G2) can be obtained as G1 (X) G Egec,, 92 (2 L(gi))
3.3 Beyond Linear Equivariant Space
Now show that the linear expression in equation can be easily extended to gen- eral non-linear models.
construct Ma,.c, (c,,c,(X,, X2)) [M (X1) + + 22, ( (2, + M 12.Gr, (X)), that satisfies the equivari- ant constraint in equation 2
That is, given models M M2, M12, M21,
Suppose the output is [Y,, Y2], then M,CCC (X,) goes from X; to Y,, whereas the cross-layer G, (X:) goes from X; Yj- is casy to construct as Bg (X,) = ले. Ener, 91 M,;,;,)) since we know from previous works such as equituning (Basu et al., 2023b) and frame averaging (Puny et al, 2021) that this averaging
leads to a universal approximator of equivariant functions, hence is an expressive equivariant design. The design is of M.j.,, (X,). We define the cross-layer M.S 13GG..G (X.) as
Σ 0, Σ M(g.x,)), M M. (xi) |G,|G,]| 0 aEEG,
(5)
where M is the pre-trained model. One can verify M M.G.co (X,) is invariant-symmetrr with respect to (Gr, Gj). The design of the model Ma ([X X2)) illustrated in Fig. la
Universality Thm. 3 shows that M.S. M. (X,) is universal approximator of invariant-symmetrie functions. Note that there are alternate choices of designs for this layer that are equivariant but do not provide the same universality guarantees, hence, are not as expressive. One such design is M.S. (X,) = |.ा||; Eg 9, 9,),), which is equivalent to M.S.C. (X.) if M a linear layer. Hence, going beyond lincar layers requires additional design choices. Hence, Thm. confirms that our choice of the
layer is expressive.
We use the definition of universality used by Yarotsky 2022 as stated in Def.
Definition 1. A function M X' + y is a universal approrimator of a continuous function f X + y if for any compact set E X, e > 0, there exists a choice of parameters of M such that [lf(z) M e for all K.
JG X + y be ony continuous function that invariant invariant-symmetrii to (G1, G2). Let M X + y be universal approrimator of fis- Here X,Y are such that if x E X.y E y, then 9 92y X2y E y for all 91.92 92 E G1,G2, so that the invariant- symmmerrtt-yymetrc property is well-defind. Then, we claim that MIS is universal opprosimator of fls G..G.. G1
Theorem 3. Let ETS
5

---

Published in Transactions on Machine Learning Research (10/2024)
Computational complexity Assuming M is large model, the bottleneck of computation of M (.. is proportional to the number of forwarded passes done through M. Thus, the computational complexity of M G.). is O[] + (G2]). This is in comparison to equituning that has O(]G1 x [G2l) computational complexity for the same task.
3.4 Equivariant Network for Large Discrete Product Groups
Given a product group of the form G G1 G2, we design the G-equivariant model M G2xC; as () (M G)²²)
Bq xG2 (X) = [[M (x Sym Ea (1 Sym
(6)
where [.] represents the concatenation of two elements, M is any model equivariant to G5. e.g. equizero Basu et. al. 2023a) applied to some pretrained model M for zeroshot equivariant performance. Note that []] can be replaced by other operations such summation that preserves the equivariance of the individual elements being summed. For the rest of the work, we restrict ourselves to summation even though the general formulation is more general. The (-),," and (+)G, Sym operations are inspired from the invariant- symmetric layers obtained in $3.2 Here, X denotes Gy-invariant feature of X, i... (g192Xu)63 Xu)ae Nu. and (9192-Xo) /av = 92 for all 01 G1.92 G2, where Xo the canonical representation of with respect Sym
to G. (Y)),. E.g., (Y)), =
Sym for all ga G., Sym denotes the symmetrization of Y with respect to G., i.e. (Y)), 9Y(Y))
G..| Dg Y:Y is a valid symmetrization of Y.
1
Intuitively, M& xG. works as follows: the first term (M&2 Ey (x)))) Sym captures the Gi-invariant and G2- equivariant features of X and the second term captures the Gz-invariant and Gi-equivariant features of X. Combining the two features gives an output that is equivariant to both G₁ and G2. Note that combining these two features requires the commutativity assumption in $3.1 Discussion on generalizing this design the product of N groups is given in Appendix C.2 We now prove that Ma GGxG, is equivariant. to G1 x G2.
Thcorem 4. M Gixx,,(8 (X) defined in equation is equivariant to G, G2. That is, MG (0192X) Ey GixGs(X)
M1
Computational complexity Note that the computational complexity of equation 6 is O( ([G]]+|G2]] (G2l) when the bottleneck is the forward pass through M, e.g., when M is large pretrained model.
4 Applications
We first look multi-image classification in $4.1 application of the first design. The rest of the applications focus on the second design, where the goal is to design equivariant. networks for large product groups on a single imput from pretrained models. Please note that the semi-direct product between the groups is equivalent to direct product for the experiments based on language generation and compositional generalization because the groups are acting on disjoint sets.
4.1 Multi-Image Classification
Here we consider the multi-image classification problem, where the input consists of multiple images and the output is label, which is invariant to certain transformations, such rotations, made to the input images. We perform experiments using two datasets: Caltech101 (Li et al. 2022 and 15Scene (Fei-Fei & Perona 2005)
We construct our equivariant CNN using the first design in $33 which we call multi-CNNN, and compare its performance to a non-equivariant CNN. Multi-GCNN first passes each image in the input through equiv- ariant convolution followed by densely connected blocks constructed using group averaging like in equitune. Additionally, features from different blocks are fused via the invariant symmetric channels while maintaining necessary equivariance properties. Finally, invariant outputs are taken in the final layer.

---

Published in Transactions on Machine Learning Research (10/2024)
4.2
Compositional Generalization in Languages
Compositionality in natural language processing (Dankers et al.] 2022) is oflen thought to aid linguistic generalization (Baroni 2020). Language models, unlike humans, are poor at compositional generalization, as demonstrated by scveral datasets such as SCAN (Lake & Baroni 2018). SCAN is a command-to-action translation dataset that lests compositional generalization in language models. Previous works (Gordon et al. 2020 Basu et al. 2023b al have considered two splits Add Jump and Around Right that can be solved using group equivariance. But each of these splits only required groups of size two. Hence, we extend the SCAN dataset using the context-free grammar (CFG) of the dataset. We add production rules for up and down taken as an additional dimension to the original dataset. We refer to the extended dataset SCAN-IT, which has splits that require slightly larger groups of sizes up to eight solve the compositional generalization task. More background on the original SCAN dataset, extended version SCAN-II, and approaches to solve it are discussed in Appendix D.1
4.3 Intersectional Faireess in Natural Language Generation
We consider the problem of inherent bias present in natural language generation (NLG), specifically, for GPT-2 Radford et al. (2019). We consider the framework of [Sheng et al. (2019) to test. biases present in language models. We aim to design one single model that reduces social biases amongst each pair of the demographics ["man', "woman"), ["Black', "White"], and ["straight', "gay"] and their intersections. This is important since even though we reduce bias amongst two pairs of demographics such as ["man', "woman'] and [BBlack,, "White"), it does not guarantee fairness for demographics at the intersection such as "Black woman". This is the indersectionality problem of fairness (Ovalle et al.) 2023 Wang et al. 2022) that arises since numerous social dimensions form identity (Akerlof & Kranton 2010
Group-theoretic fairness for the interscctions (2033b) using products of groups. This extends the solution to fairness by Basu et al. (2023b) from using separate groups to attain fairness for each pair of demographics to using one single product group that addresses the intersectionality problem in addition to providing fairness to individual pairs of demographics. Using this product group with multi-cquitume gives a complexity proportional to the sum of group sizes rather than their products, making our method very scalable. For designing our equivariant models, we use the same group actions as Basu et al. (2023b), provided in Appendix D.2 for completeness.
be guaranteed by extending the framework of Basu et al.)
Similar to Basu et al. (023b), we evaluate method using the regard classifier of [Sheng al. (201 A regard classifier is similar to a sentiment classifier but designed specifically for fairness studies by finetuning BERT (Devlin et al. 2019) on a human-curated fairness dataset. [Sheng et al. (2019) provide two different sets of five contexts to generate lext, called the respect task and occupation task. The respect context is concerned with the respect of the demographic in general, e.g., it has context such as "The X was known for', where X is replaced by the name of the demographic. Similarly, the occupation context is concerned with the occupation corresponding to these demographics.
4.4 Robust Image Classification using CLIP
Here we perform robust image classification using pretrained CLIP (Radford et al.) 2021 models for 90 rota- tions and flips. Previous work Basu et al. (2023a) design separate robust models for these transformations. Here, we provide a single model that is robust to both using our multi-group design from (3.ppiie to both equitune and equizero. We consider the Imagenet-Va V2 (Recht et al. 2019 and CIFAR100 (Krizhevsky et al.) image classification datasets. The application of our method from 3.4 to CLIP is pretty straightforward and is described in Appendix D.3
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Mean (standard deviation) test accuracies for multi multi-image classification on the Caltech-101 dataset. N denotes the number of images present as input. Train augmentations corresponding to each of the N inputs are shown as an ordered sequence. Here R. means random 90º rotations and means no transformation. Fusion denotes the use of invariant-symmetric layers.
Model
CNN
Multi-GCNN
Fusion N
Dataset
Train Aug.
0.436 (0.002)
0.451 (0.009)
0.65 (0.006)
0.693 (0.006)
2
RR
0.548 (0.004)
0.587 (0.022)
0.651 (0.003)
0.688 (0.009)
III
0.472 (0.007)
0.5 (0.009)
0.713 (0.019)
0.739 (0.008)
Caltech101
3
RRR
0.62 (0.010)
0.65 (0.008)
0.702 (0.015)
0.73 (0.013)
IIII
0.501 (0.010)
0.529 (0.015)
0.71 (0.014)
0.724 (0.021)
4
RRRR
0.644 (0.006)
0.683 (0.015)
0.72 (0.005)
0.746 (0.01)
5 Experiments and Results
5.1 Multi-Image Classification
Experimental setting We use the Caltech-101 and 15-Scene datasets. For multi-input network with N inputs, we partition the train and test datasets for each label in tuples of N. We add random 90 rotations to the test images, and for training, we report results both with and without the transformations. This tests the efficiency gained from equivariance and the robustness of models, similar to Basu et al. (2023b). For each dataset, we report results on multi-input image classification with N inputs, where N = {2, 2,4. 4}. We call the multi-input equivariant CNN based on the design from $3.3 as multi-GCNNS. Further details on the model design are given in Appendix E. We train each model for 100 epochs and a batch size of 64, an SGD optimizer with learning rate of 0.01, momentum of 0.9, and a weight decay of 0.001.
Results and observations Tab. and show the test accuracies and Caltech-101 and 15Scene datasets, respectively. Clearly, multi-GCNN outperforms CNN across both datasets as well as the number of inputs used. Moreover, we find that the models using the invariant symmetric layer described in $3.3 generally outperform the ones without. This illustrates the benefits of early fusion using the invariant symmetric layers.
5.2 Compositional Generalization in Language
Experimental setting We work the SCAN-II dataset where have train dataset and three different test dataset splits. The train dataset is such that each of the test splits requires equivariance to different product groups. The product groups are made of three smaller each of size two, and the largest product group considered is of size eight. Hence, performance on these splits shows benefits from equivariance to different product groups. Details of the dataset construction are given in Appendix [D.T] We consider the same architectures as Basu et al. (2023b a). i.c., LSTMs, GRUs, and RNNs, each with a single layer with hidden units. Each model was pretrained on the train set for 200k iterations using Adam optimizer (Kingma) & Ba 2015) with learning rate of 10-4 and faccer-forcing ration 0.5 (Williams & Zipser 1989 We test the non-equivariant pretrained models, along with equituned and multi-equituned models, where equitune and multi-equitune use further 10k iterations of training on the train set. For both equitune and multi- equitune, we use the largest product group of size eight for construction. We use the cross-entropy loss as our training objective.
Results and obsorvations Fig. shoow the results of pretrained models, and finctuning results of equitune and multi-qquitune on the various test splits. We find that pretrained models fail miserably on the test sets even with excellent performance the train set, confirming that compositional generalization is not trivial
8

---

Published in Transactions on Machine Learning Research (10/2024)
LSTM
GRU
RNN
1.2
10-
0.2
2.8
Text Accurecc
Text Accuracy
Text Accurecy
0.3
30-
0.4
ылпр: 24mLUp_ 34T.4p_MMii...m t.mup
Junp wwn.ur_juppp.u Jef tam,up
jurp
auw.jp.urp....mmmm t.mLPP Splt
Spit
(a)
(b)
(c)
Figure 2: Multi-Fquittning for SCAN for (a) LSTM (b) GRU (c) RNN Models. Models were finetuned for 10K iterations with relevant groups for each task. Comparisons are done with pretrained and equi-tuned models. Results are over three random soeds.
other
positive
neutral
negative
Gender-Respect
Color-Respect
Respect
68-
ger-
08
65-
64
63
61
NAVE
wWWWA.
11
149 White
MW WWM glack
61
Women
Woman
Man Womes
Black whis
Black
Warhe
MA
GPT2
Equitune
Multi-qqiituee
GPT2
Equitune
Muti-Equitune
OFF
PuT COUTUE
(a)
(b)
(c)
Figure 3: The plots (a), (b), and (c) show the distribution of regard scores for the respect task for the set of demographic groups gender, race, and an intersection of gender, race, and sexual orientation respectively. For GPT2 we observe clear disparity in regard scores amongst different demographic groups. Fach bar the plots correspond to 500 generated samples. Equitune and Multi-Equitune reduces the disparity in the regard scores.
achieve for these models. We note that multi-oquitune performs competitively equitune and clearly outperforms non-equivariant models.
5.3 Intersectional Fairness in NLG
Experimental setting We closely follow the experimental setup of Basu el al. 2023b) and Sheng et al. (019 There are two tasks provided by Sheng et al. (2019): respect task and occupation task. Each task consists of five contexts shown in Tab. 5 For each context and cach model, such as GPT-2, and GPT-2 with equitune (EquiGPT2) or multi-equitune (MullEqqiGPPT), we generate 100 sentences. We use both equitune and multi-equitune with the product group corresponding to the product of the demographics ["man', "woman"], ["Black", "White"], and ("straight", "gay"). Here, focus debiasing for all the demographic pairs with one single model each for equitune and multi-cquitune. Quite directly, it also addresses the problem of intersectionality. These sentences are classified as positive, negative, neutral, or other by the regard classifier Sheng et al. 2019).
Results and observations Fig. and 5 show some results corresponding to the respect task and occu- pation task, respectively, for various demographics and their intersoctions. The rest the plots are provided in Fig. and We find that EquiGPT2 and MultiEquiGPT2 both reduce the bias present across the various demographics and their demographics with one single product group of all the demographic pairs.

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Perplexity Scores for GPT2, EquiGPT2, and MultiEqiiGP12. Equi- and MultiEqu-GPP22 show negligible performance drops on Wikitext-2 and Wikitext-103 test compared GPT2
Dataset
GPT2
EquiGPT2
MultiEqiGPP2
Wikitext-103
28.23
29.29
29.56
Wikitext-2
23.86
24.64
24.88
ImagenetV2 too-1 accurac les
RN101 top-1 accuracies
VIT-8/16 top-1 accuracies
70
70
No Darifommttinn
motra motel
Hesralnod model
03
Ramoom 90" rocaions and Tip.
Equitune
Equiure
Mutlequttnee
Nub-crputunz
60
55-
50
00
50
70o3
40
40-
30
30
20
20 Imagenet
Imagenet
(a)
(b)
(c)
Figure 4: (a) shows that CLIP is not robust to the transformations ions of 90 rotations (rot90) and flips. (b) and (c) show that muitie and multi-qquizero are competitive with equitune and equizero, respectively, for zero-shot classification using RN101 and ViT-B/16 encoders of CLIP for the product of the transformations rot90 and flips, even with much lesser compute.
In Tab. we show the benefits memory obtained from using MultiEquiGPt2 compared to EquiGPT2, which close the difference in the sum and product of the sizes of the smaller groups. Further, in Tab. 2 verify that MultiEquGGPT2 has negligible drop in perplexity on the test sets of WikiText-22 and 103 compared to GPT2 and close to EquiGPT2.
5.4 Robust Image Classification using CLIP
Experimental setting We use the CLIP models with various Resnet and ViT encoders, namely, RN50, RN101, ViT-B/32, and ViT-B/16. We test the robustness of the zero-shot performance of these models on the Imagenet-V2 and CIFAR100 datasets for the combined transformations of rot90 (random 90º rotations) and flips. We make comparisons in performance amongst original CLIP, and equitune, equizero, multi-oquitune, and multi-equizero applied to CLIP.
Results and observations Fig. 4a and Da. show that the CLIP models vulnerable to simple trans-
formations such random rotations and flips as was also observed in Basu et al. (2023a) ). Fig. 4b Fig, 4c Fig. 9b and Fig. [ show the robustness results for RN101, ViT-B/16, RN50, and ViT-B B B/32 respectively. We find that across all models and datasets, multi-oquitune and multi-cquizero perform competitively to eq- uitune and equizero respectively. Moreover, Tab. S we find that multi-equitune take less memory compared equitune as expected from theory. That is, multi-equitune consumes memory approximately proportional |G1| GGl 6, whereas equitune consumes memory proportional |G₁| |G2l 8, where |Gi| for 90P rotations and |G2l 2 for flips.
6 Conclusion
We introduce two efficient podellaggostic multi multi-group equivariant network designs. The first design aims at neural networks with multiple inputs with independent group actions applied them. We first characterize the entire linear equivariant space for this design, which gives rise uwvarian-syymeetn layers as its sub- component. Then generalize this to non-linear layers. We validate its working by testing it multi-input
10

---

Published in Transactions on Machine Learning Research (10/2024)
image classification. Finally, inspired by this invariant-symmetris design, introduce second design for single input with large product groups applied to it. This design is provably much more efficient than naive model agnostic designs. We apply this design to several important applications including compositional gencralization in language, intersectional fairness in NLG, and robust classification using CLIP.
Ethics statement Our fairness algorithm provides intersectional fairness in group-theoretic sense as defined in $D22 It aims to reduce bias in natural language generation. But our algorithm is dependent on equality and neutral sets taken from Basu et al. (2023b ab. which are constructed by people. Hence, these constructions of sets ncod be constructed responsibly deployed for public usc. Our evaluation for lairness based on regard scores computed using the methods of Sheeng et al. (2019). Basu et al. (2023b) show that the regard classifier itself may contain bias. Hence, though the regard classifier acts
great evaluation metric for academic purposes, better evaluation metric needs to be constructed if it is deployed for evaluating sentences in practice.
Reproducibility statement All proofs to theoretical claims are provided in $B Details of dataset constructed for compositional generalization experiments are given in D Detailed experimental settings for each experiment are provided in 5 and
References
George A. Akerlof and Rachel E. Kranton. Identity Economics. Princcton University Press, 2010.
Ekin Akyurek and Jacob Andreas. Lexicon learning for few shot sequence modeling. In Proceedings of the 59th Annual Meeting of the Associalion for Compulational Lingwistics and the 11th International Joint Conference Natural Language Processing, pp. 4934 4946, 2021.
Jacob Andreas. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Associalion for Compulalional Linguislics, pp. 1656-7566, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. MiXiv:1907.02898 2019.
Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, and Yaron Lipman. Frame averaging for equivariant shape space learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 631-641, 2022.
Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks. Philo- sophical Transactions of the Royal Society B, 375(1191122019037, February 2020.
Sourya Basu, Jose Gallego-Posada Francesco Viganò, James Rowbottom, and Taco Cohen. Equivariant mesh attention networks. Transactions on Machine Learning Research, 2022.
Sourya Basu, Pulkit Katdare, Prasanna Sattigeri, Vijil Chenthamarakshan, Katherine Driggs- CaaggelC Payel Das, and Lav R. Varshney. Efficient equivariant transfer learning from pretrained models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 4213- 4224, 2023a.
Sourya Basu, Prasanna Sattigeri, Karthikeyan Natesan Ramamurthy, Vijil Chenthamarakshan, Kush R. Varshney, Lav R. Varshney, and Payel Das. Equi-tuning: Group equivariant fine-tuning of pretrained models. Proceedings of the AAAT Conference on Artificial Intelligence, 2023b.
Moulik Choraria, lbtihal Ferwana, Ankur Mani, and Lav R. Varshney. Learning optimal features via partial invariance. In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the International Conference Machine Learning, pp. 2090-990099 2999, 2016.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Taco S. Cohen and Max Welling. Steerable CNNs. In Proceedings of the International Conference Learning Represenlations, 2017.
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. The paradox of the compositionality of natural language: A neural machine translation case study. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 4154-4175, 2022.
Pim De Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In Proceedings of the International Conference on Learning Represen- lalions, 2020.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J. Guibas. Vector neurons: A general framework for -qquivvaan networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200-12209, 2021.
Jacob Devlin, Ming-Woo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186, 2019.
Alexandre Agm Duval, Victor Schmidt, Alex Hernandez-Garcia, Santiago Miret, Fragkiskos D. Malliaros, Yoshua Bengio, and David Rolnick. FAEnet: Frame averaging equivariant GNN for materials modeling. In Proceedings of the International Conference on Machine Learning, pp. 9013-9033, 2023.
Li Fei-Fei and Pictro Perona. A Bayesian hicrarchical model for learning natural scene categorics. In Proceedings of the 2005 IEEE Conference on Compuler Vision and Pallern Recognilion, pp. 524-5025 -31,,,1, 2005.
Mare Finzi, Gregory Benton, and Andrew G. Wilson. Residual pathway priors for soft equivariance con- straints. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 30037 30049, 2021a.
Marc Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing oquivariant multilayer perceptrons for arbitrary matrix groups. In Proceedings of the International Conference on Machine Learning, pp. 5318-3322 2021b.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, SE(3))ttrnssoommrss 3D cotostranslation equivariant attention networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 1970-1981, 2020.
J Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the International Conference on Machine Learning, pp. 1272, 2017.
Jonathan Gordon, David Lopex-Paz, Marco Baroni, and Diane Bouchacourt. Permutation equivariant models for compositional generalization in language. In Proceedings of the International Conference on Learning Representations, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelorating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning, pp. 448 456, 2015.
Yichen Jiang, Xiang Zhou, and Mohit Bansal. Mutual exclusivity training and primitive augmentation induce compositionality. In Proceedings of the 2022 Conference Empirical Methods in Natural Longuage Processing, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakoo, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7333):583 589, 2021.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Sékou-Oumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua Bengio, and Siamak Ravanbakhsh. Equiv- ariance with learned canonicalization functions. In Proceedings of the International Conference on Machine Learning, pp. 15546 15566, 2023.
Jinwoo Kim, Tien Dat Nguyen, Ayhan Suleymanzade, Hyeokjun An, and Scunghoon Hong. Learning prob- abilistic symmetrization for architecture agnostic equivariance. arXiv:2306.02288 028666 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2015.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. Proceedings of the International Conference on Machine Learning, pp. 2747-2755, 2018.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-100 (Canadian Institute for Advanced Research). URL http://www. cs.
wtt.scss.toooooooo toronto..o...o edw--kriz/iiaaar html
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of eequenco-to-scquene recurrent networks. In Proceedings of the International Conference on Machine Learning, pp. 2873 2882, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11))2278-2333 2324, 1998.
Fei-Fei Li, Marco Andrecto, Aurelio Ranzato, and Pictro Perona. Caltech 101, Apr 2022.
Zhaoyi Li, Ying Wei, and Defu Lian. Learning to substitute spans towards improving compositional gener- alization. arXiv:2306.028400 2023.
Kaitlin Maile, Dennis George Wilson, and Patrick Forré. Equivariance-uware architectural optimization of neural networks. In Proceedings of the International Conference on Learning Representations, 2023.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning sets of symmetric elements. Tn Proceedings of the International Conference on Machine Learning, pp. 6734 6744, 2020.
Arnab Kumar Mondal, Siba Smarak Panigrahi, Sékon-Oumar Kaba, Sai Rajeswar, and Siamak Ravanbakhsh. Equivariant adaptation of large pre-trained models. arXiv:2310.01647, 2023.
Artem Moskalev, Anna Sepliarskaia, Frik J. Bekkers, and Arnold Smeulders. On genuine invariance learning without weight-tying. In Proceedings of the International Conference on Machine Learning, 2023.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann machines. Proceedings of the 27th International Conference on Machine Learning, pp. 807-814, 2010.
Anaclia Ovalle, Arjun Subramonian, Vagrant Gautam, Gilbert Gee, and Kai-Wei Chang. Factoring the ma- trix of domination: critical review and reimagination of intersectionality in AI fairness. arXiv:2303.17555, 2023.
Gaurav Patel and Jose Dolz. Weakly supervised segmentation with cross-modality equivariant constraints. Medical Image Analysis, 77:102374, 2022.
Omri Puny, Matan Atzmon, Edward Smith, Ishan Misra, Adilya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In Proceedings of the International Conference Learning Represendations, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 18))99, 2019.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pp. 8748 8763, 2021.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter-sharing, In Proceedings of the International Conference Machine Learning, pp.
2017.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Proceedings of the International Conference on Machine Learning, pp. 5389-5400, 2019.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In Proceedings of the International Conference Machine Learning, pp. 9323-9332, 2021.
Emily Sheng, Kai- Wei Chang, Prem Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp- 3407-3412, 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15(1):
1958, 2014.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Ri- ley. Tensor field networks: Rotation-and translation-equiiariant neural networks for 3D point clouds. arXiv:1802.06219, 2018.
Angelina Wang, Vikram V. Ramaswany, and Olga Russakovsky. Towards intersectionality in machine learn- ing: Including more identities, handling underrepresentation, and performing evaluation. In Proceedings of the 2022 ACM Conference on Fairness, Accountabilly, and Transparency, pp. 336 336-349, 2022.
Maurice Weiler and Gabriele Cesa. General E(2)-equivariant steerable CNNs. In Proceedings of the 32nd International Conference on Newral Information Processing Systems, 2019.
Ronald J Williams and David Zipser. A learning algorithm for continually rumning fully recurrent neural networks. Neural Compulation, 1(2):270 280, 1989.
Jianke Yang, Robin Walters, Nima Dehmamy, and Rose Yu. Generative adversarial symmetry discovery. In Proceedings of the International Conference on Machine Learning, 2023.
Jingfeng Yang, Le Zhang, and Diyi Yang. SUBS: Subtree substitution for compositional semantic parsing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 169 174, 2022.
Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approzi- mation, 55(1):407-474, 2022.
A Additional Definitions
Groups and group actions A group is set G accompanied by binary operation such that the four axioms of a group are satisfied, which are a) closure: 91 92 E G for every 91,92 E G. b) identity: there exists In (92 Is) and d) inverse: for every E G,
E G such that e.g- g. there exists -1 such that g g- g g =e. e. When clear from context, we write 9i simply as 91
c) associativity: (91 x 92)
A group action of a group on space X, is given as a : G X + X such that a) a(c, = x for all alg1 x) for all 91.92 G, x E X, where e is the identity element of G. When clear from context, we write agg, x) simply as gr.
I E X' and b) a(g,,
14

---

Published in Transactions on Machine Learning Research (10/2024)
B Proofs
Proof to Thm. 1 To prove the equivariance property, we want LG Caa,s, bX2])
[a( (())) + C, (X + T (aXi)) (axx)) for any a E Gr,b G2. Recall from defini- tions of equivariance and invariance-yymmetyy the following equalities.
(aX1) = al (g((): (X1);
(7)
(bX2) bL. Eq (X2):
(8)
G, (X2) = al. G. (X2),
(9)
L G, (Xi) = bL (X1),
(10)
for any a G1, b G2. Here, equation and equation hold by definition of these equivariant layers.
It follows Eu (aXi) + G. (bX2) al ((LE) L. (bX2)), since LG (x1) from equa-
tion and G. (bX2) = aL (bX2) from equation 9 Similarly, it follows L + L,, (aX1) = b(LG. (X2) + LG (aX1)), which concludes the proof.
Proof to Lem. Let be d x matrix and we want find the dimension of the space of matrices L such that the fixed point equation P(gz)= x L x P(g1) = L holds for all 91 G1 and 92 G2, where P(9:) denotes the permutation malrix corresponding to ft- Thus, this fixed point equation. From Maron et al. (2020), the dimension of this null space can be obtained by computing the trace of the projector function onto this space. One can verify the projector here is given by where @ is the Kronecker product. From the properties
want to compute the dimension of the null space of
Tofav,G Ggfmm = |লा| Egieg, Enyeg, of the trace function, we know Tr(P(g:) & P(gz)) Tr(P(g))) x Tr(P(g2)), which concludes the proof. ]
Proof to Thm. g The dimension of the linear layer LG,,G G, ([X1, = E(G1) + E(G2) + IS(G1, G2) + IS( G1), since we have two equivariant layers that have dimensions F(G,) and F(G2), respectively, and two -ymmetrri layers, which have dimensions LS(G,, G2) and LS(G2, G1), respectively. Recall the definitions of E(-) and IS(..) from equation and equation respectively.
Now we compute the dimension of any linear layer satisfying the equivariant, constraint in equation and show it matches the dimension of La G2 X2)). To that end, first note that the projector onto this equivariant space is ললन| Egre,, Eoseg, (P(g)) @ P(zz)) & (P(11) @ P(gz)), where 4 & denote the Kronecker sum and Kronecker product, respectively. Further, we know from Maron et al. (2020) that the dimension of the equivariant space, say E(GI, G2), is given by the trace of the projector onto this space. Thus,
E(GI, G2) = G1
Σ
Σ
Tr((P(gi) @ P(az)) @ (P())) @ P(æ2))
EIEG, BEEG2
Σ
Σ
Tr((P(g:) @ P(g))) x Tr((P(g1) e P(gz))
(11)
GGillG
gEE s2EGs
Σ
Σ
(Tr(P(g))) +Tr(22)))))
(12)
=G
EI mcGGs
Σ
Σ
(Tr(P(g:))+ +Tr(p())))
|G
EI SrEGs
Σ
Σ
|GillG2 EI S3EG2
Σ Tr(P(g)))2 + G2
Σ Tr(P(g)))
Σ Σ (T((P())))
G1
G
GEEG:
BzEG,
G gzEGa
E(( E(G)) + 1S( G2) + IS(G2, Gi),
(13)
15

---

Published in Transactions on Machine Learning Research (10/2024)
where equation I2 holds because the trace of the Kronecker sum of two matrices is the sum of the traces of the two matrices, equation 11 holds because the trace of the Kronecker product of two matrices is the product of the traces of the two matrices. Finally, equation 13 follows from the definitions of E(-) and IS(..).
Thus, we have proved that G, ([X1. X2]) equivariant, hence, lies the space of linear equivariant functions for the constraint in equation 2 Further, LG G2 (2)) has the exact same dimension as the linear equivariant space of equation 2 Hence, La, ([X,, X2]) characterizes the entire linear equivariant space of equation 2
]
Proof to Thm. We know M is a universal approximator of fe Hence, for any K E X, > 0, there exists choice of parameters of M such that [[M(r) o,),o,()))l S for all x K.
Define Ksym EG. 91 which is also compact set. Thus, there exists choice of parameters for M such that [[M(x) - f., (x)]l S e for all a E Ksym.
For the same > 0, Ksym defined above, we now compute || (④) - f (x)] using the definition of M Mo (1) from equation and show that is less than equal to e, concluding the proof. We have M (x) - fdh,ca (x)l
1
Σ 92 Σ
M(gia) - 1/ f,,,,,())l
(14)
Gi|lGa
B2EG2
SIEG,
G1
1
92
M(gia)
G. G2
Σ
92 Σ
f19 (9x)
(15)
9/ S2
SIEG1
g2EGs
9IEG:
Σ
Σ
[MM(gia) /1 (g)x))
(16)
|1
R2EG2 EGE
Σ
Σ e
(17)
Gi Ga
Ga EtE
(18)
where equation 14 follows from the definition of equation equation
15 follows because o.),
(x)
188 Gz) (91z) for all E G1,92 G2, equation 16 follows from the triangle inequality and the assumption
|gael| = for all g2 E G2. Finally, equation 17 follows because [|M(giz) - fd. (g(x)) S for all e KSym
Proof to Thm. 4 We first prove (M (11192x)) /av) Sym
9192(M Eg ((X) ))))))
Sym
We have
((9 (2
Ino Sym )) G1
Eg ( Inn) Sym
lcn ))G,
(19)
(oeM Eq ((X) [nn ))G,
Sym
(20)
Σ hg2M (X Inv)
Eg Cg
(21)
hEG,
gh
Σ
hg2
MygMen Ey
(x
G.
hEG,
91
Σ
gahMBy Ey
G2
(X(:.))
(22)
hEG1
hM Fq
(X Fru
9192
hEG
91 Eq Go
(X Inv Sym
Ci
))
16

---

Published in Transactions on Machine Learning Research (10/2024)
where equation 19 follows from the definition of the invariant operator in $3.4 equation 20 follows from
the Cz-equivariance of M equation 21 follows from the definition of symmetric output in 33.4 Finally, equation 22 follows from the commutativity assumption in $3.1
C General Design for a Product of N Groups
Here we provide extensions of our two designs in 3.3 and 3.4 to a product of N groups in C and CC.2
respectively.
C.1 N- Inpup Group Equivariant Models
We extend the design in $3.3 to N inputs Xi...., Xx with group G, acting independently on X,, respectively, Suppose the outputs are Yi...., YN and given models M,, My processing X; and contributing to Yi, Yj- respectively. Then, the equivariant model using M,s Mys for i, E [I,...,NN} consists of an equivariant and an invariant-symmetric component.
for N = i.c., for imput i, have M, (Xx), which is The oquivariant component remains the invariant-symmetuo components, where the invariant-symmeetie equivariant to G, Additionally, Y; has N-1 component is J.G, G, (X,)) It is trivial to see that Y; is equivariant with respect to G, acting on X; since the equivariant component M, (X₂) and M M (Xgz are all equivariant. Hence, the sum/concatenation of equivariant functions gives an equivariant. function.
C.2 Large Product Group Equivariant Models
Extension to N product groups for the model design in equation 6 is trivial and described next. Given a product group of the form G = (Gi X .... (.. x GN) ... ). we design the G-equivariant model
By
M Gxx.x.. (X) i (E....,
Σ
(M Sym
(23)
where MG; F5q is any model equivariant to G₁, e.g, equizero (Basu al 2023a applied to some pretrained
model M for zcroshot equivariant performance, and GY represents the product of all the smaller groups except GG It is easy to check that M X--- (X) is equivariant to (G1 x x... .G. x ..)...).
The intuition for this design is the same for G = G. G2, i.c., (ME. (X Ine )) Daya, preserves the equivariant GiG
Sym
features with respect to G; and invariant features with respect to the rest of the product, which is finally merged with other cquivariant features by taking features symmetric with respect to G., Here, obviously, the summation over i can be replaced by any other permutation invariant invariant/equien /equivariant functions such as max or concatenation.
Additional Details on Applications
D.1 Compositional Generalization in Language
Original SCAN splits The original SCAN split considered in works related to group equivariance primar- ily dealt with the Add Jump and AheAnd Right splits. The Add Jump split consists of command-action pairs such that the command "jump' never appears in the sentences in the training set except for the word "jump" itself. However, similar verbs such as "walk' or "run" appear in the datasct. But the test set does contain sentences with "jump" in them. Thus, to be able to generalize to the Lest. set, a language model should be able to understand the similarity between the words "jump" and "walk'. Gordon et al. (2020) showed that this can be achieved using group equivariance and that group equivariance can help in composi- tional generalization. Similarly, the Around Right split has train set without the phrase "around right" in any of its sentences, but the phrase is contained in its test set. Moreover, the train set also contains phrases
17

---

Published in Transactions on Machine Learning Research (10/2024)
like "around left', thus, to perform well on the test set, the models must understand the similarity between "left' and "right". Thus, like Add Jump, the Around Right task can also be solved using group equivariance. Note that in both these cases, the groups of interest are size two each. Thus, to better illustrate the benefits of our multi-group equivariant networks and to use group equivariance in more practical compositional gen- eralization lask, we exlend the datasel to larger group of size eight. This new exlended dataset, SCAN-IT, is constructed using similar context-free grammar (CFG) as SCAN. Before discussing the construction of SCAN-II, we review some different methods used in the literature solve SCAN and how they differ from our multi-group approach.
More related works Several works have explored solving the compositional generalization task of SCAN using data augmentation such as Andreas 2020); Yang et al. (2022) ; Jiang et al. (2022) : Akyurek &
Audreas (2021): ; Li et al. (2023). Equivariance, as we know, provides the benefits of augmentations tions while
also providing guarantees of generalization. Hence, several works have also explored group equivariance
perform the compositional generalization task on SCAN such as Gordon et al. (2020); Basu et al. (2023b a).
Here the method of Gordon et al. (2020) only works when trained from scratch, whereas the methods of Basu et al. (2023bla a) work with pretrained models but use a frame equal to the size of the entire group, Hence, group equivariant methods for finetuning pretrained models for compositional generalization have been restricted to small groups. We use our efficient multi-equitune design with larger groups to achiove competitive performance to equitune in terms of compositional generalization on our new splits of SCAN, while being computationally efficient.
SCAN-II splits Tab. B and Tab. 4 show the context-free grammar and commands-to-aation conversions for SCAN-II. Note "turn up" and "turn down' are new commands added to SCAN-II useful for testing compositionality to larger product groups. In SCAN-IT, we have a single train dataset and four splits of test datasets: jump, turn Jeft, turn_up, and turn_ up je__jjm__turr tern_left. Here, jump, turn lert_ and turn_up require cquivariance to the pair of commands ["jump", "walk"], [" "down"), and ["left', "right"], respectively, along with equivariance in the corresponding actions to perform well on the test sets. turn up_ jump turn left requires equivariance to the product of the groups required for the other test sets.
Table 3: Phrase-structure grammar generating generating SCAN-II commands. The indexing notation allows
infixing: D[i] is to be read as the ith element directly dominated by category D
C S and S
V +
D + +turn up
C SS after S
V +U
D turn down
C S
D +U left
U - walk
+ V twice
D +U right
look
S V thrice
D +U up
U -rrnn
S V
D +U down
U +jump
V DD[1] opposite [[2]
D -turn left
V DD[1 around D[2]
D -turn right
D.2 Intersectional Fairness in NLG
(2023b) used with language models Here we define the group-theoretic fairness framework of Basu et al. (LM) such as GPT2. Then, we discuss how the framework changes upon extension product groups. First, for each list of demographic groups, we define set of list of words & called the equality words set, and set of words A called the neutral words sets. The equality set È represents the words corresponding to cach demographic, c.g., for the list of demographic groups ["man", "woman'], the equality words set can be [[*man", " "woman"], ["boy", "girl"), ["king', "queen']] The neutral set W represents the words that are neutral with respect to any demographic, e.g. ["doctor", "nurse", "student"]. Given a vocabulary of the LM, the words are partitioned between & and A in this setting. There is a more general setting called relaxed-equitune in Basu et al. (2023b) where the words in the vocabulary are distributed into three sets E, Nr and G. Here, & and N are defined the same as in equitune, but consists of all the words that do not
18

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Double brackets [[ denote the function translating SCAN-1I linguistic commands into sequences of actions. Symbols x and denote variables limited to the set fwalk, look, run, jump). The lincar order of actions reflects their temporal sequence
[walk]] WALK
[u opposite left]] [turn opposite left]] [u]
[lokk] LOOK
[u opposite right] [turn opposite right] [u]
[run] RUN
Iu opposite up] [[tunn opposite upl [u]]
[jump] = JUMP
[u opposite down] [turn opposite down] [u]]
[turn left]] LTURN
[turn around left] = LTURN LTURN LTURN LTURN
[turn right] = RTURN
[turn around right] = RTURN RTURN RTURN RTURN
[turn up] = UTURN
[turn around up] = UTURN UTURN UTURN UTURN
[turn down] DTURN
[turn around down] = DTURN DTURN DTURN DTURN
[u left] = LTURN [u]
[u around left] LTURN [u]] LTURN [u] LTURN [u] LTURN [u]
[u right] RTURN [u]]
[u around right] RTURN [u] RTURN [u] RTURN [u] RTURN [u]
[u upl] = UTURN [u]]
[u around up] UTURNN [u] UTURN [u] UTURN [u] UTURN [u]
[u down] = DTURN [u]
[u around down] = DTURN [u] DTURN [u] DTURN [u] DTURN [u]
[turn opposite left] = LTURN LTURN
[x twice] = [x] [x]
[turn opposite right] = RTURN RTURN
[x thrice] = [x]] [x]] [x]
[turn opposite up] UTURN UTURN
[x1 and x2] = [x1] [x2]
[turn opposite down] DTURN DTURN
[x] after x2] [x=] [x]]
Table 5: The contexts corresponding to the respect task and occupation task by Sheng et al. 2019 Here X is replaced by some specific demographic, e.g., "man", "woman', " "Black man", etc.
Respect task
Occupation task
The X was known for
The X worked as
The X was described as
The X had a job as
The X was regarded as
The X had part-time job as
The X was thought of
The X earned money by
The X was well-known for
The X started working as
obviously belong to either & or N. In this work we focus on equitune since all the methods developed for large product groups here trivially carry over to the implementation of relaxed-equiitne.
Now we review the group actions equitune for a single list of demographics of length d, such as ["man", "woman"] has length d 2. Given a cyclic group of length d, G {e, "e...g..... ,..,,11),, , it acts on the vocabulary as follows. The group action of a cyclic group is completely defined by the group action of its generator, in this case, the element g G simply makes cyclic shift of size one in the equality set & and leaves the neutral set N invariant. For example, if G = (e.g) and & = [["man", "womnn')), ["boy", "girl"), ["king", "queen"]], then gË = [["man", "woman'], ["bay", "girl"), ["king', "queen']]
Previous works such as equitune and X-equitune have only focused on debiasing one list of demographic groups, but debiasing demographics at the intersection remains to be addressed. For example, debiasing the marginal demographics ['man', "woman"] and ["Black', "Whitc"] does not guarantec debiasing for demo- graphies at the intersection such "Black woman". Debiasing at the intersection is possible if we provide equivariance to product groups corresponding to the two lists of demographics. Thus, using multi-equitune, we aim to provide debiasing corresponding to the product group, but using significantly lesser compute compared to an implementation for the same product group using equitune.
The implementation multi-equitune here is very simple since all the group actions are on the vocabu- lary space and are disjoint. That is, the first step of canonicalization can be performed independently for each group, which are then passed through respective equivariant architectures. Finally, the outputs are symmetrized on disjoint output. vocabulary before they are averaged.
19

---

Published in Transactions on Machine Learning Research (10/2024)
D.3 Robust Image Classification using CLIP
As mentioned in $3.4 for two groups G₁, G2, the multi-group architecture is given by MGNGG, NG2 (X) =
M
Sym + (M Eg (X ( Sym (x )).
Ga Na, Suppose G1 is the group of 90º rotations and G2 is the group of flips. Then, for a given image X first, we compute X for i E (1.2), which is computed by appropriately canonicalizing X with respect to G, using the technique of Kaba et al. (2023). Kaba et al. (2023) also requires a small auxiliary network equivariant to G2 which is constructed by equituning a small randomly initialized matrix. MEJ is constructed by directly using the equilune transform (without any fineluning) on the vision encoder of CLIP. Further, since we just need invariant features from CLIP, we simply obtain invariant features from the output of M by pooling along the orbit of G,. Morcover, since the features obtained are invariant, the 0&tm operator leaves the output unchanged. Finally, for equitune, we simply average the outputs from (XLee) Sym
G.
Whereas for equizero there are two minor modifications to the method described above: a) Meg is obtained by applying the equizero transform instead, i.e., a max is taken over the outputs with respect to the inner product with CLIP text embeddings, b) another max is taken over outputs (M to the inner product with CLIP text embeddings.
(M (xer.) with respect
E
Additional Details on Experimental Settings
E.1
Multi-Image Classification
The 15Scene dataset contains a wide range of scene enviromments of 13 categories. Each category includes 200 to 400 images with an average size of 300 250 pixels. Similarly, Caltech101 contains pictures of objects from 101 categories. Each category includes 40 to 800 images of 300 x 200 pixels. We use the cross-entroopy loss as our optimization objective for training our models.
The multi-GCNN consists of three components: an equivariant Siamese block, an invariant-sy fusion block, and finally a linear block. The Siamese block is a Siamese network made of two convolutional layers, each with kernel size 5, and channel dimension 16. Each convolution is followed by a ReLU (Nair & Hinton 2010), max pool, and a batch norm [loffe & Szegedy] 2015). It is followed by fully connected layer with a hidden size computed by flattening the output of the convolutional layers and output size 64. The output is passed through ReLU, dropout (Srivastava et al.] 2014). and batch norm. Finally, this block is made cquivariant using the equitune transform (Basu et al.) 2023b). The N inputs are passed through this Siamese layer parallelly. The fusion block is built identically to the Siamese block, excepl, we make it invariant instead of equivariant. The fusion block also takes the inputs parallelly. Fusion is performed by adding the output of the fusion layer corresponding to input i multiplied by a lcarnable weight to all the features corresponding to the other inputs. Following this, we perform invariant pooling and pass it through the linear block to get the final output. The linear block consists of two densely connected layers with a hidden size of 64. Further, The non-equivariant CNN is constructed exactly as the multi-GCNN network except that no equituning operation is performed anywhere for equivariance or invariance.
use ReLU and dropout between the two densely connected layers.
All multi-image classification experiments were done on a single Nvidia A100 GPU with S0GB memory in a compute cluster. The total GPU hours required for these experiments is less than 24 hours.
F
Additional Results
This section gives some additional results that referred to in the main text.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Mean (and standard deviation) test accuracies for multi-image classification on the 15-Scene dataset. N denoles the number of images present as input. Train augnentations corresponding to each of the N inputs are shown as an ordered sequence. Here R. means random 90" rotations and means no transformation. Fusion denotes the use of invariant-symmotric layers.
Model
CNN
Multi-GCNN
Fusion
X
checkmark
X
checkmark
Dataset
Train Aug.
0.432 (0.017)
0.434 (0.018)
0.679 (0.044)
0.712 (0.08)
2
RR
0.606 (0.026)
0.593 (0.014)
0.688 (0.012)
0.705 (0.016)
III
0.467 (0.01)
0.479 (0.007)
0.694 (0.038)
0.768 (0.02)
15-Scene
3
RRR
0.612 (0.028)
0.632 (0.05)
0.74 (0.014)
0.775 (0.028)
IIII
0.488 (0.012)
0.479 (0.028)
0.777 (0.021)
0.765 (0.032)
4
RRRR
0.661 (0.012)
0.71 (0.046)
0.762 (0.019)
0.765 (0.008)
Table 7: Memory consumplion between equitune and multi-qquitune and GPT2 for a product group of the form G1 x G2 x G3, where |G,] = for {13 Note that ideally, equitune would consume memory proportional
|G1] [G2] |Gy] = and multi-oquitune would consume memory proportional [G1 + [G2l |Gyl 6. Our results show slightly more memory consumed by equitune compared multi-qquitune expected for these groups. We batch size of for the following measurements.
Model
GPT2
Equitune
Multi-Equitune
Memory Consumption (MiB)
1875
2753
2345
positive
neutral
negative
Gecuer-Occupation
Color-Occupation
Occupation
09
68
66
62-
02
61
01-
Woman
Woman
Wornen
glack white
glack white
glack write
GPT2
Equitune
Equiune
Equitune
Equitune
(a)
(b)
(c)
Figure 5: The plots (a), (b), and (c) show the distribution of regard scores for the occupation task for the set of demographic groups gender, race, and an intersection of gender, race, and sexual orientation respectively. For GPT2 we observe clear disparity in regard scores amongst different demographic groups. Each bar the plots correspond 500 generated samples. Equitune and Multi-Equituni reduces the disparity in the regard scores.
Table 8: Memory consumption (in MiB) between equitune and multi-equitume for the group of random 902 rotations and flips. Here the product group is of the form G1 x G2, where G 4, 2. Note that ideally, equitune would consume memory proportional to |G1] x |G2| 8 and multi-equilune would consume memory proportional to [G1l + [G2l = 6. Our results show slightly more memory consumed by equitune compared to multi-oquitune as expected for these groups. We use a batch size of 32 for the following measurements.
Method Dataset
RN50
RN101
ViT-B/32
VIT-B/16
Equitune
5161
5199
2853
4633
Multi-Equitune
4389
4425
2663
4023
21

---

Published in Transactions on Machine Learning Research (10/2024)
negalive
texpality-Respect
Texuality-Occupatioo
10
1.0
0.
0.9
0.8-
0.8
Regard Classiter Score
0.7
Regard Clarsifer Score
0.6
01
0.0
Gay
Straight
Gay
Straight
Gay
Straight
Gay
Straight
Gay
Straight
Gay
Straight
GPT2
Equitune
Multi-Equitune
GPT2
Equitune
Multi-Equitune
(a)
(b)
Figure 6: The plots (a) and (b) show the distribution of regard scores for the respect task and the occupation task respectively. For GPT2 we observe clear disparity in regard scores amongst diffcrent demographic groups. Each bar in the plots correspond to 500 generated samples. Equitune and Multi- Fulti--quitune reduces the disparity in the regard scores.
other
positive
neutral
negative
Respect
Respect
Respect
10
10
08-
3 0d
60
28
00
MMY
N0
196707
Worh
Warnon
sprght
0P1
092
Couture
(a)
(b)
(c)
Figure The plots (a), (b), and (c) show the distribution of regard scores for the respect task for three different intersectional demographics of gender, race, and the intersection of gender, race, and sexual orien- tation. For GPT2 we observe clear disparity in regard scores amongst different demographic groups. Fach bar in the plots correspond to 500 generated samples. Equitune and Multi-Equitune reduces the disparity in the regard scores.
22

---

Published in Transactions on Machine Learning Research (10/2024)
Occupation
Occupation
Occupation
69
68
07
64
04-
02-
1309
Cack
1PAT
Gr
Kart
Rach
Srrgte
OM
(a)
(b)
(c)
Figure The plots (a), (b), and (c) show the distribution of regard scores for the occupation task for three different intersectional demographics gender, race, and the intersection gender, race, and sexual orientation. For GPT2 we observe clear disparity in regard scores amongst different demographic groups. Each bar in the plots correspond to 500 generated samples. Equitune and Multi Multi-Fquituue reduces the disparity in the regard scores.
CIFAR100
RN30 accuracies
VIT-3/2 accuracies
70
Randon ratelans
trrrffrmmttion
55
Frotral mocel Equituns
Pretrolned model
Equituee
Math-equhune
Milh--gultin
60
45
PM nutlouuuzeer
Eavcaro
60
Equinero
NuBl-20/200
35
Top
25
30
30
20-
RN500
RN101
VIT-W32
VIT-3/16
Imagenet V
CIFAR100
Imagenat V2
CIFARLOO
(a)
(b)
Figure 9: (a) shows that CLIP is not. robust to the transformat ions of 90" rotations (rol90) and flips. (b) and (c) show that multi-equitune and multi-equizero zero-shot classification using RN50 and ViT-B/32 encoders of CLIP for the product of the transformations rot90 and flips, even with much lesser compule.
competitive with equitune and equizero, respectively, for
23

---

Published in Transactions on Machine Learning Research (10/2024)
Memory size
Accuracy Group size
Multi Equde Namary
Hemery
Memory (in GB)
Accuracy
30
Equiaro Acsurass
Eqquzero
Pretratned Model Accurscy
10
200
100
150
200
Group size
Group size
(a)
(b)
Figure 10: Plots comparing the performance of equizero and multi-oquizero on the Imagenct dataset with random rotations using pretrained CLIP models. (a) Shows that the memory required for equizero increases linearly with increase in group size, whereas for multi-equizero, the memory required is proportional to the sum of the sizes of the smaller groups that comprise the larger group. (b) Shows that multi-equizero, just as equizero, is able to benefit from equivariance and significantly outperforms the non-equivariant model.
G Efficiency Vs. Performance Trade-Off
Here, we discuss the trade-off between efficiency and performance between equit and our multi-equitune algorithm in equation B That is, for a product group of the form G1 x G2, we provide better intuition how
we reduce the computational complexity from O |G2l) to O( |+ |G2)). At the same time, we explain how exactly we get some drop in performance of the network with benefits computational complexity.
For simplicity, here the formulation for multi-equitune for product group of the form of Gi x G2 M MGxG, (X)
focus the invariance
with commutative group actions here. Recall
(Me)(X (X (x Sem + +MME, (x (X ))
Sym For the invariance case, the expression simply becomes Mfay
NG, (X) =
G (X!ne) + M ( From Sec. 3.4 recall that X denotes G--invariant feature of X. Thus, one way of writing XIve X [G Eg.ecr J.X. Using this definition of XTav we have
Mau G,NG, (X)
1
Σ
M Σ 9192X) +
Σ M( Σ X1 9IEG.
(24)
G G2
G1 G2
B2EG2
S.EG,
g1EG2
Now, if M is linear, we can write equation 24 as
Mfae Gi (X)
Σ Σ M(9192X) BIEG1 92EG2
(25)
G |
First note that equation 24 has a computational complexity of O( + |G2l), and that of equation 24 is
O(G,, x |G2)), where computational complexity here refers to the number forward passes of the model M. On the other hand, equation 25 the exact expression for equitune operation for the product group G1 G2, when M generalized to general functions. Further, we know that equitune is universal approximator of equivariant functions Basu et al. (2023b). However, even though the Envariantssymmettii layer in equation 5 is universal approximators of invariant-symmetti functions, multi-oquitune is not universal approximator of equivariant functions. Thus, even though equation 24 and equation 25 are exactly identical when M is linear, they provide different expressivity when M is not linear, which is the general case we consider.
Finally, we emphasize that this drop expressivity of equation 24 negligible when M itself a large pretrained model as seen in Fig. and Moreover, equation 24 provides computational benefits over equation 25 for product groups.
24

---

Published in Transactions on Machine Learning Research (10/2024)
show that multi-equizero performs much better than the non-equivariant pretrained CLIP model on the Imagenet dataset with random rotation transformations. Further, we also note that the memory required for equizero increases linearly with an increase in group size. In contrast, for multi-equizero, the memory required is proportional to the sum of the sizes of the smaller groups that comprise the larger group. For our experiments here, we used the product groups c2 c2 Cg2 02 x c, Cg x cs, 0g x cg x cg2 and 02 C3 C5 x C7, where Cp the cyclic group of size n.
In Fig. 10,
25


Reviews:
[{"review_id": "zeDMDPnaEt", "reviewer": "Reviewer_5Wqe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- S1. The proposed approach is technically sound in terms of equivariance and expressive power, and is original, as far as I can confirm. The extension from equivariant linear maps [1, 2] to the proposed approach is interesting.\n- S2. The experiments verify that the proposed approach works, i.e. achieves equivariance while inheriting the benefits of pretrained models, in several problems involving product groups. The intersectional fairness experiment is practically interesting and could be appealing to the audience in the NLP domain as well.\n- S3. The ablation results on the invariant-symmetric component (Tables 1 and 6) are informative and adds support to the proposed approach.\n\nWeaknesses\n\n- W1. The algorithm given in Section 3.4 requires obtaining the canonical representation $X_0$ of the input $X$ with respect to the whole group $G$. Given this, the motivation to use the proposed approach is unclear since we can achieve equivariance to $G$ easily and efficiently by computing $g$ such that $X_0 =g\\cdot X$ and compute $f_\\theta'(x)\\coloneqq g^{-1}f_\\theta(X_0)$ following [3]. The authors argue in Section 2 that canonicalization does not provide good zero-shot performance, but the claim is not consistent with the literature [4] and not supported within the experimental results. Similarly, the authors claim in Section 1 that \"Rather, we are interested in validating the computational benefits gains as the size of the product groups grows while leveraging the performance gains from equivariance.\" overall, this applies identically to canonicalization as it uses a single forward pass [3, 4]; yet, I was not able to clearly tell the advantages of the proposed approach compared to canonicalization from the main text or experimental results.\n- W2. In order for the algorithms presented to be useful in practice, there should be practical problems involving large discrete groups described as a product of a large number of very small component groups. Examples of such practical applications are not clear from the main text; some examples I can think of are processing a large number of independent images jointly, but the relevant experiment in Section 5.1 only experiments with up to 4 images and it is questionable which particular applications the proposed approach might find uses. Another example I can think of is processing a lot of small tabular data.\n- W3. For the methods in Section 3.3 and Section 3.4, shouldn't the computational complexity be $O(N(|G_1|+\\cdots+|G_N|))$ since the invariant symmetric layers are computed for all pairs of component groups $G_i$, $G_j$? Also, while the paper makes use of the term \"computational complexity\", I think it is more accurate to use the term \"the number of model forward passes\".\n- W4. In the end of Section 3.2, the authors state without proof that symmetrized linear layers span the full space of linear equivariant layers. While this seems intuitively correct, a formal proof or reference would be necessary for completeness.\n\nMinor comments\n- In Section 3.1 (page 3), there is a typo: $G_N, ..., G_N$ -> $G_1, ..., G_N$\n\n[1] Maron et al. On learning sets of symmetric elements (2020)\n\n[2] Maron et al. Invariant and equivariant graph networks (2019)\n\n[3] Kaba et al. Equivariance with learned canonicalization functions (2023)\n\n[4] Mondal et al. Equivariant adaptation of large pretrained models (2023)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the group averaging approach for turning a non-equivariant (pretrained) neural network $f_\\theta:X\\to Y$ to be equivariant to a symmetry group $G$ by $f_\\theta'(x)\\coloneqq\\frac{1}{|G|}\\sum_{g\\in G}g^{-1}\\cdot f_\\theta(g\\cdot x)$. While group averaging requires $|G|$ forward passes of the given network, this paper proposes a method to scale it to large discrete groups $G$ expressed by a (semi)direct product of smaller groups $G_1, ..., G_N$ which is argued to require $|G_1|+\\cdots+|G_N|$ forward passes. In Section 3.2, the authors first consider the linear case, when each component group $G_i$ acts exclusively and independently on a subset of data dimensions. By extending the approach of [1, 2] to the considered problem setup, the authors show that any linear equivariant layer to such a product group can be decomposed into a combination of: (1) linear equivariant layers for each component group, and (2) linear \"invariant-symmetric\" layers for each pair of the component groups. The authors then state that each linear equivariant or invariant-symmetric layer can be written as a group averaging of an arbitrary linear layer, and then switch the (arbitrary) linear layer into an arbitrary non-linear neural network, which gives the non-linear approach in Section 3.3 which is compatible with pretrained neural networks. This approach inherits the forward pass structure of the linear case, and consequently is argued to require only $|G_1|+\\cdots+|G_N|$ forward passes of the (pretrained) neural network. Then, in Section 3.4, the authors relax the assumption that each component group $G_i$ acts exclusively and independently on a subset of data dimensions (at the cost of requiring canonicalization of input and increased complexity). The authors demonstrate the proposed approach on multi-image classification (corresponding to the method given in Section 3.3), compositional generalization and intersectional fairness in languages (corresponding to Section 3.4), and making CLIP robust to 90-degree rotations and flips, and show that \"early fusion\" [1] performed by invariant symmetric components is beneficial for performance.\n\n[1] Maron et al. On learning sets of symmetric elements (2020)\n\n[2] Maron et al. Invariant and equivariant graph networks (2019)", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the weaknesses, I believe the modifications that answer the following questions would be critical in strengthening the paper:\n\n- (from W1) Why should one favor the proposed approach compared to canonicalization, while it is simple and also can address the problem of efficiently performing symmetrization of a pretrained model? How well does/would the proposed approach perform compared to canonicalization?\n\n- (from W2) What would be the practical problems that involve large discrete groups and could be efficiently solved by the proposed approach (while other approaches such as equitune fail)? How well does/would the proposed approach perform on these problems?\n\n- (from W3-4) Clarifications on the theoretical results and arguments related to W3-4.", "broader_impact_concerns": "I have not found particular concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7Yh4UkRyVQ", "reviewer": "Reviewer_xngP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThis paper is mostly well written, and I enjoyed reading it and learning about the topic. The intersectional fairness application is great. \n\n\n\nWeaknesses: \n\nThe method is motivated by operating on large product groups, but then when it comes time to do this, the paper backs off. The experiments do not contain any large product -- it seems something like <4 groups and each group has <4 transformations. The paper does clarify that \"we emphasize that our goal is not to scale the models to extremely large groups. Rather, we are interested in validating the computational benefits gains as the size of the product groups grows\", but my expectation when I read this is to see some kind of plot showing that as the number of transformation groups increases, computational complexity stays tractable, and some advantage grows over prior methods. This does not seem to be present.  \n\n\nThe applications that don't have to do with intersectional fairness seem contrived and unhelpful. In particular, I don't know what to do with \"multi-image classification where each view is transformed independently with transformations such as rotations\" in Caltech-101.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for converting pre-trained models into ones that are equivariant to certain groups of transformations, for the special case where the groups are products of subgroups. The paper considers two such cases: in the first, the model takes multiple inputs, and the model should be equivariant to some group of transformations for each input; in the second, the model takes a single input but the space is such that there exists some outer product of transformations that must be accounted for.  The key idea lies in implementing a layer which is invariant-symmetric with respect to two groups, and combining this with a group-equivariant layer, to create a layer which is multi-group-equivariant. The result is a model which can naturally handle products of equivariances, which is useful in handling things like intersectional fairness (e.g., equivariant wrt race AND gender simultaneously).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Is it feasible to measure and demonstrate some computational advantages over equitune? The equitune paper had some computational complexity experiments, comparing to finetuning (Table 3 in their supp). I imagine something like that would be great in this paper. \n\nI would have liked to see some optimization objectives, more like equitune. (Even if they are the same as the ones in equitune.) \n\nI recommend avoiding sentences like \"The resulting design has an interesting invariant-symmetric (IS) fusion layer between channels.\" It does not look good to call your own method interesting.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "b2jO4aaHQn", "reviewer": "Reviewer_qafY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- the paper raises an important topic and I think it would be of interest to the community\n- the idea seems interesting and the theoretical justification seems convincing as I read it\nWeaknesses:\n- the structure should be improved to reflect the differences between the proposed model and equitune, in a way outlined  in Point 2 below\n- reproducibility: the model, as I detail below in the requested changes, is not reproducible as it is presented. It does not describe the hardware and software for the experimental setup\n- It would improve the paper if it were to explicitly discuss the limitations of the proposed models\n- correctness: to back up claims about the performance, the experimental section needs to contain confidence intervals", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses the problem of design of model-agnostic equivariant networks.\nThe authors propose two models: the multi-input invariance-symmetry (IS) layer and an architecture inspired by equitune; the second model builds upon the first one and targets large product groups on single input.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper can be of interest to the community but needs to improve upon the evidence. \n1. “N separate equivariant models equivariant to the groups GN,...,GN” G1, …, GN?\n2. Section 3.2 jumps straight into Equation 1. It would be better to motivate it by contrasting it with the original equitune model and what motivates this particular formulation. It might bring more clarity if it were shaped as a comparison with the original equitune model. \n4. In Table 1, Table 2 and Table 6, is is possible to show the confidence intervals? It’s not clear whether the performance difference is any significant. \n5. *“The main contribution of our work is to provide an efficient model agnostic equivariant method that works with pretrained models. The efficiency arises by dividing large groups into small product groups and providing a method to symmetrize over the smaller groups that gives equivariance with respect to the larger group.”* I understand that the efficiency is specified in terms of computational complexity in Section 3.4; however it would be good to put the absolute numbers to see how much wall clock time this operation takes, at least or some experiments for the first and the second model\n6. In Section 5.3, it is important that the authors clearly define the set up of the model in terms of reproducibility. It is also unclear whether the results of debiasing are statistically significant, and how do they compare to the work by Basu et al. (2023b) and Sheng et al. (2019) as well as other debiasing baselines. \n7. Reproducibility: while the architecture of the network is specified in the appendix E1, the training procedure and the hardware used are not clear.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a scalable method for making non-equivarient neural networks equivariant to symmetry group transformations, particularly for product groups. The method extends from linear to non-linear networks, ensuring compatibility with pre-trained models while maintaining efficiency. Its effectiveness is demonstrated in various tasks like multi-image classification, intersectional fairness, and robustness to rotations and flips.

The reviewers generally agree that the proposed approach and experiments, such as intersectional fairness, are of interest to the research community. The theoretical justification provided is convincing and well-supported. The authors have adequately addressed many of the reviewers' concerns in their rebuttal, including clarifications on advantages over related work, theoretical results, and reproducibility. As a result, all reviewers and AC recommend acceptance.
Decision: Accept (poster)