Efficient Equivariant Transfer Learning from Pretrained Models
Authors: Sourya Basu
Pulkit Katdare
Prasanna Sattigeri
Vijil Chenthamarakshan
Katherine Rose Driggs-Campbell
Payel Das
Lav R. Varshney
Venue: NeurIPS 2023 poster (2023)
Keywords: zero-shot learning
equivariant machine learning
equivariant fine-tuning
pretrained models
TL;DR: We present efficient equivariant transfer learning algorithms for non-equivariant pretrained models.

Abstract: Efficient transfer learning algorithms are key to the success of foundation models on diverse downstream tasks even with limited data. Recent works of Basu et al. (2023) and Kaba et al. (2022) propose group averaging (equitune) and optimization-based methods, respectively, over features from group-transformed inputs to obtain equivariant outputs from non-equivariant neural networks. While Kaba et al. (2022) are only concerned with training from scratch, we find that equitune performs poorly on equivariant zero-shot tasks despite good finetuning results. We hypothesize that this is because pretrained models provide better quality features for certain transformations than others and simply averaging them is deleterious. Hence, we propose λ-equitune that averages the features using importance weights, λs. These weights are learned directly from the data using a small neural network, leading to excellent zero-shot and finetuned results that outperform equitune. Further, we prove that λ-equitune is equivariant and a universal approximator of equivariant functions. Additionally, we show that the method of Kaba et al. (2022) used with appropriate loss functions, which we call equizero, also gives excellent zero-shot and finetuned performance. Both equitune and equizero are special cases of λ- equitune. To show the simplicity and generality of our method, we validate on a wide range of diverse applications and models such as 1) image classification using CLIP, 2) deep Q-learning, 3) fairness in natural language generation (NLG), 4) compositional generalization in languages, and 5) image classification using pretrained CNNs such as Resnet and Alexnet.

Efficient Equivariant Transfer Learning from Pretrained Models
Sourya Basu
Pulkit Katdare University of Illinois at Srbana-Champar
University of Illinois at Champa-Cn
Prasanna Sattigeri IBM Research
Vijil Chenthamarakshan
Katherine
IBM Research
University of Illinois at Urbana- Chaama-Chaampain
Payel Das
Lav R. Varshney University of Illinois at Urbana- Urbana-Champiin Champaign
IBM Research
Abstract
Efficient transfer learning algorithms are key to the success of foundation models on diverse downstream tasks even with limited data. Recent works Basu et al. (2023) and Kaba et al. (2022) propose group averaging (equitune) and optimization- based methods, respectively, over features from group-transformed inputs to obtaín equivariant outputs from non-equivariant neural networks. While Kaba et al. (2022) are only concerned with training from scratch, we find that equitune performs poorly equivariant zero-shot tasks despite good finetuning results. We hypothesize that this because pretrained models provide better quality features for certain transformations than others and simply averaging them is deleterious. Hence, we propose X-equitune that averages the features using importance weights, Às. These weights are learned directly from the data using small neural network, leading to excellent zero-shot and finetuned results that outperform equitune. Further, we prove that X-equitune is equivariant and universal approximator of equivariant functions. Additionally, we show that the method of Kaba et al. (2022) used with appropriate loss functions, which call equizero, also gives excellent zero-shot and finetuned performance. Both equitune and equizero are special cases of X- equitune. To show the simplicity and generality of our method, validate a wide range of diverse applications and models such as I) image classification using CLIP, 2) deep Q-learning, 3) fairness in natural language generation (NLG), 4) compositional generalization in languages, and 5) image classification using pretrained CNNs such as Resnet and Alexnet.
Introduction
Group-equivariant deep learning leverages group equivariance an inductive bias to design efficient and reliable neural networks. Popular examples include convolutional neural networks (CNNs) equivariant translations (LeCun al.. 1989), group convolutional neural networks (GCNNs) equivariant to general discrete groups (Cohen & Welling, 2016), and recently Alphafold2 equivariant to 3D rotations (Jumper et al., 2021). But these methods cannot leverage pretrained models.
With the increase in open sourced large pretrained models, it is now crucial to develop efficient transfer learning algorithms that leverage group equivariance. Basu al. (2023) proposed equitune, an equivariant finetuning method that uses group averaging over features extracted from pretrained
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

arg
photo
Original Imagee
A M0,,, M
4.
4
4.
Cat
T
Ted Encoder
Aphoto of anm infject)
ATM
Transfommed Image
Figure I: Implementation of X-equitune on CLIP. Weighted average of image features corresponding to transformed inputs are computed, which is further used for computing text-image similarity scores.
models. Several other methods were proposed get equivariant output from non-equivariant backbone architectures, e.g. some form of averaging (Puny et al., 2021; Atzmon et al., 2022) or optimization over certain proxy loss functions (Kaba al., 2022). But these latter methods were originally designed for training from scratch and not much is known about their finetuning abilities.
Here, we find that equitune performs poorly zero-shot tasks. Our first main contribution is to show that the optimization method of Kaba et al. (2022) when used with appropriate proxy loss functions provides excellent zero-shot and finetuning performance. We call this method equizero.
The results from equizero suggest that pretrained models provide better quality features for some group transformations than others. Thus, we propose X-equitune, which learns importance weights directly from the data and uses them to perform a weighted group averaging. We show X-equitune outperforms equitune and is competitive with equizero for zero-shot tasks. Moreover, for finetuning, X-equitune often outperforms both equitune and equizero. This constitutes our second main contribution.
To validate our methods, we provide experiments on a diverse set of pretrained models and datasets. We show zero-shot performance of equizero on: 1) image classification using CLIP, 2) deep Q- learning, 3) fairness in natural language generation (NLG), and 4) compositional generalization in languages. For CLIP and deep Q-learning, we used the naturally available loss functions, namely, similarity scores and Q-values, respectively. For fairness in NLG and compositional generalization, we closely follow the setup of Basu al. (2023), and we use regarl scores Sheng et al. (2019) and the negative of the maximum of the output probability distribution, respectively, as the loss function.
We first show results of X-equitune for image classification using CLIP, finding that X-equitune performs competitively with equizero and outperforms equitune for both zero-shot and finetuning tasks. Then, we show simple case where finding a good proxy loss function for equizero is non- trivial: image classification using pretrained CNNs. Here, we find that equizero performs even worse than equitune but X-equitume easily outperforms equitune and equizero. The organization of the paper is summarized below:
$3 provides details of X-equitune and equizero and proves number of their properties.
$4 provides overview of the applications considered in the experiments and how equivariance methods are used there.
$5 provides experimental details and results for all the applications.
2
Background
Here discuss relevant concepts in group equivariance and group equivariant transfer learning.
Group Equivariance group (G. is set accompanied by a binary operator '.' "hat satisfy the axioms of group, namely i) closure; g h e G for all g, h G; ii) associativity: (g-h) h)- k = =g-- gh----);

---

iii) identity: there exists E G. such that e e. for all g G; iv) inverse: for every element g E G, there exists g such that g g =
= e. We write g h as gh for brevity.
Given a set X, we define a group action of G on X' as rx G x X + X' such that it satisfies two axioms, namely i) identity: (x) = x for all E X, where e E G is the identity; ii) compatibility: [9 I'x (h, m)) = ['x x), for all h E G, x E X. We write (gx( simply as ga for brevity.
A model M: X + 3 is equivariant to G under the group action of G on X' and y if M(gr)
gM()) for all E G, x E X. This essentially means that any group transformation g to the input [x(g.2) 2) should reflect with an equivalent group transformation of the output I (g M(z)).
Equivariant Finetuning Recently, Basu et al. (2023) proposed a finetuning method called eq- uituming that starts with potentially non-equivariant model M and produces model Mg that is equivariant to G. Equituning converts pretrained model into an equivariant version by minimizing the distance of features obtained from pretrained and equivariant models. The output of an equituned model is given by
1 1 Eg- Ma(x) = |G|
(I)
DEG
While the averaging in equation 1 is shown to be useful for finetuning, we find it leads to poor equivariant zero-shot learning. This could be because the pretrained model outputs high quality features only for some of the transformed inputs. Hence, averaging them directly leads to low quality zero-shot performance. This can be avoided by using weighted averaging as discussed in Sec. 3.
Optimization- Based Canonical Representation On the other hand, Kaba et al. (2022) show that group equivariant output, MQPT(z)) can be obtained by optimizing a (non-equivariant) loss function ((M(g)) with respect to group elements g E for any x E X' as shown below.
MOPT (x) 900=0-1mm 10=99'MM0 MM)a),,
where 9. = arg min geG (M(gz) with 1 y - R being an injective proxy loss function and assuming the minima is unique. However, the purpose of this formulation in Kaba et al. (2022) is only to obtain an equivariant representation for training from scratch. Moreover, competitive zero-shot or finetuning performance is obtained Kaba et al. (2022) using this method. We show that the choice of / plays an important role in its zero-shot and finetuning performance, even outperforming equituning. This method is obtained as a special case of X-equituning introduced next.
Additional Related Works Group equivariance plays a key role in geometric deep learning (Bron- stein et al., 2021) for designing efficient domain specific neural networks. Several elegant architectures have been proposed for equivariant image classification (Cohen & Welling, 2016, 2017: Romero & Cordonnier, 2020), reinforcement learning (Mondal et al., 2020; van der Pol et al., 2020; Mondal et al., 2022; Wang et al., 2022), graph (Satorras et al., 2021; Keriven & Peyré, 2019; Gasteiger et al., 2021) and mesh (De Haan et al., 2020; He al., 2021; Basu et al., 2022) processing, natural language processing (Gordon et al., 2019; Li et al., 2022), and data generation (Dey et al., 2021). These architectures need to be trained from scratch, which is not always desirable.
Frame averaging produces equivariant output from non-equivariant architecture backbones (Puny al., 2021; Atzmon al., 2022; Duval al., 2023). Most work here focuses on finding good frames, which are equivariant subsets of groups, for specific groups, and not for general groups And not much is known about their performance with pretrained models. Kaba et al. (2022) also give a canonicalization baaaeddaaaat---addedddd method that uses an equivariant auxiliary network for constructing equivariant networks out of non-equivariant backbones and is used for training from scratch. But this work requires an additional equivariant network and appropriate trainable parameterization of the group actions, which is presented only for certain groups of interest. Further, this work is not concerned with equivariant performance of pretrained models. Zero-shot group equivariance was also recently used by Muglich al. (2022) for zero-shot coordination in partially observable Markov decision processes (POMDPs). In contrast, our work aims to be provide efficient equivariant transfer learning algorithms that are general in terms of considered tasks and groups, and does not require additional equivariant networks.
3

---

3 X-Equitune
We propose X-equitune, where unequal weights are assigned features obtained from transformed inputs. This is a simple generalization of equitune in equation and the optimization-balt basde approach in equation 2, where the goal assign higher values better features. Like these previous methods, X-equitune is equivariant and universal approximator of equivariant functions.
The main idea of X-equitune that given pretrained model M. the features M(ga) for any fixed are not all equally important for all E G. We denote by A(gz) the importance weight of feature ((gz) for g E G.v X. We assume G finite, just as Basu al. (2023). Suppose À : X' + R known priori, and denote the X-equituned model as M Then want minimize
min (
(g) (
BEG
s.t.
(g = gM (I) for all g E G.
We call the solution to equation 3 X-equitune, given by
G
M2 (x) = E \(gx) Σ g M(gx) DEG
(4)
3.1 Special Cases
When A(gx) = 1 for E G. then equation is the same as equation 1. Further, we get equation when À is indicator function Xggx) 1 (g=g.)> where g- arg min ((M(g)) with / y such that the minimization well defined. We use M denote the equizero model.
The first main contribution of our work is experimental. We show there are good loss functions for equizero for several diverse tasks that easily outperform equitune by choosing the best features.
The second main contribution to show that À-equitune outperforms equitune and is competitive with equizero even when good loss functions are known. Moreover, when the loss functions not trivial equizero performs even worse than equitune, but X-cquitune casily outperforms both.
3.2
Canonical---Equitume
Here, provide an extension of the X-equitune algorithm of equation 4 to continuous groups. The idea is to combine the canonicalization method of Kaba et al. (2022) with X-equitune leading to an expressive equivariant network with weighted averaging over features with different group actions applicd to them.
Definition 1
Given a (continuous) group G, a non non-equivariant function M X + Y, and an equivariant auxiliary function (from the setting of Kaba et al. (2022)) X + G. lambda functionn À + Rt, and set of group elements ( 0,), i.e. 0, E G. we define the camonical---equiiee operators as
Ma (x) = Eoco A(Oh(z)-II) DE0
(5)
(a = Eose ((0h(z)-1z) (MMMMa)
(6)
In Thm. I, we show that the canonical-x-equivaaiiae network is equivariant to the group G
Theorem 1. Me.enull)) and M (n) are, respectively, equivariant and invariant to
3.3 Properties
Now show in Thm. 2 that equation 4 is equivariant with respect G. Theorem 2 (Equivariance). 2 defined in equation 4 is equivariant to G.

---

Table 1: Inference times of equitune, equizero, and X-equitune for the c4 group for various CLIP models on CIFARIOU We use batch size 32 on a single Nvidia A100 GPU.
Model
Time (sec.)
Memory (MB)
Equitune
Equizero
X-Equitune
Equitune
Equizero
X-Equitune
RN50
14.15
14.10 10.27
23.5 16.08
3703
3703 2587
3941
ViT-B/32
11.00
2589
2915
We define universal approximator in Def. 2. Then, Thm. 3 proves that X-equitune is universal approximator of equivariant functions for groups where |[9|| = for g E G. This includes a wide range of groups including the permutation group, the SO(n) groups of special orthogonal groups. etc. This condition the same as the class of groups considered in Basu et al. (2023).
Definition 2 (Universal approximator). A model M + + is a universal approximator of continuous function J X + y if for any compact set K. C X and e >> 0, there exists choice of parameters for M such that (z)) - M(x)]l S e for all K.
Theorem 3 (Universality). Let fo X' y be any continuous function equivariant to group G and ler + R+ be any positive scalar function. And let M: X + y be a universal approximator of fa X Here X is such that if E X, then gr E X to ensure the equivariance of fG is well-defind. Then, M2 is a universal approximator of fc.
Computational Complexity Note that equitune, equizero, and X-equitune have the same compute complexity. In practice, in Tab. I we find that equitune and equizero have exactly the same time and memory consumption, whereas X-equitune takes little more time and memory because of the additional À network. We illustrate on RN50 and ViT-B/32 models of CLIP using the same text encoder, but different image encoders. RN50 uses Resnet50 based image encoder, whereas ViT-B/32 uses vision transformer based image encoder.
Beyond Zero-Shot Learning Let us emphasize that even though the equizero model in equation is not differentiable due to the arg max, we can still use simple gradient estimators known in the literature. One popular estimator the equizero output in equation 2 would be written as M (z) M(x) + (MQ (z) - M(x)) deettcc (), where detach () indicates that no gradient flows through the term (MC (z) M(x)). practice,
ttroughhhhhhhhh estimator (Bengio al., 2013), where the
we found it to be slightly better to use Mc(2) instead of M(x) and write M (x) = Ma (x)
(M& (ar Ma (M))))) (ee))d)ccccc)) $5.1.3 illustrates few-shot learning and finetuning using equizero and compares with equituning.
4 Applications
First we present several applications in $4.1 where finding a loss function for equizero easy. This naturally leads to excellent equivariant zero-shot results outperforming equitune. Then, in $4.2 we provide two applications in image classification to show 1) the benefits and drawbacks of equizero compared to equitune and X-equitune, and 2) that X-equitune consistently performs well avoiding the drawbacks of equizero.
4.1 Equizero Applications
Here we provide applications where equizero achieves excellent zero-shot performance, namely: 1) deep Q-learning. 2) fairness NLG,, and 3) compositional generalization in languages.
Equizero Reinforcement Learning Recent works, such as van der Pol et al. (2020); Mondal al. (2020), have developed RL algorithms that leverage symmetries in the environments that helps improve robustness and sample efficiency. But no existing work efficiently uses group equivariance on top of pretrained RL models. Here we apply equizero and equitune on top of pretrained models inspired from the group symmetries found in van der Pol et al. (2020). We find that equitune outperforms non-equivariant pretrained models but equizero outperforms both. We simply use the Q-values the proxy loss function as described in I with more details.
5

---

Fairness in NLG We seek to reduce the social biases inherent in language models (LMs), focusing on GPT2 (Radford et al., 2019). We consider the group- theoretic fairness setup of Sheng et al. (2019) and Basu et al. (2023). We take the sets of demographics, namely ['man', "woman'], J, ['straight', 'gay'l, and ['black', 'white"]. For each demographic group, Sheng et al. (2019) proposed two tasks, called respect task and occupation task, where each task consists of five context phrases. A language model (LM) is given these contexts to generate sentences. These generated sentences are then classified as 'positive', 'negative', 'neutral', or 'other' by a regard classifier also proposed by Sheng et al. (2019). These outputs are called regard scores. A regard classifier is BERT-based model similar to a sentiment classifier but more specific for faimess tasks. We use equizero using the regard scores as the proxy loss function, so as to maximize positivity in the generated texts, while guaranteeing group-theoretic fairness. To that end, we propose two algorithms that help in reducing existing biases across demographic groups.
Equizerol.M and R R-Equizerol.MM Basu et al. (2023) define EquiLM and R- R-FquiL.M, that use sequential version of equitune to perform group transformed averaging to achieve fairness across demographic groups. While EquiLM and R- R-EquiLM generate debiased outputs, they do not produce positive regard scores, which is desirable to reduce toxicity in generated text. We propose EquizeroLM and R- R-EquizeroLM which use equizero to maximize regard scores, ensuring both fair and less toxic. Further details provided in $C.2.
Zero-Shot Compositional Generalization We show compositional generalization capabilities of equizero on the SCAN dataset (Lake & Baroni, 2018). SCAN measures compositionality using a language-to-acte -octtion translation task. E.g., if the model learns that the phrase "Jump", "Run", "Run Twice" translate to the actions "JUMP", "RUN", "RUN RUN" from the train set, then, SCAN tests whether the model also learns that "Jump Twice" translates to "JUMP JUMP". Such a reasoning however common in human beings is hard to find in language models.
We apply equizero on two tasks in SCAN, Add Jump and Around Right. Gordon et al. (2019) solved these tasks by constructing sophisticated group equivariant networks from scratch and training them. Basu et al. (2023) used the same group actions as Gordon et al. (2019), but used equituning on pretrained non-equivariant models for a few iterations and obtained comparable results. But, as we note, equitune has poor zero-shot performance. We show that equizero using negative of maximum probability from the output as the loss function gives much better zero-shot performance. Using gradient estimators described in $3.3, we also compare the finetuning performance of equizero against equitune. Details of group actions used are given in $C.3
4.2 X-Equitune Applications
Here consider two important applications: CLIP- and CNN-based image classification. For CLIP, it is easy to find a loss function for equizero that provides better results than equitune. But for the simple case of CNN-based classification it is non-trivial to find such loss function. Since X-equitune does not require a loss function, it performs better than equitune in both cases. Equizero only performs well for CLIP, but fails miserably for CNN-basee classification.
CLIP-Based Image Classification CLIP is a pretrained model consisting of image and text encoders that give impressive zero-shot classification performance across variety of unseen datasets. But, in Fig. 4a and 7, we find that CLIP is not robust to simple transformations such as rotation by multiples of 90º or random flips. This trend is seen across different image encoders like RN50, RN101 (Radford et al., 2021), ViT-B/32 and ViT-B/16 (Dosovitskiy et al., 20.1). This can be avoided by making the model in/equi-variiai 1 that equitune does not produce good zero-shot performance. We show in $5.2.1 that using equizero with image-text similarity score as loss function provides much better zero-shot results than equitune. Later, in 1, we show that X-equitune achieves better zero-shot performance than equitune without any loss function. Finally, when finetuned, we find that X-equitune tends to perform the best, possibly because it does not need gradient estimators like equizero, and because it uses weighted averaging to obtain better features than equitune.
such transformations, e.g.. by using equitune. But we show in
CNN-based image classification For image classification using pretrained CNNs such Resnet and Alexnet, we note that finding a good loss function for equizero non-trivial. As such, we consider two loss functions 1) negative of the maximum probability as it worked well with the SCAN

---

Sibwerd
Canpuse
350
300
250
200
150
Prtnaned Agent Equitue
Prstranedd Agett
Agent
Ezuerr
Figure 2: Comparison of zero-shot performance of equizero to equituning and a von-equivaraint pretrained model are shown in (a), (b), and (c) for Gridworld, Cartpole, and Acrobot, respectively. Equizero outperforms both equituning and non-equivariant pretrained model. Results over five seeds.
task in $4.1 and 2) the entropy of the output distribution since it correlates with the confidence of the model (Wang et al., 2018). But, equizero with these loss functions performs even worse than equitune. Further, find that X-equitune easily outperforms both equitune and equizero.
5 Experiments
Here, we provide experimental results for equizero and X-equitune in $5.1 and $5.2, respectively, for all the applications described in $4. Additional experiments for manonical---equiieee are provided in $. D.5. The code for this paper is available at https: // //ttttb..tttt///////////g cww. htam//auusuurra//// /lambda__ equitune.
5.1 Zero-Shot Performance using Equizero
5.1.1 Equizero Reinforcement Learning
We first pretrain Deep Q-learning nets (DQNs) for each of the Gridworld, Cartpole, and Acrobot environments using the default architecture from Raffin et al. (2021) with 103k parameters. We pretrained all the models using learning rate 10-4, We used training time steps as 100k, 100k, and 70k for Gridworld, Cartpole, and Acrobot, respectively. These number of steps were chosen to obtain the best models by varying the time steps from 50k to 100k in multiple of 10k for fixed seed.
Experimental Setting:
Results and Observations: Fig. 2 show the evaluation performance of equizero and compare it with equituning and non-equivariant pretrained models. We find that equituning performs better than non-equivariant models and equizero outperform both of them. The results are over five seeds.
5.1.2 Fairness in Natural Language Generation
postive
nepacive
100 5U0
08
angl
200
672 S200F2
(a)
(b)
(c)
Figure 3: Plots (a), (b), and (c) show the regard scores for GPT2, EquiGPT2, R-EquiGPT2, Equize- roGPT2, and R- R-EquizeroGPT2. In equitune, if negativity is present in some demographics, it gets redistributed in the other demographics, which undesirable. Equitune is only able to debias, whereas, equizero models not only debiases the texts but also makes the regard scores more positive.

---

Table 2: Zero-shot performance of non-equivariant models, equituned, and equizeroed models for LSTM on SCAN. LSTMs were trained for 200K iterations. We find that equizero outperforms other methods using non-equivariant pretrained models. Results are over three random seeds.
Add Jump
Around Right
Model
Group
Val. Acc.
Test Acc.
Group
Val. Acc.
Test Acc.
LSTM
99.1 (0.3)
0.0 (0.0)
98.9 (0.7)
0.4 (0.7)
EquiLSTM
Verb
62.5 (1.5)
7.1 (1.3)
Direction
80.5 (4.5)
28.2 (12.9)
EquizeroLSTT
Verb
98.4 (0.9)
75.2 (1.5)
Direction
98.7 (5.8)
81.7 (2.4)
Experimental Setting: We use GPT2 (Radford al., 2019), with 117M parameters our pretrained model. We consider the lists of demographic groups ['man', 'woman") J, ['hitee', 'black"), J, and ['straight', 'gay'l J. We compare our method against EquiGPT2 and R-EquiGPT2 (Basu et al., 2023). For the equizero models, we use beam length m as 5, as described in $C.2. We limit the sentence lengths be 15 for all models. We generated 500 sentences for each of the models and demographic groups by varying the seeds for both respect and occupation context.
Results and Observations: Fig. 3 and 9 compare the regard scores for all the considered models. We observe that equizero models are not only able to debias among various demographics like 'man' and 'woman', but also reduces the toxicity/ negativity of the scores. Debiasing seen from the equality of the scores amongst the demographics considered for equivariance. And reduction in toxicity is observed by noticing that the regard scores are more positive, Like equituning (Basu al., 2023), equizero models show high quality of generated texts. Sample generations from the demographic groups l'straight', 'gay'] are shown in Tab. 3, 6, 7, and 5 for all the models. Note in Tab. and 6, that even with perfect equivariance, where the word 'straight' simply gets replaced by 'gay', the regard scores are very different. This shows the presence of bias in the regard classifier itself as was also observed by Basu et al. (2023).
5.1.3 Compositional Generalization using Equizero
Experimental Setting: We evaluate the performance of our algorithm on the SCAN Dataset (Gordon al., 2019) on the Add Jump and Around Right tasks. All the recurrent models (RNN, LSTM, GRU) and their equivariant counterparts contain single hidden layer of 64 units. For all training processes, we use the Adam optimizer (Kingma & Ba, 2015) and reacher-foreing ratio 0.5 (Williams & Zipser, 1989). For pretraining and finetuning, we use 200k and 10k iterations respectively. For pretraining, we use learning rate of 10-4, whereas for finetuning, we used learning rates 10-5 and 2 10-5 for Add Jump and Around Right, respectively. Results are over three seeds.
Observation and Results: Tab. 2 shows that equizero outperforms equituned and non-equiva models sero performance with LSTMs. In $D, we observe similar results for RNNs and GRUs in Tab. 9 and 8, respectively. We use gradient estimators discussed in $3.3 for performing few-shot learning using equizero. For few-shot learning, we find in Fig. 10 in $D that equizero is competitive with equitune for small iterations, but as the number of steps increase, equitune is the better algorithm. This is expected since the gradients are computable for equitune, but only approximated in equizero. Tab. 10, 11, and 12 in $D provide the results for finetuning using equitune and equizero for 10k iterations. We find the equitune is slightly better when finetuning for 10k iterations. This shows equizero is better for zero-shot and few-shot learning, but for large iterations, equitune is preferable.
5.2 Zero-Shot and Finetuning Performance using X-Equitune
5.2.1
Equi/Invariant Image Classification using CLIP
Experimental Setting: We first perform zero-shot image classification on Imagenet-V2 and CI- FARI00 using CLIP. We use two transforms, random 90º rotations and flips, for testing their robustness. We encode class labels using the 80 text templates provided in Radford al. 2
Obtained from https: :tttps::/giiee //github...tttt. wwt.c/om/////eeee coon/oppnai/CLII
8

---

65-
Imageoet tpp-
Amagandd
piimagere
Biy
63-
100- slunne
20
10
10
35
1945
VIT4/16
RN50
RAN022
VIT-016
RA.50
FVV131
VITA/1
(a)
(b)
(c)
Figure 4: In (a) note that zero-shot performance of CLIP drops significantly when we add random rotations flips to the input. This trend is across all image encoders, i.c. ResNet (RN50 and RN101) and VIT (ViT-B/32 and ViT-B/16). In (b) and (c) we show classification results on Imagenet- V2 with random 90 rotations and flips, respectively. We observe equizero outperform equitune and original CLIP for all image encoders.
We then evaluate finetuning capabilities of equitune, equizero, and X-equitune on CIFARI00 with random 90 rotations. Here we choose À to be a two-layered feed-forward network, which is described in detail along with learning rates used, in $D.4. The input this A-network are the features from frozen CLIP encoder. First, the X-network is trained with the model kept frozen. Then, while finetuning the model, the network is frozen. We show results for both Resnet and ViT backbone trained for 1000, 2000, 3000, 4000 finetuning steps.
Results and Observations: Fig. 4 shows test accuracies for Imagenet-V2 with random 90 rotations and flips. We observe that the pretrained model's performance reduces drastically when transformations are applied to the dataset. Whereas both equitune and equizero are relatively robust to such transformations. Moreover, equizero outperforms both equitune and the pretrained model. Similar observations made for CIFAR100 in Fig. 8 in $D.
In Fig. 5a and 1a we plot the test accuracies of X-equitune on CIFARTO0 for both variants of Resnet and VIT backbones. We observe that X-equitune performs better than both equitune and equizero (with finetuning) on Resnets.
ViT-B/16, we observe that X-equitune easily outperforms both equitune and equizero (with finetuning). On ViT-B/32, we find that X-equitune outperforms equitune but equizero outperforms both equitune and X-equitune. Thus, X-equitune performs competitively with equizero, even in applications where good loss functions are known.
5.2.2 Equi/Invariant Image Classification using Pretrained CNNs
We now evaluate the performance of X-equitune on rot90-CIFARI0 dataset. Here, each image is rotated randomly by multiple of 90%, We use pretrained Alexnet and Resnet, trained extensively over CIFARI00. For the X-equituning, we choose as a two layered feed-forward network with hidden layer of dimension 100 with input features extracted by this pretrained Alexnet or Resnet. Along with A, we also perform linear probing wherein last two layers for the classification problem is being learned using learning rate of 10
Experimental Setting:
Observation and Results: In Fig. 5b and llb we see that X-equitune outperforms equitune and equizero. Moreover, equizero performs both Alexnet and Resnet pretrained modules.
worse than equitune. This trend is consistent across
Limitations, Societal Impact, and Conclusion
Limitations and Societal Impact: Our results focus finite groups; extension continuous groups requires further work in parameterization of continuous groups (cf. Benton et al. (2020)) group decomposition (cf. Basu al. (2021); Maile et (2023)). Our work on fairness in NLG aims to debias foundation models and thereby lead to positive societal impact. But we use equality and neutral sets obtained from previous human-made works. We believe there scope for optimizing the design of such sets using neural networks for more general demographic groups
Conclusion: We present X-equitune and its special case equizero that outperform equitune on cquivariant zero-shot and finetuning tasks. We show that both methods are equivariant, universal,

---

Top-1 accuracies for RN on rot90' CIFAR100
Top-l Accuratios for Rux100 -CIFAROO
0.50
45
Top-1 Accuracy
0.00
0.50
Equitune
23-
RN50 Arequiture
0.50
A-oquitune
RN101 Equizero
Equiture
0.40
equizero (wth entropyl
IN100 Avequituee
0.40
equizero with max probabilit)
15
3000
Finetuning steps
Epochs
(a)
(b)
Figure 5: In (a) and (b), we plot accuracies of pretrained CLIP and Resnet, respectively, for equitune, equizero, and X-equitune. For CLIP, X-equitune performs competitively on CIFARI00 with equizero for zero-shot and outperforms for finetuning. For Resnet, equizero performs even worse than equitune on CIFARIO, whereas X-equitune outperforms both equizero and equitune.
computationally efficient, and are easy to implement. Equizero performs well when good proxy loss functions are available for downstream tasks, which we show is easy to find across several diverse tasks. X-equitune outperforms equitune and is competitive with equizero without the need for any proxy loss. We consider diverse tasks: i) deep Q-learning, ii) fairness in NLG, iii) compositional generalization in languages, iv) CLIP-based classification, and CNN-based classification. Experimental results validate the superiority of X-equitune and equizero over equitune on zero-shot and finetuning tasks.
Acknowledgment
Discussion with Moulik Choraria on the X-equitune finetuning for CLIP appreciated. portion of the work was supported by the Department of Energy (DOE) award (DE-SC0012704) This work was also supported in part by ZJU-UIÜC Joint Research Center Project No. DREMES 202003, funded by Zhejiang University.
References
Atzmon, M., Nagano, K., Fidler, S., Khamis, S., and Lipman, Frame averaging for equivariant shape space learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 631-641, 2022.
Basu, S., Magesh, A., Yadav, H., and Varshney, L. R. Autoequivariant network search via group decomposition. arXiv:2104.0m 044404848 2021.
Basu, S., Poslago-Posada, J., Viganò, F., Rowbottom, J., and Cohen, T. Equivariant mesh attention networks. Transactions on Machine Learning Research, 2022.
Basu, S., Sattigeri, P., Ramamurthy, K. N., Chenthamarakshan, V., Varshney, K. R., Varshney, L. R., and Das. Equi-tuning: Group equivariant fine-tunie tinettuning of pretrained models. In AAAI Conference on Artificial Intelligence, pp.
2023.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. 1308.3333 2013.
Benton, G., Finzi, M., Izmailov, P., and Wilson, Learning invariances in neural networks from training data. Advances in Neural Information Processing Systems, 33:17605-17616 2020.
Brockman, G., Cheung, V., Pettersson, L.,, Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAl Gym, 2016.
10

---

Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv:2104.13475. 2021.
Cohen, and Welling, M. Group equivariant convolutional networks. International Conference on Machine Learning, pp. 2990-2999999 2999, 2016.
Cohen, T. and Welling, M. Steerable CNNs. In International Conference on Learning Representations, 2017.
De Haan, P., Weiler, M., Cohen, T., and Welling, M. Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In International Conference on Learning Representations, 2020.
Dey, N., Chen, A., and Ghafurian, S. Group equivariant generative adversarial networks. In International Conference Learning Representations, 2021.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Duval, A., Schmidt, V., Garcia, A. H.. Miret, S., Malliaros, F. D., Bengio, Y., and Rolnick, D. FAENet: Frame averaging equivariant GNN for materials modeling. arXiv:2305.05577. 2023.
Finzi, M., Welling, M., and Wilson, A. G. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International conference on machine learning, pp. 3318-3328, 2021.
Gasteiger, J., Becker, F, and Günnemann, S. GemNet: Universal directional graph neural networks for molecules. Advances in Neural Information Processing Systems, 34:6790-6802, 2021.
Gordon, J., Lopez-Paz, D., Baroni, M., and Bouchacourt, D. Permutation equivariant models for com- positional gencralization in language. In International Conference on Learning Representations, 2019.
He, L.., Dong, Y., Wang, Y., Tao, D., and Lin, 7. Gauge equivariant transformer. Advances in Neural Information Processing Systems, 34, 2021.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool,
K., Bates, R., Židek, A., Potapenko, A., et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583-588) 2021.
Kaba, S.-O., Mondal, A. K., Zhang, Y., Bengio, Y., and Ravanbakhsh, S. Equivariance with learned canonicalization functions. arXiv:2211.06453 2022.
Kalashnikov, D., Varley, J., Chebotar, Y., Swanson, B., Jonschkowski, R., Finn, C., Levine, S., and
Hausman, K. MT-Opt: Continuous multi-task robotic reinforcement learning at scale. 2021.
Keriven, N. and Peyré, G. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems, 2019.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representutions, 2015.
Lake, B. and Baroni, M. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pp. 2873-2882, 2018
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E.,, Hubbard, W., and Jackel, I., D. Backpropagation applied handwritten zip code recognition. Neural Computaaioon 1(4):541-555 1989.
Li, Y., Wang, X., Xiao, J., and Chua, T.SS. Equivariant and invariant grounding for video question answering. Proceedings of the 30th ACM International Conference on Multimedia, pp. 4714- 4722, 2022.

---

Maile, K., Wilson, D. G., and Forré, P. Equivariance-aver architectural optimization of neural networks. In International Conference on Learning Representations, 2023.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing Atari with deep reinforcement learning. arXiv: 2013.
Mondal, A. K., Nair, P., and Siddiqi, K. Group equivariant deep reinforcement learning. MiXiv:2007.03337, 2020.
Mondal, A. K., Jain, V., Siddiqi, K., and Ravanbakhsh, S. EqR: Equivariant representations for data-efficient reinforcement learning. In International Conference on Machine Learning, pp. 15908-15926, 2022.
Muglich, D., de Witt, C. S., van der Pol, E., Whiteson, S., and Foerster, N. Equivariant networks for zero-shot coordination. In Advances in Neural Information Processing Systems, volume 35, pp. 6410-6423, 2022.
Puny; O., Atzmon, M., Smith, E. J., Misra, I., Grover, A., Ben-Hamu, H., and Lipman, Y. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations, 2021.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, 2021.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Emestus, M., and Dormann, N. Stable-baseliness Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(1): 12348-12353 2021.
Romero, D. W. and Cordonnier, J.-B. Group equivariant stand-alone self-attention for vision. In International Conference on Learning Representations, 2020.
Satorras, G., Hoogeboom, E., and Welling, M. E (N) equivariant graph neural networks. In International Conference Machine Learning, 2021.
Schulman, J., Zoph, B., Kim, C., Hilton, J., Menick, J., Wenga, J., et al. ChatGPT: Optimizing language models for dialogue. OpenAl blog.
Sheng. E., Chang, K.-W., Natarajan, P., and Peng, N. The woman worked as babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3407-3412, 2019.
Silver, D., Schrittwiesrr, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L.,, Lai, M., Bolton, A., Chen, Y., Lillicrap. T. P., Hui, F,, Sifre, I.., van den Driessche, G., Graepel, T., and Hassabis, Mastering the game of go without human knowledge. nature, 550(7676):
354-359, 2017.
van der Pol, E., Worrall, D., van Hoof, H., Oliehoek, F., and Welling, M. MDP homomorphic networks: Group symmetries in reinforcement learning. Advances in Neural Information Processing Systems, 33:4199-4210, 2020.
Wang, D., Walters, R., Zhu, X., and Platt, R. Equivariant Q learning in spatial action spaces. In Conference Robot Learning, pp. 1713-1723, 2022.
Wang, X., Luo, Y., Crankshaw. D., Tumanov, A., Yu, F., and Gonzalez, J. E. IDK cascades: Fast deep learning by learning not to overthink. In Uncertainty Artificial Intelligence, 2018.
Williams, R. J. and Zipser, D. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2::270-280, 1989.
Yarotsky, D. Universal approximations of invariant maps by neural networks. Constructive Approxi- mation, 55(1):407-474, 2022.
12


Reviews:
[{"review_id": "aOLQ28KCFO", "reviewer": "Reviewer_TNLv", "summary": "This paper proposes lambda-equitune based on previous works, which averages the features using learned weights by data and small neural network. This paper provides detailed theoretical analysis to prove that the proposed lambda-equitune is equivariant and a universal approximator of equivariant functions. Diverse experiments are conducted to show the generality of the method.", "questions": "Pease refer to the weaknesses part.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Efficient transfer learning from foundation models to downstream tasks is an important tasks. This work make interesting improvements on previous work by introducing important weights trained from data and small neural network.\n2. Diverse experiments results are provided, including image classification, deep Q-learning and natural language generation. The proposed lambda-equitune method shows good results on majority of the tasks.\n3. Detailed theoretical analysis for equitune is provided. The writing and illustration is clear. \n", "weaknesses": "1. The novelty against the previous work of Basu et al. (2023). Since extra parameters and fintuning process are introduced, the contribution of this work could be further explained.\n2. According to figure 1, both original data and transformed data are used for inference of the proposed method, it is very natural to conduct embedding from features or from results, how about these embedding methods compared with the proposed method?\n3. The setting of the image classification experiments is a bit naive. Since the paper focuses on efficient transfer learning, there are many more meaningful and realistic transfer learning tasks in computer vision domain rather than flip or rotation of 90 degree.\n4. Minors: the format of reference part should be adjusted in Page 12. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our contribution interesting.\n\nWe first clarify a few points from our paper that the reviewer might have misunderstood, then we address the weaknesses pointed out by the reviewer.\n\nReviewer: ``This paper proposes lambda-equitune based on previous works, which averages the features using learned weights by data and small neural network.\"\n\n**Response:** Please note that we not only provide the general framework of \\lambda-equitune but also provide an important special case called equizero that does not require any additional neural network or even any additional data.\n\nWe would like to point out that using equizero with appropriate loss functions (often naturally available, e.g. image-text similarity scores in CLIP) we outperform equitune in several experiments such as \na) improving the robustness of CLIP b) deep Q-learning c) Debiasing and detoxifying (improving the regard scores) LLMs, using regard scores as the loss/score function d) improving compositional generalization capabilities of RNNs, GRUs, and LSTMs\n\nWe use $\\lambda$-equitune in cases where searching for a loss function is non-trivial. We illustrate one such case in the paper: classification with CNNs. \n\nNow, we address the weaknesses pointed out by the reviewer. \n\n\nReviewer: ``The novelty against the previous work of Basu et al. (2023). Since extra parameters and fintuning process are introduced, the contribution of this work could be further explained.\"\n\n**Response:** There are two main novelties of this work that benefit our work compared to Basu et. al: \na) equizero: equivariant zero-shot learning using no additional data or small neural networks. This outperforms equitune for zeroshot learning on several diverse downstream tasks.\nb) \\lambda-equitune: where we use a small neural network to weigh the features obtained from the pre-trained model corresponding to different transformed inputs.\n\nFrom Tab. R1, note that the number of added trainable parameters is negligible. It shows that using a tiny fraction of extra parameters for performing the weighted averaging can be highly beneficial for extracting equivariant features from pretrained models.\n\nTable R1. Number of additional trainable\n| Exp. name       | added trainable params| pretrained params | frac of added params|\n| ----------- | ----------- |----------- | ----------- |\nCLIP (RN50)\t\t|\t112.5k\t     \t|    25.6 M               |\t0.0043\t     |\nCLIP (RN101) \t\t|\t61.3k\t\t|    44.5 M\t|\t0.0013\t|\nCLIP (ViT-B/32 and ViT-B/16)|\t61.3k\t \t|    86M\t\t|\t0.0007\t|\nResnet\t\t\t|\t66.6k\t\t|    11.6M\t|              0.005\t|\nAlexnet\t\t\t|\t934.k\t\t|     61.1M\t|              0.0150\t\t|\n\n\n\nReviewer: \"According to figure 1, both original data and transformed data are used for inference of the proposed method, it is very natural to conduct embedding from features or from results, how about these embedding methods compared with the proposed method?\"\n\n**Response:** \nUnfortunately, since the reviewer has not provided any reference for the embedding methods they are referring to, we are unable to provide a detailed comparison with these methods. We would urge the reviewer to kindly share some references so that we could try to provide some comparison.\n\nPlease note that several of our experiments use embedding from the CLIP model and use image-text similarity scores to obtain the classification scores. \n\nIn Fig. 4a and 7, we find that existing CLIP embedding-based classifiers that do not use group equivariance are not robust to transformations such as flip or rotation of 90 degrees. Then, in Fig. 4b, 4c, and 8, we show how equizero leverage group equivariance to provide robustness to such transformations, moreover, it outperforms other equivariant methods such as equitune without using any additional data or learnable parameters. Experiments on lambda equitune that use CLIP-based embedding techniques are provided in Fig 5, 11.\n\nMoreover, since our formulation of both lambda-equitune and equizero are model agnostic, it is easy to extend them to other embedding based methods not considered in this work, such as in [1]. In [1], we can simply make both the CLIP and the cache models equivariant to guarantee equivariance of this embedding based technique.\n\nWe believe this convinces the reviewer that a) we have already provided comparisons with CLIP-based embedding methods, b) the generality of our method allows to incorporate our method into any other embedding based methods.\n\n[1] Zhang et. al. \"Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification\", arXiv:2207.09519v1\n\nReviewer: \"The setting of the image classification experiments is a bit naive. Since the paper focuses on efficient transfer learning, there are many more meaningful and realistic transfer learning tasks in computer vision domain rather than flip or rotation of 90 degree.\"\n\n**Response:** \nNote our method is completely general and provably equivariant for any group. Our experiments focusing on robustness to flips/rotations for pretrained CLIP/CNN models can be easily generalized to any other discrete groups, when applicable.\n\nWe emphasize that our work is not restricted to computer vision. Besides, we also show efficient transfer learning in several other domains such as Fairness in NLG (e.g., debiasing GPT2), deep Q-learning, compositional generalization in languages.\n\nMoreover, recent work [2] shows that equivariance to seemingly naive groups such as rot90/flip can provide improvement in performance where the actual transformations in data are much more complicated and not explicitly known.\n\n[2] Wang et. al. \"The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry\", ICLR 2023\n\nReviewer: \"Minors: the format of reference part should be adjusted in Page 12.\"\n\n**Response**: We thank the reviewer for pointing this out. We will make sure to fix this in the updated version.\n"}, {"review_id": "Y8SwQIHiTl", "reviewer": "Reviewer_DNsq", "summary": "This paper proposes an extension of symmetrization approach (Yarotsky 2018; Puny et al., 2021; Kaba et al., 2022; Basu et al., 2023) for achieving invariance and equivariance to (small and finite) symmetry groups, with a focus on empirical demonstration of zero- and few-shot transfer learning from non-equivariant pretrained architectures for a range of applications involving different group symmetries. The technical contribution that allows zero- and few-shot transfer is the introduction of some score (rank) function denoted lambda that weights all possible transformed inputs, and using it to turn standard group averaging (Yarotsky 2018) into weighted averaging in a way that the equivariance (and universality) of symmetrization is still guaranteed. The key idea is that an appropriate choice of the score function allows symmetrization to work favorably for the underlying pretrained model (by assigning higher weights to \"important\" group transformations), which can allow few-shot transfer learning. The authors name this approach lambda-equitune. In particular, choosing the score function as an indicator on argmin of some loss function allows for a canonicalizing symmetrization (Kaba et al,., 2022) that empirically allows zero-shot transfer, which the authors name equizero. The authors experimentally demonstrate the proposed algorithm in a range of applications including reinforcement learning, fairness in language models, compositional generalization, and image recognition under 90-degree rotations and flips.\n\nYarotsky et al., Universal approximations of invariant maps by neural networks (2018)\n\nPuny et al., Frame Averaging for Invariant and Equivariant Network Design (2021)\n\nKaba et al., Equivariance with Learned Canonicalization Functions (2022)\n\nBasu et al., Group Equivariant Fine-Tuning of Pretrained Models (2023)", "questions": "Q1. For the experiments mentioned as few-shot (e.g., Figure 10), how many shots are used?\n\nQ2. In case of zero-shot learning without parameter updates, what kind of practical advantage could we expect from the universality result (Theorem 2)?", "limitations": "The authors have addressed potential limitations and negative societal impact in Section 6.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper aims to address an important and original problem of few-shot or zero-shot transfer of non-equivariant pre-trained deep neural networks to solve equivariant problems. While equivariant transfer learning from non-equivariant model has been investigated by some prior and concurrent work (Basu et al., 2023; Kim et al., 2023), few-shot or zero-shot transfer has not been demonstrated in literature as far as I know. The methodology is clearly motivated and explained, and I think this offers a nice way to steer the behavior of pretrained models towards equivariance with additional controllability offered by the score function or loss function, as demonstrated in the fairness experiment where fairness jointly with high regard score is achieved.\n\nS2. The score function lambda for lamdba-equitune and the loss function for equizero doesn't seem to have to respect group symmetry, which is an advantage as it allows for wider range of choices. While this has been theoretically shown in Kaba et al., 2022, as far as I know this is the first work to empirically utilize the property, since Kaba et al., 2022 proposed but did not experiment with optimization approach.\n\nS3. The applications demonstrated with experiments are quite comprehensive, ranging from reinforcement learning to language generation and visual recognition. Also, the experimental results overall seems to support the main claims of the paper.\n\nBasu et al., Group Equivariant Fine-Tuning of Pretrained Models (2023)\n\nKim et al., Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance (2023)\n\nKaba et al., Equivariance with Learned Canonicalization Functions (2022)", "weaknesses": "W1. One major ambiguity I find in the paper is that, while the title of the paper and some parts in the main text mention few-shot learning, it seems the experiments concern zero-shot learning or fine-tuning with a fair amount of data. This was confusing to me given that fine-tuning is not equivalent to few-shot learning. Am I missing something?\n\nW2. The approach is only applicable to small, finite groups, due to the requirement of evaluating the score function (lambda) for all possible transformed inputs. This is in contrast to some concurrent work (Kim et al., 2023) that extends to combinatorial or continuous groups, and can be considered a limitation of the proposed algorithm at the current state.\n\nW3. For the fairness in language generation, I think there is a limitation in the proposed algorithm that the considered words (upon which group transformations are defined) are implicitly assumed to be not separated by the tokenizer of the language model. This might not be generally true for modern language models, as a tokenizer can choose to split an equality word into non-equality substrings (a potential example is waitress -> wait + ress). In this case, fairness would not be achieved as expected using vocabulary permutation transformations.\n\nKim et al., Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer finds our problem original and contributions advantageous.\n\nWe first provide some clarifications, then address the weaknesses.\n\nClarifications:\n\nReviewer: \"S2: The score  ... choices. While ... shown in Kaba et al., 2022, ... experiment with optimization approach.\"\n\n**Response:** To the best of our knowledge, Kaba et. al. only show that the loss function need not be equivariant in the case of the optimization approach where there is an arg min (cf. eqn. 4, eqn. 6 in Kaba et. al.). Instead, our setup simply performs weighted averaging. Of course, the setting of Kaba et. al. can be generalized to ours. However, this generalization is not shown in Kaba et. al.  \n\nWeaknesses:\n\nReviewer: \"One major ambiguity ... title of the paper .... Am I missing something?\"\n\n**Response:** We thank the reviewer for pointing out the confusion in title. We agree that the focus of the paper can be better described as a combination of zero-shot learning and finetuning rather than few-shot learning. We admit our number of training samples, even though much smaller than those used by pretrained models, should technically be called finetuning rather than few-shot learning. We used the umbrella term \"few-shot learning\" to explain the plethora of application across zero-shot learning and finetuning. \n\nWe wanted to emphasize that while equituning takes several iterations on the data to obtain good results, equizero and lambda-equitune can perform much better than equitune with only a few iterations or none (equizero). \n\nWe are happy to change the title to \"Efficient Equivariant Transfer Learning from Pretrained Models\", a better description for our contributions. We apologize for any caused confusion.\n\nReviewer: \"... applicable to small, finite groups... concurrent work (Kim et al., 2023)....\"\n\n**Response:** Our framework can be easily extended to continuous groups by using it with canonicalization of Kaba et. al., which is an alternative to the method of Kim et. al. (*please note this work appeared only after the Neurips deadline*).\n\n**Main idea:** combine canonicalization from Kaba et. al. with $\\lambda$-equitune leading to expressive equivariant network with weighted averaging over features with different group actions applied to them.\n\n**Def**: Given a (continuous) group $G$, a non-equivariant function $M:X\\mapsto Y$, and equivariant auxiliary function (from the setting of Kaba et. al.) $h: X \\mapsto G$, lambda functions $\\lambda: X \\mapsto R^+$, and a set of group elements $\\Theta$ = {$\\theta_1, \\ldots, \\theta_k$}, i.e. $\\theta_i \\in G$, we define the canonical-$\\lambda$-equitune operators as\n$M^{ \\lambda }_{G, equi}(x) = $\n\n $(\\sum_{ \\theta \\in \\Theta } \\lambda (\\theta h(x)^{-1}x) h(x) M( \\theta h(x)^{-1}x) )/( \\sum_{ \\theta \\in \\Theta }  \\lambda (\\theta h(x)^{-1} x) )$\n\n\n$M^{ \\lambda }_{G, inv}(x) = $\n\n $(\\sum_{ \\theta \\in \\Theta } \\lambda (\\theta h(x)^{-1}x) M( \\theta h(x)^{-1}x) )/( \\sum_{ \\theta \\in \\Theta }  \\lambda (\\theta h(x)^{-1} x) )$\n\n\n**Thm**: $M^{\\lambda}_{G, equi}(x)$ is equivariant to $G$.\n\n**Proof**: First note $h(gx) = g h(x)$. \nThus, we have $\\lambda(\\theta h(gx)^{-1} gx) = \\lambda(\\theta h(x)^{-1} g^{-1} gx) = \\lambda(\\theta h(x)^{-1} x)$. \nHence, $\\lambda(\\theta h(gx)^{-1} gx)$ is invariant to actions of $G$.\n\nFinally, $M^{ \\lambda }_{G, equi}(g x)$\n\n$=(\\sum_{ \\theta \\in \\Theta } \\lambda (\\theta h(gx)^{-1}gx) h(gx) M( \\theta h(gx)^{-1}gx) )/( \\sum_{ \\theta \\in \\Theta }  \\lambda (\\theta h(gx)^{-1} gx) )$\n\n$=(\\sum_{ \\theta \\in \\Theta } \\lambda (\\theta h(x)^{-1}g^{-1}gx) g h(x) M( \\theta h(x)^{-1}g^{-1}gx) )/( \\sum_{ \\theta \\in \\Theta }  \\lambda (\\theta h(x)^{-1} g^{-1}gx) )$\n\n$=g (\\sum_{ \\theta \\in \\Theta } \\lambda (\\theta h(x)^{-1}x) h(x) M( \\theta h(x)^{-1}x) )/( \\sum_{ \\theta \\in \\Theta }  \\lambda (\\theta h(x)^{-1} x) )$\n\n$=g M^{\\lambda}_{G, equi}(x)$.\n\nThe proof for invariance of $M^{\\lambda}_{G, inv}(x)$ follows similarly.\n\n**Exp. results are provided in the Tab. global response pdf**\n\nFor exp., we use $G = $SO(2), the invariant regression function from Finzi et. al. as our task. We define $M$ as an MLP with 5 layers. $h$ is constructed using a fixed function that sums $x_1, x_2$ and computes the corresponding SO(2) rotation matrix from it. $\\lambda$ is a small MLP with 3 layers, but much smaller number of neurons in them. We adjust the num. of params. in models to make sure both when using $\\lambda$ and without, we have similar number of parameters.\n\nWe use a train and test size of 10000, 10000, batch size 500, learning rate $5*10^{-3}$, epochs 100, num. of seeds 5. \n\nFinzi et. al.. \"A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups.\" ICML, 2021.\n\nReviewer: \"For the fairness... limitation...tokenizer of the language model.\"\n\n**Response:** We agree with the reviewer that the tokens might not be words.\n\nAs discussed in limitations in lines 318-319, we think there is a scope for optimizing these equality and neutral sets instead of using human-made sets of words. \n\nIn future, one can directly learn sets of tokens that maximizes regard scores of LLMs, e.g., using RLHF. \n\nNote, our current formulation still give empirically good fairness results for GPT2 (cf. Fig. 3 and 9) using BPE tokenizers.\n\nQuestions:\n\nReviewer:\"..., how many shots are used?\"\n\nPlease note, as discussed above, our motivation is to show efficiency of finetuning (using few iterations) in this Fig. We are happy to update the title and figure labels. Apologies for confusion.\n\nReviewer:\"In case of zero-shot learning without parameter updates, what kind of practical advantage could we expect from the universality result (Theorem 2)?\"\n\n**Response:** It says that if the non-equivariant model is well-trained and is a good approximator to a certain function, then so are the obtained equivariant zero-shot results. This result ensures that obtained equivariant model is still an expressive equivariant model and not overconstraining the pretrained model.\n"}, {"review_id": "jesKi6wFLD", "reviewer": "Reviewer_vsSr", "summary": "This paper proposed an equivariant few-shot learning method from pretrained models, namely λ-equitune, averaging the features using importance weights, i.e. λs. These weights are learned directly from the data using a small neural network, leading to excellent zero-shot and finetuned results that outperform equitune. This work further proves that λ-equitune is equivariant and a universal approximator of equivariant functions, and shows that equitune and equizero (the method of Kaba et al. (2022) used with appropriate loss functions) are special cases of λ-equitune. The authors conduct a series of analyses and experiments, validating the simplicity and generality of the proposed method on a wide range of diverse applications and models.", "questions": "In Equation (3), maybe M_G^λ (g,x) is M_G^λ (gx)?\n\n", "limitations": "Limitations are provided in the paper. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1). The idea of this work is novel for equivariant few-shot learning from pretrained models.\n\n(2). A wide range of diverse applications and experiments validate the claim of the work.\n", "weaknesses": "(1). I think Sub-Section '3.2 Properties' is somewhat confusing and should be analyzed in detail to ensure logical consistency. For instance, what is the connection between Theorem 1 and Definition 1, Theorem 2? Is Theorem 1 the foundation of Definition 1 and Theorem 2? Mathematical tools such as definitions, theorems, lemmas, etc. are used to summarize and abstract the theory of the entire method, requiring detailed descriptions to construct their connections.\n\n(2). Using importance weights to the features averaging in λ-Equitune need to be discussed, including its benefits against to existing equivariant finetuning methods. Could learnable λ weights adapt the feature outputs from pretrained models? Is this a key to λ-Equitune obtaining excellent results in few-shot learning?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work novel and appreciating the diversity of our experiments. \n\nWe address the weaknesses and the questions raised by the reviewer below:\n\nReviewer: \"I think Sub-Section '3.2 Properties' is somewhat confusing and should be analyzed in detail to ensure logical consistency. For instance, what is the connection between Theorem 1 and Definition 1, Theorem 2? Is Theorem 1 the foundation of Definition 1 and Theorem 2? Mathematical tools such as definitions, theorems, lemmas, etc. are used to summarize and abstract the theory of the entire method, requiring detailed descriptions to construct their connections.\"\n\n**Response:** We thank the reviewer for pointing out the confusion in the explanation of the theoretical results in Sec. 3.2. We clarify the theoretical contributions below and will update it to the revised version.\n\nTheorem 1 is a standalone theorem showing that the proposed method called lambda equitune is provably equivariant to the considered group G.\n\nDefinition 1 provides a definition of universality popularly used in several group equivariant neural networks papers, Yarotskiy (2022), Ravanbakhsh (2020). Theorem 2 uses the definition of universality from Definition 1 in its proof of showing that lambda equitune is a universal approximator of equivariant functions. The proof to Theorem 2 is provided in the appendix.\n\nWe hope this improves the exposition of the results in Sec. 3.2.\n\n\nReviewer: \"Using importance weights to the features averaging in λ-Equitune need to be discussed, including its benefits against to existing equivariant finetuning methods. Could learnable λ weights adapt the feature outputs from pretrained models? Is this a key to λ-Equitune obtaining excellent results in few-shot learning?\"\n\n**Response:** Indeed the idea behind lambda equitune is that the learnable lambda weights adapt such that they are high for the \"good\" features and low for \"bad\" features. Here \"good\" features are the features that contribute towards better performance and similarly, the \"bad\" features do not help get good performance. Existing equivariant finetuning methods such as equitune simply average all the obtained features from different transformed inputs, which is deleterious because not all features contribute equally to the performance of the model. \n\nThe reason why certain transformations of the input yield better results can be explained through the example of using CLIP on transformed imagenet and CIFAR100 in Fig. 4a and 7, respectively. Note in both the figures that CLIP shows better results when the input images are provided in the upright form. Whereas, when the input images are rotated or flipped, their performances drop significantly, thus showing that the features obtained from non-upright images are not as useful aa the ones obtained from the upright images.\n\nNow, when a dataset contains images that have random rotations and flips, the lambda weights can automatically find out which transformed images contribute the most to the performance (in this case the upright images) and weight them the most leading to better performance than equitune of Basu et. al.\n\nWe hope this explains the importance of the lambda weights better. We would add this to the revised version of the paper to help improve the clarity.\n\nReviewer: \"In Equation (3), maybe M_G^λ (g,x) is M_G^λ (gx)?\"\n\n**Response:** We thank the reviewer for pointing out this typo. We will update this in the revised version of the paper.\n\nRavanbakhsh. \"Universal equivariant multilayer perceptrons.\" ICML 2020.\n\nYarotsky. \"Universal approximations of invariant maps by neural networks.\" Constructive Approximation (2022)\n\n\n"}, {"review_id": "HgA8za4QL3", "reviewer": "Reviewer_8HWc", "summary": "The paper introduces $\\lambda$-equitune, an innovative method that refines existing strategies for achieving equivariant outputs from non-equivariant neural networks. $\\lambda$-equitune employs importance weights for feature averaging, which outperforms the group averaging method equine. The authors also present equizero, another approach enhancing zero-shot and fine-tuned performance. The effectiveness of these methods is validated across diverse applications and models, including image classification, deep Q-learning, and natural language generation.", "questions": "(1) This paper mentioned that equizero performs well when good agent loss functions are available for downstream tasks. Can the authors elaborate on the process of selecting or designing these proxy loss functions? How did they ensure that the selected loss functions accurately capture the desired goals of each task?\n\n(2) Could the authors discuss the potential implications of their research in real-world applications and any limitations or considerations when deploying the proposed methods in practical scenarios?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The paper introduces a novel approach to addressing the limitations of existing equivariance methods by incorporating importance weights for feature averaging. This creative combination of ideas contributes to the originality of the work.\n\n**Quality**: The research is of high quality, backed by rigorous theoretical justifications and empirical evaluations. The authors provide compelling evidence to support the claims made, ensuring the reliability and robustness of their proposed method.\n\n**Clarity**: The paper is exceptionally well-written, presenting complex concepts in a clear and concise manner. The organization of the paper facilitates understanding, and the inclusion of theoretical proofs enhances its clarity.\n\n**Significance**: The significance of the paper lies in its broad applicability across multiple domains and models, including image classification, deep Q-learning, and natural language generation. The improved zero-shot and fine-tuned results achieved by $\\lambda$-equitune and equizero highlight their potential to advance the field of transfer learning.", "weaknesses": "One weakness of the paper is the limited exploration of continuous groups, as the focus is primarily on finite groups. The authors acknowledge the need for further work to extend their methods to continuous groups but do not provide concrete solutions or insights in this regard. This limitation restricts the proposed algorithms' generalizability and applicability to real-world applications involving continuous transformations. Addressing this weakness by offering a more detailed discussion on approaches for handling continuous groups would enhance the paper's relevance and broaden its potential impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer finds our approach novel, innovative, and applicable in multiple domains and models. We address the weaknesses and questions raised by the reviewer below:\n\nweaknesses:\n\nReviewer: \"...limited exploration of continuous groups...\"\n\n**Response:** We thank the reviewer for pointing out the significance of exploring continuous groups to our setting.\nWe have only considered discrete groups for our work since we were mainly motivated by our considered applications that only required discrete groups.\n\nTo this end, we first point out that the work of Kim et. al. (that appeared on arxiv after the Neurips submission deadline), as pointed out by Reviewer DNsq, generalizes our framework to continuous groups using symmetric probabilistic averaging leading to equivariance in expectation for continuous groups. \n\nWe would also like to note that extending our framework to continuous groups is simple and there could be many ways to generalize our framework to continuous groups. \n\nAs such, we show that it also can be done by combining the setting of Kaba et. al. with our $\\lambda$-equitune setup. **We have provided the extension with proof of equivariance in the response to Reviewer DNsq and the experimental results are in the pdf corresponding to the global response.**\n\nIt shows that $\\lambda$-equitune can be used to weigh the features corresponding to different transformations of the inputs obtained from canonicalization differently, leading to improved performance.\n\nWe hope this convinces the reviewer that our work can be extended beyond discrete groups easily.\n\nQuestions:\n\nReviewer: \"This paper mentioned that equizero performs well when good agent loss functions are available for downstream tasks. Can the authors elaborate on the process of selecting or designing these proxy loss functions? How did they ensure that the selected loss functions accurately capture the desired goals of each task?\"\n\n**Response:** We thank the reviewer for raising this important question. We found that several machine learning tasks contain naturally available loss functions that can act as a proxy of their performance, e.g. CLIP has similarity scores, and Q-learning has Q-values. Maximizing these scores naturally leads to better performance as it is part of their original optimization tasks. The task of fairness in GPT2 is similar to that of the RL task, where we assign values describing the performance of the models. For GPT2 the scores are directly obtained from the regard scores of Sheng et. al. For compositional generalization in languages, the task is similar to classification and we chose the maximum of the probability as the score since it shows the confidence of the model in making the prediction (alternately one could also use the entropy of the probability vector, which is often used to measure confidence score of prediction). We found this score to work well for compositional generalization. The only surprising case that we found was that of classification using CNNs, where simple averaging of equitune seems to outperform loss /score functions such as entropy/maximum of probability, which shows that there are cases where finding good loss functions might be non-trivial. This, in addition to noting that the loss function can be replaced with learnable weights as score function leads to our general method, called lambda-equitune. \n\nHence, we conclude that several machine learning tasks have naturally available loss/score functions that can be exploited for equizero. However, lambda-equitune can always be employed to obtain good equivariant finetuning results irrespective of the domain of application.\n\nReviewer: \"Could the authors discuss the potential implications of their research in real-world applications and any limitations or considerations when deploying the proposed methods in practical scenarios?\"\n\n**Response:** We believe equizero and lambda-equitune when used with strong pretrained models would definitely make them robust, e.g. uses in robotics or object recognition.\nRegarding the fairness studies in our work, the current version focuses on debiasing using the metric and setup of Sheng et. al. and using the equality and neutral word sets from Basu et. al. that were human-made. We believe that for deployment of this application requires further testing and improvement in both the used regard scores and the construction of the equality and neutral sets. In particular, both the regard scores, and the equality and neutral word sets need to be constructed such that they satisfy any requirements of the application where it is deployed.\n\nBasu et al., Group Equivariant Fine-Tuning of Pretrained Models (2023)\n\nKim et al., Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance (2023)\n\nKaba et al., Equivariance with Learned Canonicalization Functions (2022)\n\nSheng, et. al. The woman worked as a babysitter: On biases in language generation. (EMNLP-IJCNLP), 2019.\n"}]

Author rebuttal:
We sincerely thank the reviewers for their valuable comments and suggestions regarding our paper titled, “Equivariant Few-Shot Learning from Pretrained Models”. Our paper is motivated by the need to efficiently utilize pretrained models for a variety of downstream applications which benefit from equivariance. To that end, we propose equizero, a zero-shot model that is equivariant and with much better performance than equituning. This is because equizero chooses the best features from pretrained models using a proxy loss function, unlike equitune that simply performs averaging over features. We also show important theoretical properties of this equivariant model like universality, which says that our method still preserves the ability to approximate any equivariant function. We further demonstrate diverse downstream applications of equi-zero over wide-varying tasks like image classification, reinforcement learning, compositional generalization and equivariant CLIP. We also propose $\lambda$-equitune, which is a relaxed generalization of the equizero algorithm and does not require any proxy loss whatsoever. Experiments validate the efficiency of $\lambda$-equitune and equizero over equitune. We next try to alleviate a few key concerns pointed out by reviewers

**Key Concerns**

**Approach limited to finite groups:**

We address the concerns raised by reviewers DNsq and 8HWc regarding the limitations of our approach to finite groups.

Reviewer DNsq also refers [3], which extends our framework to continuous groups by performing symmetric probabilistic averaging. 

First, we gently point out that [3] **appeared online only after the Neurips deadline**. Further, we would like to show there are alternate simple methods to extend $\lambda$-equitune to continuous groups. We describe one such method here.

We simply use the canonicalization method of [1] in conjunction with $\lambda$-equitune to obtain $\textit{canonical}$-$\lambda$-$\textit{equitune}$, which is a) equivariant to continuous groups and b) weighs different features using importance weights to obtain expressive equivariant network.

This idea is based on constructing equivariant frames of [2]. [1] uses a frame of size exactly one, whereas we use a frame of larger size and weigh the features corresponding to different frame elements based on their importance.

A formal definition of our method and its proof of equivariance is provided in the response to reviewer DNsq. 

Further, we conduct an experiment on the SO(2) invariant regression task of [4, Sec. 7. 1]. The results of this experiment are provided in the attached table. It shows that canon-$\lambda$-equitune clearly outperforms non-equivariant model as well as canonicalization method of [1].

Thus, a simple extension of $\lambda$-equitune leads to equivariance to continuous groups.

[1]: Kaba et. al., “Equivariance with Learned Canonicalization Functions”, ICML 2023

[2]: Puny et. al., “Frame Averaging for Invariant and Equivariant Network Design” ICLR 2022

[3]: Kim et. al., “Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance”, ArXiv 2023

[4]: Finzi et. al. "A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups." ICML. 2021.

**Ambiguity about title**

We agree that the focus of the paper can be better described as a combination of zero-shot learning and finetuning rather than few-shot learning. It should be technically called finetuning rather than few-shot learning. We used the umbrella term "few-shot learning" to explain the plethora of application across zero-shot learning and finetuning. However, we admit that changing the title to a better name would be appropriate.

As such, we are happy to change the title of our paper to "Efficient Equivariant Transfer Learning from Pretrained Models", which would be a better title to describe the combination of zeroshot and finetuning. We apologize for any caused confusion and thank the reviewer for pointing this out. We believe this will help readers to understand our work better.

**Image classification experiment for $\lambda$-equitune is a bit naive**

First, we would again emphasize that our method is completely general and provably equivariant for any transformation for which the group actions on the input/output are well-defined. Thus, our experiments focusing on robustness to flips/rotations for pretrained CLIP/CNN models can be easily generalized to any other discrete groups.

Secondly, we emphasize that our work is not restricted to computer vision. Apart from experiments on computer vision (CLIP and CNN-based equivariant/robust classification), our work also shows efficient transfer learning in several other domains such as Fairness in natural language generation (e.g., debiasing and detoxifying GPT2), equivariant deep Q-learning, compositional generalization in languages.

Moreover, recent work [5] shows that equivariance to seemingly naive transformations such as rot90/flip can provide robustness/improvement in performance where the actual transformations in data are much more complicated and not explicitly known. This shows that our experiments with equivariance to seemingly simple transformations can be useful in much more complicated scenarios and could motivate further investigation in this direction in future works.

[5]: D.Wang et. al., The Surprising Effectiveness of Equivariant Models in domains with Latent Symmetry, ICLR 2023

Meta-review: The submission looks into the problem of turning a pre-trained and non-equivariant network into an equivariant network for a downstream equivariant task. It introduces an approach called $\lambda$-equitune which performs a weighted averaging of transformed input embeddings. The approach generalizes equivariant finetuning and optimization-based canonical representation, and special cases of $\lambda$-equitune are shown to be equivalent to the two. Empirical results are presented on a range of applications including reinforcement learning, group-theoretic fairness in language, zero-shot compositional generalization, and image classification.

Reviewers note that the submission tackles an important task (TNLv, DNsq)–equivariant transfer learning in zero-shot and finetuned settings–presents a detailed theoretical analysis of the proposed approach (TNLv, 8HWc), and provides empirical support showing promising results on a diverse set of tasks (TNLv, DNsq, vsSr, 8HWc).

The most salient concerns raised by reviewers are:

* The proposed approach's novelty in the context of Basu et al. (2023) (TNLv). This concern is addressed by the authors' clarification of their contributions in the rebuttal.
* The image classification setup appears naive (TNLv). The authors emphasize in their response that their method is "completely general and provably equivariant for any transformation for which the group actions on the input/output are well-defined", and that the experiments can be generalized to any other discrete group, when applicable. This addresses Reviewer TNLv's concern.
* The term "few-shot learning" in the title and text is not reflective of the experiments carried out in the paper (DNsq). The authors acknowledge the concern in their response and offer to change the paper's title and wording to address it.
* The approach is only applicable to small, finite groups (8HWc, DNsq). In their response to Reviewer DNsq, the authors outline a strategy for extending their framework to continuous groups through the canonicalization of Kaba et al. (2022) and provide empirical results on this strategy. This addresses the reviewers' concerns.

All reviewers note in their replies to the authors that they are in favor of acceptance.
Decision: Accept (poster)