Meta-learning Symmetries by Reparameterization
Authors: Allan Zhou
Tom Knowles
Chelsea Finn
Venue: ICLR 2021 Poster (2021)
Keywords: meta-learning
equivariance
convolution
symmetry

Abstract: Many successful deep learning architectures are equivariant to certain transformations in order to conserve parameters and improve generalization: most famously, convolution layers are equivariant to shifts of the input. This approach only works when practitioners know the symmetries of the task and can manually construct an architecture with the corresponding equivariances. Our goal is an approach for learning equivariances from data, without needing to design custom task-specific architectures. We present a method for learning and encoding equivariances into networks by learning corresponding parameter sharing patterns from data. Our method can provably represent equivariance-inducing parameter sharing for any finite group of symmetry transformations. Our experiments suggest that it can automatically learn to encode equivariances to common transformations used in image processing tasks.

Published as a conference paper at ICLR 2021
META-LEARNING SYMMETRIES BY REPARAMETERI- ZATION
Allan Zhou, Tom Knowles, Chelsea Finn Dept of Computer Science, Stanford University (ayz, tknowles, obf intiiimanoorddeeeee
ABSTRACT
Many successful deep learning architectures are equivariant to certain transfor- mations in order conserve parameters and improve generalization: most fa- mously, convolution layers are equivariant to shifts of the input. This approach only works when practitioners know the symmetries of the task and can manually construct an architecture with the corresponding equivariances. Our goal is an approach for learning equivariances from data, without needing to design custom task-specific architectures. We present a method for learning and encoding equiv- ariances into networks by learning corresponding parameter sharing patterns from data. Our method can provably represent equivariance-inducingg parameter shar- ing for any finite group of symmetry transformations. Our experiments suggest that it can automatically learn to encode equivariances to common transforma- tions used in image processing tasks. We provide our experiment code at phtttps: //githubs com/ A annaangnhoo/mmtttaa learni ng- symmsyymettt r ries
INTRODUCTION
In deep learning, the convolutional neural network (CNN) (LeCun et al. 1998) is prime example of exploiting equivariance to a symmetry transformation to conserve parameters and improve gen- eralization. In image classification (Russakovsky et al., 2015 Krizhevsky et al., (012) and audio processing (Graves and Jaitly] 2014 Hannun et al.) 2014) tasks, we may expect the layers of deep network to learn feature detectors that are translation equivariant: if we translate the input, the output feature map also translated. Convolution layers satisfy translation equivariance by definition, and produce remarkable results these tasks. The success of convolution's "built in" inductive bias suggests that we can similarly exploit other equivariances to solve machine learning problems.
However, there are substantial challenges with building in inductive biases. Identifying the correct biases to build in is challenging, and even if we do know the correct biases, it is often difficult to build them into neural network. Practitioners commonly avoid this issue by "training in" de- sired equivariances (usually the special case of invariances) using data augmentation. However, data augmentation can be challenging in many problem settings and would prefer to build the equiv- ariance into the network itself. For example. robotics sim2real transfer approaches train agents that are robust to varying conditions by varying the simulated environment dynamics (Song al. 2020). But this type of augmentation is not possible once the agent leaves the simulator and is trying to learn or adapt to a new task in the real world. Additionally, building in incorrect biases may actually be detrimental to final performance (Liu et al. 2018b).
In this work we aim for an approach that can automatically learn and encode equivariances into neural network. This would free practitioners from having to design custom equivariant architectures for each task, and allow them to transfer any learned equivariances to new tasks. Neural network layers achieve various equivariances through parameter sharing patterns, such the spatial parameter sharing of standard convolutions. learnably represent sharing patterns. We leverage meta-learning to learn the sharing patterns that help a model generalize on new tasks.
this paper we reparameterize network layers to
The primary contribution of this paper is approach to automatically learn equivariance-induee parameter sharing, instead of using custom designed equivariant architectures. We show theoreti- cally that reparameterization represent networks equivariant to any finite symmetry group. Our

---

Published as a conference paper at ICLR 2021
experiments show that meta-learning can recover various convolutional architectures from data, and learn invariances to common data augmentation transformations.
RELATED WORK
A number of works have studied designing layers with equivariances to certain transformations such as permutation, rotation, reflection, and scaling (Gens and Domingos, 2014 Cohen and Welling 2016 Zaheer et al., 2017; Worrall et al., 2017. Cohen et al.J [2019] Weiler and Cesa [019] Worrall and Welling 2019). These approaches focus on manually constructing layers analagous to standard convolution, but for other symmetry groups. Rather than building symmetries into the architecture, data augmentation (Beymer and Poggio 1995] Niyogi et al. 1998) trains a network to satisfy them. a hybrid approach that pre-trains basis of rotated filters in order to define roto-translation equivariant convolution. Unlike these works, we aim to automatically build in symmetries by acquiring them from data.
Diaconu and Worrall (2019)
Our approach is motivated in part by theoretical work characterizing the nature of equivariant layers for various symmetry groups. In particular, the analysis of our method as learning a certain kind of convolution is inspired by Kondor and Trivedi (2018), who show that under certain conditions all linear equivariant layers are (generalized) convolutions. Shawe-TTyyorr (1989) and Ravanbakhsh et al.] (2017) analyze the relationship between desired symmetries in a layer and symmetries of the weight matrix. [Ravanbakhsh et al. (2017) show that we can make a layer equivariant to the permutation representation of any discrete group through a corresponding parameter sharing pattern in the weight matrix. From this perspective, our reparameterization is way of representing possible parameter sharing patterns, and the training procedure aims to learn the correct parameter sharing pattern that achieves a desired equivariance.
Prior work on automatically learning symmetries include methods for learning invariances in Gaus- sian processes (van der Wilk et al. 2018) and learning symmetries of physical systems (Greydanus) et al. 2019; Cranmer al. 2020). Another very recent line of work has shown that more general Transformer (Vaswani et al. 2017) style architectures can match or outperform traditional CNNs on image tasks, without baking translation symmetry (Dosovitskiy al. 2020). Their results sug- gest that Transformer architectures can automatically learn symmetries and other inductive biases from data, but typically only with very large training datasets. One can also consider automatic data augmentation strategies (Cubuk et al. 2018 Lorraine et al. 2019) as way of learning symmetries, though the symmetries are not embedded into the network in a transferable way. Concurrent work by Benton et al. (2020) aims to learn invariances from data by learning distributions over transforma- tions of the imput, similar to learned data augmentation. Our method aims to learn parameter sharing of the layer weights which induces equivariance. Additionally, our objective for learning symme- tries is driven directly by generalization error (in a meta-learning framework), while the objective in Benton et al. (2020) adds regularizer to the training loss encourage symmetry learning.
Our work is related to neural architecture search (Zoph and Le 2016 Brock et al.) 2017 Liu et al. 801 Elsken al., 2018), which also aims automate part of the model design process. Although architecture search methods are varied, they are generally not designed to exploit symmetry or learn cquivariances. Evolutionary methods for learning both network weights and topology (Stanley and
Mikkulainen] 2002: Stanley et al. 2009) are also not motivated by symmetry considerations.
Our method learns to exploit symmetries that are shared by collection of tasks, form of meta- learning (Thrun and Pratt 2012 Schmidhuber 1987; Bengio et al.) 1992 Hochreiter et al. 200T) We extend gradient based meta-leaming (Finn et al., 2017 Li et al.
2017 Antoniou et al. 2018)
to separately learn parameter sharing patterns (which enforce equivariance) and actual parameter values. Separately representing network weights in terms of sharing pattern and parameter values is a form of reparameterization. Prior work has used weight reparameterization in order to "warp" the loss surface (Lee and Choi 20188 Flennerhag et al., (019) and to learn good latent spaces (Rusu et al. (0018) for optimization, rather than to encode equivariance. HyperNetworks (Ha et al. 2016 Schmidhuber 1992) generate network layer weights using a separate smaller network, which can be viewed as nonlinear reparameterization. albeit not that encourages learning equivariances. Modular meta-lcarning (Alet et al., (018)) is a related technique that aims to achieve combinatorial generalization on new tasks by stacking meta-learned "modules," each of which is neural network.
2

---

Published as a conference paper at ICLR 2021
This can be parameter sharing by re-using and combining modules, rather than using layerwise reparameterization.
PRELIMINARIES
In Sec.
3.1 we review gradient based meta-learning, which underlies our algorithm. Sections 3.2
and 3.3 build up formal definition of equivariance and group convolution (Cohen and Welling 2016 a generalization of standard convolution which defines equivariant operations for other groups such as rotation and reflection. These concepts are important for theoretical understanding of our work as a method for learning group convolutions in Sec. 4.2
3.1 GRADIENT BASED META LEARNING
Our method is gradient-based meta-learning algorithm that extends MAMI. (Finn et al. 2017), which we briefly review here. Suppose we have some task distribution p(T), where each task dataset split into training and validation datasets (D D. J. For model with parameters 0, loss and learning rate a, the "inner loop" updates 0 on the task's ' training data: 0' = 0 - a ( D In the "outer loop." MAML meta-learns good initialization
by minimizing the loss 6' on the task's validation data, with updates of the form 6 + 0 ") ae L(0', D Although MAML focuses on meta-learning the inner loop initialization such the inner learning rate In our method, we meta-learn a parameter sharing pattern at each layer that maximizes performance across the task distribution.
one can extend this idea to meta-learning other things
3.2 GROUPS AND GROUP ACTIONS
Symmetry and equivariance is usually studied in the context of groups and their actions sets; refer to Dummmi and Foote (2004) for more comprehensive coverage. A group G is a set closed under some associative binary operation, where there is identity element and each element has inverse. Consider the group (Z, +) (the set of integers with addition): we can add any two integers obtain another, each integer has an additive inverse, and 0 is the additive identity.
A group G can act on a set X through some action p G Aut(X) which maps each G to some transformation on X. p must be homomorphism, i.e. p(gh) p()) for all g, h E G, and Aut(X) X) is the set of au- tomorphisms on X (bijective homomorphisms from X to itself). As shorthand gæ = (z) for any I E X. Any group can act itself by letting X G: for (Z, +), we define the action gr = g + x for any 9. E Z.
W y
X
a b C a b a b (weights)
YI
X
Va
Ya
write
X
(fibeer)
X2
Ya
The action of group G V is called a representation. which we denote G GI. Recall GT((V) is the set of invertible linear maps on V. Assume the vec- tors v V are discrete, with components v[i]. If we already have G's action on the indices, natural corresponding representation is defined (T()))][i == v(g- 'i]. As a concrete example, consider the representation
vector space
Figure Convolution as translating filters. Left: Standard 1-D convolution slides a filter w along the length of input æ. This operation is translation equivari- ant: translating will translate y. Right: Standard con- volution is cquivalent
fully connected layer with parameter sharing pattern: each row contains translated copies of the filter. Other cquivariant layers will have their own sharing patterns.
G (Z,+) for infinite length vectors. The indices are also integers, so the group is acting itself as defined above. Then (I(g)o({i = vlg-11 vfi g] for any g. Z. Hence this representation shifts vectors by translating their indices by g spaces.
3.3 EQUIVARIANCE AND CONVOLUTION
A function (like neural network layer) is equivariant to transformation if transforming the function's imput is the same as transforming its output. To be more precise, we must define what those transformations of the input and output are. Consider neural network layer R - R
3

---

Published as a conference paper at ICLR 2021
Assume we have two representations N1, 22 of group G on R" and R", respectively. For each g E G, (i(g) transforms the imput vectors, while T2(g) transforms the output vectors. The layer is G-equivariant with respect to these transformations if (( (g)v) T2(g))(v), for any g G, v. E IR". If we choose T22 = id we get ((m((g))) = Φ(v), showing that invariance is a type of equivariance.
Deep networks contain many layers, but function composition preserves equivariance. So we achieve equivariance in each individual layer, the whole network will be equivariant. Pointwise nonlinearities such as ReLU and sigmoid are already equivariant to any permutation of the input and output indices, which includes translation, reflection, and rotation. Hence we are primarily focused on enforeing equivariance in the linear layers.
Prior work (Kondor and Trivedi 2018) has shown that linear layer & is equivariant the action of some group if and only if it is a group convolution, which generalizes standard convolutions to arbitrary groups. For a specific G. we call the corresponding group convolution "G-convolution" to distinguish it from standard convolution. Intuitively, G-convolution transforms filter according to each 9 E G. then computes a dot product between the transformed filter and the input. In standard convolution, the filter transformations correspond to translation (Fig. [ G-equivariant layers convolve an input v E R" with a filter v E R". Assume the group G {9i Im is finite:
<w) = -w M = Σ [+ Σ (11) (1)
(1)
In this work, we present a method that represents and learns parameter sharing patterns for existing layers, such as fully connected layers. These sharing patterns can force the layer to implement various group convolutions, and hence equivariant layers.
ENCODING AND LEARNING EQUIVARIANCE
To learn equivariances automatically, our method introduces flexible representation that can encode possible equivariances, and an algorithm for learning which equivariances to encode. Here we describe this method, which we call Meta-learning Symmetries by Reparameterization (MSR).
4.1 LEARNABLE PARAMETER SHARING
As Fig. shows, fully connected layer implement standard convolution if its weight matrix is constrained with a particular sharing pattern, where each row contains a translated copy of the same underlying filter parameters. This idea generalizes equivariant layers for other transformations like rotation and reflection, but the sharing pattern depends on the transformation. Since we do not know the sharing pattern priori, we "reparameterize" fully connected weight matrices represent them in a general and flexible fashion. A fully connected layer & R" - Rm with weight matrix W E m x is defined for input x by <xx) Wx. We can optionally incorporate biases by appending dimension with value "1" to the input We factorize W as the product of a "symmetry matrix" U and vector v of k "filter parameters":
vec(W) Uv, V VER E
For fully connected layers, we reshapcTthee vector vec(W') E Rmm into a weight matrix W E Rm Intuitively, U encodes the pattern by which the weights W will "share" the filter parameters v. Crucially, we can now separate the problem of learning the sharing pattern (learning U) from the problem of learning the filter parameters v. In Sec. 4.3 we discuss how to learn U from data.
The symmetry matrix for each layer has mak entries, which can become too expensive in larger layers. Kronecker factorization is a common approach for approximating a very large matrix with smaller ones (Martens and Grosse 2015 Park and Oliva 2019). In Appendix [A we describe how we apply Kronecker approximation to Eq. and analyze memory and computation efficiency.
In practice, there are certain equivariances that expensive meta-leam, but that know to be useful: for example, standard 2D convolutions for image data. However, there may be still other symmetries of the data (i.e., rotation, scaling, reflection, etc.) that still wish to learn
We will the atypical convention that vec stacks matrix entries row-wise, not column-wise.
4

---

enforce various equivariances, which will be preserved regardless of Published as a conference paper at ICLR 2021
U
vec (W)
w
(<(e)
0
0
1
reshape
0
((g)
1
parameters
layer
weights
group
symmmetry
represontation matrix
Figure 2: We reparameterize the weights of each layer terms symmetry matrix can enforce equiv- ariant sharing patterns of the filter parameters v. Here we show U that enforces permutation equivariance. More technically, the layer implements group convolution on the permutation group Sy: U"s block submatrices (e), (g) define the action of cach permutation filter Note that U need not be binary in general.
Algorithm 1: MSR: Meta- Training
vec W) Uv
input: (Tr)==1 p(T): Meta-training tasks
brver loop
Imput:
I
Uses:
input: (U,v): Randomly initialized symmetry
X
matrices and filters.
(other lugers)
input: a, 17. Inner and outer loop step sizes.
while not done do
v'a LV
sample minibatch (T:
(Ti N
Outer loop
forall T; E (T:) i= do
Input:
losss
U
L cll
{D D + T;; // task data
(other lugers)
6, - V.C,U,v,D D(");
v v ad;;
inner step / outer gradient x/
Figure 3: For each task, the inner loop updates the filter parameters u the task using the inner loop loss. Note that the symmetry matrix U does not change in the inner loop, and is only updated by the outer loop.
G, + a v(U,
( outer step */
U 4 U n G;
automatically. This suggests "hybrid" approach, where bake-in equivariances know be useful, and learn the others. Indeed, we can directly reparameterize a standard convolution layer by reshaping vec(W) (W) into convolution filter bank rather than weight matrix. By doing so bake in translational equivariance, but we can still learn things like rotation equivariance from data.
4.2 PARAMETER SHARING AND GROUP CONVOLUTION
By properly choosing the symmetry matrix of Eq we can force the layer to implement arbitrary group convolutions (Eq. 1 by filter Recall that group convolutions generalize standard convolu- tion to define operations that are equivariant to other transformations, such as rotation. Hence by choosing U properly the value of c.
Proposition 1 Suppose G is a finite group (91..... Jm. There exists UG RmMxx such that for any u E R", the layer with weights vec(W) U implements G-convolution on input x E IR". Moreover, with this fixed choice of UG, any G-convolution can be represented by weight matrix vec(W)- = T for some v E T
Intuitively, U can store the symmetry transformations T(g) for each g E G. thus capturing how the filters should transform during G-convolution. For example, Fig. 2 shows how U can implement convolution on the permutation group Sz. We present a proof Appendix
Subject to having correct UG, is precisely the convolution filter in G-convolution. This will motivate the notion of separately learning the convolution filter v and the symmetry structure in the inner and outer loops of meta-learning process, respectively.
5

---

Published as a conference paper at ICLR 2021
Synthetic Problems MSE (lower better)
Small train dataset
Large train dataset
Method
1: = 1
k = 2
== 5
k
k 2
== 5
MAML-FC
3.4 + .60
2.1 + .35
1.0
3.4 .49
2.0
.27
1.1 1.1
MAML-LC
2.9 .53
1.8 .24
1 .08
2.9 = .42
1.6
.89 .08
MAML-Conv
.00 + .00
.43 .09
411: .04
.00 = .00
.53 .08
.99 1 .04
MTSR-FC (Ours)
3.2 49
.86 1: .06
.12 = .03
107:.02
.07 .01
MSR-Joint-FC (Ours)
.25 + .16
.12 +
.04
.21
+
.03
.01 .00
.08 0.80002
.12 .02
MSR-FC (Ours)
.07 + .22
.07 +
.02
.16
.02
.00 .0
.09
Table 1: Meta-test MSE of different methods synthetic data with (partial) translation symmetry. "Small" vs "large" train datasct refers the number of examples per training task. Among methods with non-convolutional architectures, MSR-FC closest to matching actual convolution (MAML-Cony performance on translation equivariant (k: = 1) data. On data with less symmetry (k = 2, 5), MSR-FC outperforms MAML-Conv and other MAML approaches. MSR-Joint is an ablation of MSR where both U and v of Eq. 2 are updated on task train data, rather than just MTSR is ablation of MSR where train the reparameterization using multi-tas learning, rather than meta-leaming. Results are shown with 95% confidence intervals over test tasks.
4.3 META-LEARNING FOUIVARIANCES
Meta-learning generally applies when want learn and exploit some shared structure in dis- tribution of tasks p(7). In this case, we assume the task distribution has some common underlying symmetry: i.e., models trained for each task should satisfy some set of shared equivariances. We extend gradient based meta-learning to automatically learn those equivariances.
Suppose we have L- L-layer network. We collect each layer's symmetry matrix and filter parameters: U.V+ V. + {U_,... U²}, }, {w², ...,.... vzj. Since we aim to learn equivariances that are shared across the symmetry matrices should not change with the task. Hence, for any T₁ p(T) the inner loop fixes U and only updates v using the task training data:
v avv-av,LCC
where L is simply the supervised learning loss, and is the inner loop step size. During meta- training. the outer loop updates U by computing the loss on the task's validation data using v':
d U U a C(U, vD.a)
(4)
We illustrate the inner and outer loop updates in Fig. 3 Note that in addition to meta-learning the symmetry matrices, we also take outer updates averaged over batches of tasks, as we describe in Alg.
also still meta-learn the filter initialization
in prior work. In practice
After meta-training is complete, we freeze the symmetry matrices U. On new test task Ta p( P(T), we use the inner loop (Eq. 3 to update only the filter v. The frozen U enforces meta-learned parameter sharing in each layer, which improves generalization by reducing the number of task- specific inner loop parameters. For example, the sharing pattern of standard convolution makes the weight matrix constant along any diagonal, reducing the number of per-task parameters (see Fig.
CAN WE RECOVER CONVOLUTIONAL STRUCTURE?
We now introduce series of synthetic meta-learning problems, where each problem contains regres- sion tasks that are guaranteed to have some symmetries, such as translation, rotation, or reflection. We combine meta-learning methods with general architectures not designed with these symmetries in mind to see whether each method can automatically meta-learn these equivariances.
5.1 LEARNING (PARTIAL) TRANSLATION SYMMETRY
Our first batch of synthetic problems contains tasks with translational symmetry: we generate out- puts by feeding random input vectors through 1-D locally connected (LC) layer with filter size 3 and no bias. Each task corresponds to different values of the LC filter, and the meta-learner must minimize mean squared emror (MSE) after observing single input-output pair. For each problem we constrain the LC filter weights with a rank E (1,2,5} factorization, resulting in partial translation symmetry (Elsayed et al. 2020). In the case where rank k 1, the LC layer

---

Published as a conference paper at ICLR 2021
is equivalent to convolution (ignoring the biases) and thus generates exactly translation equivari- ant task data. We apply both MSR and MAML to this problem using a single fully connected these models have translation equivariance built in and
layer (MSR-FC and MAML-FC), must meta-learn to solve the tasks efficiently. For comparison, we also train convolutional and locally connected models with MAML (MAML-CCnn and MAML-LC). Since MAML-Cony has built in translation equivariance, we expect it to at least perform well on the rank k = 1 problem. We also ran two ablations of MSR that use the same reparam- eterization (Eq. [ but vary the training procedure, In MSR- Joint, we allow U and v to be jointly updated in the inner loop, instead of only updating v in the inner loop. Hence MSR- Joint is trained identically to MAMI,, but with reparameter- ized weights. MTSR is an ablation of MSR that trains using multi- task learning instead meta-learning. Given data from training task T, MTSR jointly optimizes U (shared symme- try matrix) and (1) (task specific filter parameters) using the MSE loss. For a new test task we freeze the optimized and optimize newly initialized filter using the test task's 's train- ing data, then evaluate MSE on held out data. Even though the true filter that generates the data has width 3. for MSR and MTSR we initialize the learned filter v to be the same size as the input, per Prop.[] [ In principle, these methods should auto- matically meta-learn that the true filter is sparse, and to ignore the extra dimensions in Appendix further explains the experimental setup.
Figure 4: After observing transla- tion equivariant data, MSR enforces convolutional parameter sharing on the weight matrix. An example weight matrix is shown above.
Table D shows how each method performs on each of the synthetic problems, with columns denoting the rank l of the problem's 's data. The "small large train dataset" results differ only in that the latter contains 5 or 10 times more examples per training task, depending on k. On fully translation equivariant data
1), MAML-Conv performs best due to architecture having built in trans- lation equivariance. MSR-FC is the only non-convolutional architecture to perform comparably to MAML-Conv for & 1. Fig. shows that MSR-FC has learned produce weight matrices with convolutional parameter sharing structure, indicating it has "learned convolution" from the data. Appendix C.T visualizes the meta-learned U, which we find implements convolution as Sec. 4.2 predicted. Meanwhile, MAML-FC and MAML-LC perform significantly worse they are unable to meta-learn this structure. On partially symmetric data (k 2, k 5). MSR-FC performs well due to its ability flexibly meta-learn even partial symmetries. MAML-Conv performs worse here since the convolution assumption is overly restrictive, while MAML-FC. and MAMI--IC are not able to meta-learn much structure. MSR-Joint-FC performs comparably to worse than MSR-FC across the board. Note that following prior work (Li et al. 2017). 2017). all methods use meta-learned inner learning rates on parameters that change in the inner loop. For MSR-Joint-FC we observe that the meta-learned inner loop learning rates corresponding to U are significantly smaller than the inner learning rates corresponding to v. suggesting that U is changing relatively little in the inner loop (see Appendix T).le MTSR-FC performs significantly worse than MSR-FC with small training datasets, but performs comparably with large datasets. This indicates that although our reparame- terization can be trained by either multi-task learning meta-learning, the meta- meta-learning approach (Alg. I is more efficient learning from less data.
5.2 LEARNING EQUIVARIANCE TO ROTATIONS AND FLIPS
We also created synthetic problems with 2-D synthetic im- age inputs and outputs, in order to study rotation and flip equivariance. We generate task data by passing randomly generated inputs through single layer E(2)-equivariant steerable CNN (Weiler and Cesal 2019) configured to be cquivariant to combinations of translations, discrete rota- tions by increments of 45", and reflections. Hence our synthetic task data contains rotation and reflection in ad- dition to translation symmetry. Each task corresponds to different values of the tata-ggnn network's 's weights. We apply MSR and MAML. to single standard con-
Rotation/Flip Equivariance MSE
Method
Rot
Rot+Flip
MAML-Conv
.544
.507
MSR-Conv (Ours)
.004
.001
Table 2: MSR learns rotation and flip equivariant parameter sharing top of standard convolution model, and thus achieves much better generalization error on meta-test tasks compared to MAML on rotation and flip equivariant data.

---

Published as a conference paper at ICLR 2021
volution layer, which guarantees translation equivariance. Each method must still meta-learn rotation and reflection (flip) equivariance from the data. Table 2 shows that MSR easily learns rotation and rotation+reflection equivariance top of the convolu- tional model's built in translational equivariance. Appendix C22 visualize the filters MSR produces, which we see rotated and/or flipped versions of the same filter.
6 CAN WE LEARN INVARIANCES FROM AUGMENTED DATA?
Practitioners commonly use data augmentation train their models to have certain invariances. Since invariance is a special case of equivari- ance, we can also view data augmentation as a way of learning equivariant models. The downside is that we need augmented data for each task. While augmentation is often possible during meta-training, there are many situations where it is impractical at meta-test time. For example, in robotics we may meta-train a robot in simulation and then deploy (meta-test) in the real world, a kind of sim2real transfer strategy (Song al. 2020). During meta-training we can augment data using the simulated environ- ment, but we cannot do the same meta-test time in the real world. Can we instead use MSR to learn equivariances from data augmentation at training time, and encode those learned equivari- ances into the network itself? This way, the network would preserve learned equivariances on new meta-test tasks without needing any additional data augmentation.
Algorithm 2: Augmentation Meta- Seta-Training
input: {Te) 1 Meta-training tasks
input:
Any meta-leamerr
input: AUGMENT: Data augmenter
forall Ti E (T:) do
(D Dr + Te: // task data
split
D.ae + AUGMENT (Duul))
T. + (D'r Draa)
META -RRAI (T.EE))
Alg. describes our approach for meta-learning invariances from data augmentation, which wraps around any meta-learning algorithm using generic data augmentation procedures. Recall that each task is split into training and validation data T, = {D(r Dr We use the data augmentation procedure to only modify the validation data, producing new validation dataset Di for each task. We re- re-assemble each modified task T, + {D! Dy So for each task, the meta-learner observes unaugmented training data, but must generalize to augmented validation data. This forces the model to be invariant to the augmentation transforms without actually seeing any augmented training data.
We apply this augmentation strategy to Omniglot (Lake et al., 2015) and Minilmagenet (Vinyals) et al. (016)) few shot classification to create the Ongnigoo and Sug-Miinlmaggnne benchmarks. Our data augmentation function contains a combination of random rotations, flips, and resizes (rescaling), which we only apply task validation data as described above. The problem is set up analogous to (Finn et al. 2017): for each task, the model must classify images into one of either 5 20 classes (n-way) and receives either or 5 examples of each class in the task training data (k-shot). Unlike Finn et al. (2017) our Aug-Omniglot and Aug-Minilmagennt benchmarks contain transformed task validation data.
We tried combining Alg. A with our MSR method and three other meta-learning algorithms: MAML (Finn al. 20017). ANIL RRaghu al. 2011), and Prototypical Networks (ProtoNets) (Snell al., 2017). While the latter three methods all have the potential to learn equivariant features through Alg. 2 we hypothesize that since MSR enforces learned equivariance through its symmetry matrices it should outperform these feature- metallarnttnnnn methods. We also paired MAML with a model that has built in equivariance the group D8 (45*-incremen) rotation and reflections) using the E2-CNI library (Weiler and Cesal 2019). We call this baseline "MAML+D8"". Appendix D.3 describes the experimental setup and methods implementations in more detail.
Table 3 shows each method's meta-test accuracies on both benchmarks. Across different settings MSR performs either comparably to the best method, or the best. MAML and ANIL perform sim- ilarly to each other, and usually worse than MSR, suggesting that learning equivariant or invariant features is not as helpful as learning equivariant layer structures. ProtoNets perform well on the eas- ier Aug- -mn-iiotlo benchmark, but evidently struggle with learning transformation invariant metric space on the harder Aug Migiilmmaaeeet problems. MSR even outperforms the architecture with built in rotation and reflection symmetry (MAML+D8) across the board. MSR's advantage may be due
8

---

Published as a conference paper at ICLR 2021
Aug-Ommn Hug-Omniglot
Iug-Minilmagenet
5 way
20 way
way
Method
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
MAML
87.3 0.5
98.6
0.3
67.0 + 0.4
79.9 + 0.3
42.5 = 1.1
61.5 - 1.0
MAML (Big)
89.3 + 0.4
94.8 + 0.3
69.6 0.4
83.2
37.2 = 1.1
63.2 = 1.0
ANIL
86.4 0.5
0.3
47.5
79.8 0.3
43.0 + 1.1
62.3 + 1.0
ProtoNets
92.9 0.4
0.2
85.1 0.3
94.3
34.6 1 0.5
54.5 0.6
MAML +
94.6 + 0.4
96.4 0.3
82.6 0.3
85.1
0.3
14.9 - 1.2
56.8 + 1.1
MSR (Ours)
95.3 + 0.3
97.7 + 0.2
84.3
0.2
92.6
+ 0.2
45.5 + 1.1
65.2 + 1.0
Table 3: Meta-test accuracies on aug-Omiiglot and aug-Minimaagent few-shot classification. These bench- marks test generalization to augmented validation data from un-augmented training data. MSR performs com- parably to or better than other methods under this augmented regime. Results are shown with 95% confidence intervals over test tasks.
to the additional presence of scaling transformations in the image data; we are not aware of archi- tectures that build in rotation, reflection, and scaling equivariance at the time of writing. Note that MSR's reparameterization increases the number of meta-learned parameters at each layer, so MSR models contain more total parameters than corresponding MAML models. The "MAML (Big)" results show MAML performance with very large models containing more total parameters than the corresponding MSR models. The results show that MSR also outperforms these larger MAML models despite having fewer total parameters.
7 DISCUSSION AND FUTURE WORK
We introduce a method for automatically meta-leaming equivariances in neural network models, by encoding learned equivariance-induenng parameter sharing patterns in each layer. On new tasks, these sharing patterns reduce the number of task-specific parameters and improve generalization. Our experiments show that this method can improve few-shot generalization on task distributions with shared underlying symmetries. We also introduce a strategy for meta-training invariances into networks using data augmentation, and show that it works well with our method. By encoding equiv- ariances into the network as a parameter sharing pattern, our method has the benefit preserving efficiently.
learned equivariances
tasks so it can learn
Machine learning thus far has benefited from exploiting human knowledge of problem symmetries, and we believe this work presents step towards learning and exploiting symmetries automatically. This work leads to numerous directions for future investigation. In addition to generalization ben- efits, standard convolution is practical since it exploits the parameter sharing strueture improve computational efficiency, relative to a fully connected layer of the same input/output dimensions. While MSR can improve computational efficiency by reparameterizing standard convolution layers, it does not exploit learned structure to further optimize its computation. Can we automati- cally learn or find efficient implementations of these more structured operations? Additionally, MSR is focused on learning finite symmetry groups, while approximating infinite ones (e.g., learning 45%. increment rotation symmetry as an approximation to continuous rotation symmetry). Unfortunately, the number of parameters increases with the resolution of the approximation, so further research would be useful in discovering more scalable methods of approximating and learning continuous symmetries. Finally, our method is best for learning symmetries which are shared across a distribu- tion of tasks. Further research on quickly discovering symmetries which are particular to a single task would make deep learning methods significantly more useful many difficult real world prob- lems.
ACKNOWLEDGEMENTS
We would like to thank Sam Greydanus, Archit Sharma, and Yiding Jiang for reviewing and cri- tiquing earlier drafts of this paper. This work was supported in part by Google. CF a CIFAR Fellow.

---

Published as a conference paper at ICLR 2021
REFERENCES
F. Alet, T. Lozano-Pérez, and L. P. Kaclbling. Modular meta-learning. arXiv: 10166, 2018.
arXiv preprint
A. Antoniou, H. Edwards, and A. Storkey. How to train your maml. arXiv: (8t.:098000052 2018.
arXiv preprint
Bengio, Bengio, J. Cloutier, and Geesei. On the optimization synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, volume 2. Univ. of Texas, 1992.
G. Benton, M. Finzi, P. Izmailov, and A. G. Wilson. Learning invariances in neural networks. arXiv preprint arXiv:2010.1188 2020.
D. Beymer and T. Poggio. Face recognition from one example view. In Proceedings of IEEE International Conference Computer Vision, pages 500-507. IEEE, 1995.
A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv:1708.05344, 2017.
T. Cohen and M. Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-2999, 2016.
T. S. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling. Gauge equivariant convolutional networks and the icosahedral cnn. arXiv preprint TrXivv1902.000 04615, 2019.
M. Cranmer, S. Greydanus. S. Hoyer, P. Battaglia, D. Spergel, and S. Ho. Lagrangian neural net- works. arXiv preprint arXiv:2003. 046600, 2020.
E. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, and Q V. Le. Autoaugment: Learning augmentation policies from data. coir 1ss/1805.09501 (2018). arXiv preprint arXiv:1805.05001, 2018.
Deleu, T. Würff, M. Samiei, J. Cohen, and Y. Bengio. Torchmeta: Meta-Learning li- brary for Py PyTorch, 2019. URL https::/ //ap::://aaa /tt////rxivvv wrr..i..r///b//1900000 06576 Available at:
N. Diacomu and D. E. Worrall. Learning to convolve: A generalized weight-tying approach. arXiv preprint arXiv: 11900006663. 2019.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recogmition at scale. arXiv preprint urXiv:2010.11929 2020.
D. S. Dummit and R. M. Foote. Abstract algebra, volume 3. Wiley Hoboken, 2004.
G. F. Elsayed, P. Ramachandran, J. Shlens, and S. Kornblith. Revisiting spatial invariance with low-rank local connectivity. arXiv preprint arXiv:2002.02959 2020.
T. Elsken, J. H. Metzen, and F. Hutter. Neural architecture search: A survey. arXiv preprint arXiv:1808.00 053.777777 2018.
C. Finn, P. Abheel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep net- works. In Proceedings of the 34th International Conference on Machine Learning-Volurr Volume 70, pages 1 1126-1135. JMLR. org, 2017.
S. Flennerhag, A. A. Rusu, R. Pascanu, H. Yin. and R. Hadsell. Meta-learning with warped gradient descent. arXiv preprint arXiv:I909.000 1909000025, 2019.
R. Gens and P. M. Domingos. Deep symmetry networks. In Advances in neural information pro- cessing systems, pages 2537-2545, 2014.
A. Graves and N. Jaitly. Towards end-to-end speech recognition with recurrent neural networks. In International conference machine learning, pages 1764-1772, 2014.
10

---

Published as a conference paper at ICLR 2021
E. Grefenstette, B. Amos, D. Yarats, P. M. Htut, A. Molchanov, F. Meier, D. Kiela, K. Cho, and S. Chintala. Generalized inner loop meeaalearrnnn. arXiv preprint arXiv:1910001727, 2019.
S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pages 15353-15365 2019.
D. Ha, A. Dai, and Q. Le. Hypernetworks. arXiv preprins 1rXiv: 2016.
A. Hamnun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh. S. Sen- gupta, A. Coates, et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv: 1412.5566,, 2014.
S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pages 87-94. Springer, 2001.
D. P. Kingma and J. Ba. arXiv: 1412.690, 2014.
Adam: A method for stochastic optimization.
arXiv preprint
T. G. Kolda and B. W. Bader. Tensor decompositions and applications. SIAM review, 51(3):455-500, 2009.
R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups, 2018.
A. Krizhevsky, Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through proba- bilistic program induction. Science, 35016266):1332-1333, 2015.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)) 22224,,,,,, 1998.
Y. Lee and Choi. brasee meta-lcarning with learned layerwise metric and subspace. arXiv preprint arXiv: 0111800105558.. 2018
Z. Li, Zhou, F. Chen, and H. Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.098833 2017.
H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. arXiv: 1806.0050 2018a.
arXiv preprint
R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. In Advances Neural Information Processing Systems, pages 9605-9616, 201 8b.
J. Lorraine, P. Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differ- entiation. arXiv preprint arXiv:1911.022 2019.
J. Martens and R. Grosse. Optimizing neural networks with kronecker-factored approximate curva- ture. In International conference on machine learning, pages 2408-2417, 2015.
P. Niyogi, F. Girosi, and T. Poggio. Incorporating prior information in machine learning by creating virtual examples. Proceedings of the IEEE, 86(11):2196-2209 1998.
E. Park and J. B. Oliva. Meta-curvature. In Advances in Neural Information Processing Systems, pages 3309-3319, 2019.
A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards under- standing the effectiveness of maml. arXiv preprint 19X00:1909001177 2019.
S. Ravanbakhsh, J. Schneider, and B. Poczos. Equivariance through parameter-sharing In Proceed- ings of the 34th International Conference on Machine Learning Volume 70, pages 2892-2901 JMLR org, 2017.
11

---

Published as a conference paper at ICLR 2021
O. Russakovsky, J. Deng, H. Su, J. Krause. S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2111-52, 2015.
A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta- learning with latent embedding optimization. arXiv preprint (rxXiv/807.009900 2018.
J. Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The meta-meta-... hook.) Diploma thesis, Institut f. Informatik, Tech. Univ. Munich, 1:2, 1987.
Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
Shywo-Taylor. Building symmetries into feedforward networks. In 1989 First IEE International Conference on Artificial Neural Networks, Publ. No. 313), pages 158-162. IET, 1989.
J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in neural information procexsing systems, pages 4077-4087, 2017.
X. Song, Y. Yang, K. Choromanski, K. Caluwaerts, W. Gao, C. Finn, and Tan. Rapidly adaptable legged robots via evolutionary meta-learning. arXiv preprint arXiv:2003.01239 2020.
K. O. Stanley and R. Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.
K. O. Stanley, D. B. D' Ambrosio, and J. Gauci. A hypercube-barce based encoding for evolving large- scale neural networks. Artificial life, 15(2):185-212, 2009.
Thrun and L. Pratt. Learning to learn. Springer Science & Business Media, 2012.
M. van der Wilk, M. Bauer, S. John, and J. Hensman. Learning invariances using the marginal likelihood. In Advances in Neural Information Processing Systems, pages 9938 198-9948, 2018.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008, 2017.
O. Vinyals, C. Blundell, T. Lillicrap, Wierstra, al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
M. Weiler and G. Cesa. General (2)-equivar steerable cnns. In Advances in Neural Information Processing Systems, pages 14334-14345, 2019.
D. Worrall and M. Welling. Deep scale-spaces: Equivariance over scale. In Advances in Neural Information Processing Systems, pages 7364-7376. 2019.
D. E. Worrall, J. Garbin, Turmukhambetov, and G. Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 5028-5037, 2017.
M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. In Advances in neural information processing systems, pages 3391-3401, 2017.
B. Zoph and Q V. Le. Neural architecture search with reinforcement learning. arXiv preprint 111.01661. 2016.
12

---

Published as a conference paper at ICLR 2021
A APPROXIMATION AND TRACTABILITY
A.1 FULLY CONNECTED
From Eq. 2 we see that for a layer with m output units, n input units, and l filter parameters the symmetry matrix U has mnk entries. This is too expensive for larger layers, so in practice, we need a factorized reparameterization
reduce memory and compute requirements when & is larger.
For fully connected layers, we use Kronecker factorization to scalably reparameterize each layer. First, we assume that the filter parameters v E R&l can be arranged matrix R*x!, Then we reparameterize each layer's weight matrix W similar to Eq. 2 but assume the symmetry matrix is the Kronecker product of two smaller matrices:
vec(W) = (U1 @ U2)vec(V), U1 E Raxi UU R
(5)
Since only store the two Kronecker factors U1 and U2. we reduce the memory requirements of U from markl to mk + nl. In our experiments we generally choose E RR*x= so Ui E Rax= and U2_ m Then the actual memory cost of each reparameterized layer (including both and v) is m² + n² + mn, compared to mn for a standard fully connected layer. So in the case where m n. MSR increases memory cost by roughly constant factor of 3.
After approximation MSR also increases computation time (forward and backward passes) by roughly constant factor of compared to MAMI.. A standard fully connected layer requires single matrix-matrix multiply y = WX in the forward pass (here Y and X are matrices since inputs and outputs are in batches). Applying the Kronecker-vee trick to Eq. givves:
W = U = vec(W) = (U1 @ (2)
(6)
So rather than actually forming the (possibly large) symmetry matrix U1 X U2. we can directly
construct W simply using 2 additional matrix-matrix multiplies W U.VUT. Again assuming V E Rm and m = n, each matrix in the preceding expression is approximately the same size as W.
A.2 2D CONVOLUTION
When reparameterizing 2-D convolutions, we need to produce a filter (a rank-4 tensor W e XXWW.. We assume the filter parameters stored in rank 3 tensor V R and factorize the symmetry matrix U into three separate matrices Un E RCoxP, U2 E RCix and U3 E HWx, A similar Kronecker product approximation gives:
W V x1 U1 X2 U2 X3 Us, WE R.RRC.XCHHW
W = rW))
W WERCIXCHHX
(8)
where X, represents n-mode tensor multiplication (Kolda and Bader, 2009). Just as in the fully connected case, this convolution reparameterization equivalent to Kronecker factorization of the symmetry matrix U.
An analysis of the memory and computation requirements of reparameterized convolution layers proceeds similarly to the above analysis for the fully connected case. As we describe below, in our augmented experiments using convolutional models each MSR outer step takes roughly 30% 40% longer than a MAML outer step.
In practice, for any experiment where reparameterize a standard 2-D convolution with weights W E xCxxCCxxx H W we choose = C., = C, and s = HW. Equivalently, we choose V e RC.XC Although not necessary, this choice conveniently makes the matrices U1, U2 and Ug into square matrices, which we can initialize to identity matrices at the start of meta-learning.
B PROOF OF PROPOSITION 1
To show the connection with the existing literature, we first present a slightly generalised definition of G-convolution that more common in the existing literature. We instead model input signal
13

---

Published as a conference paper at ICLR 2021
a function f X R some underlying space X. We then consider finite group G {9i..... In of symmetries acting transitively on X. over which we desire G-equivariance. Many (but not all) of the groups discussed in (Weiler and Cesa 2019) are finite groups of this form.
is proven by [Kondor and Trivedi 2018) that a function & is equivariant to if and only if is a G-convolution this space. In the domain of finite groups, of this notion: a finite "G cross-correlation" of f with a filter vb ; X + R. This is defined by (Cohen and Welling 2016) as:
consider slight simplification
( (f ())
Σ TEX
f(x)
(9)
We can now connect this notion with the linear layer, as described in our paper. First, in order for a fully connected layer's weight matrix W to act on function f. we must first assume that has finite support fæi...., 2e J-i.... f() is only non-zero at these s points within X. This means that f can represented as "dual" vector E R given by f, - fx;)), on which W can act
=(g")
=(g')
=(g")
We aim to show a certain value of UG E IReex allows arbitrary G cross-correlations and only G cross-corrllttions-- to be repre- sented by fully connected layers with weight matrices of the form
vec(W) = UCV,
(10)
where u E R is any arbitrary vector of appropriate dimension. The reshape specifically gives W R which transforms the vector e R
Figure The theoretical con- volutional weight symmetry matrix for the group (g) & NT -radian 2
With this in mind, we first use that the action of the group can be represented matrix transformation this vector space, using the matrix representation n:
C4. where 9 rotation of a 3x3 image (N e {0, 1, 1,2,3]. Notice that the image is flattened into length 9 vector. The matrix π ((g) describes the action of Nu radian rotation on this im- age.
(g))) =
(11)
where notably =(g) E R
We consider G E R and E R. Since RR also treat v as a the "dual" vector of a function X + R with support (#1, ......... F.y, described by 0(x;) tg. We can interpret é convolutional filter, just like vp in Eq. W then acts on just as it acts on F, namely:
[g)o]] (2z))
(12)
Now, we define UG by stacking the matrix representations of g; E G:
(91)
UG
:
(13)
[(g))
which implies the following value of W
T(
W = SV) = reshape
(14)
[gnn)
(g )
"This definition avoids notions such lifting of X G and the possibility of more general group repre- sentations, for the sake simplicity. We recommend Kondor and Trivedi (018) for a more complete theory of G-convolutions.
"This is using the natural linear algebraic dual of the free vector space (.1,..., Fs).
14

---

Published as a conference paper at ICLR 2021
Meta-learned LRs on synthetic problems
Variable
Small train dataset
Large train dataset
k=
=
k = 5
h: = 1
==
1: 5
U
-0.017
-0.011 +0.326
00.052
-0.009
-0.021
-0.039
+0.241
+0.401
+0.241
+0.307
+0.312
Table 4: In the ablation "MSR-Joint-FQ" of Sec. 4 we jointly updated U and v in the inner loop with meta- learned inner loop learning rates for each. This is contrast with standard MSR, where only is updated in the inner loop (also with meta-learned learning rate), and U is only updated in the outer loop. The inner learning rates were initialized at 0.02 for all variables. The table shows the inner loop learning rates at the end of training. The relative magnitudes suggest that v is being updated significantly more than U in the inner loop.
This then grants that the output of the fully connected layer with weights W is:
(W = E ((((:))))s
(15)
Using that f has finite support f ., JeJ. and that (T(())))) = =ūgg, zzy). we have that:
(WT), i(g 22))(()))))))++))))))))))))))))) Σ x))x).
(16)
Lastly, we interpret WaJ function (f) mapping each 9i G to its 1th component:
(f)(.) = (WJ); Σ x)(()
(17)
xEX
which is precisely the cross-correlation as described in Eq. 9 with filter s/ = ô. This implies that @c must be equivariant with respect to G. Moreover, all such Geequivariant functions are G cross- correlations parameterized by v. with UG fixed as in Eq. 13 we have that W UC v can represent all G-equivariant functions.
This means that if v is chosen to have the same dimension the input, and the weight symmetry matrix is sufficiently large, any equivariance to finite group can be meta-leamed using this ap- proach. Moreover, in this case the symmetry matrix has very natural and interpretable structure, containing a representation of the group in block submatrices this structure seen in practice in our synthetic experiments. Lastly, notice that corresponds (dually) to the convolutional filter, jus- tifying the notion that we learn the convolutional filter in the inner loop, and the group action in the outer group.
In the above proof, we' ve used the original definition of group convolution (Cohen and Welling
2016) for the sake of simplicity. is useful to note that slight generalization of the proof applies for more general equivariance between representations, as defined in equation (3.3 (i.e. the case when
(g)) an arbitrary linear transformation. and necessarily of the form (gg) fg))(z) f(g lz).)
This is subject to a unitarity condition on the group representation (Worrall and Welling 2019).
Without any modification to the method, arbitrary linear approximations to group convolution can be learnt when the representation is not permutation of the indices. For example, non axis-aligned rotations can be easily approximated through both bilinear and bicubic interpolation, whereby the
value of pixel æ after rotation is linear interpolation of the 4 or 16 pixels nearest to the "true" value of this pixel before rotation g -zx. Practically, this allows us to approximate equivariance to 45 degree rotations of 2D images, for which there don't 't exist representations of the form in Eq. 12
C FURTHER SYNTHETIC EXPERIMENT RESULTS
C.I VISUALIZING TRANSLATION EQUIVARIANT SYMMETRY MATRICES
Fig. 6 visualizes the actual symmetry matrix U that MSR-FC meta-learns from translation equiv- ariant data. Each column is one of the submatrices (i) corresponding the action of the discrete
15

---

Published as conference paper at ICLR 2021
Meta-learned symmetry matrix representations
m(D)
m(1)
m22)
mm3)
m(4)
mt5)
m(6)
mt7)
miB)
m(9)
m(10) mil1 m(22) m(33 m114) m(15) m116) mi177 m(18) m(19)
Figure 6: The submatrices of the meta-learned symmetry matrix of MSR-FC on the translation equivariant problem [Sec.(5.1] Intensity corresponds each entry's 's absolute value, We see that the symmetry matrix has been meta-learned to implement standard convolution: each x(i) translates the size filter u E R$ by spaces. Note that in actuality the submatrices are stacked on top of each other in U in Eq. 13 but display them side-by -ide for visualization.
Synthetic problem data quantity
k
k
2
k
Rot
Rot+flip
No. train tasks
400
800
800
8000
8000
No. test tasks
100
200
200
2000
2000
Examples/train task (Small)
2
2
4
20
20
Examples/train task (Large)
20
20
20
Train examples/test task
1
1
Table 5: The amount of training and test data provided each method in the synthetic experiments of Table and Table [ The row indicates that on the test tasks,, methods were expected to solve each problem using a single example from that task.
translation group element Z the filter In other words, MSR automatically meta-learned to contain these submatrices (i) such that each (() translates the filter by spaces, effectively meta-learning standard convolution! In the actual symmetry matrix the submatrices top of each other as in Eq. [3 but we display each submatrix side-by-side for easy visualization. The figure is also cropped for space: there are total of 68 submatrices but show only the first 20, and each submatrix is cropped from 70 x 3 to 22 x 3.
stacked
C.2 VISUALIZING ROTATION AND FL.IP EQUIVARIANT FILTERS
In Sec. 5.2 we ran three experiments reparameterizing convolution layers to meta-learn 90" rotation, 45 rotation, and 45 rotation+flip equivariance, respectively. Figure 7 shows that MSR produces rotated and flipped versions of filters in order to make the convolution layers equivariant to the corresponding rotation or flip transformations.
D EXPERIMENTAL DETAILS
Throughout this work implemented all gradient based meta-learning algorithms in PyTorch using the Higher (Grefenstette et al. 2019) library.
D.1 TRANSLATION SYMMETRY SYNTHETIC PROBLEMS
For the (partial) translation symmetry problems we generated regression data using single locally connected layer. Each task corresponds to different weights of the data generating network, whose entries we sample independently from a standard normal distribution. For rank & locally connected filters sampled & width-3 filters and then set the filter value at each spatial location to be random
16

---

Published as a conference paper at ICLR 2021
linear combination of those k filters. Table 5 shows how many distinct training and test tasks generated data for. For each particular task, we generated data points by randomly sampling the entries of the input vector from standard normal distribution, passing the input vector into the data generating network, and saving the input and output as pair.
trained each method for 1, 000 outer steps task batches of size 32, enough for the training loss to converge for every method in every problem. We used the Adam (Kingma and Ba 2014) optimizer in the outer loop with learning rate .0005. Like most meta-learning methods, MAML and MSR split each task's examples into support set (task training data) and a query set (task validation data). On training tasks MAML and MSR used 3 SGD steps on the support data before computing the meta- meea-traiiing objective on the query data, while using SGD steps on the support data of test tasks. We also used meta-learned per-layer learning rates initialized to 0.02. At meta-test time we evaluated average performance and error bars on held-out tasks.
MSR and MAML training: During meta-training
MTSR training: We reparameterize fully connected layers into symmetry matrix U and filter c, similar to MSR. MTSR maintains single shared U, but initializes a separate filter 2(0) for each training task Te- Given example data D; from Tú we jointly optimize (U, w(() using the loss C(U, v(), D.). In practice each update step updates U and all {o(*) in parallel using the full batch of training tasks. Given test task we initialize new filter v alongside our already trained U. We then update v on training examples from the test task before evaluating on held out examples from the test task. We use 500 gradient steps for each task at both training and test time, again using the Adam optimizer with learning rate 0.001.
We ran all experiments on a single machine with a single NVidia RTX 2080Ti GPU. Our MSR-FC experiments took about 9.5 (outer loop) steps per second, while our MSR-Com experiments took about 2.8 (outer loop) steps per second.
D.2 ROTATION+ FLIP SYMMETRY SYNTHETIC PROBLEMS
The setup of the rotation and rotation+flip symmetry problems is very similar to that of the transla- tion symmetry problems. Here we generated regression data using a single E(2)-steerable (Weiler and Cesa 2019) layer. Each task again corresponds to a particular setting of the weights of this data generating network, whose entries are sampled from a standard normal distribution for each task. We generate examples for each task similarly to above, and Table [ shows the quantity of data available for training and test tasks.
MAML and MSR training setups here similar to the translation setups, but reparameterize the filter of standard convolution layer to build in translation symmetry and focus on learning rotation/flip symmetry. Unlike the translation experiments, here we use I SGD step in the inner loop for both train and test tasks, and initialize the learned learning rates to 0.1.
D.3 AUGMENTATION EXPERIMENTS
To create Aug- Onnigloi and Aug- Minilmeemt,,,nt,, we extended the Omniglot and Minilmagenet benchmarks from TorchMeta (Deleu et al. 2019). Each task in these benchmarks is split into support (train) and query (validation) datasets. For the augmented benchmarks we applied data augmenta- tion to only the query dataset of each task, which consisted of randomly resized crops, reflections. and rotations by up to 30". Using the torchvision library, the augmentation function is:
# Data augmentaiion applied to ONLY the query set. size 28 # Omniglot image size. 84 for Minilmagenet. au augment_fn Compose( GandomResizedCropp (28. scale= 1.0)).
Random V Verticcallli (p=0.5))
RandomHHrrizomaalllpppp---- (p=0.))
RandomRotation (30, resample=Image BBLLIEER))
For the augmented Omniglot and Minilmagenet 1-shot experiments, MAML used exactly the same convolutional architecture (same number of layers, number of channels, filter sizes, etc.) prior
17

---

Published as a conference paper at ICLR 2021
work Omniglot and Minilmagenet (Vinyals al. 2016 Finn et al. 2017). For MSR repa- rameterize each layer's s weight matrix or convolutional filter using the Kronecker approximation (Appendix(A) such that the reparameterized layer has the same number of input and output neurons as the corresponding layer in the MAMI. model.
For Minilmagenet 5-shot, experimented with increasing architecture size via more channels and/or larger filters, which yielded better accuracies on meta-validation tasks. For MSR, MAML, and ANIL we increased the number of output channels from 32 to 128 and increased the kernel size from to in the first convolution layers. We then inserted a x 1 convolution layer with 64 output channels right before the linear output layer. For the ProtoNet architecture we similarly increased the output channels at each layer from 32 to 128, but found that keeping the kernel size at worked best.
For "MAML (Big)" experiments we increased the architecture size of the MAML model to exceed the number of meta-parameters (symmetry matrices filter parameters) in the corresponding MSR model. For Minilmagenet 5-Shot we inserted an additional linear layer with 3840 output units before the final linear layer. For Minilmagenet 1-Shot we increased the number of output channels each of the 3 convolution layers from 32 to 64, then inserted additional linear layer with 1920 output units before the final linear layer. For the Omniglot experiments we increased the number of output channels at each of the 3 convolution layers to 150.
For all experiments and gradient based methods we trained for 60, 000 (outer) steps using the Adam optimizer with learning rate .0005 for Minilmagenet 5-shot and .001 for all other experiments. In the inner loop we used SGD with meta-learned per-layer learning rates initialized to 0.4 for Om- niglot and .05 for Minilmagenet. We meta-trained using single inner loop step in all experiments, and used 3 inner loop steps at meta-test time. Although MAML originally meta-trained with 5 in- ner loop steps on Minilmagenet, we found that this destabilized meta- titt-trrining on our augmented version. We hypothesize that this is due to the discrepancy between support and query data in our augmented problems. During meta-training we used task batch size of 32 for Omniglot and 10 for Migeent At meta-test time we evaluated average performance and error bars using 1000 held-out meta-test tasks.
We ran all experiments on machine with a single NVidia Titan RTX GPU. For our we ran two experiments simultaneously on same machine, which likely slowed each invididual experiment down. Our MSR method took about 0.6 steps per second, whereas the MAML baseline took about 0,86 steps per second. For Aug- aug-Minimagenet we ran one experiment per machine. MSR took 4.2 steps per second, while MAML took 5.6 steps per second on these experiments.
18

---

Published as a conference paper at ICLR 2021
fitters 90* rot equivartance
Filters for rot eculvariance
tilera produced for 4s* rot flpp scuivorianee.
Figure 7: MSR produced convolution filters, after meta-learning 90" rotation, 45 rotation, and 45" rotation+flip equivariance the Sec. 5.2 experiments. Notice that MSR learns to achieve the corresponding equivariance by producing rotated//fipped versions of the same filter.
19


Reviews:
[{"review_id": "noutxvzC6hP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work (i) meta-learns equivariances in neural networks by reparameterizing fully connected layers into a symmetry matrix and filter parameters, \nand (ii) meta-learns invariances from augmented data.\n\nStrengths:\n+ The work implements (i) by a layer and custom inner loop optimizer using the higher-order optimization library [1]\nand (ii) by augmenting benchmark datasets [2].\n+ The work learns partial translational symmetry, euqivariance to rotation and flips\n+ The work performs important ablation studies, such as testing reparameterization with and without meta-learning by using multi-task learning instead.\n\n\nWeaknesses:\n- Mostly toy examples\n\n- Rather than baking-in inductive biases into the network by encoding equivarainces, \nother approaches such as the vision Transformer [3] learn inductive biases from data:\nlearning local, medium, and long range connections, discovering architectures which supersede CNNs, learning filters, \nand demonstrating that CNNs are a curve on an attention distance vs. network depth plane.\nAdding a reference to this line of work may improve the introduction.\n\n- Minor changes:\nFigures 1,2, and 3 may be improved.\nAlgorithm 2 may be sufficiently described in words.\nTypos on page 14 may be fixed.\nlines 9-10 should read \"chosen to have\" and \"theoretically meta-learned\"\nline 32 should read \"each $\\pi(i)$ translates the filter\"\n\n\n[1] Generalized inner loop meta-learning, Grefenstette et al, 2019.\nhttps://github.com/facebookresearch/higher\n\n[2] Torchmeta: A meta-learning library for PyTorch, Wurfl et al, 2019.\nhttps://github.com/tristandeleu/pytorch-meta\n\n[3] An image is worth 16x16 words: Transformers for images recognition at scale, Dosovitskiy et al, 2020\nhttps://arxiv.org/pdf/2010.11929.pdf\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cmJiSMaf1ig", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "OVERVIEW:\nThe authors present a meta-learning approach for network equivariance where the key idea is that equivariance to a finite group of transformations can be achieved by identifying the sharing pattern of weights. Their proposition claims that a fully connected layer $\\phi: \\mathbb{R}^n \\rightarrow \\mathbb{R}^m$ with weights $W$ can be factorized into a symmetry matrix $U$ and filter parameters $v$ where the symmetry matrix encodes desired group-convolutions: $\\text{vec}(W) = U v, \\hspace{1em} v \\in \\mathbb{R}^k, U \\in \\mathbb{R}^{mn \\times k}$. Within the meta-learning framework, they learn both $U$ and $v$ as part of the outer and inner steps respectively. They demonstrate that they are able to recover the translational equivariance baked into traditional convolutions using this approach including the expected symmetry pattern (shown in Fig. 4). They are also able to demonstrate this two more scenarios: (i) for a group of translation, discrete $45^\\circ$ rotation and reflection, and (ii) for Augmented-Omniglot and Augmented-MiniImagenet, providing empirical evidence that their proposed approach learns meaningful information for network equivariance.\n\nPROS:\n- I really liked the idea of equivariance captured as a symmetry pattern (or weight sharing) and being able to learn it from data using meta-learning. This is very interesting and exciting with the ability to \"learn\" rather than \"hope\" that equivariance is learned from appropriately augmented data. \n- A proof of Proposition 1 is presented in the Appendix which is a nice contribution in my opinion.\n- I liked the visualizations of the symmetry pattern for translation equivariance and the filters for discrete rotation equivariance (in appendix). They are visual evidence of achieving what is expected from the proposed approach.\n- The experiment in Section 5.2 is helpful in backing up the claim of \"hybrid\" equivariance where you are able to learn translations + discrete rotations. This is achievable by appropriate filter design for simpler groups (mostly 2D) like in Harmonic-Nets [Worrall et al] but can get complicated for interesting groups like SO(3) & SE(3).\n\nCONS:\n- The biggest concern for me are that the experiments are largely on synthetic data which has been randomly generated (Sections 5.1 and 5.2). The only real data experiment is on Aug-Omniglot and Aug-MiniImagenet for a few-shot classification task. Equivariance has been demonstrated to be effective on real data in two setups that I am aware of: (a) 3D Model classification with spherical convolutions like in Cohen & Welling, [Esteves et al](https://arxiv.org/abs/1711.06721), (b) Azimuth and scale estimation on Google Earth images [Henriques and Vedaldi](http://proceedings.mlr.press/v70/henriques17a.html). Applying the proposed method to one of these experimental setups will be very helpful in convincing the readers of their use on real data.\n- Is the proposed approach learning local equivariance (like with harmonic filters) or global equivariance (like with warped convolutions where transforming the image into polar coordinates gives it rotation and scale equivariance)?\n- From the proof of Proposition 1, it becomes clear why the restriction to a finite group. The question I have is what is needed to move into more general (continuous) groups like SO(2), SO(3), SE(3)? Will an approximation (into some finite pattern with tiling) be good enough? I think that it is a possible future research direction but I am interested in knowing your thoughts about it.\n- The methods the authors compare against are meta-learning methods which makes sense given the broader framework the work is in. However, I would like a comparision with other network equivariance via filter design algorithms. I don't expect better results but some discussion of comparable performance without the handcrafted design or the ability to learn equivariance for groups without a handcrafted filter available will be a huge plus.\n\nREASON FOR RATING:\nI like the paper and it makes a very good contribution in an important area. It is however held back by the lack of experiments on real data and comparision with other established equivariance works leading to my current rating.\n\nUPDATE:\nI have read the author feedback and other reviews/discussions. I have updated my rating to 8 from 7 reflect it.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QeYEbyGmlh7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose MSR, a parametrization of convolutional kernels that allows for meta-learning symmetries shared between several tasks. Each kernel is represented as a product of a structure matrix and a vector of the kernel weights. The kernel weights are updated during the inner loop.  The structure matrix is updated during the outer loop.  \n\nStrengths\n1. The paper is interesting and is easy to read. The figures help a lot in understanding the discussed ideas. \n2. The authors demonstrate that the proposed method outperforms the baseline meta-learning models. They empirically prove that MSR indeed learns valuable symmetries from the set of tasks and the provided data.\n3. The related work as well as the experimental part allow for a clear positioning of the proposed approach. It demonstrates a valuable connection between meta-learning and building equivariant models.\n\nI did not find any major weaknesses in the presented paper.\n\nQuestions\n1. A matrix $W$ of size $8\\times8$ can be reparametrized in several ways. The corresponding vector $v$ can be of size $1, 2, 4, \\dots 64$. Do you consider the size of the vector as a hyperparameter? If so, how to choose it? \n2. If we consider the case of the exact flip symmetry, then the length of $\\text{vec}(W)$ must be even. One half encodes the original weight and the other half encodes the flipped weight. So we end up with a constraint between the shape of the matrix and the structure of the symmetry group. The same argument applies to all other symmetry groups. What happens if the constraint is not satisfied? Can we learn a flip symmetry for $W$ of size $7 \\times 2$? How $U$ will look in this case?\n\nI enjoyed reading the paper. It is insightful, well-written, and demonstrates several valuable results both theoretical and experimental. \n\n### Decision\nThe authors answered all my questions. My decision stays the same.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QbagW2O08yI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nThe paper presents a meta-learning algorithm to learn/encode equivariance into deep nets. The main idea is to decompose the model parameters into two parts, a spatial sharing pattern, and the trainable weights. When transferring to a new task, the sharing pattern is fixed and only the remaining trainable weights are tuned. They authors motivated this approach stating that data augmentation may not be practical for robotics application which requires training in the real-world. For experiments, they consider a synthetic dataset where they can recover the equivariance and also k-shot classification tasks on datasets augmented with crops, rotations, and reflections.\n\n### Decision:\nI recommend a borderline reject for this paper. I have some questions with the claimed contribution, and details in the experimental section that should be answered before I can recommend an acceptance.\n\n### Supporting Arguments:\n1.\tOverall, I think the paper is interesting and relevant to the community. However, there are some questions that should be addressed.\n2.\tThis paper demonstrates that through their reparametrization, parameter-sharing, the network can be \"equivariant to any finite symmetry group\". This result is similar to the work by Ravanbakhsh et al., (2017), which they also show the sharing-patterns encode equivariance properties. Due to its relevance, I believe a more careful discussion should be included and not just cited as “theoretical work has characterized the nature of equivariant layers for various symmetry groups”.\n\n3.\tFor the experiments, I think a necessary to have a baseline that treats both U,v as trainable parameters. I wonder if it is necessary to train U and v separately on train and val; Maybe the performance gain comes just from the fact that U is trainable, i.e., this model architecture benefits learning. \n\n4.\tWhat are the reported +/- in the Tables? Is it the standard deviation over several random initialization runs? \n\n5.\tJust to confirm, “test-sets” in Aug-Omniglot and Aug-MiniImagenet are augmented as well? If yes, are there still performance gain without the augmentation?\n\n6.\tAre the baselines in Table 3 trained with data-augmentation as well? The paper states “benchmarks are identical to prior work (Finn et al., 2017)”; does that mean without data augmentation? Also, what happens if both train and val sets are augmented for Alg. 2?\n\n7.\tThere seem to be some learning rate, step sizes, and architecture tuning as described in supplementary materials. How are these hyperparameters searched? What metric is being used to pick them and what ranges were considered?\n\n### Additional feedback:\n- Figure 2. Is a little bit blurry?\n- Maybe also mention the issue of  data augmentation with real-world data in the introduction? It wasn’t clear to me the challenges with data augmentation until much later in the paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to meta-learning symmetries. While several approaches have recently emerged with similar goals, and sometimes greater convenience and empirical performance, the proposed approach has some interesting characteristics, such as changing properties of the architecture to extrapolate these symmetries. There was a quite a spread of opinions about the paper, the empirical results were not strong, and updates to the paper focused on helpful text additions, but did not substantively improve the evaluation or experiments. Notwithstanding, the paper is conceptually interesting, there are no major flaws, and there is sufficient support for it.
Decision: Accept (Poster)