Equivariance-aware Architectural Optimization of Neural Networks
Authors: Kaitlin Maile
Dennis George Wilson
Patrick Forré
Venue: ICLR 2023 poster (2023)
Keywords: equivariance
neural architecture search
geometric deep learning

Abstract: Incorporating equivariance to symmetry groups as a constraint during neural network training can improve performance and generalization for tasks exhibiting those symmetries, but such symmetries are often not perfectly nor explicitly present. This motivates algorithmically optimizing the architectural constraints imposed by equivariance. We propose the equivariance relaxation morphism, which preserves functionality while reparameterizing a group equivariant layer to operate with equivariance constraints on a subgroup, as well as the $[G]$-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization. We further present evolutionary and differentiable neural architecture search (NAS) algorithms that utilize these mechanisms respectively for equivariance-aware architectural optimization. Experiments across a variety of datasets show the benefit of dynamically constrained equivariance to find effective architectures with approximate equivariance.

Published as a conference paper at ICLR 2023
EQUIVARIANCE-AWARY ARCHITECTURAL OPTIMIZA- TION OF NEURAL NETWORKS
Kaitlin Maile
Dennis G. Wilson
IRIT, University of Toulouse kaitlin. mai ee@1 rit.
ISAE-SUPAERO, University of Toulouse
dennis wi ssne sae- see-suuprro.. fr
Patrick Forré
University of Amsterdam p.d. forre@uva. nl
ABSTRACT
Incorporating equivariance to symmetry groups as constraint during neural net- work training can improve performance and generalization for tasks exhibiting those symmetries, but such symmetries are often not perfectly nor explicitly present. This motivates algorithmically optimizing the architectural constraints imposed by equivariance. We propose the equivariance relaxation morphism, which preserves functionality while reparametrizing group equivariant layer operate with equivariance constraints on subgroup, as well the [G]-mixed equivariant layer, which mixes layers constrained to different groups to enable within-layer equivariance optimization. We further present evolutionary and dif- ferentiable neural architecture search (NAS) algorithms that utilize these mech- anisms respectively for equivariance-aware architectural optimization. Experi- ments across a variety of datasets show the benefit of dynamically constrained equivariance find effective architectures with approximate equivariance.
INTRODUCTION
Constraining neural networks to be equivariant to symmetry groups present in the data can improve their task performance, efficiency, and generalization capabilities (Bronstein al.) 2021] as shown by translation-eque cunnuiaaiooneuuiaiiiaantt convolutional neural networks (Fukushima & Miyake 1982 LeCun et al.) 1989) for image-basee based tasks (LeCun et al.) 1998). Seminal works have developed general theories and architectures for equivariance in neural networks, providing blueprint for equivariant oper- ations on complex structured data (Cohen Welling 2016 Ravanbakhsh et al., 2017 Kondor & Trivedi 2018, Weiler et al., 2021). However, these works design model constraints based an explicit equivariance property. Furthermore, their architectural assumption of full equivariance in every layer may be overly constraining; e-g., in handwritten digit recognition, full equivariance 180" rotation may lead to misclassifying samples "6" and "9". WWiler Cesa (2019) found that local equivariance from final subgroup convolutional layer improves performance over full equiv- ariance. If appropriate equivariance constraints are instead learned, the benefits of equivariance could extend to applications where the data may have unknown or imperfect symmetries.
Learning approximate equivariance has been recently approached via novel layer operations (Wang al. 2022 Finzi et al., 2021 Zhou al., 2020; Yeh al. 2022 Basu al. 202I). Separately, the field of neural architecture search (NAS) arms to optimize full neural network architectures (Zoph & Le 2017, Real et al., 2017 Elsken et al. 2017 Liu et al. [0018] Lu et al. 2019). Existing NAS meth- ods have not yet explicitly optimized equivariance, although partial or soft equivariant approaches like Romero & Lohit (2022) and van der Ouderaa et al. (002)) approach custom equivariant architec- tures. An important aspect of NAS is network morphisms: function- peretion-reeeevingg architectural changes Wei et al. 2016) which can be used during training to change the loss landscape and gradient de- scent trajectory while immediately maintaining the current functionality and loss value (Maile al.) 2022). Developing tools for searching over a space of architectural representations of equivariance would permit NAS algorithms be applied towards architectural optimization of equivariance.

---

Published as a conference paper at ICLR 2023
Contributions First, present two mechanisms towards equivariance-aware architectural opti- mization. The equivariance relaxation morphism for group convolutional layers partially expands the representation and parameters of the layer to enable less constrained learning with a prior on symmetry. The mmixed equivariant layer parametrizes a layer as a weighted sum of layers equiv- ariant to different groups, permitting the learning of architectural weighting parameters.
Second, we implement these concepts within two algorithms for architectural optimization of partially-equivariant networks. Evolutionary Equivariance-Awarr NAS (EquiNAS utilizes the equivariance relaxation morphism in a greedy evolutionary algorithm, dynamically relaxing con- straints throughout the training process. Differentiable Equivariance-Aww plements mGi-mixed equivariant layers throughout network to learn the appropriate approximate equivariance of each layer, in addition their optimized weights, during training.
NAS (EquiNASD) im-
Finally, we analyze the proposed mechanisms via their respective NAS approaches in multiple im- age classification tasks, investigating how the dynamically learned approximate equivariance affects training and performance over baseline models and other approaches.
RELATED WORKS
Approximate equivariance Although no other works on approximate equivariance explicitly study architectural optimization, some approaches are architectural in nature. We compare our tributions with the most conceptually similar works to our knowledge.
The main contributions Basu al. (2021) and Agrawal & Ostrowski 2022 are similar to pro- posed equivariant relaxation morphism. Basu et al. (2021) also utilizes subgroup decomposition but instead algorithmically builds up equivariances from smaller groups, while our work focuses laxing existing constraints. Agrawal & Ostrowski (2022) presents theoretical contributions towards network morphisms for group-invariant shallow neural networks: in comparison, our work focuses on deep group convolutional architectures and implements the morphism in a NAS algorithm.
The main contributions of Wang et al. (2022) and Finzi et al. (2021) are similar our proposed [G]- mixed equivariant layer. [Wang et al. (2022) also uses a weighted sum of filters, but uses the same group for each filter and defines the weights over the domain of group elements. |Fizzi et al. (202T) uses an equivariant layer in parallel to a linear layer with weighted regularization, thus only using two layers in parallel and weighting them by regularization rather than parametrization. [Mouli & Ribeiro (2021) also progressively relaxes equivariance constraints, but with regularized rather than parametrized constraints.
In more diverse approaches. Zhou et al. (2020) and Yeh et al. (2022) represent symmetry-inducing weight sharing via learnable matrices. Romero & Lohit (2022) and van der Ouderaa et al. (2022) learn partial or soft equivariances for each layer.
Neural architecture search Neural architecture search (NAS) aims to optimize both the architec- ture and its parameters for a given task. Liu et al. (2018) approaches this difficult bi-level optimiza- tion by creating a large super-network containing all possible elements and continuously relaxing the discrete architectural parameters to enable search by gradient descent. Other NAS approaches include evolutionary algorithms (Real et al. 2017: Lu et al. 2019 Elsken et al. (017) and reinforce- ment learning (Zoph & Le 2017). which search over discretely represented architectures.
3 BACKGROUND
We assume familiarity with group theory (see Appendix A. For discrete group G, the 1hh G- equivariant group convolutional layer (Cohen & Welling 2016) of a group convolutional neural network (G-CNN) convolves the feature map f: G + R from the previous layer with filter
with kernel size represented as learnable parameters s) G CCx, For each output channel
We identify the correlation and convolution operators as they only differ where the inverse group element placed and refer hoth "convolution" throughout this work.
2

---

Published as a conference paper at ICLR 2023
A
B
equavoriacee
equaaaracce
anlend equiveriont loyer
Leatneble parmmdeers
Equivariance morphism
Training
equívariant operation
Expandedd
oculariant
Figure 1: Visualizing the equivariance relaxation morphism and the [G]-mixed equívariant layer, using the C4 group. In (A), the learnable parameters of Ca-equivariant convolutional layer are expanded using each group element action the expanded filter can be used in standard convo- lutional layer. The equivariance relaxation morphism reparametrizes the layer to be architecturally constrained C2 equivariance, initialized to functional C4 equivariance. In (B), convolutional oper- ations equivariant to subgroups of C4 are summed with learnable architectural weighting parameters.
d E [Ci], where [C] == {I...., C}, and group element E G, the layer's output is defined as:
C1-1
ka v)a(o) = Σ
Ja(h)baclo hh).
REG
The first layer is special case: the input to the network needs to be lifted via this operation such that the output feature map of this layer has a domain G. In the case of image data, an image x with C channels may be interpreted space to real number for each channel, where the tt channel of x referred to as Te- The input is x: Z + RC,, so the layer is instead lifting convolution:
function 2² + EC mapping each pixel in coordinate
Co
[ G vla(g)
Σ Σ
(2)
yE22 c=l
We present our contributions the group convolutional layer case, although similar claims apply for the lifting convolutional layer case.
4 TOWARDS ARCHITECTURAL OPTIMIZATION OVER SUBGROUPS
We propose two mechanisms to enable search over subgroups: the equivariance relaxation mor- phism and a mmi-m equivariant layer. The proposed morphism, depicted in Figure [ and described in Section 4.1 changes the equivariance constraint from one group to another subgroup while preserving the learned weights of the initial group convolutional operator. The mGle cquivariant layer, shown in Figure [B) and presented in Section 4.2) allows for a single layer to represent equivariance multiple subgroups through weighted
4.1 EQUIVARTANCE RELAXATION MORPHISM
The equivariance relaxation morphism reparametrizes G-equivariant group (or lifting) convolu- tional layer to operate over a subgroup of G. partially removing weight-sharing constraints from the parameter space while maintaining the functionality of the layer.
Let G' < G be a subgroup of G such that G' \ G is finite. Let R be a system of represen- tatives of the left quotient (including the identity element), so that G" G (G'r r E R}, where G'r = [g'r I' E G'}. Given a G-equivariant group convolutional layer with feature map f and filter s/. we define the relaxed feature map f: G + ([(C-1x(R)) and relaxed filter
bb G" (CC) (R)) follows. For e [CI-1], R. d [Ci]:
Flam)(og)) = falg's) s), (aa)) (t-Igg)
(3)
(4)
3

---

Published as a conference paper at ICLR 2023
We define the equivariance relaxation morphism from G to G' as the reparametrization of as v (Eq. (4) and reshaping of as f (Eq. 3). We will show that the new layer, [f * (aa (g'), is equivalent to [f *g Vla(g't) down to reshaping. Since the mapping G' x R + G, (g',t) gtt bijective, every g can uniquely be written as g = g't with g' E G' and t E R. For g G, G'g g E G" G has unique representative i E R with G'g = GlI, and g' :- gt E G'. Similarly, E may be written as h = hts with unique h E G' and 8 E R. With these preliminaries, we get:
[/ ka () = [s *a Vla(g)
(5)
Se(h)aaalg h))
BEG
CL-1
E
N' SER
E
Σ
(8)
NEG e=1 *ER
f x (5
(d.1) (g').
which shows the claim. Thus, the convolution of f with e is equivariant to G but parametrized as G' d'-equivariant group convolutional layer, where the representatives are expanded into independent channels. This morphism can be viewed as initializing G' G'-equivariant layer with pre-trained prior of equivariance to G. maintaining any previous training.
Standard convolutional layers translational symmetry over pixel space. Regular group convolutions are often implemented by relaxation to the translational symmetry group by expanding the filter via the appropriate group actions, allowing a standard convolution implementation from a deep learning library to be used. The equivariance relaxation morphism generalizes this concept any subgroup. This, as well as how the equivariance relaxation morphism is implemented, is discussed further in Appendix B
special case of group-equivariant layers, where the group is
4.2 [G]-MIXED EQUIVARIANT LAYER
Towards learning equivariance, we additionally propose partial equivariance via a mixture of lay- ers, each constrained to equivariance different groups, applied in parallel to the same input then combined via weighted sum. The equivariance relaxation morphism provides a mapping of group elements between group and a subgroup. For set of groups [G]. such as a subgroup lattice of some group G, we define [G]-mixed equivariant layer as:
[f (d,1) (g)
Σ zG fo
(g)
(10)
(d,t)
GEIG]
f G Σ
(g),
(11)
GEIG
(d,a)
where each element 20 of [z] = {zg [G]} is architectural weighting parameter such that
Ece[c| 2G = 1, G" is a subgroup of all groups in [G], each element et of [ is a filter with a
domain of G, and v/G is the transformation of &/ from domain of to G' as defined in Equation Thus, the layer parametrized by [8)] and [z]. computing weighted sum of operations that are cquivariant to different groups of [G]. The layer may be equivalently computed by convolution of the input with the weighted sum of transformed filters, shown Equation II We provide further implementation details in Appendix B
5 EQUIVARIANCE-AWAN NEURAL ARCHITECTURE ALGORITHMS
We present two NAS methods that utilize the presented mechanisms for discovering appropriate equivariance during training: Evolutionary Equivariance-Aware NAS (EquiNAS and Differen-
4

---

Published as a conference paper at ICLR 2023
Algorithm Evolutionary equivariance aware neural architecture search.
procedure EqUINAS symmetry group G)
Initialize population with G-equivariant group convolutional network.
for each generation do for each network in population do
Add children network with relaxed equivariance constraints into population.
for each network population do Partially train network on dataset.
Select Pareto-efficient and high accuracy networks as new population.
return population
tiable Equivariance-Awar NAS (EquiNASp). Both methods optimize an architecture while learn- ing weights, yielding a final trained network adapted to equivariances present in the training data. However, they differ in NAS paradigm and approximate equivariance representation: EquiNAS g, in Section 5.1 searches for networks composed of layers each fully equivariant to possibly different groups, while EquiNASp. in Section 5.2 searches for smooth mixtures of equivariant layers.
5.1 EVOIJUTIONARY EQUIVARIANCE- NAS
Towards finding the optimal full equivariance per layer, the equivariance relaxation morphism pre- sented in Section 4.1 is applied as the genetic operator in an evolutionary hill-climbing algorithm. NAS (EquiNAS g algorithm, given in Algorithm similar to other evolutionary NAS methods such as Elsken et al. (2017) with pareto selection as in Falanti jet al. (2022). population of networks, which starts with an individual with all layers equivariant to the largest possible group, undergoes mutation via equivariance relaxation and selection based on accuracy and parameter count to optimize neural architecture while learning network parameters. See Appendix A.2 for further background evolutionary NAS.
The Evolutionary Equivariance-Awar
In each generation, candidate networks are evaluated based on maximizing validation accuracy and minimizing parameter count: the pareto-domines individuals with highest accuracy kept, then additional high-accuracy individuals are added if necessary until the desired parent population size reached. Offspring are generated from each parent separately by mutation using the relaxation morphism. This preserves the weights of the parametrized equivariance during mutation, allow- ing for the continuous training of networks over evolution by inheritance from parent individuals. Specifically, mutation reduces single layer's parametrized equivariance to subgroup within the constraint that each layer has parametrized equivariance to subgroup of all preceding layers. This constraint yields local equivariance properties for the network, as shown in Weiler & Cesal (2019) and [Elsayed al.) (2020) to be empirically favorable in image classification tasks. The resulting individuals are each trained independently for a given training time, and then this process repeats.
The second objective of minimizing parameter count is intended to advance efficient networks, such as those with large symmetry groups. Accuracy-basse selection alone would necessarily prefer larger networks as mutation via the equivariance relaxation morphism results in two networks with identical performance but different size, the relaxed network having more parameters, until train- ing; potentially short-term increases in validation accuracy after training would then result in the selection of individuals with more parameters. Thus, the proposed strategy of selecting both pareto- dominant and high-aces high-accuracy individuals is intended to maintain a diverse yet efficient population without succumbing overly greedy selections too early.
5.2 DIFFERENTAALEE EQUIVARIANCE-AWAR NAS
In a contrasting paradigm, the [G] -mixed equivariant layer presented in Section 4.2 allows for smoothly searching across a spectrum of equivariance for each layer via a differentiable NAS algo- rithm. Our Differentiable Equivariance- Awarre Aware NAS (EquiNASp) algorithm, defined in Algorithm 2 is inspired by DARTS (Liu et al. 2018) with significant changes detailed in the following para- graphs. EquiNAS simplifies the bilevel optimization of the architecture weighting parameters Z and filter weights Vi into alternating independent updates, computing the gradient update for Z with the current, rather than optimal, & for the current architecture encoded by Z, to boost search effi- ciency with minimal performance loss compared higher order approximations (Liu al. 2018..
5

---

Published as a conference paper at ICLR 2023
Algorithm 2 Differentiable cquivariance neural architecture scarch.
procedure B (Set of groups [G])
Initialize network with [G]-mixed equivariant layers, parametrized by v and Z.
while not converged
Update Z by V 2 C.( (V, Z) Update V by Vv Z( V Z). return trained network
In most differentiable NAS search spaces, the desired output architecture is discretized select a subset of architectural options within constraints, then the weights are re-initialized and trained within the static architecture. In our formulation, this is not necessary as any mixed operation can be equivalently expressed as single layer equivariant to any group G" that is a common sub- group all groups of the mixed operation (Eq. [T]: in our experimental case, this is a standard ienislation-quivariinn convolutional layer, so the final model can be equivalently expressed as a standard convolutional model with encoded partial equivariance. Thus, the final optimized architec- ture and trained weights are output from the single search process. We explore the standard NAS paradigm, where weights are reinitialized and trained in the final static architecture, in Appendix [D]
In order to enforce that the scaling of each filter does not confound the architecture weighting pa- rameters, we use the weight normalizing reparametrization (Salimans & Kingma 2016) and do not update the scalar norm parameter of each filter after initialization.
We do not use disjoint datasets for updating and Z, but rather draw one batch for V and another for Z independently and randomly from the same training split. This allows for a standard dataset split and to use the validation set for hyperparameter tuning.
These two NAS approaches present adaptations of two standard types of NAS, evolutionary and differentiable, to the search for optimal partial equivariance. We next study empirically the two EquiNAS methods three datasets, one with known rotational symmetry and two with unknown but visually significant rotational and reflectional symmetry.
EXPERIMENTS
We focus on the regular representation of groups and show experiments with reflectional and up to 4-fold rotational symmetry groups applied to image classification tasks. Examples of symmetry groups acting on pixel space, which corresponds to ZZ, include T(2), which consists of discrete translations in both dimensions; the cyclical groups C. which consist of n-fold rotations; and the dihedral groups D,, which consist of reflections with n-fold rotations, where n E {1, 4} for exact symmetry without interpolation. The p4 group consists of discrete translations and multiples of 90 rotations and may be represented as T(2) x C.. The p4m group consists of discrete translations, reflections, and multiples of 90º rotations and may be represented T(2) D4. As standard convo- lutional layers are already equivariant to T(2), we refer to layers also equivariant to n-fold rotations with or without reflections as D, -n-equivarian,, respectively. So, Ci equivariant convolutional layer is standard cunslation-eqiiaaainn the set of potential groups for mutation in EquiNAS and as [G] in EquiÑAS..
convolutional layer. We use (C₁, D1, C22, D2, C1, D1} as
We present experiments on image classification for variety of datasets. The Rotated MNIST dataset (Larochelle al. 2007, RotMNIST) is version of the MNIST handwritten digit dataset but with the images rotated by any angle. This task serves as a simple investigational study with known symmetry, while the following two tasks are more realistic and complex. The Galaxy10 DECals dataset (Leung & Bovy 2019 Galaxy10) contains galaxy images in 10 broad categories. The ISIC 2019 dataset (Codella et al. 2018, Tsscandl et al.] 2018 Combalia et al. 2019, ISIC) contains dermascopic images of 8 types of skin cancer plus null class. For Galaxy 10 and ISIC, down- sample the images to 64 x 64 due to computational constraints, which adds notable difficulty to the tasks. These tasks exhibit varying levels of rotational and reflectional symmetry, motivating architectural optimization to determine the most effective application of equivariance constraints.
Across all experiments, the architectures designed to have consistent channel dimensions once expanded to a standard cunslation-eqivarrin convolutional layer for each layer across models. Thus, constrained equivariance larger symmetry group results in fewer learnable parameters. A

---

Published as a conference paper at ICLR 2023
Method
RotMNIST
Galaxy
ISIC
EquiNASE
1.78 + 0.04
20.3 0.9
31.0 + 0.4
RPP (Finzi al. 2021
2.18 + 0.04
24.3 2.8
32.2 1.7
D4 baseline
1.78 + 0.11
50.8 17.0
32.1 + 2.4
C4 baseline
1.64 + 0.22
29.6 = 5.5
32.9 + 1.0
C baseline
5,02 + 1.15
31.6 - 4.8
33.2 1.5
C1 (prior: D2)
1.93 0.05
27.8 = 5.3
31.9 + 1.5
C1 (prior: Da)
3,40 0.07
25.9 = 2.3
31.4 + 2.6
C (prior: C4)
2.96 0.05
30.7 7.1
32.5 + 1.1
Table 1: Test enror (lower is better) in percent of incorrect classifications across tasks and ap- proaches. Statistics are aggregated over the final selected population of individuals for EquiNAS; and across 5 random seeds for all other methods. The best and second best average per task are highlighted. See Figures 6 in Appendix D for individual results and additional experiments.
RPP baseline
10
[D,, Do C. Co.
Final Test Error (%)
Parameter Count
[C, C. Ca. C.
[D D4 Da.
Ca baseline
[D D.. D4 Da. D. Da. Cal
DD Da. Da. Da. De. De Da D.)
10
20
30
40
50
Generations
Figure 2: Historical parameter counts of all selected individuals for EquiNAS on RotMNIST. The architectures of the final selected population are labeled. Each history is colored by the final test accuracy, measured of each individual upon removal. For other tasks, see Figure in Appendix D
layer constrained C4 equivariance has IC& Dal = times many independent channels and as many parameters as a layer constrained to D3 equivariance. This is notably different paradigm than other works that equate parameter counts across architectures with different equivariance properties.
As baseline comparisons, we train and test G-CNNs with static architectures. In addition to the static baselines, re-implement the residual pathway priors (RPP) approach by Finzi et al. (202T) as C₁ equivariant layer with regularization in parallel with Da equivariant convolutional layer.
Further experiment details such as architecture details and other hyperparameters are Appendix C For each paradigm of experiments, we present results in the following subsections, with general discussion in Section Additional ablation and random search baselines are in Appendix D
6.1 EVOLUUIONNARY EQUIVARIANCE-AWARR NAS
The classification test errors are listed in Table The advantages of equivariance search methods are most apparent the Galaxy benchmark. While EquiNAS outperforms most baselines on RotM- NIST and all baselines ISIC, it has similar performance and some final architectures to the D baseline for both tasks. However, the D₂ baseline fails at the Galaxy 10 task, demonstrating that the same equivariant architecture not always be naively applied. Both search methods, EquiNAS and RPP, outperform all baseline models on Galaxy 10, and by a large margin for EquiNAS
The evolutionary progress RotMNIST is shown in Figure the selected population maintains a fully equivariant network in every generation. The final selected population originates from two main lineages, staying fully equivariant until the last generations and the other diverging from

---

Published as a conference paper at ICLR 2023
Method
RotMNIST
Galaxy10
ISIC
EquiNASp
2.29 0.27
21.8 1.2
32.8 0.6
RPP (Finzi al. 2021
2.89 + 0.27
22.0 1.8
31,5 0.9
D4 Baselme
2.97 1.50
22.5 +: 2.0
32.0 1.0
C4 Baseline
2.43 0.54
22.2 +
32.8 1.0
C Baseline
3,97 + 0.75
26.5 + 1.5
32.9 = 3.1
Table 2: Test eπror (lower is better) in percent incorrect classifications across tasks and ap- proaches. The best and second best average errors per task are highlighted. See Figures g in Appendix D for individual results and additional experiments.
Layer
Layer
Layer
Layer 4
Layer
Layer 6
Layer
1.0
RoMNNST
0.5
0.0
10000
10000
10000
20000
10000
10000
10000
1.0
Calany10
[2]
0.5
0.0
0.0
15000
15000
15000
15000 0
15000
15000
15000
15000
Training Stops
Figure Architecture weighting parameters by layer for one selected trial from RoMMNIST, Galaxy 10, and ISIC. For other trials, Figures 10
12 in Appendix
the fully equivariant network midway through, showing that training with dynamically constrained parametrizations can produce performant models.
In addition to the normally initialized static baselines, we also train and test baselines that are initial- ized with priors to larger symmetry groups. These are implemented by initializing all layers to be constrained to the prior symmetry group, then using the equivariance relaxation morphism on each layer. EquiNAS g scarches for relaxation schedules that yield trained priors on equivariance, while these additional baselines yield untrained priors. The results in Table Table show that, the C' -q-qqiivrr networks generally improve with either equivariance prior, while the C4 equivariant networks per- form better with D& equivariance initialization only when the D4 constrained baselines also work well. The untrained prior methods do not perform as well as EquiNAS on RotMNIST, showing the benefit of investing baselines with priors have better performances than their constrained baseline counterparts.
training time the constrained equivariance. For the other tasks, the
6.2 DIFFERENTIABLE EQUIVARIANCE NAS
The classification test errors are listed in Table EquiNAS achieves better test accuracy than the other comparable methods on RotMNIST and Galaxy10. 10. Due to differences in training protocol, only comparisons of relative rankings with Table [ are possible: baseline methods accuracies fol- low similar ranking patterns, suggesting the benefit of general CA equivariance for RotMNIST and Galaxy10 10 and general D4 equivariance, including RPP, for ISIC. In this training protocol notably with adaptive optimizers, the results more consistent across methods and trials.
The dynamics of architecture weighting parameters for one exemplary trial per task are shown in Figure 3 The general trend of less constrained layers toward the output supports the conjecture of local equivariance being beneficial. However, this effect is less consistent for ISIC, the only task where EquiNASp did not exceed baselines, possibly indicating less inherent symmetry. As seen in
8

---

Published as a conference paper at ICLR 2023
Appendix D the final mixing of architectures for ISIC included high level of C1, indicating that feature analysis outside of these symmetry groups is important for this benchmark.
Previous differentiable NAS works often used regularization of network size even architecture weighting parameters themselves to encourage efficient architectures with a single highly weighted choice for each layer. However, algorithm shows strong preference for single, more equivariant and thus more expressive layer, notably to D4 or C4 equivariance, without such regularization. This may be due to the bilevel optimization dynamics: more constrained layers may be able make more effective updates and thus become favorable compared to the lagging larger layers.
7 DISCUSSION
To our knowledge, this is the first work which proposes search methods for networks with dynami- cally constrained equivariance. Many NAS approaches separately search for architecture and then reinitialize and retrain the weights, while our two proposed approaches find an optimal architecture with trained weights in single process, notably with dynamically constrained weights. Gradient- based tuning [Maclaurin et al. 2013) has shown the benefit not only of optimizing hyperparameters but also of dynamically adjusting them during training (Lichtarge et al. 2022) Dynamically con- strained weights can reap the benefits of specialization and generalization over the course of training.
Our two equivariance-awan NAS approaches have distinct approaches: EquiNAS searches for architectures composed of discretely equivariant layers, while EquiNAS searches for continuous mixtures of equivariance within each layer. The EquiNAS algorithm avoids many known problems in differentiable NAS such as the discretization gap that occurs when searching over continuous relaxation of a discrete architectural search space (Xie et al. 202T), such as that of EquiNAS Towards searching for discretely equivariant layers using the G -ixxd equivariant layer, proximal NAS algorithms use techniques such as projection (Yao et al., 2020) and stright-thhouug estimation Li et al. 2022) to avoid the discretization gap and thus may be effective for this application.
EquiNAS is innately greedy: at each selection step, the population evaluated by known current performance rather than unknown final performance, biased to architectures that train quickly. Net- works with more equivariance constraints tend to learn faster, but cquivariance relaxation may yield large gradients for newly unconstrained parameters and thus fast increases in performance. Further work could utilize metrics for final performance, such as proxies (White et al. 2022).
The theoretical and algorithmic contributions of this work are applicable beyond the image classifi- cation experiments presented to architectures with parametrized equivariance to any discrete group. We leave the extension to other group representations and domains as future work, such the con- tinuous case via careful analysis of the regular representation, still given G' \ G is finite.
Our proposed equivariance- or architectures for datasets with hypothesizable symmetry. EquiÑAS may particularly work well on tasks that benefit from local equivariance, determined by analyzing the architecture weighting parameters from first applying the more efficient EquiNAS as well as for finding good discrete architectures within which to retrain weights, based on the ablation and random comparisons. We thus recommend EquiNASp for practical applications if the final model is not restricted to discrete equivariance, which case can be used to inform design decisions for applying EquiNAS
NAS problems be practically applied find effective models
Beyond NAS, the equivariance relaxation morphism could be used in other applications such as fine-tuning and distillation. Layers of pre-trained equivariant network could be expanded via equivariance relaxation before fine-tuning on the same or new task. Similarly, network could be distilled to a wider architecture for additional performance benefits.
Conclusion We present two mechanisms towards equivariance-avvee architectural optimization, the equivariance relaxation morphism and the [G]-mixed equivariant layer, and two NAS algorithms that respectively implement these mechanisms evolutionarily as EquiNAS and differentiably as EquiNASp. We investigate how dynamic equivariance achieved by these algorithms affects the training and performance of models across multiple image classification tasks of varying complexity and assumed symmetry, demonstrating that these techniques search for performant architectures and weights even on noisy tasks. The proposed mechanisms and algorithms are extendable beyond vision tasks any architecture with parametrized equivariance any discrete group.

---

Published as a conference paper at ICLR 2023
REFERENCES
Devanshu Agrawal and James Ostrowski. A classification of G-iivariaat shallow neural networks. Advances in Neural Information Processing Systems, 35, 2022. 2
Sourya Basu, Akshayaa Magesh, Harshit Yadav, and Lav R Varshney. Autoequivariant network search via group decomposition. arXiv preprint arXiv:2 104,048884 2021.
Michael Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. (0. 13478, 2021. [
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion analy- sis toward melanoma detection: A challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), hosted by the International Skin Imaging Collaboration (ISIC). In International Symposium on Biomedical Imaging, pp. 168-172. IEEE, 2018. 6
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Confer- ence on Machine Learning, pp. 2990-2999, PMLR, 2016.
Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern, Susana Puig, et al. BCN20000: Der- moscopic lesions in the wild. arXiv preprint arXiv: 1908. 02288,, 2019.
Gamaleldin Elsayed. Prajit Ramachandran, Jonathon Shlens, and Simon Kornblith. Revisiting spa- tial invariance with low-rank local connectivity. In International Conference on Machine Learn- ing. pp. 1868-2877) PMLR, 2020. 5
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. In NIPS Workshop Meta-Learning, 2017.
Andrea Falanti, Eugenio Lomurno, Stefano Samele, Danilo Ardagna, Matteo Matteucei, et al. POP- NASv2: An efficient multi-objective neural architecture search technique. In International Joint Conference on Newral Networks, 2022. 5
Mare Finzi, Gregory Benton, and Andrew G Wilson. Residual pathway priors for soft equivariance constraints. Advances Neural Information Processing Systems, 34:30037-30049, 2021.
7
Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and Cooperation in Neural Nets, pp- 267-285. Springer, 1982. [
Israel Herstein. Topics in algebra. John Wiley & Sons, 2006
Risi Kondor and Shubhendu Trivedi. On the generalization of cquivariance and convolution in neural networks to the action of compact groups In International Conference on Machine Learning, pp- 2747-2755. PMLR, 2018.
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empir- ical evaluation of deep architectures on problems with many factors of variation. In International Conference on Machine Learning, pp. 473-480. PMLR, 2007. 6
Yann LeCun, Bernaard Boser. John S Denker, Donnie Henderson, Richard E Howard, Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied handwritten ZIP code recogmition. Neural Computation, 1(4):541-551, 1989.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86 86(11):2278-2222 1998.
Henry W Leung and Jo Bovy. Deep learning of multi-element abundances from high-resolution spectroscopic data. Monthly Notices of the Royal Astronomical Society, 483(3):3255-322) 3277,,777,40 2019.
10

---

Published as a conference paper at ICLR 2023
Yanyu Li, Pu Zhao, Geng Yuan, Xue Lin, Yanzhi Wang, and Xin Chen. searchh Efficient neural architecture search via channel pruning and structural reparamcterization. In International Joint Conference on Artificial Intelligence, 2022.
Jared Lichtarge, Chris Alberti, and Shankar Kumar. Simple and effective gradient-based tuning of sequence-to-sequence models. International Conference on Automated Machine Learning Workshop Track, 2022. 9
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2018. 2
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf. NSGA-Net: neural architecture search using multi-objective genetic algo- rithm. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 419-427, 2019. 2
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- hyperparameter optimiza- tion through reversible learning. In International Conference on Machine Learning, pp. 2113- 2122. PMLR, 2015. y
Kaitlin Maile, Hervé Luga, and Dennis G Wilson. Structural learning in artificial neural networks:
A neural operator perspective. Transactions on Machine Learning Research, 2022. D
Chandra Mouli and Bruno Ribeiro. Neural networks for learning counterfactual g-invariances from single environments. In International Conference on Learning Representattions 2021. 2
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter- sharing. In International Conference on Machine Learning, pp. 2892-2901. PMLR, 2017. |
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning, pp. 2902-2911, 2. PMLR, 2017. 1
David W Romero and Suhas Lohit. Learning partial equivariances from data. Advances in Neural Information Processing Systems, 35, 2022. 2
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in Neural Information Processing Systems, 29, 2016.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data. 5(1):
1-9, 2018. 6
Tycho FA van der Ouderaa, David W Romero, and Mark van der Wilk. Relaxing equivariance constraints with non-stationary continuous filters. Advances Neural Information Processing Systems, 35, 2022. 1
Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly sym- metric dynamics. In International Conference on Machine Learning. PMLR, 2022.
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In International
Conference on Machine Learning, pp. 564-572. PMLR, 2016. [
Maurice Weiler and Gabriele Cesa. General E())eqquiaa steerable CNNs. Advances in Neural Information Processing Systems, 32, 2019. 5
Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolu-
tional networks isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106.000 06020, 2021.
Colin White, Mikhail Khodak, Renbo Tu, Shital Shah. Sébastien Bubeck, and Debadeepta Dey. A deeper look at zero-cost proxies for lightweight NAS. In ICLR Blog Track, 2022, URL https:
git.tb. httpoloo//2220222 zero- http://eeooooooooooooooe
9

---

Published as a conference paper at ICLR 2023
Lingxi Xie, Xin Chen, Kaifeng Bi, Longhui Wei, Yuhui Xu, Lanfei Wang, Zhengsu Chen, An Xiao, Jianlong Chang, Xiaopeng Zhang, et al. Weight-sharing neural architecture search: battle to shrink the optimization gap. ACM Computing Surveys (CSUR), 54(9):1-37, 2021. g
Quanming Yao, Ju Xu, Wei-Wei Tu, and Zhanxing Zhu. Efficient neural architecture search via proximal iterations. In AAAI Conference on Artificial Intelligence, pp. 6664-6671, 2020. 9
Raymond A Ych, Yuan- -ing Hu, Mark Hasegawa- and Alexander Schwing. Equivariance discovery by learned parameter-sharing. In International Conference on Artificial Intelligence and Statistics, pp. 1527-1545. PMLR, 2022.
2
Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization. In International Conference Learning Representations, 2020.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning, In International Conference on Learning Representations, 2017.
A ADDITIONAL BACKGROUND
A.1 SYMMETRIES IN NEURAL NETWORKS
A symmetry of an object is mapping of the object onto itself such that structure is preserved. A symmetry group G a set of such mappings along with a binary operation :: G x G - G, known the group product, that satisfies axioms for closure, associativity, the identity, and the inverse (Herstein (006). A group G acts on a set X' via the group action G x X' + X, (g, x) + g.z that satisfies axioms for identity and compatibility: X' called G-space.
Equivariance is the property of a mapping such that transformation of the input results in equiv- alent transformation of the output. Formally, a mapping h: X + between two G-spaces is G-equivariant if for all g E G and x E X' we have: ((g.z) = g.h(z). For example, an image seg- mentation neural network should be (qquqquivrri; shifting the input should result in the same shift in the output.
Invariance is a special case of equivariance, where the output of the function is completely inde- pendent of transformation of the input. Formally, mapping h: X' + Z is G-invariant if for all g E G and æ X' we have: h(g-r) = har).. For example, an image classification network should be ((2)-invaraan:: shifting the input should not change the output. Symmetries leave objects invariant.
For two groups and H with group products "G and "и respectively where H acts on G with group action ., the (outer) semi-direct product G' x H of H acting on G' is group composed of the set of elements G x H with group product (g, h) (g', H') = (g a (h.g'), ) h . h²) and inverse (g, h)- (h 99 hh-1 h)
A subgroup H of G is a nonempty subset with the same group product that also fulfills the group axioms. Then, g.l {g h/h H} and Hg {h- g/h H} denote the left coset and right coset, respectively, of with representative g.
A.2 NEURAL ARCHITECTURE SEARCH
Evolutionary algorithms are optimization methods inspired by evolution in biology. where individ- uals in population compete with their phenotypic traits in order to pass on their genotypic traits to offspring. The population is the current collection of individuals. Each individual is an instance of the object be optimized and has a genotype that is decoded into a phenotype. In this case, each individual is a neural network, with genotype that encodes the parametrized equivariance group of each convolutional layer, represented as a vector of integers. The individual continues training on the task before competing against other individuals to be selected as parent to mutate to generate the next population. Each parent itself is kept for the next population, as well as each valid child that is generated via the equivariance relaxation morphism, such that they are functionally equivalent to their parent at initialization (although with a different architecture) and thus have the same fitness before training. Each individual in this population is partially trained, such that these children di- verge from their siblings and parent, so that the next set of parents may be selected and this process repeats.
12

---

Published as a conference paper at ICLR 2023
Pareto dominance can be used in multi-objectiin optimizations to select the next parent population. For a population of individuals each scored in n objectives, an individual is pareto-optimat optimal if no individual has at least one strictly better score for objective without that of any other objectives being strictly worse. The pareto front is the set formed by all pareto-optimal individuals.
B IMPLEMENTATION DETAILS
Group convolutional layers The implementation of regular group convolutional layers can be viewed as a special case of our proposed equivariance relaxation morphism. With the preliminaries given in Section 4. and the case G" = T(2). and v are computed such that ((.,)) = felg's) and (d),),(( (g') va,c(tt'g's g) for each g' T(2), [Ci-1], s,t R. and d E [Ci].
Let Se |R] The learnable parameters of the G--equivariant 1th layer with Ct output channels, corresponding to vú, are stored as tensor of size C x Ct-1 x Sc, x K2 x K,. The filter transformation expands this filter tensor by performing the action of each r E R on another copy of the tensor to expand its shape along new dimension, resulting in a tensor of size C XSS xC XS, x K2 x K₁, which is reshaped CiSc, CC-18G, x K2 K₁. The imput tensor to the 1 layer, corresponding to f. is in the shape of B X x - x Wi-1, which is reshaped to B x 1CCC-1Sc x X x W and convolved with the expanded filter. The output of shape x CiSG, x HI Wi reshaped to B x C x Sc, x Hi x W.
Equivariance relaxation morphism To implement the equivariance relaxation morphism, the new filter tensor is initialized by applying Equation 4 such that result of applying the preceding filter transformation is equivalent. Our implementation of group actions relies on group channel indexing to represent the order of group elements: to ensure this is consistent before and after the morphism, the appropriate reordering of the output and input channels of the expanded filter are applied upon expansion. The new filter tensor has a shape of Cj x Ci-1/8 S K₁ K₁, The [G]- mixed equivariant layer is built on top of this implementation, also using proper input and output channel reordering between layers to ensure correct mixing of group channels.
C EXPERIMENTAL DETAILS
Architecture backbone For both EquiNAS and EquiNAS experiments, we use the same back- bone architecture, such that the static baselines have the same architecture across experiments. The architectures have lifting layer followed by 7 group convolutional layers, for a total of convolu- tional layers. After 4 layers, the channel count doubles, from 16 to 32 for D₁ equivariant layer and scaling up for smaller symmetry group equivariance constraints. An average pooling layer is placed after every other layer for all architectures and additionally after the fifth and seventh convolutional layers for Galaxy10 and ISIC. After the final group convolutional layer is group-dimension aver- age pooling followed by two linear layers to the output dimension. Every convolutional and linear layer except the output layer is immediately followed by batchnorm then ReLU.
Hyperparameters The hyperparameters for each algorithm are selected such that baselines only differ by training time and optimizers. The learning rates were selected by grid search over baselines on ROMMNIST. For all experiments in Sections 6.]] we use a simple SGD optimizer with learning rate 0.1 to avoid confounding effects such as momentum during the morphism. For EquiNAS the parent selection size is 5, the training time per generation is 0.5 epochs, and the number of generations is 50 for all tasks. Baselines were trained for the equivalent number of epochs. For all experiments in Section 6.2 we use separate Adam optimizers for V and Z, each with a learning rate of 0.01 and otherwise default settings. The total training time is 100 epochs for RotMNIST and 50 epochs for Galaxy10 10 and ISIC. For RPP, we C1 Ci-cquivariant layer with an L2 regularization parameter 1e-6 in parallel with a Du-equivariant layer without regularization.
For ROMMNIST and MNIST, we the standard training and test splits with batch size of 64, reserving 10% of the training data as the validation set. For Galaxy10, we set aside 10% of the dataset the test set, reserving 10% of the remaining training data the validation set. For ISIC, we set aside 10% of the available training dataset as the test set, reserving 10% of the remaining data the validation set and the rest training data. For the latter two datasets, we resize the
13

---

Published as a conference paper at ICLR 2023
images to 64 x 64 due computational constraints and batchsize of 32. The validation sets were previously used for hyperparameter tuning: for experimental results, they are only used for the experiments in Section 6. as necessary for the E algorithm. No data augmentation is performed, although the datasets are normalized.
D ADDITIONAL EXPERIMENTS AND FIGURES
Layer
Layer
1.0
0 0.5
0.0
5000
5000
1.0
0 0.5
0.0
5000
5030
5000
3.0
0.5
0.0
5000
5000
3.0
6 0.5
0.0
5000
5000
0.5
0.0
5000
5000
5000
2.0
2 0.5
0.0
5006
5000
Training Steps
Figure 4: EquiNAS, on classification of six augmentations of MNIST, where each row is trial on MNIST augmented to exhibit symmetry to the labeled group.
To explore the symmetry discovery of EquiNASp. apply it six augmentations of the MNIST dataset (LeCun et al. 1998), where each augmentation applies the group actions of each group in {C1, C22 C1. Dj. D2, Dr} respectively. The resulting architecture dynamics of this experiment shown in Figure 4 showing that less augmented versions still have some inherent symmetry, while more augmented versions induce stronger architectural changes towards layers that equivari- ant to larger groups. Across all augmentations, earlier layers tended towards more constraints to cquivariance.
As ablation studies and comparisons, we implement two kinds of random search for each NAS method. The first ablates smart architecture search: EquiNAS Random Select works described in Algorithm|] but with random parent selection (instead of pareto-front selection) and EquiNASp Random works as described Algorithm 2 but with random Z updates (instead of gradient de- scent) by shuffling Z gradients. The second is more akin to standard NAS random search: for the evolutionary paradigm, we train randomly selected static architectures in the discrete architecture search space for the same training time and selecting the top 5 by validation accuracy, and for the differentiable paradigm, we train randomly selected static architectures in the continuous archi- tecture search space and selecting the top 5 by validation accuracy. 30 and 25 were respectively calculated to be approximately the same compute cost as the trials EquiNAS and EquiNAS p. These are labeled as "Random Static" for both the evolutionary and differentiable paradigms. Since Random Static trains static architectures while EquiNAS, ø and EquiNAS, dynamically search for both architectures and parameters, take the best 5 architectures for each and retrain their param-
14

---

Published as a conference paper at ICLR 2023
eters from scratch as in the standard NAS paradigm, labeled as EquiNAS Retrain and EquiNAS Retrain, respectively, for fair comparison to Random Static.
The results of these additional experiments compared against those of our main algorithms and baselines in Figures
and
Shown in Figure EquiNAS outperforms EquiNAS Random Select, showing the benefit of using informed selection to guide the relaxation of equivariance constraints over training. Additionally, E g Retrain outperforms the Random Static baseline, showing that using compute in an informed search is more beneficial than just randomly searching the space of static architecture constraints.
EquiNASp finds competitive architectures on average and can find architectures which outperform baseline choices like architectures fully equivariant to Cj or D4. From comparing EquiNAS Re- train to EquiNAS results in Figure [8, retraining resulting architecture is not consistently better or worse than using the final weights from EquiNASD. showing that there isn't disadvantage to training weights during search and avoiding the additional cost of post-search training step.
The use of randomized loss information in Random Static in Figure g shows that an informed search for architecture hyperparameters is generally useful. However, experiments on the ISIC benchmark demonstrates that the architecture search can be deceptive and that random loss information can outperform informed loss. This motivates exploration into the use of noise during the search process for architectural parameters.
A sampling of random continuous architectures in Random Static in Figure 8 shows that random ar- chitectures can perform well on problems where fully equivariant architectures like the D2 baseline already perform well. However, the Galaxy 10 problem, the D₁ and C₁ baseline have high vari- ance, suggesting that fully equivariant architecture is sub-optimal. On this baseline, EquiNAS greatly outperforms search of random architectures, demonstrating that EquiNAS, D discover the appropriate equivariance for a specific dataset over fixed or randomly selected architectures.
The search space for EquiNASp is already well-formed for random architectures, compared to the discrete search space of EquiNAS This is enabled by the [G]-mixed equivariant layer, which contribution of this work. Random non-mixed equivariant architectures do worse all three benchmarks compared to random architectures which use the [Gi-mixed equivariant layer. This can explain why the E results are closer to random baselines than the EquiNAS g results, as the search space permits easily finding the appropriate mix of equivariances compared to a discretized search space.
Our methods enable searching for both architecture and parameters concurrently in single training process. This approach is more efficient than NAS approaches that only search for architectures, re- quiring retraining process within the resulting architecture for evaluation. However, these ablation and random search comparisons show that our algorithms may get performance gains from adding retraining phase with tradeoff of further compute cost.
15

---

Published as a conference paper at ICLR 2023
RotMNIST
Galaxy10
ISIC
Test Error (%6
Telt Error (%6
Test Error (%6
40
ERuWAS, Pardoo Safeett
EquINAS, Natrain
EquiNASs
Ramibm Statte
RPp
maniine
Bassdine
Bazstine
Uirior: Da)
teries Upvior: Cul
EqUINAS, Retrain EqUIASS, Fundom Random Statie
EquINAS
RPp
Dataiin Bassline
Bamaiin
(ever: ou levies (erior:
EquiNAS, Retrain ERUINAS Furdom Salect Rando Statie
EquINAS
RPe
Datadine
Racaline
Bansline
lprior Da)
Leries (erfor:
Figure 5: Test errors for experiments of Section 6.1
RotMNIST
Galaxy10
ISIC
EquINAS
36
(train)
RPP (test)
60
baseline
baseline
C₁ baseline
(prior:
lest Lrror (8)
Error (86)
(prier:
G (prior: Cs)
Test
30
20
28
10f
10f
10"
Parametor Count
Parameter Count
Parameter Count
Figure 6: Test errors against stored parameter count for experiments of Section 6.1 For EquiNAS g parameter counts of the final models are shown, although training begins with the same parameter count as the D2 baselines as shown in Figures[landj Although RPP trains with a higher parameter count, the parameters may be stored in single filter per layer using Equation TT for testing.
16

---

Published as a conference paper at ICLR 2023
1e6
RPP
70
G baseline
60
[Ca, Ca, Ca. Ca C.
[Ca C.. Ca. Ca
Ci
Final Test Error (9)
50
Parameter Count
Go Cu
CCa. Ca.
30
baseline
20
D baseline
10
20
30
40
50
Generations
(a) Galaxy10 10
1e6
8
RPP
baseline
37
36
Paramotor Count
Final Test Error
35
D4 Cy.
34
Dr.
[D,, Dr, D., Da, Dr. D,1
C. baseline
32
(Dt, D. Da. Da. Cal [Du, Dr. Da Dr Dr. Dr. Co Ci]
31
30
baseline
Generations
(b) ISIC
Figure 7: Historical parameter counts of each selected individual for EquiNAS (see Figure 2 for RotMNIST).
RotMNIST
Galaxy10
ISIC
28
37
26
Test Error (%)
T
35
24
Error
34
Test
22
20
31
EquitA5 Ratrai
EquiNASo
EquiNASp Bandom
Random Static
D2 Bateline
C. Baseline
G Baseline
EquiNASe EquiRAS, Retrain
EmuntdS Random z
Random Static
Di Barsline
RPp
Ca Baseline
Ci Bassllne
EquiRAS, Retrain EquiNASS Random Z
EquiNASo
Random Static
De Baseline
Basuline
Baseline
Figure 8: Test errors for experiments of Section 6.2
17

---

Published as a conference paper at ICLR 2023
RotMNIST
Galaxy10
ISIC
38
DquNAAs [tect)
tquilibip Itain)
37
26
Susaline
Base me
Base
Test Entor 198)
335
Enror
34
Test
+
22
33
flo
20
30
10P
10'
104
10'
10a
10'
Parameter Count
Parameter Count
Parometer Count
Figure 9: Test errors against stored parameter count for experiments Section 6.2 EquiNAS and RPP train with higher parameter count, the parameters may be stored in single filter per layer using Equation [] testing.
Although
Layer
Layer
Layer
Layer 4
Layer 5
Layer 6
Layer
Layer
0.25
0.00
1.00
0.75
0.00
1.00
0.75
0.0
0.25
0.00
Www.-
10000
10000
10000
10000
10000
10000
10000
10003
Training Steps
Figure 10: Architecture weighting parameters by layer for all trials on RotMNIST.
18

---

Published as a conference paper at ICLR 2023
Layer
Layer
Layer
Layer 5
Layer
Layer
Layer
1.00
0.75
0.50
0.25
0.00
1.00
0.50
0.25
0.00
1.00
0.75
0.0
0.00
15000
15:00
15000
15000
15000
Training Stops
Figure 11: Architecture weighting parameters by layer for all trials Galaxy 10.
Layer
Layer
Layer
Layer
Layer 5
Layer 6
Layer
Layer
1.00
0.75
0.50
0.79
0.00
1.00
0.50
0.25
0.00
1.00
0.75
0.50
15000
15000
15000
15000
15000
15000
Training Steps
Figure 12: Architecture weighting parameters by layer for all trials on ISIC.
19


Reviews:
[{"review_id": "fAo8AGe7yWO", "reviewer": "Reviewer_7fUJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors explore technique towards leveraging equivariances to subgroups rather than the entire group. First they propose equivariance relaxation morphism $-$ which partially removes weight sharing constraints on the entire group (without losing out on functionality) and the mixed layer allows to learn partial equivariance as a weighted sum. The authors then propose two equivariance aware NAS algorithms - one evolutionary and the other differentiable - and show good experimental results in both cases.", "summary_of_the_review": "Currently the strengths of the paper out weigh the weaknesses. I would be happy to increase my scores if the authors can answer the raised questions.", "strength_and_weaknesses": "**Strengths:**\n1. The formulation of partial/ approximate equivariances/ invariances $-$ as a mixture over subgroups is interesting and an important problem to tackle.\n2. Strong experimental results for both their learning paradigms.\n3. For a person with sufficient group theory background - the paper is easy to understand.\n\n**Weaknesses and correspnding questions/ suggestions**:\n1. Lack of comparison of the evolutionary paradigm with a similar approach [1] which searches over the subspace lattice  to identify te groups to be equivariant/ invariant to.\n2. It is hard to see how this extends to non-finite groups which are not compact. Would suggest that the authors comment on this.\n3. While the paper is understandable for someone who has the necessary group theory background - it is almost impossible for general audience - would recommend the authors add necessary background and preliminaries to the appendix.\n4. Studies in out of distribution settings are missing (for e.g. train only contains images rotated by 0-15 degrees, but test has those between 60-180 degrees) - this is will give us more insight into the weights learned.\n\n**Minor:**\n1. Please make sure the title on openreview matches the title on the paper\n\n**References**\n1. Mouli, S. Chandra, and Bruno Ribeiro. \"Neural Networks for Learning Counterfactual G-Invariances from Single Environments.\" ICLR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "epzPpf3JLE", "reviewer": "Reviewer_6SKq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two approaches to optimizing the weights and architecture of networks with approximate equivariance.  The first approach EquiNAS_E uses an evolutionary algorithm to progessively weaken the symmetry constraints in the layers of an equivariant NN as it trains.  The second EquiNAS_D uses differentiable weights, optimized alternatingly with the normal network weights, which combine different kernel with different levels of symmetry constraints.  Both methods are tested on image classification tasks.", "summary_of_the_review": "This method seems quite interesting to me due to the fact that there are datasets in which strict equivariance may not perform well due to being too strict of an inductive bias or because the constraints interfere with optimization.  Moreover, equivariant networks have a great deal of hyperparameters in the amount of symmetry they exhibit at various levels, so a more sophisticated approach than random searching is worthwhile.  The proposed idea for relaxing symmetry constraints is appealingly simple and reasonable.  However, the experimental results are not that convincing as the proposed method is often outperformed by equivariant networks or by relaxed networks with equivariant initialization.  In the case EquiNAS_D does significantly outperform on Galaxy10, it has very high variance.  Since the relaxation method also adds parameters, it is not clear when improved performance is due to parameter count vs. too strong symmetry constraints.  ", "strength_and_weaknesses": "## Strengths \n- Equivariant networks have more hyperparameters to tune due to the choice of symmetry group.  For example, in E(2)-CNN (https://arxiv.org/abs/1911.08251), the authors experiment with symmetry groups $C_n$ and $D_n$ for many values of n before finding n=12 to work best.  This paper proposes a method for automatically determining the correct level of symmetry.\n- Moreover, on many problems end-to-end strict symmetry is not desirable if the symmetry in the domain is only approximate or if symmetry constraints interfere with optimization.  The proposed method adds to the relatively few methods which learn relaxed symmetry constraints and adapt to the level of symmetry in the data.\n- I found the equivariance relaxation morphism to be a simple and effective idea for relaxing symmetry constraints in networks on a per-layer basis.\n- The [G]-mixed equivariant layer also seems like a reasonable idea for selecting the degree of equivariance by differentiably choosing weights over kernels with different levels of equivariance.  In theory this allows the model to learn a specific equivariance or an approximate equivariance which is in-between.  It also allows the network to reduce the symmetry constraint in later layers, which is what is done effectively by CNNs in practice by downsampling with stride.  \n- Experimental results on the learned subgroup weights (Fig. 2) reveal interesting trends showing that the networks do select for higher equivariance in early layers and lower equivariance in later layers which matches what has found to be effective in practice for equivariant networks. \n \n\n## Weaknesses / Questions \n- I am not sure of the framing of the method.  I am not an expert on NAS and am open to correction, but it seems to me both proposed methods find a trained network, not an architecture, since they are training the network weights and selecting an architecture at the same time.  The final architecture may not have great performance if trained from scratch.  In this sense, it seems the method is more a special training procedure than NAS method.   EquiNAS_D, in particular, is simply an architecture in which some weights are optimized alternatingly. \n- Given the evolutionary algorithm starts with a single network, how do we know to what extent its evolution is guided by architecture versus the specific weights in the mutated networks?  In experiments, I think both population level variance and initialization variance should be accounted for.  \n- One of the biggest issues is the fact that the relaxation morphism increases the number of parameters making it difficult to distinguish performance increases are from relaxing constrains or increasing parameter count.  The authors do discuss this issue and suggest their optimization strategy over both parameter count and accuracy helps.  It would be better, however, if the number of parameters could be preserved by the relaxation. \n- The experimental results are not completely convincing of when this method would add significant value over simply doing hyperparameter search over equivariant networks.  In Table 1, an equivariant baseline is best on rotMNIST and ISIC, and a non-rotationally equivariant network with equivariant initialization is best for Galaxy10.  In table 2, EquiNAS_D does do best for rotMNIST, but the score is much lower than the best values in E(2)-CNN after full tuning of the best equivariant method.  In Galaxy10, EquiNAS_D does outperform, but with very high variance.  \n- I have some issues with the clarity of the description of 4.2.  The condition that all groups in $[G]$ be subgroups or supergroups of all other groups in $[G]$ does not appear to be met by the expermental example since $D_2$ is not a subgroup or supergroup of $C_4$.  It's also not clear to me why this condition is necessary.  It is a fairly strict condition, although I think it would include useful cases.  I am also a bit unclear on the form of the input signal $f$.  In order to be convolved with $\\tilde{\\Psi}$ and have $G$-equivariance for each $G$ in $[G]$, $f$ should be defined over a supergroup of all $G \\in [G]$.  Is this the case?   I am correct that if $z_G$ is a one-hot vector on the group $G$, then the operation is $G$-equivariant? \n\n## Questions \n- If the evolution algorithm can only mutate in the direction of symmetry constrain relaxation, doesn't this bias the algorithm towards relaxing all the symmetry constraints? \n- Does this work have a relation to diffstride (https://arxiv.org/abs/2202.01653) in that stride is typically a hyperparameter that corresonds to a choice of the subgroup of the translation group and can be shown to differentiably optimized? \n- After equivariance is broken in a given layer, there isn't much point in principle to persevering it downstream.  Are mutations automatically applied to all following layers or did your method discover this?  If so, I think that is interesting enough to be considered a strength. \n\n## Minor \n- In Eqn 1, if you reverse the order of $f$ and $\\Psi$ and drop the sum over c and indices $c,d$, the equation can be simplified using matrix multiplication. \n- 4.1, para 2, \"neutral element\" -> \"identity element\" \n- 4.1 may benefit from writing as a proposition\n- 4.1, Para 3, Line 5,  $h \\in G'$ -> $h' \\in G'$ \n- Sec 5., Line 4, typo \"optimizffe\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ONH_eijaoX", "reviewer": "Reviewer_W8Dc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two methods for equivariance-aware neural architectural search (NAS). The motivation for this work stems from the fact that symmetries present in a data set might often be imperfect or non-explicitly known. The first method is based on the proposed equivariance relaxation morphism, a procedure that equivalently reparameterizes a group convolution layer to operate with equivariance constraints on a subgroup; an evolutionary NAS based on such morphism is later adopted. The other method, named $[G]$-mixed equivariant layer, linearly combines layers equivariant to different groups, whose coefficient is search through a differentiable NAS. Thoughtful experiments on three datasets are conducted to demonstrate the performance.", "summary_of_the_review": "I think the paper is interesting. I am willing to adjust my rating based on the authors' response to my questions.", "strength_and_weaknesses": "**Strength**\n1. The paper is well motivated. Most G-CNN literature assume that the symmetry group of the learning task is explicitly known and perfect. The paper instead uses a equivariance-aware NAS procedure to search for the optimal architecture for (potentially) unknown and imperfect group transformations.\n2. The paper is mostly well-written.\n3. The experiments are detailed and thorough.\n\n**Weakness**\n1. The equivariance relaxation morphism, which relies on reparameterizing the group convolution on a subgroup, seems to be limited to discrete group. Can this be extended to continuous groups?\n2. Moreover, the relaxation morphism is also restrictive in the sense that the number of the unstructured channel (c_out) cannot be adjusted after changing to a subgroup. This inevitably increases the model size as group size decreases, which is indeed the case for EquiNAS_D.\n3. The clarity of the paper in section 5.1 can benefit from more explanation on the technical terms such as population, pareto selection, pareto front, etc.\n4. In MNIST_rot, the review would like to see what happens when $C_8$ and $D_8$ are included in the picture.\n5. If I understand correctly, the reported EquiNAS_E result on Galaxy10 in Table 1 might be misleading. Figure 4(a) shows that only one NAS output in the end achieves significantly smaller test error compared to all the remaining results.\n6. Are the models involved in EquiNAS_D significantly larger compared to the baseline? After all, a linear combination of layers equivariant to different groups is taken. Does that mean the model is slow to train and (especially) test?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7PNIb_VXlXX", "reviewer": "Reviewer_RfYJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to search for neural architectures whose layers have equivariance that can exploit symmetries in the data. They propose two methods—network morphisms combined with evolutionary search and mixture relaxations combined with differentiable NAS—to search the space of symmetries. The methods are evaluated on some image classification tasks.", "summary_of_the_review": "I am currently leaning against acceptance to the rather limited empirical evaluation.\n\n# Post-rebuttal\n\nGiven the additional evaluations, with new datasets and comparisons with baselines, I think the paper has sufficient experimental backing / interesting methodology to be accepted. ", "strength_and_weaknesses": "### Strengths:\n1. The problem of finding the correct equivariance that can exploit symmetries in the data is important for the development of general-purpose NAS methods.\n2. The proposed approaches are simple and natural procedures.\n3. The writing is fairly clear and the experimental results are interesting.\n4. Code will be released upon acceptance.\n\n### Weaknesses:\n1. The experimental evaluation is somewhat limited. Given the fairly direct extension of prior work (morphisms and weight-sharing) to actually developing the search algorithms, I would expect more emphasis to be placed on experimental depth. This could either come in the form of evaluation on interesting benchmarks where we might expect finding symmetries to outperform regular CNNs (e.g. Tu et al. (2022)) and/or more detailed analysis on more involved search spaces of whether the proposed methods are recovering the “right” symmetry for the dataset, even if synthetic.\n2. The authors do not consider the random search baseline for their method. More generally, it would be useful to follow the NAS reproducibility checklist here (Lindauer & Hutter, 2020).\n\n### References:\n- Lindauer, Hutter. Best practices for scientific research on neural architecture search. JMLR 2020.\n- Tu, Roberts, Khodak, Shen, Sala, Talwalkar. NAS-Bench-360: Benchmarking neural architecture search on diverse tasks. NeurIPS 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster