Group Equivariant Subsampling
Authors: Jin Xu
Hyunjik Kim
Tom Rainforth
Yee Whye Teh
Venue: NeurIPS 2021 Poster (2021)
Keywords: group equivariance
group invariance
convolutional architecture
autoencoders
representation learning

Abstract: Subsampling is used in convolutional neural networks (CNNs) in the form of pooling or strided convolutions, to reduce the spatial dimensions of feature maps and to allow the receptive fields to grow exponentially with depth. However, it is known that such subsampling operations are not translation equivariant, unlike convolutions that are translation equivariant. Here, we first introduce translation equivariant subsampling/upsampling layers that can be used to construct exact translation equivariant CNNs. We then generalise these layers beyond translations to general groups, thus proposing group equivariant subsampling/upsampling. We use these layers to construct group equivariant autoencoders (GAEs) that allow us to learn low-dimensional equivariant representations. We empirically verify on images that the representations are indeed equivariant to input translations and rotations, and thus generalise well to unseen positions and orientations. We further use GAEs in models that learn object-centric representations on multi-object datasets, and show improved data efficiency and decomposition compared to non-equivariant baselines.

Group Equivariant Subsampling
Jin Xu
Hyunjik Kim?
Tom Rainforth
Yee Whye Teh
Department of Statistics, University of Oxford, UK. DDeepMind, UK.
Abstract
Subsampling used in convolutional neural networks (CNNs) in the form of pooling or strided convolutions, to reduce the spatial dimensions of feature maps and to allow the receptive fields grow exponentially with depth. However, it is known that such subsampling operations are not translation equivariant, unlike convolutions that are translation equivariant. Here, we first introduce translation equivariant Drbssamppinguussmpling layers that can be used to construct exact translation equivariant CNNs. We then generalise these layers beyond translations to general groups, thus proposing group equivariant subsumplingguussamplii use these layers to construct group equivariant autoencoders (GAEs) that allow to learn low-dimensional equivariant representations. We empirically verify on images that the representations are indeed equivariant to input translations and rotations, and thus generalise well to unseen positions and orientations. We further use GAEs in models that learn object-centric representations on multi- object datasets, and show improved data efficiency and decomposition compared to non equivarrit baselines.
We
Introduction
Convolutional Neural Networks (CNNs) known to be more data efficient and show better generalisation on perceptual tasks than fully-connected networks, due to translation equivariance encoded in the convolutions: when the input image/feature map is translated, the output feature map also translates by the same amount. In typical CNNs, convolutions are used in conjunction with subsampling operations, in the form of pooling strided convolutions, spatial dimensions of feature maps and to allow receptive field to grow exponentially with depth. Sassammplinnuuusmplinnggg operations are especially necessary for convolutional autoencoders (ConvAEs) (Masci et al., 2011) I) because they allow efficient dimensionality reduction. However, is known that subsampling operations implicit in strided convolutions or pooling layers are not translation equivariant (Zhang, 2019), hence CNNs that use these components are also not translation invariant. Therefore such CNNs and ConvAEs are not guaranteed to generalise to arbitrarily translated inputs despite their convolutional layers being translation equivariant.
reduce the
Previous work, such as Zhang (2019); Chaman and Dokmanić (2020), has investigated how to enforce translation invariance on CNNs, but does not study equivariance with respect to symmetries beyond translations, such rotations reflections. In this work, first describe subsampiing/upsampiinn operations that preserve exact translation equivariance. The main idea is to sample feature maps on an input-dependent grid rather than fixed one as in pooling or strided convolutions, and the grid is chosen according to sampling index computed from the inputs (see Figure 1). Simply replacing the in standard CNNs with such translation equivariant uubsamplig/lupsamping operations leads to CNNs and transposed CNNs that can map between spatial inputs and low- dimensional representations translation equivariant manner.
"Corresponding author: <jin. www.tattttt...... ax.ac.ub uk>
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

0
0
shift right by 1 unit
shift to right by unite
Figure 1: Equivariant subsampling on 1D feature maps with scale factor 2. The input feature map has length 8, and initially we sample from odd positions determined by Equation (1) (top). When the original feature map is shifted to the right by 1 unit (bottom left), the sampling index becomes 1, so we instead sample from even positions. When the feature map is shifted to the right by 2 units (bottom right), we again sample from odd positions, but the outputs have been shifted to the right by 1 unit correspondingly.
We further generalise the proposed subsampling/upsampling operations from translations to arbitrary groups, proposing group equivariant subsampping/uussappiing In particular we identify subsampling as mapping features on groups G to features on subgroups K (vice versa for upsampling), and identify the sampling index coset in the quotient space G/K. See Appendix A for a primer on group theory that needed to describe this generalisation. We note that group equivariant subsampling is different to coset pooling introduced in Cohen and Welling (2016), which instead gives features on the quotient space G/ K, and discuss differences in detail in Section 4. Similar to the translation equivariant aubsampling/upsampiing. group equivariant subsampling/upsampiing can be used with group equivariant convolutions to produce group equivariant CNNs. Using such group equvariant CNNs can construct group equivariant autoencoders (GAEs) that separate representations into an invariant part and an equivariant part.
While there is growing body of literature group equivariant CNNs (G-CNNs) (Cohen and Welling, 2016, 2017; Worrall et al., 2017; Weiler et al., 2018b,a; Thomas et al., 2018; Weiler and Cesa, 2019a), such equivariant convolutions usually preserve the spatial dimensions of the inputs (or lift them to even higher dimensions) until the final invariant pooling layer. There a lack of exploration on how to reduce the spatial dimensions of such feature maps while preserving exact equivariance, to produce low-dimensional equivariant representations. This work attempts to fill this gap. Such low- dimensional equivariant representations can be employed in representation learning methods, allowing various advantages such as interpretability, out-of-distributtin generalisation, and better sample com- plexity. When using such learned representations downstream tasks such as abstract reasoning, reinforcement learning, video modelling, scene understanding. it is especially important for represen- tations to be equivariant rather than invariant in these tasks, because transformations and how they act on feature spaces are critical information, rather than nuisance as in image classification problems.
operations that preserve translational equivariance. (ii) We generalise the proposed aubsamplinguusamppingg op- erations arbitrary symmetry groups. (iii) We use equivariant subsampiinguussamping operations to construct GAEs that gives oindiinesioonal equivariant representations. (iv) We empirically show that representations learned by GAEs enjoys many advantages such interpretability, out-of of-oofrrriibutiin generalisation, and better sample complexity.
In summary, make the following contributions: (i) We propose subsampling/upsampe
2 Equivariant Subsampling and Upsampling
2.1 Translation Equivariant Subsampling for CNNs
In this section we describe the proposed translation equivariant subsampling scheme for feature maps in standard CNNs. Later Section 2.2, we describe how this can be generalised to group equivariant subsampling for feature maps on arbitrary groups.
Standard subsampling Feature maps in CNNs can be seen as functions defined on the integer grid, e.g. & for ID feature maps, and 2 for 2D. Hence we represent feature maps as f f + Rd where

---

d is the number of feature map channels. For simplicity, we start with ID and move on to the 2D case. Typically, subsampling in CNNs is implemented as either strided convolution or (max) pooling,
and they can be decomposed as
CONV = SUBSAMPLING" o CONV`
MAXPOOL & = SUBSAMPLING MAXPO01
where subscripts denote kernel sizes and superseripts indicate strides. E IN is the scale factor for SUBSAMPLING, and this operation simply restricts the imput domain of the feature map from cZ. without changing the corresponding function values.
Translation equivariant subsampling In our equivariant subsampling scheme, we instead restrict the input domain to cZ + i, the integers = i mod G, where is a sampling index determined by the input feature map. The key idea is to choose i such that it is shifts by (mode) when the input is translated by I, to ensure that the same features are subsampled upon translation. Let i be given by the mapping & : Iz + cZ. Iz denotes the space of vector functions on Z and Z cZ is the space of remainders upon division by
i = Bc(s)= = mod (arg max (I(
zez
where || ||1 denotes L' -orm (other choices of norm are equally valid). Other choices for De are equally valid as long they satisfy translation equivariance, ensuring that the same features are subsampled upon translation of the input:
Lo(f(- - t)) = mod (ff()) + ttc).
Note that this holds for Equation (1) provided the argmax unique, which we assume for now (see Appendix B.1 for discussion of the non-unique case). We can decompose the subsampled feature map defined on cZ + into its values and the offset index i, expressing as [fo, i| E (ZzzZ//c))
where fo is the translated output feature map such that fo(cz) = f(cz + i) for x E Z.
The subsampling operation described above, which maps from I> to [L,z,//cl is translation equivariant: when the feature map I is translated to the right by / E Z, one can verify that So will be translated to the right by cl i+t] J. and the sampling index for the translated inputs would become mod + f, c). We provide an illustration for = 2 in Figure I, and describe formal statements and proofs later for the general cases in Section 2.2.
Multi-layer case For the subsequent layers, the feature map fo is fed into the next convolution, and the sampling index i is appended to a list of outputs. When the above translation equivariant subsampling scheme is interleaved with convolutions in this way, we obtain an exactly translation equivariant CNN, where each subsampling layer with scale factor ck produces sampling index i E Z/caZ.. Hence the equivariant representation output by the CNN with L subsampling layers final feature map fL and a L-tuple of sampling indices (i1. ...). This tuple can in fact be expressed cquivalently as a single integer by treating the tuple as mixed radix notation and converting to decimal notation. We provide details of this multi-layer case in Appendix B.2, including rigorous formulation and its equivariance properties.
Translation equivariant upsampling As counterpart to subsampling, upsampling operations increase the spatial dimensions of feature maps. We propose an equivariant upsampling operation that takes in feature map ʃ E I.z and sampling index E 7/cZ, and outputs a feature map fa E Iz, where we set fu( + i) = f(cr) and 0 everywhere else. This works well enough in practice, although in conventional upsampling the output feature map is often smooth interpolation of the input feature map. To achieve this with equivariant upsampling, we can additionally apply average pooling with stride 1 and kernel size >
2D Translation equivariant subsampling When feature maps are 2D, they can be represented as functions on Z?. The sampling index becomes 2-element tuple given by:
(x", "² arg (xx(e
(i,j) (mod c), mod c))
and we subsample feature maps by restricting the input domain to c72 + (i,j). The multi-layer construction and upsampling analogous to the ID-case.
3

---

2.2 Group Equivariant Subsampling and Upsampling
In this section, we propose group equivariant subsampling by starting off with the ID-translation case in Section 2.1, and provide intuition for how each component of this special case generalises to arbitrary discrete groups G. We then proceed to mathematically formulate group equivariant subsampling, and prove that it is indeed G-equivariant.
Feature maps groups First recall that the feature maps for the DD-translation were defined as functions on 2, or f E Iz for short. To extend this to the general case, we consider feature maps f as functions group G, i.. f E Ig {f G + V where V is vector space, is done in e.g. group equivariant CNNs (G-CNNs) (Cohen and Welling, 2016). Note that translating feature maps f on Z by displacement is effectively defining new feature map f'(+ f(. u). In the general case, we say that the group action on the feature space is given by
[T()))(g) = f("
(3)
where π is group representation describing how G acts on the feature space.
Recap: translation equivariant subsampling Recall that standard subsampling that occurs in pooling or strided convolutions for ID translations amounts to restricting the domain the feature map from Z to eZ, whereas equivariant subsampling also produces sampling index i Z/cZ, an integer mod c, and that this is equivalent to restricting the input domain to cZ + i. i is given by the translation equivariant mapping 4. Iz + c//ZZ. We translate the input domain back to cZ, and represent the output of subsampling as [fe, i] E (I.z, 7 /cZ), where fo is the translated output feature map and ft(cr) f(e i) for x E Z.
Group equivariant subsampling Similarly in the general case. for a feature map Ic. standard subsampling can be seen as restricting the domain from the group G to a subgroup K, whereas equivariant subsampling additionally produces sampling index pK E G/K, where the quotient space G/K = {gK: g E G} is the set of (left) cosets of K in G. Note that we have rewritten i as p to distinguish between the ID translation case and the general group case. This is equivalent to restricting the f to the coset pK. The choice of the coset pK is given by equivariant map (ug) K for u. g E G), such that
Ia + G/K (the action of G on G/K is given by u(gK) pK = (f). This restriction of f to pK can also be thought of as having an output feature map fo on K and choosing a coset representative element P E pK, such that fo(k) fpk)). This choice of coset representative is described by function s G/K K - G, such that p = s(pK). The function s called section and should satisfy s(pK)K = pk.
Now let us formulate subsampling and upsampling operations So and S, 1 mathematically and prove its G--quivariance. Let Ix- = {f K + V'} be the space of feature map on K. 5 takes in feature map f Ia and produces a feature map fo E Ix and a coset in G/ K. In reverse, the upsampling operation S.1k takes in feature map in Ik. coset in C/K, and produces a feature map in Ig- We use a section 8 G/K - G to represent a coset with a representative element G, and point out that equivariance holds for any choice of 8.
Formally, given an equivariant map & IG + G/K (we will discuss how to construct such a map in Section 2.3), and fixed section s : G/K + G such that p = s(pK), the subsampling operation Sp Ia + Ix x G/K K is defined as:
pk ){), [6(K) = f(kk) for l: E K
[f,, pk] = S
(4)
while the upsampling operation S.16 Ix GG/K + IG is defined as:
fg = f 9) if K else 0 fa = S. (J.KK)
(5)
"This is not to be confused with the space of Mackey functions in, e.g., Cohen et al. (2019), and rather it is the space of unconstrained functions on G.

---

Proposition 2.2. If the action of group G on the space Ig and IK G/K are specified by n x' (as defined in Equations (3) and (6)), and 4 Ia + G/KK is an equivariant map, then the operations Sp.k G and S. GG K as defined in Equations (4) and (5) are equivariant maps between Ig To make the output of the upsampling dense rather than sparse, one can apply arbitrary equivariant smoothing functions such as average pooling with stride and kernel size 1, to compensate for the fact that we extend with Os rather than values close to their neighbours. In practice, we observe that upsampling without any smoothing function works well enough.
The statement the equivariance of Sotk and Sutk requires we specify the action of G on the space Zk x G/K, which we denote as T For any E G.
p' = upK, fb = ==xd"--up)) 1h, p p'k] = n' (fo pk]
(6)
Lemma 2.1. x' defines valid group action of G the space Ix G/K.
We can now state the following equivariance property (See Appendix for a proof):
and LK x G/K.
In fact, we can also prove the converse (See Appendix D):
Proposition 2.3. If S Ia IK G/K (as defined in Equation (4)) is an equivariant map. then the corresponding + Ia + G/K must be equivariant.
The above implies that t must depend the input feature map f-
2.3 Constructing +
We use the following simple construction of the equivariant mapping $ IG - G/K K for subsam- pling/upsampling operations, although any equivariant mapping would suffice. For an input feature map f E Ig, we define
pk = pf) = (arg max (K
gEG
Provided that the argmax is unique, it is easy to show that (up) == (x)((),,, hence + equivariant. In practice one can insert arbitrary equivariant layers to before and after we take the norm || || to avoid a non-unique argmax (see Appendix F). Note that the argmax function alone may not be noise-robust. In Appendix E.2, empirically show that applying smoothing equivariant layers before taking the argmax would improve the stability the output sampling indices.
Non-unique argmax case When the input feature map f E Ia has inherent symmetries, i.e. there exists u E G, u o e, such that f = TTu) one cannot avoid a non-unique argmax Equation (7). That is because if there is unique argmax g" such that g" arg maxgeG [[) || would have:
f(u 9²) f(g")
max [ gEG
Therefore u "''"" is also a valid argmax, hence the argmax is not unique. For symmetric inputs, the equivariant map & would give set of sampling indices (cosets) rather than single one. instead consider including this set of sampling indices in Zeg, and let group acts on this set, it can be shown that the exact equivariance would still hold. In practice, we uniformly sample a sampling index from this set to perform subsampling, and the subsampled feature maps will be the same for all sampling indices from this set because the inputs are symmetric. This complexity is unavoidable because an equivariant map that maps the feature map to single coset does not exist in this case. However, perfectly symmetric inputs are very rare for wealdworld applications and we only encounter this problem for synthetic data.
3 Application: Group Equivariant Autoencoders
Group equivariant autoencoders (GAFs) are composed of alternating G-convolutional layers and cquivariant subsumpling/upsaapping operations for the encoder/decoder. One important property
5

---

of GAEs is that the final subsampling layer of the encoder subsamples to a feature map defined on the trivial group {e}, outputting a vector (instead of a feature map) that is invariant. For the ID-translation case, suppose the input to the final subsampling layer is feature map f defined on Z. Then the final layer produces an invariant vector (o) = f(iz. where it arg m [lf( I|1- Note that there is no scale factor ez here. Intuitively we can think of this as setting the scale factor eL = DC. Hence the encoder of the GAE outputs a representation that is disentangled into an invariant part Zinv fo( (the vector output by the final subsampling layer) and an equivariant part Aeq (i1,...,r.))
For the general group case, instead of specifying scale factors as in Section 1. we specify sequence of nested subgroups G Go 2 G1 2 2 GL = {e), where the feature map for layer I is defined on subgroup Gr. For example, for the p4 group G = Z x C4, we can use the following sequence for subsampling:
C, 2 27 C. 2 47 x Cx 2 87 X C2 {e). Note that for the final two layers of this example, we are subsampling translations and rotations jointly.
We lift the input defined on the homogeneous input space to Ig (see Appendix A.3 for details on homogeneous spaces and lifting), and treat fo E Ia as inputs to the autoencoders. The group equivariant encoder ENC can be described follows:
Ut; PtGi] Stite Gt (G-CNN D1)
[Zinv; Req] = [sr((e) where 1..... L and G-CNN,()) denotes G-convolutional layers before the lth subsampling layer. The decoder DEC simply goes in the opposite direction, and can be written formally as:
Gr)]
(8)
f is defined on GL = {e} and fL()) = Zunv
fi-1 G-CNN GL (Ji, prGi))
where 1 = L,..., and f fo gives the final reconstruction.
Recall from Section 2.1 that the tuple (fi....,,2z .z)) can be expressed equivalently single integer. Similarly, the tuple (p1 P.G.P. PL Gz) can be expressed as a single group element in G. We show in Appendix B.2 that the action implicitly defined the tuple via Equation (6) simplifies elegantly to the left-action on the single group element in G.
We now have the following properties for the learned representations (see Appendix D) for a proof): Proposition 3.1. When ENC and DEC are given by Equations (8) and (9), and the group actions are specified as in Equation (3) and Equation (6), for any 9 E G and f E Ig, we have
[Zaws 9 Zegl = ENC(x(g)f)
x x DEC (Zims 9 Zeq)
4
Related Work
Group equivariant neural networks The equivariant subsampling/upsampiing that we propose deals with feature maps (functions) defined on the space of the group G or its subgroups K, which transform under the regular representation with the group action. Hence our equivariant subsam- pling/upsampling is compatible with lifting-basee group equivariant neural networks defined on discrete groups (Cohen and Welling, 2016; Hoogeboom et al., 2018; Romero and Hoogendoorn, 2020; Romero et al., 2020) that define mapping between feature maps on G. We also discuss the extension of group equivariant subsampling to be compatible with those defined on continuous/L.ie groups (Cohen et al., 201 8a; Esteves et al., 2018; Finzi et al., 2020; Bekkers, 2020; Hutchinson et al., 2021) in Section 6. This is in contrast to group equivariant neural networks that do not use lifting and use irreducible representations, defining mappings between feature maps on the input space X. (Cohen and Welling, 2017; Worrall et al., 2017; Thomas et al., 2018; Kondor et al., 2018; Weiler et al., 2018b,a; Weiler and Cesa, 2019a; Esteves et al., 2020; Fuchs et al., 2020).
Coset pooling In particular, Cohen and Welling (2016) propose coset pooling, which is also method for equivariant subsampling. Here a feature map f on G is mapped onto feature map 4 (f) on G/K K (as opposed to K, for our equivariant subsampling) follows:
((f( = P
(10)

---

such that the feature values on the coset gK are pooled. For the ID-rranslation case, where G Z, cZ, this amounts to pooling over every cth pixel, which disrupts the locality of features as opposed to our equivariant subsampling that preserves locality, and hence is more suitable to use with convolutions for translation equivariance. See Figure 2 for a visual comparison. As such, the p4-CNNs in Cohen and Welling (2016) use standard max pooling with stride=2 rather than coset pooling for Z², and coset-pooling is only used in the final layer to pool over feature maps across 90-degree rotations, achieving exact rotation equivariance but imperfect translation equivariance. In our work, we use translation equivariant subsampling in the earlier layers and rotation equivariant subsampling in the final layers to achieve exact aoto-translation equivariance.
X
X
X
TRKE
coset pooling
eqy. ds boomppiing
X
X
X
X
mod 2
Figure 2: Coset (max) pooling vs. equivariant subsampling.
Unsupervised disentangling and object discovery GAEs produce equivariant (Zey) and invariant (zmv) representations, effectively separating position and pose information with other semantic information. This relates to unsupervised disentangling (Higgins et al., 2017: Chen et al., 2018; Kim and Mnih, 2018; Zhao et al., 2017) where different factors of variation in the data are separated in different dimensions of low-dimensional representation. However unlike equivariant subsampling, there no guarantee of any equivariance in the low-dimensional representation, making the resulting disentangled representations less interpretable. Works on unsupervised object discovery (Burgess et al., 2019; Greff et al., 2019; Engeleke et al., 2020; Locatello et al., 2020) learn object-centric representations, and we showcase GAEs in MONet (Burgess et al., 2019) where replace their VAE with a V-GAE in order to separate position and pose information and learn more interpretable representations of objects in data-efficient manner.
Shift-invariance in CNNs As early as Simoncelli et al. (1992), it has been discussed that shift- invariance cannot hold for conventional subsampling. Although standard subsampling operations such as pooling strided convolutions are not exactly shift invariant, they do not prevent strong performance on classification tasks (Scherer et al., 2010). Nonetheless, Zhang (2019) integrates anti- aliasing to improve shift-invariance, showing that it leads to better performance and generalisation on classification tasks. Chaman and Dokmanić (2020) explore a similar strategy to our equivariant subsampling by partitioning feature maps into polyphase components and select the component with the highest norm. However, unlike the proposed group equivariant subsanpping/uusampiing which tackle general equivariance for arbitrary discrete groups, both works focus only translation invariance.
Equivariant/invaiian autoencoders GAEs learn exact dinensiiiii equivariant representa- tions under the autoencoding framework, and this has also been explored in previous work. Locatello al. (2019) constructs autoencoders that are equivariant to the D12 group (30' rotations and reflec- tions) but not to translations, and the dimension of their learned representation grows with the group size. Lohit and Trivedi (2020) explores a rotation-invariant encoder on spheres with global pooling layer and a rotation-invarian loss function, without inserting subsampling layers between convolu- tional layers. Moreover, unlike the work above that focuses on group equivariant neural networks, Hinton et al. (2011); Sabour al. (2017); Kosiorek et al. (2019) learn equivariant representations with capsule networks. However, general capsule networks do not come guaranteed exact equivariances or invariances.
Experiments
In this section, we compare the performance of GAEs with equivariant subsampling to their non- cquivariant counterparts that use standard hoopaappigg/ussapplngg in object-centric representation learning. We show that GAEs give rise to more interpretable representations that show better sample complexity and generalisation than their non-equivariant counterparts. In Appendix E. we show that we can also observe generalisation performance gains when using group equivariant subsampling for classification tasks.
7

---

Bouss (1leg)
Stap
St 12
Sc+30
Input
Input
Reconstruction
Reconstructed movie with replaced 2
Figure 3: (Left) Manipulating reconstructions by modifying the equivariant part Zoq- The second column are the original reconstructions, which match the inputs well. The subsequent columns are reconstructions decoded from modified žeq- We transform Eey with sequence of group elements, and show the resulting reconstructions. (Right) Manipulating reconstruction shape by modifying žinv-
(G- (G-))oonlluiionall autoencoders (G) are composed alternating [G-)convolutional layers and subsampling/upsaapling operations with final MLPs applied to the flattened feature maps. We categorize models by the types of equivariance preserved by the con- volutional layers. We consider three different discrete symmetry groups: pl (only translations), p4 (composition of translations and 90 degree rotations), p4m (composition of translations, 90 degree rotations and mirror reflection). The baseline models are: ConvAAE-p (standard convolutional autoen- coders), GConv AE-p4, GConv AE-p4m, where the corresponding equivariance is preserved in the (G-)convolutional layers but not in the subsampiing/uusaappiig operations. The equivariant counter- parts of these baseline models are GAE-pl, GAE-p4, GAE-p4m, where the subsampling/upsamee operations are also equivariant. For baseline models, we use a scale factor of for all subsam- pling/upsampling layers. For GAEs, we subsample first the translations, then rotations, followed by reflections, all with scale factor 2. e.g. for GAE-p4m. the feature maps at each layer are defined on the following chain of nested subgroups: 72 x (C4 C2) 2 (27)² 2 x (G, x C2) 2 (47)² x (C4 C2) > (8Z)² x (C, C2) 2 (16Z)² x (C2 C2) 2 (e). As Cohen and Welling (2016), we rescale the number of channels such that the total number of parameters of these models roughly match each other.
Models and Baselines
Data To demonstrate basic properties of GAEs and compare sample complexity under the single object scenario, we use Colored-dSprite (Matthey et al., 2017) and a modification FashionMNIST (Xiao al., 2017), where first apply zero- prrd-padding to reach size of 64 x 64, followed by random shifts, rotations and coloring. For multi-object datasets, we use Multi-dSprites (Kabra et al., 2019) and CLEVR6 which is variant of CLEVR (Johnson et al., 2017) with up objects. All input images are resized to a resolution of 64 x 64.
See Appendix F and our reference implementation for more details hyperparameters and data preprocessing. Our implementation is built upon open source projects Harris et al. (2020); Paszke et al. (2019); Yadan (2019); Weiler and Cesa (2019b); Engelcke et al. (2020); Hunter (2007); Waskom (2021).
5.1 Basic Properties: Equivariance, Disentanglement and -iitti--iitiitttioon Generalization
The encoder-decoder pipeline in GAEs is exactly equivariant. In Figure 3, we train GAE-p4m on 6400 examples from Colored-dSprites, and we show how to manipulate reconstructions by manipulating the equivariant representation Zeq (left). If an image is encoded into [ Req], then decoding [Zine Zeq] will give & where f is the reconstruction of x. When the input has perfect symmetries (c.g. squares, cllipses Figure 3), Zeg is obtained by sampling from a set of sampling indices but different sampling indices in this set would give the same reconstruction (see Section 2.3).
Equivariance
Disentanglement The learned representations in GAEs are disentangled into an invariant part Piny and an equivariant part zeq- In Figure 3 (left), we vary the equivariant part while the invariant part remains the same. Figure (right), we show the frames of movie of heart, and show its reconstruction after replacing Ainv representing a heart with that of an ellipse. Note that the ellipse shape undergoes the same sequence of transformations as the heart.
"https://githin wtt.///////iooo gwt.://///guuubbbngggg...i

---

UP
30' 0.37
ConvAE-p1
GConvAE-p4
GAE-p1
GAE-p4
Figure 4: Generalisation to out out-of- distribution object locations and poses. During training, we constrain shapes to the top-left quarter, and the orientation to be always less than 90 degrees. On the right, we compare the error of reconstructions of different models generalise on objects at unseen locations in the first row, and how they generalise to unseen orientations in the second row.
Colored SSprits
FashionMNIST
0.010
0.010
ConvAE-p1
GConvAE.p4
GConvAE-p4m
GAE-p1
0.006
0.006
MSE
GAE-pA
MSE
GAE GAE-p4m
0.004
0.004
0.002
0.002
0.000
0.000
1k
2k
3k
4k
5k
Training Set Size
Training Set Size
Figure 5: Reconstruction error on single object datasets
Out-of-distribution generalisation GAEs can generalise to data with unseen object locations and poses. We train an GAE-p4 on 6400 constrained training examples, where we only use examples with locations in the top-left quarter and orientations within [0, 90] degrees, as shown in Figure 4. During test time, we evaluate mean squared error (MSE) of reconstructions on unfiltered test data to see how models generalise to unseen location and poses. Both ConvAE-pl and GConv cannot contrast, GAE-pl generalise to
generalise well to object poses out their training distribution. any locations without performance degradation but not to unseen orientations, while GAE-p4, which encodes both translation and rotation equivariance, generalises well to locations and orientations. We only heart shapes for evaluation, because the square and ellipse have inherent symmetries (see Section 2.3).
5.2 Single Object
Since GAFs are fully equivariant and can generalize to unseen object poses, it natural to conjecture that such models significantly improve data efficiency when symmetry-transfomee are also plausible samples from the data distribution. We test this hypothesis on Colored-dSprites and transformed FashionMNISTT and the results are shown in Figure On both datasets, equivariant autoencoders significantly outperform their non-equivariant counterparts for all considered training sizes. In fact, shown in the figure, equivariant models trained with a smaller training size is often comparable to baseline models trained on a larger training set. Furthermore, the results demonstrate that it beneficial consider symmetries beyond translations in these problems: for both non- equivariant and equivariant models, variants that encode rotation and reflection symmetries consistently show better performance compared models that only consider the translation symmetry,
data points
5.3 Multiple Objects
In multi-object scenes, it is often more interesting to consider local symmetries associated with objects rather than the global symmetry for the whole image. To exploit object symmetries in

---

Table I: Reconstruction error MSE (x x) (mean(stddev) across 5 seeds) on multi-object datasets
Dataset
Multi-dSprites
CLEVR6
Training Set Size
3200
6400
12800
3200
6400
12800
MONet
2.661 (0.38))
(.385(
0.326(0.076)
(.679)
0.562(0.057)
0.546(0.056)
MONei-GAE-pl 0.659(0.103
0.355(0.025)
0.264(0.042)
(0.064)
0.432(0.052)
(.386))
MONet-GAEEpd
(.563(0.195)
0.317(0.060)
0.461 (0237(
0.414(0.022) (.418().018)
Table 2: Foreground segmentation performance in terms of ARI (mean(stddev) across seeds)
Dataset
Multi-dSprites
CLEVR6
Training Set Size
3200
6400
12800
3200
6400
12800
MONet
0.597(0.022)
0.747(0.049)
0.891 0.891(0.000)) (0.009)
0.878(0.023) (.025)
0.865(0.033)
0. 0.762(0.049) MONet-GAE-pl
O. 0.889(0.013) (0.04)
0.921 (0.015)
0.917(0.032) 0.920(0.022
MONet-GAE-pd
0.753(0.089)
(0.02))
(.902)).025)
0.878(0.055)
0.914(0.012) 0.910(0.011)
We excluded 2 outliers here as the baseline MONet occasionally fails during late-phase training.
image data, one needs to first discover objects and separate them from the background, which is a challenging problem on its own. Currently, GAEs do not have inherent capability to solve these problems. In order to investigate whether our models could improve data efficiency in multi-object settings, we rely on recent work on unsupervised object discovery and only use GAFs to model object components. More specifically, we explored replacing component VAEs in MONet (Burgess et al., 2019) with V-GAEs (probabilistic version of our GAEs, where a standard Gaussian prior is put on Finv and Zeq remains deterministic), and train models end-to-end. Again we study the low data regime to show results on data efficiency.
We train models on consider two evaluation metrics: mean squared error (MSE) to measure the overall reconstruction quality, and adjusted rand index (ART), which is a clustering similarity measure ranging from 0 (random) to (perfect) to measure object segmentation. As in Burgess et al. (2019), we only use foreground pixels to compute ARI. Component VAEs in MONet use spatial broadcast decoders (Watters et al., 2019) that broadcast the latent representation to a full scale feature map before feeding them into the decoders, and the decoders therefore do not need upsampling. It has the implicit effect of encouraging the smoothness of the decoder outputs. To encourage similar behaviour, we add average pooling layers with stride and kernel size 3 to our equivariant decoders. As shown in Table 1, using GAEs to model object components significantly improves reconstruction quality, which is consistent with our findings in single-object scenario. As shown in Table 2, using GAEs to model object components also leads to better object discovery in the low data regimes, but this advantage seems to diminish as the dataset becomes sufficiently large.
and CLEVR6 with training set sizes 3200, 6400 and 12800. We
6 Conclusions, Limitations and Future Work
Conclusions We have proposed subsamplinguupsampling operations that exactly preserve transla- tion equivariance, and generalised them to define exuci group equivariant subsampling/ussampligg for discrete groups. We have used these layers in GAEs that allow learning low-dimensional repre- sentations that can be used to reliably manipulate pose and position of objects, and further showed how GAEs can be used to improve data efficiency in multi-object representation learning models.
Limitations and Future work Although the equivariance properties of subsampling layers also hold for Lie groups, we have not discussed the practical complexities that arise with the continuous case, where feature maps are only defined on finite subset of the group rather than the whole group. We leave this as important future work, as well as application of equivariant subsampling for tasks other than representation learning where equivariance/invarie classification, localization (See Appendix E. for a preliminary exploration of classification tasks). Another limitation is that our work focuses on global equivariance, like most other works in the literature. An important direction to extend to the case of local equivariances e.g. opject-specific symmetries for multi-objec scenes.
is desirable e.g. object
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Adam R. Kosiorek for valuable discussion. We also thank Lewis Smith, Desi Ivanova, Sheheryar Zaidi, Neil Band, Fabian Fuchs, Ning Miao, and Matthew Willetts for providing feedback on earlier versions of the paper, and anonymous reviewers for their constructive suggestions during the review process. JX gratefully acknowledges funding from Tencent Al Labs through the Eeford-Tencent Collaboration on Large Scale Machine Learning.
References
Bekkers, (2020). B-spline CNNs Lie groups. In ICLR.
Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, L., Botvinick, M., and Lerehner, A. (2019). Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11399.
Chaman, A. and Dokmanić, 1. (2020). Truly shift-invariant convolutional neural networks. arXiv preprint arX1::001.112214.
Chen, T. Q., Li, X., Grosse, R., and Duvenaud, (2018). Isolating sources of disentanglement in variational autocncoders. In International Conference on Learning Representations.
Cohen, T. and Welling, M. (2016). Group equivariant convolutional networks, In International conference machine learning, pages 2990-2999.
Cohen, S., Geiger, M., Köhler, J., and Welling, M. (2018a). Spherical CNNs. In ICLR.
Cohen, T. S., Geiger, M., Köhler, J., and Welling, M. (2018b). Spherical CNNs. In International Conference on Learning Representations.
Cohen, T.S., Geiger, M., and Weiler, M. (2018c). Intertwiners between induced representations (with applications to the theory of equivariant neural networks). arXiv preprint arXiv: 1803.10443
Cohen, T. S., Geiger, M., and Weiler, M. (2019). A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems, 32:9145 9156.
Cohen, Welling, (2017). Steerable cnns. International Conference on Learning Representations.
Dieleman, S., De Fauw, J., and Kavukcuoglu, K. (2016). Exploiting cyclic symmetry in convolutional neural networks. In International Conference on Machine Learning, pages 1889-1898.
Dieleman, S., Willett, K. W., and Dambre, J. (2015). Rotation-in invariant convolutional neural networks for galaxy morphology prediction, Monthly notices of the royal astronomical sociery, 450(2):1441 1459,
Engelcke, M., Kosiorek, A. R., Parker Jones, O., and Posner, I. (2020). GENESIS: Generative Scene In- ference and Sampling of Object-Centric Latent Representations. International Conference on Learning Representations (ICLR).
Esteves, C., Allen-Blanchette, C., Makadia, A., and Daniilidis, (2018). Learning SO(3) cquivariant represen- tations with spherical CNNs. ECCV.
Esteves, C., Makadia, A., and Daniilidis, K. (2020). Spin-weighted spherical CNNs. In NewrIPS.
Finzi, M., Stanton, S., Izmailov, P., and Wilson, A. (2020). Generalizing convolutional neural networks for equivariance to Lie groups on arbitrary continuous data. ICML.
Fuchs, F. B., Worrall, D. E., Fischer, V., and Welling, M. (2020). TranssTrrssssmmers 3D -ro-srranlltion cquivariant attention networks. In NeurIPS.
Gens, R. and Domingos, P. M. (2014). Deep symmetry networks. Advances in neural information processing systems, 27:2537-255 2545.
Greff, K. Kaufman, R. L., Kabra, R. Watters, N.. Burgess, C., Zoran, D., Matthey, L., Botvinick, M., and Lerchner. A. (2019). Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424 2433. PMLR.

---

Harris, C. R., Millman, K. J.. van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor. J.. Berg, S., Smith, N. Kern, R.. Picus, M., Hoyer, S., van Kerkwijk, M. I.. Brett M.. Haldane, A., del Río, J. E., Wiebe, M., Peterson, P., Marcchat,,, P. Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E. (2020). Array programming with NumPy, Nature, 585(7825)3377335
Higgins, I, Matthey, L.,, Pal, A., Burgess, C.,, Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2017). VeE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations.
Hinton, G. E., Krizhevsky, A., and Wang, S. D. (2011). Transforming auto-cncoders. Internatiional conference on artificial neural networks, pages 14-51. Springer.
Hoogeboom, E., Peters, J. W., Cohen, T. S., and Welling, M. (2018). HexaConv. In ICLR.
Hunter, D. (2007). Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90-95.
Hutchinson, M., Le Lan, C., Zaidi, S., Dupont, E., Teh, Y. W., and Kim, H. (2021). Lictransformer: Equivariant self-attention for lie groups. In Proceedings of the 38th International Conference on Machine Learning (ICML).
Johnson, J., Hariharan, B., Van Der Maaten, L.. Fei-Fei, L., Lawrence Zitnick. C.,, and Girshick. R. (2017).
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 2901-2910.
Kabra, R., Burgess, C., Matthey, L., Kaufman, R. L., Greff, K., Reynolds, M., and Lerehner, A. (2019). Multi-object datasets.
Kanazawa, A., Sharma, A., and Jacobs, (2014). Locally scale-invariant convolutional neural networks. arXiv preprint urXiv:1412.5100
Kim, H. and Mnih, A. (2018), Disentangling by factorising. In International Conference on Learning Represen- tations.
Kingma, D. P. and Ba. J. (2015). Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y., editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Kondor. R., Lin, Z., and Trivedi, (2018). Clebsch-Gordan nets: a fully Fourier space spherical convolutional neural network. In NeurIPS.
Kosiorek, A. R., Sabour, S., Teh, Y. W., and Hinton, G. E, (2019). Stacked capsule autoencoders. In Advances in Neural Information Processing Systems.
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., and Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation. In Proceedings of the 24th international conference on Machine learning, pages 473-480.
Locatello, F., Bauer, S., Lucic, M., Ractsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 41 14-4122. PMLR.
Locatello, F., Weissenbom, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, (2020). Object-centric learning with slot attention. arXiv preprint arXiv:2006. 15055.
Lohit, S. and Trivedi, S. (2020). Rotation-invariant autoencoders for signals on spheres. arXiv preprint arXiv:2012.04474
Marcos, D., Volpi, M., and Tuia, D. (2016). Learning rotation invariant convolutional filters for texture classification. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2012-2017. JEEE.
Masci, J., Meier, U., Ciresan, D., and Schmidhuber, J. (2011). Stacked convolutional auto-encoders for hierarchical feature extraction. In ICANN.
Matthey, L., Higgins, L., Hassabis. D., and Lerchner, A. (2017). dsprites: Disentanglement testing sprites dataset. https///ittubb.com/eee https//ggthbb.ccmmmemmmmm/p/pppp datases.
12

---

Paszke, A.. Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga. L.. Desmaison, A., Kopf, A., Yang, E., Vitioo, Z.., Raison, M.. Tejani, A., Chilamkurthy, S., Steiner, B.. Fang, L., Bai, J., and Chintala, (2019). Pytorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché- Buc, F,, Fox, E., and Gareet,, R., editors, Advances in Neural Information Processing Systems 32, pages 8024 8035. Curran Associates, Inc.
Romero, D. W., Bekkers, E. J., Tomezak, J. M., and Hoogendoorn, M. (2020). Attentive group equivariant convolutional networks. In ICML.
Romero, D. W. and Hoogendoorn, M. (2020). Co-attentive equivariant neural networks: Focusing equivariance on transformations co- oo-ocurrring in data. In ICLR.
Sabour, S., Frosst, N., and Hinton, G. E. (2017). Dynamic routing between capsules. NIPS 17, page 3859-3869, Red Hook, NY, USA. Curran Associates Inc.
Scherer, D., Müller, A., and Behnke, S. (2010). Evaluation of pooling operations in convolutional architectures for object recognition. In Diamantaras, K., Duch, W., and Iliadis. L. S., editors, Artificial Neural Networks ICANN 2010, pages 92-101, Berlin, Heidelherg. Springer Berlin Heidelberg.
Simoncelli, E., Freeman, W. Adelson, E., and Heeger, D. (1992). Shiftable multiscale transforms. IEEE Transactions Information Theory, 38(2):587-60) 607.
Thomas. N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. (2018). Tensor field networks:
Rotation-and translation-cquivartiii neural networks for 3d point clouds. arXiv preprint arXiv: 1802.08219
Waskom, M. L. (2021). seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021.
Watters, N., Matthey, L., Burgess, C. P., and Lerchner, A. (2019). Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint urXiv:1901.0700
steerablle Weiler, M. and Cesa, G. (2019a). General e (2)-equivariant steerable cnns. In Advances in Newral Information Processing Systems, pages
Weiler, M. and Cesa, G. (2019b). General B(2)-Equivariant Steerable CNNs. In Conference on Neurul Information Processing Systems (NeurIPS).
Weiler, M., Geiger, M., Welling, M., Boomsma, W., and Cohen, T. S. 8a). 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems. pages 10381-10392.
Weiler, M., Hamprecht, F. A., and Storath, M. (2018b). Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 849-858.
Worrall, D. E., Garbin, S. J., Turmukhambetov, D., and Brostow, G. J. (2017). Harmonic networks: Deep translation and rotation equivariance. In Proceedings the IEEE Conference Computer Vision and Puttern Recognition, pages 5028-5037.
Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: novel image dataset for benchmarking machine learning algorithms.
Yadan, (2019). Hydra framework for elegantly configuring complex applications. Github.
Zhang, R. (2019). Making convolutional networks shift-invariant again. In International Conference Machine Learning, pages 7324-7334.
Zhao, S., Song. J., and Ermon, S. (2017). Infovae: Information maximizing variational autoencoders. arXiv preprint
13


Reviews:
[{"review_id": "tqpngQVeF_a", "reviewer": "Reviewer_a1LF", "summary": "This paper introduces a group-theoretic framework for equivariant subsampling. Based on a similar approach to ref [3] in the case of translations, the idea is to subsample according to intrinsic landmarks and keeping track of the shifts involved. The framework introduced can handle this procedure for general symmetry groups by seeing the subsampled space as a subgroup, and the shifts are stored as elements of the quotient of the original group and the subgroup.\nThe method is evaluated in the context of VAEs, where it is shown that an architecture using equivariant subsampling and upsampling is capable of fully disentangling covariant from invariant features, and, depending on the group used, is shown to generalize perfectly to unseen object positions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strangely, my main criticism of this paper is also the main reason I find it interesting. A more pedestrian way of describing what has been done is registration of the signal among a collection of possible transformations, according to some intrinsic signal property, while not forgetting the selected registering transformation. In this sense the group-theoretic approach is (an even somewhat restrictive) syntactic sugar. That said, I consider this a neat framework that can represent many previous approaches, that guides implementations, is shown to work in experiments, and may provide leads for extensions in the future. Also, it shows an example of a non-trivial equivariant transformation, where part of the information is absorbed into an index.\n\nIt should be noted that registration is inherently unstable no matter what procedure is used to determine the shift (see e.g. https://arxiv.org/abs/1203.1513 2.1ff). There will always be a signal exhibiting the discontinuity of the map from signal to shift parameter (consider a signal with two local maxima in the l1-norm used here, with one peak epsilon/2 lower than the other, but far away in terms of position. A perturbation increasing the lower peak by epsilon will make the registration landmark jump. In general, a map between metric spaces of different dimensionality cannot be continuous in both directions). It may be interesting to explore this a little bit to see whether there are adversarial examples with practical impact.\n\nThe groups explored here, while providing a proof of concept, are really not very interesting. I would have been much more delighted to see this framework used to tackle e.g. continuous, or at least more fine-grained discrete rotational steps. The VAE data set samples, by virtue of being one shape on a constant background, would not pose any difficulty (i.e. border effects from e.g. periodic wrap-around) in applying these.\n\nGiven the memorization of shifts, can you envision a way of being able to sample from the GAE latent? Would it require putting a probability measure on the shifts?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations were discussed.\nI do not foresee potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "mDUocFmYYB", "reviewer": "Reviewer_WgFm", "summary": "The paper proposes a method to equip discrete-group convolutional neural networks (GCNNs) with a perfectly equivariant subsampling layer. The method relies on a non-fixed sampling grid which is dynamically chosen as an equivariant function of the current input. This method enables encoder-decoder architectures where each layer is equivariant to the global action of the discrete group on the input of the model. Experimental results confirm the accurate equivariance and the improved generalization of the models.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed equivariant subsampling is a clear imrpovement over common down-sampling methods. The main idea seems intuitive and simple to implement, making it probably very easy to adopt in many existing equivariant models.\n\nI found the presentation quite clear overall and I liked the intuitive description of the subsampling idea using the 1D translations as an example. However, I think the introduction to the abstract groups setting was a bit fast; to keep the paper self contained, I think it would be useful to at least define group convolution in the main paper.\n\nThere are a number of issues that I think should be addressed more explicitly (see below).\n\nWhile the reconstruction tasks are certainly important and clearly show the benefit of using this subsampling method, I think some simple experiment on a standard benchmark such as MNIST-rot (for classification) should be included to compare the proposed method with other equivariant models from the literature.\n\nI think the discussion in B.1 should be included more prominently in the main paper. I find the stochastic alternative proposed in B.1 less satisfying. This is not necessarily an issue since I find the subsampling method proposed in the main paper a very nice contribution, but I believe this limitation should be discussed more explicitly in the main paper.\n\nThe case of symmetric inputs appears also in Fig. 3 for the ellipse and the square, so some comments about this should also be included in the experimental section.\n\nIf I understand correctly, the proposed subsampling method guarantees perfect global equivariance but can reduce the local equivariance. This is because the sampling grid depends on the global image. This means that changing a single pixel in a region of the image far away from a local patter can change the pixels selected around that local patter for the downsampling. Because the downsampling is also performed without any smoothing of the features, this can potentially lead to a completely different prediction for that pattern. Clearly, this is a worst-case scenario and it is not clear yet how this affects the stability of the model in practice.\nStill, some comments about this potential instability should be included among the limitations.\n\nFinally, I wonder how this subsampling method affects the performance of a model when using mini-batches, since different samples in the batch might require different sampling grids.\n\nEDIT: Having read the author response, I have decided to maintain my rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any issue with the authors' comments on the societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "IshKZSb--nA", "reviewer": "Reviewer_RUmB", "summary": "The authors propose subsampling and upsampling layers that are equivariant to common transformation groups. These constructions are used to design autoencoders consisting of layers that alternate between group equivariant convolutions and group equivariant subsampling/upsampling layers. The result is that the entire autoencoder is *exactly* group equivariant. Experiments on translations, translations+rotations, translations+rotations+reflections for simple images with single/multiple objects clearly show the improved performance over subsampling methods that are not group-equivariant.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The core of the paper is group equivariant subsampling method that is clever and simple to implement. It can be easily integrated with the architectures which already employ group equivariant convolutions and can be composed with each other to form a deep network. Although the simple method shown in Eqn. 11 may not always have a unique solution, as in the case of feature maps with symmetries, the authors propose a way to overcome this problem.  \n\nThe main weakness in the paper would be that in all the experiments, the feature maps are all images on the plane. I would have liked to see how easy or difficult it is to integrate these subsampling layers in the case of spherical image inputs with the feature maps defined on SO(3). And whether this would lead to improved performance in practice. It would also be useful to other researchers to add an another line in the tables with coset pooling as the subsampling operation to see how it affects final reconstruction performance. \n\nThe practical significance of this method also deserves more discussion in the paper. The experiments are okay in terms of demonstrating the main strengths of the proposed work. However, I would have like at least one experiment with a more realistic dataset, like a dataset of human faces perhaps undergoing some deformations. Related to this, what about larger transformation groups like piece-wise affine transforms or very large groups like 2D diffeomorphisms (used in facial expression modeling and medical image registration)? My guess is constructing the group-equivariant convolutions for these groups is the much bigger problem rather the group-equivariant subsampling itself. \n\nThe authors do mention that for objects with background, the first step is to remove the background before group-equivariant autoencoding and that local symmetries become more important as most scenes have several objects. These are indeed challenging tasks and perhaps this paper can provide a starting point to attack these problems. \n\nThe related work section should include other works which are similar in spirit to group equivariant autoencoders. This includes the following list of papers that is not exhaustive:\n\n1. Deforming autoencoders: Unsupervised Disentangling of Shape and Appearance ECCV 2018 https://openaccess.thecvf.com/content_ECCV_2018/html/Zhixin_Shu_Deforming_Autoencoders_Unsupervised_ECCV_2018_paper.html\n\n2. Extracting invariant features from images using an equivariant autoencoder, ACML 2018, http://proceedings.mlr.press/v95/kuzminykh18a.html\n\n3. Rotation-Invariant Autoencoders for Signals on Spheres, https://arxiv.org/abs/2012.04474\n\n4. Capsule networks", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations of the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "7FXi2rv7Et", "reviewer": "Reviewer_hKzy", "summary": "The paper proposes and evaluates equivariant subsampling and upsampling operations for group convolutional networks, which include vanilla CNNs as a special case. These operations address the issue that conventional subsampling and upsampling operations are only equivariant under the action of a subgroup of symmetries. The models have the property to disentangle the appearance of objects from their G-pose, which allows to build disentangled group equivariant autoencoders.\n\nAs a simple introduction, the authors formulate their method initially for the special case of $G = \\mathbb{Z}$ being the 1-dimensional translation group. In contrast to conventional subsampling operations, which sample the values on a grid $c\\mathbb{Z}$ with spacing $c$, the equivariant subsampling operation samples features on a shifted grid $c\\mathbb{Z} + i$, where $i = 0, \\dots, c-1$ is an integer offset that is predicted from the signal itself via some equivariant function. The tuple of subsampled features and the integer offset are shown to transform in an equivariant way under the action of the full symmetry group $\\mathbb{Z}$. If multiple subsampling layers are successively applied in a deep network, each of them adds a new offset value to a sequence of offsets. When eventually pooling to the trivial group $G=\\{e\\}$, the resulting feature is position invariant and all position information is encoded in the offset sequence. Upsampling layers are accordingly taking a low-resolution signal and an offset and embed it with this offset into a higher-resolution grid.\n\nThese constructions are in the obvious way generalized to feature maps on arbitrary discrete groups $G$: The lower resolution sampling grid is here given by a coset $pK \\in G/K$ of a subgroup $K<G$ in $G$. The group equivariant subsampling operation is therefore producing a tuple which consists of a subsampled signal on the coset and the coset (offset) itself. To upsample such a signal, it is embedded in the coset of the supergroup. Lemma 2.1 and Proposition 2.2 assert the full $G$-equivariance of these operations.\n\nThe integer offset, or, more generally, coset, needs to be computed via any equivariant map from the input signal to the space of cosets. A simple choice, made by the authors, is that coset $gK$ that consist to the argmax group element $g$ of the signal's l1-norm.\n\nThe equivariant subsampling layers are used to construct various group equivariant autoencoders (GEAs), which are evaluated against baseline autoencoders that are constructed from group convolutions but use conventional subsampling and upsampling layers. GEAs are empirically shown to be exactly equivariant and to generalize perfectly out of distribution. This is not the case for the baseline models, whose conventional subsampling and upsampling layers break the models' overall equivariance. The disentanglement of object appearance and $G$-pose in the autoencoders' latent space is in figure 3 (left) shown to allow for an independent manipulation of these factors. GEAs perform on single and multi-object reconstruction tasks significantly better than the baselines.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submission addresses an important open problem of conventional and group equivariant convolutional networks, namely the lack of equivariance in their downsampling and upsampling layers. While other common types of layers like convolutions, nonlinearities or batch normalization have equivariant counterparts, contemporary models are usually using non-equivariant or smoothing based subsampling layers, which prevents the exact equivariance of the full model. The proposed method solves this issue, allowing to build fully equivariant models. As demonstrated, this can be used to disentangle an object's pose from its appearance and generally improves the model performance.\n\nThe paper is well written and mathematically sound. All claims are proven in the appendix and demonstrated empirically. The decision to introduce the construction on the simple and practically relevant special case of $G=\\mathbb{Z}$ being the translation group helps to make the idea accessible to a broader audience. I am confident to be able to reimplement the model and reproduce the experiments.\n\nWhile any equivariant map is suitable to predict the grid offset (or coset), the authors choose the argmax of the feature vectors' norms. This choice seems to be very unstable under noise on the input, which should be investigated empirically. What happens in degenerate cases like reconstructing the top row in figure 3 (right) with $z_\\text{eq}$ taken from the bottom row, whose ellipses are $C_2$-symmetric?\nA more robust choice of equivariant map could be based on coset average pooling, followed by an argmax over cosets or on the phase offset of a harmonic basis function expansion of the features on $G$. The authors should discuss different choices of such maps and empirically investigate their stability.\n\nIt seems intuitively necessary that equivariant sub/upsampling operations are based on an offset that is computed from the signal itself - can this conjecture be proven formally?\n\nWhile the experimental section is convincingly demonstrating the merits of the method, it does not include a standard task like image classification. It would be interesting to see the performance gains on such tasks instead of only auto-encoding reconstruction errors.\n\nOverall, the submission solves an important problem of equivariant networks, and I would be happy to give it a very positive score. My low score should be understood as an incentive to extend the empirical evaluation, in particular on the choice of the equivariant offset predictor.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss limitations of the method appropriately. No immediate negative societal impacts are to be expected.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel method for subsampling along the orbit of a group action, which maintains the equi-variance property. After the rebuttal, all the reviewers agreed that this work is interesting and recommended to accept it. Thus, I will suggest to accept this paper yet the authors should incorporate the relevant suggestions of each reviewer.
Decision: Accept (Poster)