Steerable Equivariant Representation Learning
Authors: Sangnie Bhardwaj
Willie McClinton
Tongzhou Wang
Guillaume Lajoie
Chen Sun
Phillip Isola
Dilip Krishnan
Venue: Submitted to ICLR 2023 (2023)
Keywords: representation
visual
equivariance
equivariant

Abstract: Pre-trained deep image representations are useful for post-training tasks such as classification through transfer learning, image retrieval, and object detection. Data augmentations are a crucial aspect of pre-training robust representations in both supervised and self-supervised settings. Data augmentations explicitly or implicitly promote \emph{invariance} in the embedding space to the input image transformations. This invariance reduces generalization to those downstream tasks which rely on sensitivity to these particular data augmentations. In this paper, we propose a method of learning representations that are instead \emph{equivariant} to data augmentations. We achieve this equivariance through the use of \emph{steerable} representations. Our representations can be manipulated directly in embedding space via learned linear maps. We demonstrate that our resulting steerable and equivariant representations lead to better performance on transfer learning and robustness: e.g. we improve linear probe top-1 accuracy by between 1\% to 3\% for transfer; and ImageNet-C accuracy by upto 3.4\%. We further show that the steerability of our representations provides significant speedup (nearly $50\times$) for test-time augmentations; by applying a large number of augmentations for out-of-distribution detection, we significantly improve OOD AUC on the ImageNet-C dataset over an invariant representation.

Under review as a conference paper at ICLR 2023
STEERABLE EQUIVARIANT REPRESENTATION LEARN- ING
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-trained deep image representations are useful for post-training tasks such as classification through transfer learning, image retrieval, and object detection. Data augmentations are crucial aspect pre-training robust representations in both supervised and self-supervised settings. Data augmentations explicitly or implicitly promote invariance in the embedding space to the input image transformations. This invariance reduces generalization to those downstream tasks which rely on sen- sitivity to these particular data augmentations. In this paper, we propose a method of learning representations that instead equivariant to data augmentations. We achieve this equivariance through the use of steerable representations. Our rep- resentations can be manipulated directly in embedding space via learned linear maps. We demonstrate that our resulting steerable and equivariant representations lead better performance on transfer learning and robustness: e-g. we improve linear probe top-1 accuracy by between 1% to 3% for transfer; and ImageNet-C accuracy by upto 3.4%. We further show that the steerability of our representations provides sigmificant speedup (nearly 50x) for test-time augmentations; by applying large number of augmentations for out-of-distribution detection, we significantly improve OOD AUC on the ImageNet-C dataset over an invariant representation.
INTRODUCTION
Embeddings of pre-trained deep image models are extremely useful in a variety of downstream tasks such zero-shot retrieval (Radford al. [02T), few-shot transfer learning (Tian et al. 2020)
perceptual quality metrics (Zhang et al. 2018) and the evaluation of generative models (Heusel et al.
2017, Salimans et al. 2016). The pre-training
done with various supervised or seff-supervised
losses (Khosla et al.) R0200Raadoon et al. 2021 Chen et al., (020) and variety of architectures
(He al.]201) Dosovitskiy et al. 2020 Tolstikhin al.] 2021). The properties of pre-trained embeddings. such as generalization (Zhai et al. (019) and robustness (Naseer et al. 2021), are
therefore significant interest. Most current pre-training methods impose invariance to input data
augmentations either via losses (Tsuzuku et al.) 2018 Chen et al. 2020 Caron al.) 202
architectural components such as pooling (Fan et 201 For invariant embeddings, the (output) embedding stays nearly constant for all transformations of sample (e.g. geometric photometric transformations of the input). Invariance is desirable for tasks where the transformations is a nuisance variable (Lyle et al. 2020). However, prior work shows that it can lead poor performance on tasks where sensitivity to transformations is desirable (Dangovski et al. 2022 Xiao et al. (021).
Equivariance is more general property: an equivariant embedding changes (smoothly) with respect to changes at the input to the encoder (Dangovski et al., 2022). If the change is zero (or very small), we get invariance as a special case. prior work, equivariant embeddings have been shown have numerous benefits: reduced sample complexity for training, improved generalization and transfer learning performance (Cohen & Welling 2016b, [Simeonvv al., 202T Lenssen et al.) 2018 Xiao et al. 202T). Equivariance has been achieved mostly by the use of architectural modifications (Finzi ct al. groups. However, this does not cover important transformations such as photometric changes others that camnot be represented explicitly as matrix transformations. Xiao et al. (202I) and Dangovski et al. (2022) propose more flexible approaches to allow arbitrary input transformations to be represented at the embedding, for the self-super setting. However, a key distinction between these works and ours is that parameterize the transformations in latent space, allowing for steering.
2021 Cohen & Welling 2016b) and are mostly restricted to symmetries represented as matrix

---

Under review as a conference paper at ICLR 2023
Duery
Ours liquivrrian))
Stancard (Inarrart)
Query
Eoulvariant
Invoriant
Figure 1: Two examples of image retrieval, comparing our steerable equivariant model to the baseline invariant (standard) model. The top example (flowers) is for color-based steering; the bottom example (buildings) is shown for brightness breddd based retrieval. For each example, we show three query images in the left column, along with nearest neighbors the next columns (4 each the stecrable and standard models). Please see text for definitions of e(x), M (e(x)) and A (e())) Query image shown simply for illustration; we do not use that image for the retrieval. The steerable model retrieves images where the color or brightness change overrides semantics, For example, the second query on the flowers example retrieves yellow/pink neighbors and the third query retrieves purple/blue colored flowers; similarly for darker image in the second example, darker images are retrieved and brighter examples, brighter images are retrieved. The invariant model retrievals are fairly static between different color or brightness changes.
We introduce some notation. refers input sample (image). e(x; w) represents the encoder network that maps input x to the embedding e, where are the parameters of the network. We use e(x) and e(x; w) interchangeably for ease of notation. The data augmentation of sample x is represented as (x; 0), often shortened to g(x) for brevity. refers to the parameters of the augmentation, e-g.
photometric transformations is a 3-dimensional vector of red, green and blue shifts applied to the image. We denote latent space transformations as (ee, 0), taking as input the embedding e and transformation parameter 0. M may be linear (a matrix) or nonlinear function (deep network), the output of which is another vector of the same dimensions as e(x). Thus, M is mapping from the joint embedding and parameter space to embedding space.
Given this notation, if e(g(x; 0)) e(x),, i.e. the embedding does not change due to the input transformation g(),, it is said to be invariant to g. Equivariance is defined as e(g(x; 0)) = (e(x), f). If M is the identity function, then we recover invariance. The map M in latent space encourages the embedding to change smoothly with respect to the and 6, the parameters the transformation. These maps M allow us directly manipulate the embeddings e, leading us to the concept of steerability (Freeman et al.] 1991)). It has been shown that pre-trained embeddings often accommodate linear vector operations
enable e.g. nearest neighbor retrieval using Euclidean distance (Radford et al. 202T); this is coarse form of steerability. However, without more explicit control on the embeddings, is difficult to perform fine- eme-grained operations on this vector space, for example, re-ordering retrieved results by color attributes. It is not very useful in practice to steer an invariant model: the embeddings may change very little response to steering. However, enabling steerability for an equivariant embedding opens up number of applications for control in embedding space; we show the benefits in our experiments.
We introduce simple and general regularizer to encourage embedding equivariance to input data augmentations. The same mechanism (mapping functions) used for the regularization enables a simple steerable mechanism to control embeddings post-training. Our Our regularizer achieves significantly more equivariance (as measured by the metric in (Jayaraman & Grauman 2015)) than pre-training without the regularizer. Prior work (Cohen & Welling 2016a
Deng al. 2021
Zhang 2019) has introduced
2

---

Under review as a conference paper at ICLR 2023
specialized architectures to make deep networks equivariant to specific transformations such as rotation or translation. Our approach is complementary to these works: as long as a transformation is parameterized in input space, we can train mapping function in embedding space to mimic that transformation. It is agnostic to architecture. We show the benefits of our approach with applications in nearest neighbor retrieval when different augmentations are applied in embedding space, showing the benefits of steerable control of embeddings (see Fig. as an example). We also test our approach for out-of-distribution detection, transfer, and robustness: our steerable equivariant embeddings significantly outperform the invariant model. We will release our code at www. XXX. yyy
2 RELATED WORK
Data augmentation is a crucial component of modern deep learning frameworks. Pre-defined transfor- mations chained together to create new versions of every sample fed into the network. Examples of such transformations are random image crops, color jitter, mixing of images and rotations (Cubuk
let al. 2020 Hendrycks et al. 2019; Yun et al. 2019 Hendrycks et al.) 2019 Chen et al. 2020
Gidaris et al. 2018). Most data augmentation pipelines involve randomly picking the parameters of
each transformation, and defining deterministic (Cubuk al. 2020) learned order of the transfor-
mations (Cubuk et al. 2018). Adversarial training is another class of augmentations, (Xic et al. 2020
Herrmann et al. (021)) which provide pooel--daaptvv regularization. Data augmentation expands the training distribution to reduce overfitting in heavily over wver-parametrized deep networks. This provides a
strong regularization effect, and improves generalization consistently, across architectures, modalities and loss functions (Hernández-Garcia & König 2018 [Steiner et al. [221] Hou et al., [2018] [Shen]
jet al. 2020 Chen et al./ 2020 Caron et al. 2020 He et [2.1]. Data augmentations are crucial in the training of self-supervisee contrastive losses (Chen et al. 2020 Grill et al. 2020).
Most losses for deep networks implicitly or explicitly impose invariance to input data augmentations (Tsuzuku et al. 2018). For all transformations of sample (e.g. different color variations). the output embedding stays nearly constant. When this property is useful, e.g. classification under perturbations, invariance is desirable (Lyle et al., 2020). Many papers have studied invariance properties of convolutional networks to specific augmentations such as translation (Azulay & Weiss) 2018 Zhang 2019 Bruna & Mallat (013)). rotation (Sifre & Mallat 2013) and scaling (Xu et al. 2014). These architectural constructs have been made somewhat redundant in newer Transformer- based deep networks (Vaswani et al. 2017; Dosovitskiy al.) 2020: Tolstikhin et al.) 202I) which use a mix of patch tokens and multilayer perceptrons. However, invariance is not universally desirable. Equivariance is more general property from which invariance can be extracted by using aggregation operators such as pooling (Laptev et al.; 2016 Fan et al. 201T). Equivariant architectures have benefits such as reduced sample complexity training (Esteves, 2020) and capturing symmetries in the data (Smidt (022I. Rotational equivariance has been extensively studied for CNN's (Cohen & Welling 2016a Simeonov et al. 2021] Deng et al.) 202T). Convolutional networks (without pooling) constructed to have translational equivariance, although papers such as (Azulay & Weiss 2018)
try to understand when this property does not hold. A number of works have suggested specific architectures to enable equivariance e.g. (Cohen & Welling 2016b; Dieleman et | 20166) Lenssen
let al. 2018: FFinzi et al. (020) (osnovik et al. 2019) Romero et al. 2020)] Romero & Cordonnier 2020 Bevilacqua al. 2021 Smets et al., 2020). However, these architectures have not been widely adopted in spite of their useful properties, possibly due to the extra effort required to setup and train
such specialized models.
For many applications, it is also useful to be able to steer equivariant embeddings in a particular direction, to provide fnn-grained control. While it is well-understood concept in signal and image processing (Freeman et al. 1991), it is less widely applied in neural networks. Our work is inspired by that of (Jayaraman & Grauman 2015). who introduce an unsupervised learning objective to tie together ego-motion and feature learning. The resulting embedding space captures equivariance of complex transformations in 3-D space, which are hard to encode in architectures directly: they use standard convolutional networks and an appropriate loss to encourage equivariance. They show significant benefits of their approach for downstream recognition tasks over invariant baselines. The works of (Xiao et al., 2021 Dangovski et al.) 2022) are also closely related. They build explicit equivariant spaces for specific data augmentations (in their case, color jitter, rotation and texture randomization). However, they do so indirectly by increasing 'sensitivity' to transformations, and do
3

---

Under review as a conference paper at ICLR 2023
not build any Mg equivalent maps. They hence do not provide steerability. Additionally, their work is restricted to the contrastive setting, whereas our regularizer can be added to any training paradigm.
MODEL
In this paper, we work in the context of supervised training ImageNet classification models. However, note that our approach is general and easily extends to self-supervised settings, e.g. (Chen al. 2020: He et al., 202T; Chen et al.] (02T). Our standard (invariant) model is trained with cross-entropy loss, along with weight decay regularization with hyperparameter A:
LCE(x)= Σ
log Px(x; w (x) + \//w||
Here, the embedding e(x; w) is projected layer of the network). y(x) refers to the target label vector for the sample x, used for supervised learning. Vector components in both are indexed by c, which can refer to, say, the classes for supervised training. The entries of p(x) and y(x) are between 0 and 1, and they sum to 1 to form the parameters of a categorical distribution.
normalized probability space p(x; w) (the "logits"
The usual manner of training cross-entrops loss is to first apply a sequence of data augmentations to x e.g. (Cubuk et al., 2020) and then pass the transformed version of into the network. Since all transformations of x are encouraged to map to the same distribution y(x), this loss promotes invariance in p(x) and therefore also in the embedding e(x).
3.1 MEASURING EQUTVARIANCE FOR AN AUGMENTATION
In works such as Cohen & Welling 2016a); Lenssen et al. 2018). the architecture guarantees cquivariance. Our approach is agnostic to architecture, so we desire a quantitative way to measure the equivariance with respect to particular augmentation. We adapt the measure in (Jayaraman & Grauman 2015). Denoting a given augmentation by a, we use the formula:
Pa (x) [[ 0.) (gg 0a))
|e(ga (x, 0a)) - e(x)]l2
The denominator here measures invariance: lower values mean more invariant embeddings w.r.t. the augmentation a applied to input x. The numerator measures equivariance: we want the embedding of transformation Ja (x) be represented Ma(e(x).. The lower the value of pa (x), the more equivariant the embedding e(x). Note that we need the ratio in Eqn. 2 rather than just the numerator, to exclude trivial solutions where ga (x; 0.) is mapped to the same point for all 6, (for given x), and similarly for Ma (e(x)). These would indeed make the numerator small, but they are representative of invariance rather than equivariance. Dividing by the denominator ensures that (x: 0a)) be distinct from e(x) for different values of 0a. Note that M,, and Ja share the same transformation parameter 0g.-
transformation in embedding space, represented by
3.2 EQUIVARIANCE CEWARRIANCE-PROOOTIIG PROMOTING REGULARIZER
We use the numerator of Eqn. 2 to define a regularizer to promote equivariance in the embeddings:
L (x) = [[ 0.) - (x; 0a))
(3)
have separate term for each augmentation that is applied the input. As before, the transformation-specinn parameters 9₂ are shared between the embedding map Ma and the input transformation 9a- Thus, the embedding map learns to apply a transformation in embedding space that mimics the effect of applying the transformation to the input. Hence, these maps M allow us to directly manipulate the embeddings e and provide fine-grained control w.r.t transformations: a concept we call steerability. Note that it is possible to make representations equivariant but not steerable, as in Xiao et al. (2021) Dangovski et al. (2022), by not recovering the M,, 's
where we
We use the following structure for M,, A vector of continuous valued parameters 0, is first mapped to 288dimensional intermediate parameter representation using a single dense layer. This vector is then concatenated with the given embedding exx) and passed through another dense layer, to give the final output of the map which is the same dimensionality (2048) as e(x). This structure adds around % extra parameters to ResNet-50 model.
4

---

Under review as a conference paper at ICLR 2023
3.3 UNIFORMITY REGULARIZER
We observe that embeddings learned using (xr(x) lead to well-formed class clusters, but within cluster they collapse onto each other, thereby increasing invariance (and Equation (3) cannot prevent
this). To overcome this, we enforce uniformity loss (Wang & Isola 2020) which is given by:
(x) = loog
exp (e(x)))
is temperature parameter often set to a small value such as 0.1. This loss encourages the embeddings of sample x; to separate from embeddings of other samples Xj. We find that also has the effect of spreading out augmentations of the same sample, increasing the equivariance of the embeddings and reducing the likelihood of trivial solution for the mapping Ma This could alternatively have been achieved by maximizing the denominator of Equation (2). but we find empirically that strongly pushed e(x) and e(ga (x)) apart and destabilized training.
3.4 LOSS
Putting the above together, our final loss to train equivariaant/storaale model is given by:
LCEU (XE a E ( B(Lu (x) Lo (x: 0.)))
(5)
The sums are computed over different augmentations a; and a separate embedding map is learned for each augmentation for which we desire equivariance in the embedding. and / are weighting hyper- parameters. Note that the LE and Lu terms are applied to the embedding e and the cross-entropy is applied to the softmax output p.
4 EXPERIMENTS
Our models are trained on the ImageNet dataset (Deng al. 2009), on the ResNet-50 architecture (He et al. 2016). To train our steerable linear maps, we use the following augmentations:
Geometric: crop transformation parameterized by 4-dimensional parameter vector 0 geo. representing the position the top left corner (x and y coordinates) of the crop, crop height and crop width. All values are normalized by the image size (224 in our case). When ggea = [0, O, 1, 1], this corresponds to no augmentation. We denote the corresponding stecrable map as Mgeo- This augmentation encompasses random crop, zoom, resize (Chen et al. 2020). and translation (by only varying the top left corner).
Photometric: a color jitter tranformation parameterized by a 3-dimensional parameter vector ppooto- which represents the respective relative change in the values of the RGB channels. The values of Ophoto in the range of [-1, When Ophoto [0, O, 0), this corresponds to no augmentation. We denote the corresponding steerable map as M This augmentation encompasses global contrast, brightness, and hue/color transformations.
Both the standard (invariant) and our steerable equivariant models are trained with the same data augmentation. The invariant model is trained with the standard cross-entropy loss in Eqn. D foor 250 epochs, with a batch size of 4096 (other training details are in the Appendix A.2). This model achieves top-1 accuracy of 75.17% on the ImageNet evaluation set. The equivariant/steerabl model is trained with the loss in Eqn. 5 with the same learning rate schedule, number of epochs and batch size as the invariant model, with hyperparameters a=0.1 and 9=0011. As earlier, the data augmentations from SimCLR (Chen et al., (2000) are applied to a sample x and used in LCEU- In addition to this, the parameters Ophoto and 0geo are sampled uniformly at random within their pre-defined ranges. These are applied independently to generate two more of views of x, for use in (x(x)) and Ly (x). This model achieves a top-1 accuracy of 74.96% on the ImageNet evaluation set. To facilitate comparison with the equivariant model, we endow the invariant model with similar maps Mgen and Mphoto for the photometric and geometric augmentations. We train them using Eqn. but with the encoder parameters frozen.
5

---

Under review as a conference paper at ICLR 2023
4.1 EQUIVARIANCE MEASUREMENT
We use the trained maps (two maps for each model) to measure the equivariance of both the models using the measure of Eqn. 2 and report them Table [ We see that the equivariant/sterrabee model has significantly lower values than the baseline invariant model for both data augmentations, showing that pre-training with an equivariance promoting regularizer is crucial for learning equivariant and steerable representations.
Model
Pgeo +
Pphoto +
Flowers- 1
DTD 1
Pets 1
Caltech-101 1
Invariant (Standard)
0.982
0,983
84.38
64.15
91.82
86.55
Equivariant (Ours)
0,474
0.658
87.17
65.15
92.13
87.23
Table I: Left: Equivariance measure (Eqn. for the sets of augmentations. Right: Linear probe accuracy on 4 datasets: our equivariant model consistently outperforms the invariant model.
4.2 NEAREST NEIGHBOR RETRIEVAL
common use-case for pre-trained embeddings is their use in image retrieval (Xiao et al. 2021 Zheng et al. 2017). In the simplest case, given query embedding Cg. the nearest neighbors of this embedding are retrieved using Euclidean distance in embedding space over a database of stored embeddings. We test qualitative retrieval performance of the invariant and equivariant models. To mimic practical setting, we populate a database with the embeddings el ((x:) of samples x; from the ImageNet validation set. All query and key embeddings are normalized have an le norm of before being used in retrieval. Given a query sample x, we consider the following query embeddings:
e(x): Embedding of the sample x with no augmentation applied.
e(g(x)): Embedding of x after we apply a transformation g in input space.
M(exx))): Embedding map applied to embedding e(x) to steer towards egg(x)).
(ex)))) Compute the difference between e(x) and M (e(x) and add it back to (e(x))
with weight Wmi i.e. (e())) = (e(x)) + wm (M(xx - e(x)). This enables AM (e(x)) be 'pushed' further in the direction of the transformation. um is empirically chosen and set to 5 and for the equivariant and invariant models respectively for all retrieval experiments.
In addition to qualitative results, we compute the mean reciprocal rank (MRR): M MRR= - r where ri is the rank of the desired result within a list of retrieved responses single query, and n is the number of queries. MRR lies in the range 00, 1]. with 1 signifying perfect retrieval. We calculate MRR for both models, for color, crop, and brightness augmentations. Table [ieeessswww that the equivariant model achieves better ranks across all augmentations.
Model
Color
Zoom
Bright.
Color-Crop (1)
Invariant
0.619
0.356
0.444
0.126
Equivariant
0.974
0.728
0.757
0.827
Table 2: Mean Reciprocal Rank for single (left 3) and composed (right) augmentations.
4.2.1 PHOTOMETRIC AUGMENTATIONS
Results for color augmentation comparing the invariant and our equivariant/sterabll models, are displayed in Figure? 2 We observe that retrieved results for both e(g(x) and M (e(x)) change more in response to a change in query color for our steerable equivariant model than the invariant model. The color of the retrieved results for all queries for the standard model do not change appreciably, confirming invariance. This effect is even more pronounced for AM (e))).. We were unable to find any value of the parameter Wm for the invariant model that gave results qualitatively similar to the equivariant/sterablee model. In Figures toop 3 rows) and we show more examples across different classes and colors. Figure T rows) shows retrieval the setting of brightness changes. We populate the database with darkened and lightened versions using 0 photo (ő, 5, 8), where 0

---

Under review as a conference paper at ICLR 2023
Top-4 Nearest Neighbors
Query
variant)
#ix)
elgin)
Figure 2: For both invariant and equivariant/steeralle models, we show performance with types of embeddings. (x) and (e(x)) tend to be similar to each other (since Eqn. eencurreees this). For the invariant model, semantic retrieval (flowers) override visual (pink color). The equivariant model can perform better visual retrieval. By steering using A (M(x)))) we can further enhance the color component of the embedding control visual semantic retrieval.
Query
Eouivariant
Invoriant
elx)
Figure Color retrieval examples comparing our equivariant/sterrbbe model to the invariant (standard) baseline. Steerable embeddings capture both visual and semantic relationships between the query and keys. The invariant model gives the same top nearest neighbors regardless of query color.
mimic "daytime" and 5 < 0 to mimic "nighttime" versions of the images. We query for either using (e(x)). Our steerable model retrieves other images in similar lighting settings as the query, whereas the invariant model retrieves the exact same nearest neighbors for the dark and light queries.
The results demonstrate the benefit of both equivariance and steerability: applying the map to the embeddings gives both better and faster results than applying transformations to input images. Qualitatively, they also display the range of transformations and their parameters that our steerable cquivariant respresentations generalize to. More results are shown the Appendix (Figure I4.
4.2.2 IMAGE CROPPING/ZOOMING
In this experiment, we show that equivariant/steerable model preserves visual order for zooming data augmentation. Figure shows the original image and steered version (in embedding space). Each key image the dataset consists multiple zoomed versions of images from different classes. The equivariant model result maintains sensible global ordering (retrieving samples from the same class) as well as local ordering (ordering the nearest neighbors according to the level of zooming). The invariant model does not preserve local ordering. For example, the equivariant model retrievals are correctly ordered by zoom level; whereas the invariant model retrievals orders them unpredictably.
4.2.3 COMPOSED AUGMENTATIONS
More complex sequences of augmentations are easily formed by applying the map functions se- quentially. In Figure apply both photometric (color) and geometric (crop) augmentations in the database, and query using composed maps. The returned results respect both augmentations in sensible manner (although there is unique ordering). Note that the retrieved results respect
7

---

Under review as a conference paper at ICLR 2023
Query
Eoulvariant
Invoriant
M(elel)
Figure 4: Examples of retrieval with crop/zoom data augmentation. See text for details. Equivariant model retrieves the same sample, ordered correctly by zoom level (e.g. see how the dog's head progressively gets exposed). Invariant model does not preserve the zoom ordering or retrieves other samples. See Appendix for other examples.
Equivvrtant
Transforeed
Query
elgis)
Metslt
Figure 5: Image retrieval on composition of augmentations: color and crop transformations. We that along each dimension (color or crop) ordering preserved correctly.
high-level semantics (nearest neighbors belong to the class) in addition low-level attributes. We calculate MRR for this experiment as well, and report it in Table 2 (last column).
4.3 TRANSFER LEARNING
While invariance particular transformation useful for particular dataset/task, it may hinder performance on another. Thus, we expect equivariant representations perform better at transferring to downstream datasets than invariant representations. We test this by comparing the linear probe accuracy of both models Oxford Flowers- 102s (Nilsback & Zisserman 2008), Caltech-101 (Fei-Fei jet al., 2004), Oxford-IIIT Pets (Parkhi et al. 2012). and DTD (Fei-Fei et al.) (2004) (see Table T}. We that equivariant representations consistently achieve higher accuracy.
4.4
ROBUSTNESS AND OUT-OR-DISTTRIUTION DETECTION
Defocus Blur
Frost
Fog
Pixelate
0.84
0.70
0.94
0.92
0.62
0.68
0.80
0.90
0.80
0.66
0.75
Inv.
Equi.
0.70
0.88
0.86
10
70
30
40
50
60
10
20
30
40
50
60
10
70
30
40
50
8D
10
20
30
40
50
60
Augmentations
Augmentations
Augmentations
Augmentations
Figure 6: OOD for ImageNet (in-distribution) against ImageNet-C corruptions (out-of-distributtion) We use upto 60 crop augmentations. Equivariant AUC (latent) monotonically increases whereas invariant AUC (latent) stays nearly flat. Equivariant AUC's 55-15% better than that of invariant.
Invariance commonly encouraged in model pre-training to improve robustness (Zheng et al. 2016 Geirhos al. 2019 Rebuffi et al. 2021 (Hendrycks al. 2020). We test whether equivariance
8

---

Under review as a conference paper at ICLR 2023
Corruption
Equivariant
Invariant
Corruptio
Equivariant
Invariant
Gaussian Noise
37.53
33.89
Snow
33.01
30.94
Shot Noise
37.15
33.56
Frost
33.77
31.75
Impulse Noise
35.07
31.69
Fog
36.273
34.57
Defocus Blur
34.50
31.77
Brightness
39.26
37.69
Glass Blur
32.58
30.36
Contrast
40.00
38.30
Motion Blur
32.51
30.22
Elastic Transform
40.18
38.43
Zoom Blur
32.66
30.49
Pixelate
41.47
39.82
JPEG Compression
42.37
40.79
Table 3: Accuracy of models on all the corruptions from the ImageNet-C (averaged across severities).
then hurt in this setting vs invariance. We measure and compare the accuracy of the representations on various corruptions in the ImageNet-C (Hendrycks & Dietterich] 2019) dataset in Table 3 and find that the equivariant model is in fact suprisingly more robust on all the ImageNet-C corruptions. We also measure the mCE (lower better) for both models and find that our model has an mCE of 0.81 as compared to the invariant model's 0.845.
Despite better robustness, there significant accuracy loss. In this case, we want model to detect sample with corruptions as out-of -f--iiitibtttionn (OOD). Test-time data augmentation has enabled better performance on tasks such as detection of out-of-distribution. adversarial or misclassified samples and uncertainty estimation (Ayhan & Berens 2018 Bahat & Shakhnarovich [2020] Wang al. 2019). These approaches are based on the hypothesis that in-distribution images tend to exhibit stable embeddings under certain image transformations. In contrast, OOD samples have larger variations. This difference in stability can exploited to detect out-of-distribution samples. In existing work e.g. (Wang et al. 2019), multiple augmentations are applied to the input samples which are then forward propagated through the encoder. This leads to significant computational load since typically need a large number of augmentations. This becomes increasingly impractical as the number of augmentations is increased. With our steerable model we can apply these augmentations directly in embedding space, leading significant speedups. Applying 60 augmentations input and then forward propagating them a mini-batch takes 14.98 seconds. Conversely, forward propagating single sample and applying 60 mappings in embedding space takes only 0.28 and 0.02 seconds per mini-batch respectively: a nearly 50x speedup. Applying 60 augmentations at input and then forward propagating them in mini- batch takes 0.2263 seconds. Conversely, forward propagating single sample and applying 60 mappings in embedding space takes only 0.0091 and 0.0030 seconds per mini-batch respectively: a nearly 50x speedup.
We perform OOD detection using ImageNet validation set as the in-distribution dataset and ImageNet- C' as the OOD dataset. We use Mgeo to generate multiple augmentations in latent space for given image, and compute AUC curves across augmentations (see Appendix A.6 for details). Results are shown Figure for corruptions from ImageNet-C (remainder are presented in the Appendix), We see the clear benefit of applying latent augmentations for almost all corruptions and severity levels. We further see from Figure 6 that latent augmentations have an insignificant effect on the invariant model AUCs. Thus, these results demonstrate the benefit equivariant representations provide over invariant in test-time augmentations, and how steerability can be used to amplify these and obtain great computational speedups.
CONCLUSION
We have presented a method to steer equivariant representations in the direction of chosen data augmentations. To the best of our knowledge, ours is the first work to show practical approach for general deep network architectures and training paradigms. We show the benefits steerable equivariant embeddings in retrieval, robustness, transfer learning and OOD detection, with significant performance and computational improvements over the standard (invariant) model. Our method is simple to implement and adds negligible computational overhead at inference time. A limitation of our approach that it requires the learning of new maps for every new data augmentation that we would like to steer.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Murat Seckin Ayhan and Philipp Berens. Test-time data augmentation for estimation of heteroscedas- tic aleatoric uncertainty in deep neural networks. 2018.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly small image transformations? arXiv preprint 185.::/
Yuval Bahat and Gregory Shakhnarovich. Classification confidence estimation with test-time data- augmentation. arXiv preprint arXiv:2006.16700, 2020.
Beatrice Bevilacqua, Fabrizio Frasca, Derek Lim, Balasubramaniam Srinivasan, Chen Cai, Gopinath Balamurugan, Michael M. Bronstein, and Haggai Maron. Equivariant subgraph aggregation networks. CoRR, abs/21 10.02910, 2021. htt.httt///eeee wrt. 02111.01111100
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872-1888) 2013.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 008822, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR. 2020.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training self-supervised vision transformers. arXiv preprint arXiv:2104.02057, 2021.
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International conperaaetoonllcoeeeennnnnn on machine learning, pp. 2999. PMLR, 2016a.
Taco S Cohen and Max Welling. Steerable enns. arXiv preprint arXiv: 2016b.
Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 0 2018.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition Workshops, pp. 702-703, 2020.
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant self-supervised learning: Encouraging equivariance in representations. In International Conference Learning Representations, 2022. URL https
//openrevies www.wwwwwww..... net for.mmiddgggiaffffF..
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for so (3)-equivariant networks. arXiv preprint
1XXi:22004112229 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukeuoglu. Exploiting cyclic symmetry in convolu- tional neural networks. International conference on machine learning, pp. 1889-1898, PMLR, 2016.
10

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Carlos Esteves. Theoretical aspects of group equivariant neural networks. arXiv preprint arXiv:2004.05154, 2020.
Bin Fan, Fuchao Wu, and Zhanyi Hu. Rotationally invariant descriptors using intensity order pooling. IEEE transactions pattern analysis and machine intelligence, 34(10):2031-2045 2011.
Li Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference Computer Vision and Pattern Recognition Workshop, pp. 178-178, 2004, doi: 10.1109/CVPR.
2004.383.
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3165-3176. PMLR, 13-18 2020. URL https: http:::/pprceeeaanneeeeeo/////ee
mlr wttpp//ss////iiiiiii/222 html
Mare Finzi, Max Welling, and Andrew Gordon Wilson. A practical method for constructing equivari- ant multilayer perceptrons for arbitrary matrix groups. In International Conference Machine Learning, pp. 3318-3328. PMLR, 2021.
William T Freeman, Edward H Adelson, et al. The design and use of steerable filters. IEEE Transactions on Pattern analysis and machine intelligence, 13(9):891-900) 1991.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL http://///ppeeeeeeeeeeeeeeeerrrs
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self- sule-vupe learning. Advances in neural information processing systems, 33:21271-21283 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint /rxiv: 1903.12211, 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. 2019.
arXiv preprint
Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htwp::ttp/////to hattps::/apenzeeieww net/ ffrum?idd S1 ggmmxHF
11

---

Under review as a conference paper at ICLR 2023
Alex Hernández-Garche and Peter König. Further advantages of data augmentation convolutional neural networks. In International Conference on Artificial Neural Networks, pp. 35-103. Springer, 2018.
Charles Hermann, Kyle Sargent, Lu Jiang, Ramin Zabih, Huiwen Chang, Ce Liu, Dilip Krish- nan, and Deqing Sun. Pyramid adversarial training improves vit performance. arXiv preprint arXiv:2111.15121, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. gegueece-to-sequenee data augmentation for dialogue language understanding. arXiv preprint arXiv: 1807.0188 2018.
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In Proceedings of the IEFE International Conference on Computer Vision, pp. 1413-1421, 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint
arXiv:2004. 10.004.11366 2020.
Dmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Mare Pollefeys. Ti-pooling: transformation-nnvaraan pooling for feature learning in convolutional neural networks. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pp. 289-297, 2016.
Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. arXiv preprint
2018.
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy On the benefits of invariance in neural networks. arXiv preprint arXiv:2005.001 2020.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuar Hsuan Yang. Intriguing properties of vision transformers. arXiv preprint 1rXiv22005.10497, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference Computer Vision and Pattern Recognition, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Data augmentation can improve robustness. In Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https::///// http:::://ppeeevieww/ net wttp://www....ttt.oo.ooeccce.ees
David Romero, Erik Bekkers, Jakub Tomezak, and Mark Hoogendoorn. Attentive group equivariant convolutional networks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8188-88 8199, PMLR, 13-18 Jul 2020. URL https::/ fiess/vi19/romeroor
mlr
html
David W. Romero and Jean- Baaunnt Cordonnier. Group equivariant stand-alone self-attention for vision. CoRR, abs/2010.00977, 2020.
00977
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29: 2234-2242. 2016.
12

---

Under review as a conference paper at ICLR 2023
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. A simple but tough- to-beat data augmentation approach for natural language understanding and generation. arXiv preprint wrXiv:2009.13816 2020.
Laurent Sifre and Stéphane Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1233-1240, 2013.
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-cquivariant object representations for manipulation. arXiv preprint arXiv:21
2021.
Bart M. N. Smets, Jim Portegies, Erik J. Bekkers, and Remco Duits. Pde-based group equivariant convolutional neural networks. CoRR, abs/2001 2020. URL httptttss///xxl....tom org/
005/22001.09045
Tess E Smidt. Euclidean symmetry and equivariance machine learning. Trends in Chemistry, 3(2):
82-85, 2021.
Ivan Sosnovik, Michal Szmaja, and Arnold W. M. Smeulders. Scale-equivariant steerable networks. CoRR, abs/1910. 1093,, 2019. URL
11093
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270, 2021.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer ECsV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 266-282 282. Springer, 2020.
Iyy Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01600 2021.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitzzmarrin training: Scalable certification of perturbation invariance for deep neural networks. arXiv preprint trXiv:1802.040:: 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Guotai Wang, Wenqi Li, Michael Aertsen, Jan Deprest, Sébastien Ourselin, and Tom Vercauteren. Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing, 338:34-45, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2021. URL
https:
net F
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 819-828, 2020.
Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, and Zheng Zhang. Ccale-invariant convolutional neural networks. arXiv preprint arXiv:1411.6366 2014.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 6023-6032, 2019.
13

---

Under review as a conference paper at ICLR 2023
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. The visual task adaptation benchmark. 2019.
Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pp. 7324-7334, PMLR, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Liang Zheng. Yi Yang, and Qi Tian. Sift meets enn: decade survey of instance retrieval. IEEE transactions on pattern analysis and machine intelligence, 40(5):1224-1220 1244, 2017.
Stephan Zheng, Yang Song, Thomas Leung, and Ian J. Goodfellow. Improving the robustness of deep neural networks via stability training, 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4480-4488, 2016.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 VISUAL EXPLANATION OF INVARIANCE, EQUIVARIANCE AND STHERABILITY
Invariance
Equivariance
Steerability
e()e)
e( M(e())
e(x) M(e(x))
M
g
g(x)
X
g(x)
g(x)
Figure 7: The concepts of invariance, equivariance and stecrability of embeddings. Blue boxes represent (shared) encoder that takes the input to the embedding e(x). g(x) represents a transformation of x in input space; and M (e(x)) is mapping in embedding space. Equivariance is a necessary but not sufficient condition for steerability.
A.2 TRAINING DETAILS
Both the invariant and equivariant models use ResNet-50 encoder trained for 250 epochs with batch size of 4096, on 224x224 sized ImageNet images with AutoAugment (Cubuk et al.) 2018) applied on the input to the cross-entropy loss. The remaining optimization details are as follows: SGD optimizer with 0.9 nesterov momentum, 0.1 learning rate with cosine decay warmed up over 12 epochs, 0=0.1 and $=0.1.
We plot the values of the equivariance metric, p, over the course of training in Figure
34
0.5
TW
0.4
1300
0.1357
27722
0.4524
Figure 8: Values of Pphoto components over the course of training, Left: Numerator. Center:
denominator. Right: Pphoto
A.3 ABLATIONS ON HYPERPARAMETERS
In Table conduct ablations hyperparameters and 3. We can that both hyperparameters have sweet spots, above and below which the model either does not gain much equivariance, or does but at the cost of reduced accuracy. For the main paper, empirically selected a model with hyperparameter values such that the cross-entropy accuracy is not adversely reduced w.r.t to the invariance los,; and the p equivariance metric reduced (lower is better) for all augmentations.
15

---

Under review as a conference paper at ICLR 2023
Accuracy
Pgee +
Pphoto
Accuracy
Pyco +
Pphoto 4
0.0
72.52
0,9752
0,9726
le-3
74.2
0,9845
0.9914
0,05
75.01
0.5973
0.3239
0.1
75.25
0.6482
0,4744
0.01
69.51
0.9804
0.9658
0.2
74.63
0.8547
0.6345
0.1
75.51
0.984
0.9329
0.5
74.44
0,9725
0.848
1.0
75.25
0,6482
0,4744
1.0
61.11
0.9757
0.8169
5.0
67.91
0,4722
0.3134
Table 4: Left: Ablation on with 3=0.1. Right: Ablation on 3, with a=1.0
A.4 ROTATION
We add rotation to the list of augmentations and measure model accuracy and ps in Table 5t. We see that the ot (equivariance measure) is lower for our model than a standard (invariant) ResNet-50 in this case as well, and that the existing model accuracy and values for other augmentations are minimally affected. This confirms that our proposed method generalizes to rotations as well.
Model
Accuracy
Pgeo +
Pphoto +
Prot +
Invariant (Standard)
75.17
0.9819
1.03
1.037
Equivariant (Ours)
75.18
0.535
0.502
0.498
Table 5: Accuracy and Equivariance measure (Eqn. 2
with rotation augmentation
A.5 ROBUSTNESS
In Table 6 we show the accuracy of both the models on ImageNet-C dataset (Hendrycks Dietterich 2019), all 15 corruptions and 3 different severity levels. We can that the equivariant model outperforms the invariant across the board, with better accuracy on all but 5 out of 45 data points.
A.6 OUT-OF-DISTRIBUUTIN DETECTION
Test-time Augmentation Details: Here we give details of how we perform test time augmentation. We use Mgeo / Mphoto to generate multiple augmentations in latent space for a given input image. We compute the geometric mean across the set of logits generated in this manner (for given number of augmentations), and then use this average logit to compute softmax probabilities. The maximum softmax value the confidence for this sample. We use these confidences across a set ImageNet and ImageNet-C samples and probability threshold values to compute PR curve, and measure the AUC of this PR curve. We repeat this for upto 60 augmentations, and plot the AUC values across the number of augmentations. In Figure TO provide more examples of OOD detection for all 15 corruptions from ImageNet-C (Hendrycks & Dietterich) 2019), with severity level 3. In 12 of the 15 cases, the equivariant latent outperforms invariant latent space in AUC on both photometric and geometric augmentations. We display similar plots on different severity levels with the geomentric augmentation in Figures II and 13 Adding more number of augmentations may help further improve performance on the equivariant model.
We also repeat the experiment above but by applying augmentations directly on the input images. We can only apply upto 8 input augmentations, as they use significantly more memory than latent augmentations. The results are plotted in Figure g We see that (1) in general input augmentations do better than latent space augmentations but at significantly higher speed, compute and memory cost; and (2) equivariant input augmentations always better than for the invariant model. This shows the benefit of our equivariance promoting regularizer.
A.7 NEAREST NEIGHBOR RETRIEVAL
In Figures [4 15 16
we show more examples of image retrieval with color, crop/zoom, composed- color-zoom, and brightness queries respectively across different classes. Qualitatively, this displays
17
16

---

Under review as a conference paper at ICLR 2023
Equivariant
Invariant
Corruption Severity
5
Zoom Blur
62.61
39.88
8.44
61.62
34.57
4.50
Gaussian Noise
62.15
38.95
11.11
61.27
33.85
6.84
JPEG Compression
50.23
34.93
7.95
50.45
31.38
4.15
Fog
55.42
31.07
11.59
54.93
30.17
10.78
Shot Noise
57.10
13.46
5.78
56.73
12.91
5.25
Impulse Noise
63.58
27.92
7.25
61.82
24.00
5.65
Defocus Blur
49.43
32.80
19.80
48.88
30.59
18.86
Glass Blur
54.77
37.03
23.42
54.36
35.89
21.36
Motion Blur
60.66
34.55
26.53
60.12
32.66
24.05
Snow
67.60
59.41
46.28
68.07
60.63
48.00
Frost
73.62
70.06
62.73
73.36
69.89
62.21
Brightness
69.15
58.49
11.02
68.58
55.17
7.68
Contrast
68.41
51.20
9.87
68.01
48.11
8.14
Blastic Transform
68.30
59.93
46.66
67.77
59.84
45.80
Pixelate
62.42
57.54
43.91
62.87
57.44
41.07
Table 6: Accuracy of models on all the corruptions from the ImageNet-C: with multiple severities.
the range of transformations and their parameters that our steerable equivariance respresentations generalize to.
8
Accuracy
Pyco 4
Pphoto +
Flowers-100 t
DTD t
le-3
75.17
0.9824
0.9831
83.59
65.18
le-2
75.23
0.9827
0.9889
83.26
64.06
72.52
0.9752
0.9726
88.84
66.52
Table 7: Ablation with = 0, > 0
17

---

Under review as a conference paper at ICLR 2023
Gaussian Noise 3
Shot Noise 3
Impulse Noise 3
0.90
0.900
0.875
0.85
0.85
0.850
0.80
0.800
Defocus Blur
Glass Blur
Motion Blur 3
0.96
0.92
0.94
0.94
AUC
0.92
0.88
0.92
0.86
0.88
0.90
0.86
0.82
Zoom Blur 3
Snow
Frost 3
0.86
0.84
AUC
0.80
0.82
0.78
0.80
0.76
0.78
0.74
Fog
Brightness 3
Contrast 3
0.64
0.80
0.75
0.75
0.60
0.58
0.70
0.56
0.65
0.54
Elastic Transform
Pixelate
JPEG Compression
0.750
Equi. Input
0.80
Inv. Irput
Equi. Latent
0.76
0.75
0.72
0.70
0.650
0.65
0.625
50
10
20
60
30
No. of Augmentations
No. Augmentations
No. Augmentations
Figure 9; OOD Detection for ImageNet-C with geometric augmentations, applied either in input image directly in latent embedding space.
18

---

Under review as a conference paper at ICLR 2023
Gaussian Noise 3
Shot Noise 3
Impulse Noise 3
0.80
0.82
0.81
0.78
0.80
0.79
0.76
0.78
0.74
0.74
Defocus Blur
Glass Blur
Motion Blur 3
0.94
0.95
0.87
0.92
0.86
AUC
0.93
0.90
0.85
0.92
0.88
0.91
0.83
0.90
0.86
Zoom Blur 3
Snow
Frost 3
0.84
0.84
0.83
0.80
0.83
0.82
AUC
0.78
0.80
0.80
0.76
0.79
0.79
0.78
Fog 3
Brightness 3
Contrast 3
0.70
0.70
0.60
0.68
0.58
0.64
0.56
0.64
0.62
Elastic Transform
Pixelate
JPEG Compression
0.75
0.67
0.80
0.74
AUC
0.65
0.64
Equi. Photo
0.71
0.70
0.70
0.62
Equi. Geo
0.65
Inv. Geo
0.69
0.61
50
20
60
30
No. of Augmentations
No. Augmentations
No. Augmentations
Figure 10: 0OD Detection for ImageNet-C when both photometric and geometric augmentations applied. We see that both augmentations lead to improved OOD performance.
19

---

Under review as a conference paper at ICLR 2023
Gaussian Noise 1
Gaussian Noise 3
Gaussian Noise 5
0.64
0.80
0.96
0.95
0.62
0.78
0.94
0.60
0.76
0.93
0.58
0.74
0.92
Shot Noise 1
Shot Noise 3
Shot Noise 5
0.64
0.80
0.94
0.62
0.78
0.92
0.76
0.60
0.90
0.74
Impulse Noise 1
Impulse Noise
Impulse Noise
0.82
0.96
0.72
0.80
AUC
0.94
0.70
0.78
0.92
0.68
Defocus Blur
Defocus Blur
Defocus Blur
0.94
0.98
0.80
0.92
0.97
AUC
0.90
0.75
0.96
0.88
0.70
0.86
0.95
Glass Blur
Glass Blur
Glass Blur 5
0.80
0.98
0.94
0.97
AUC
0.75
Equi. Geo Inv. Geo
0.96
0.92
0.95
0.70
0.94
20
50 60
30
40
1 20 30 40
No. of Augmentations
No. of Augmentations
No. of Augmentations
Figure 11: OOD Detection for ImageNet-C with geometric augmentations and multiple severity levels.
20

---

Under review as a conference paper at ICLR 2023
Motion Blur
Motion Blur 3
Motion Blur 5
0.95
0.66
0.86
0.64
0.93
0.62
0.92
0.60
0.82
Zoom Blur 1
Zoom Blur 3
Zoom Blur 5
0.84
0.88
0.76
0.74
0.82
0.86
0.72
0.80
0.70
0.84
Snow
Snow 3
Snow 5
0.70
0.80
0.86
0.68
0.78
0.66
0.76
0.82
Frost
Frost
Frost
0.84
0.68
0.82
0.66
AUC
0.84
0.64
0.80
0.62
0.78
0.82
Fog
Fog
Fog 5
0.70
0.62
0.68
0.76
0.60
AUC
0.66
0.74
Equi.
Inv.
0.58
0.64
0.72
0.56
0.62
50 60
30
40
20 30 40 No. of Augmentations
No. of Augmentations
No. of Augmentations
Figure 12: (contd.) OOD Detection for ImageNet-C with geometric augmentations and multiple severity levels.
21

---

Under review as a conference paper at ICLR 2023
Brightness 1
Brightness 3
Brightness
0.55
0.60
0.68
0.54
0.66
AUC
0.58
0.64
0.52
0.56
0.62
0.51
0.54
Contrast
Contrast
Contrast 5
0.70
0.95
0.60
0.68
0.94
AUC
0.58
0.66
0.93
0.56
0.64
0.92
0.54
0.62
Elastic Transform
Elastic Transform
Elastic Transform
0.87
0.650
0.74
0.86
AUC
0.625
0.72
0.85
0.600
0.84
0.70
0.575
Pixelate
Pixelate 3
Pixelate 5
0.70
0.90
0.80
0.85
AUC
0.65
0.75
0.80
0.70
0.60
0.75
0.65
JPEG Compression 1
JPEG Compression
JPEG Compression 5
0.76
0.62
0.66
AUC
0.74
0.64
0.60
0.72
Equi.
0.62
Inv.
0.70
30 No. Augmentations
40 50 60
10
20
40
60
10
20
No. of Augmentations
No. Augmentations
Figure 13: (contd.) 0OD Detection for ImageNet-C with geometric augmentations and multiple severity levels.
22

---

Under review as a conference paper at ICLR 2023
Boulvartark
Invariant
AMeeod
AMeead)
Query
Eculvariant
Invoriant
(Meels)
Query
AMeood
Eoulvariant
Invariant
AMleod)
Query
Eoulvartant
Invoriant
AMeeati
Figure 14: Image Color Retrieval Examples.
23

---

Under review as a conference paper at ICLR 2023
Query
bueverint
Inveriant
Migho))
Query
Equiverent
Invecient
Atetot
Query
Atebot
Query
el290)
Query
Equivorint
Invoriant
Mix)
elgot)
Figure 15: Image Crop/Zoom Retrieval Examples.
24

---

Under review as a conference paper at ICLR 2023
Query
Equivariant
e(x)
Transformed Query
e(g(x)
Mtetx):
Figure 16: Image Color-Crop Composition Retrieval Examples.
Query
Equivariant
Inveriant
Vel
Bett
AMielsl)
AMtetvi)
AMielstl
Quary
AMtetri)
Figure 17: Image Brightness Retrieval Examples.
25


Reviews:
[{"review_id": "qfo-OWPYz3", "reviewer": "Reviewer_nTJZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes steerable equivariant embedding for representation learning. Unlike invariant embedding typically achieved through learning with data augmentation, the model jointly learns a feature embedding and a (NN-parametrized) latent space transformation. Equivariant embedding is achieved by minimizing the cross-entropy loss regularized by the equivariance-promoting regularizer and the uniformity regularizer.", "summary_of_the_review": "Even though the paper tries to address an interesting question, there are many issued identified in the *weakness* part of this review. There is also no theoretical contribution in the paper.", "strength_and_weaknesses": "**Strength**\n1. The paper is generally well-written.\n2. Extensive experiments have been conducted.\n\n**Weakness**\n1. Equation (2) is misleading. Instead of $g_a(x, \\theta_a)$ and $M_a(e(x), \\theta_a)$, the notations $g(x, \\theta_a)$ and $M(e(x), \\theta_a)$ should be used, since, if I understand correctly, the mapping $M$ and $g$ depends on $a$ only through $\\theta_a$.\n2. Instead of using $L_U$ and $L_E$ as two regularization terms, why not use $\\rho_a$ directly? The authors mentioned that pushing $e(x)$ and $e(g_a(x))$ apart destabilizes training. However, isn't pushing them apart what the authors aim to achieve?\n3. Equation (4) and Equation (5) are a bit confusing. What are $x_i$ and $x_j$ in the definition of $L_U(x)$? Is $x$ a batch of data containing $x_i$ and $x_j$?\n4. What happens if the number of transformations $a$ is large in equation (5)? For example, consider the cyclical group $C_8$; does that mean the loss $L_E$ will be the sum of eight terms?\n5. What is SimCLR at the bottom of page 5?\n6. The comparison between the invariant and the proposed equivariant embedding (especially in Section 4.1, table 1) seems unfair and somewhat pointless. After all, the invariant embedding is trained (through augmentation) to be invariant, and thus freezing the encoder while training $M$ with equation (3) will essentially learn an identity map. This surely leads to a large $\\rho_a$ defined in equation (2). What I am trying to say is that invariant embedding is indeed doing its job correctly, and it is pointless to compare it to the proposed model by forcing it to learn an $M$.\n7. In section 4.2, how is $w_n$ decided? Again, the invariant model is doing its job correctly. From a different point of view, one might say that the invariant model in Figure 2 is robustly retrieving the same figure after input transformation, whereas the figures retrieved by the proposed embedding actually seem arbitrary.\n8 What is $r_i$ in MRR? How is it calculated?\n9. More explanation is needed for Figure 2, 3, 4, 5. For the first row $e(x)$, are they the retrieved image from the same query? Does it mean the authors have trained multiple embedding models?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "efRk1RQLQZO", "reviewer": "Reviewer_EeNj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to train representations equivariant to data augmentations by optimizing two additional regularizers together with the standard cross-entropy loss. Specifically, equivariance is promoted by minimizing the L2 loss between the embedding transformed by $M_a$ and embedding obtained by feeding augmented data, and uniformity is forced to reduce invariance. Experimental results show the effectiveness of the proposed method.", "summary_of_the_review": "The idea is interesting, but the experimental setting is quite limited to show the effectiveness of the proposed method. Also, I recommend authors distinguish steerability and equivariance in a better way. Please answer my concerns above.\n\n**post-rebuttal**\n\nThe authors partially addressed my concerns, so my initial rating is unchanged. Below are additional comments:\n\n- It is not clear if the mapping on the embedding space $M$ behaves as expected. How do you guarantee that nonlinear and complex data augmentations on the data space can always be simplified to be linear transformations on the embedding space (given that the actual implementation of $M$ is essentially the summation of the linearly transformed embedding vector and augmentation parameter vector)? I think more experimental analyses other than a few number of image retrieval results are required to make the results more convincing.\n\n- Scalability on the size of model and dataset is still questionable; author response did not address this.\n\n- Together with scalability, it would be interesting to see if hyperparameters ($\\alpha$ and $\\beta$) are sensitive to the choice on the model and dataset.", "strength_and_weaknesses": "Strengths\n\n+ The idea of promoting equivariance to data augmentations seems interesting and useful.\n\nWeaknesses\n\n- Steerability is not inspired/explained well. Throughout the description on the method in Section 3, \"steer\" is never used, except when combining loss to argue that the final loss is to train a steerable model. What is steerability in deep representations, how does the proposed loss promote steerability, and how are steerability and equivariance related? It seems the authors do not distinguish steerability and equivariance well.\n\n- There are prior works about training equivariant networks. How do the authors compare them with this work? There are many properties that models can be equivariant, and it seems this work is limited to learning to be equivariant to data augmentations only.\n\n- How does the embedding map M look like? I couldn't find the actually experimented version throughout the paper.\n\n- Experimental comparisons are limited to vanilla ResNet-50 training vs. proposed method. For example, I wonder how the proposed method is compared to existing methods for enhancing robustness/detecting out-of-distribution detection in Table 3. Given that the proposed method is orthogonal to them, it would be interesting to see if the combined method can improve the performance further.\n\n- Experimental setting is limited to ImageNet-C. Experiments on standard OOD detection benchmarks might also be interesting.\n\n- I couldn't find ablation studies on the choice/combination of hyperparameters/losses.\n\n- Typo: 2nd and 3rd rows of Figure 1 have the same metric.\n\n- Typo: $e(g_a(x)$ at the end of Section 3.3.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EKuVlaXdw_", "reviewer": "Reviewer_D95j", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to learn equivariant representations that are steerable by a learned function in representation space. They empirically demonstrate the method as an improvement to ImageNet classification, where the pre-trained model performs better in transfer, retrieval, corruption, OOD detection settings.", "summary_of_the_review": "The paper demonstrates an effective way to improve equivariance of (classification) models using an augmentation-parameterized predictor. The method is intuitive and promising, but there are a few issues to be resolved before acceptance.", "strength_and_weaknesses": "Strength\n\n1. The idea is interesting and intuitive. Experiments are convincing.\n2. The retrieval visualization is compelling. It clearly shows the property of the feature space learned by the proposed method compared with invariant learning.\n3. The test-time augmentation is a useful application.\n\nWeakness\n1. The scope is limited to supervised ImageNet classification, while the self-supervised setting can be more appealing. As the paper claims, the approach “is general and easily extends to self-supervised settings”, it is relatively straightforward to try this in a contrastive learning setting (e.g., just remove the supervised loss)? If there was an issue with the contrastive learning setting, could the authors discuss it?\n2. How does this method compare with the inverse-prediction style method as in Xiao et al., 2021, Dangovski et al., 2022? Which is more effective? I feel their methods can also be easily extended to the supervised setting -- just attach a loss with an inverse predictor.\n3. How to determine alpha and beta in the loss? I have earlier noticed that increasing uniformity alone (alpha=0, beta>0) can sometimes improve transfer and robustness.\n\nMinor: The citation should use \\citet instead of \\citep in some places.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9zZwAi_Hx_", "reviewer": "Reviewer_k3c6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for learning equivariant representations with respect to photometric/geometric transformations, which has received little attention in the equivariant learning literature. The representations are also designed to be steerable so that the transformations can be directly applied in the latent space. The implementation of the method is simple and intuitive in that a mapping between two embeddings is introduced given an augmentation and regularized to be equivariant by a loss function. Since the augmentations are performed in the latent space through the mapping function, it brings significant speedup (nearly 50x) for test-time augmentations, and the equivariance of the mapping facilitates an interesting application for image retrieval, which works with image augmentations. At last, the paper demonstrates the method is beneficial to transfer learning and out-of-detection tasks.", "summary_of_the_review": "This work is novel in its idea and direction, but it significantly lacks investigations on the method itself, as I commented on the weaknesses. \nSince the authors' response addresses some of my concerns, I raise my rating, leaning toward weak accept. ", "strength_and_weaknesses": "This paper has strength in novelty since it might be the first work incorporating transformations other than matrix groups for equivariant learning. In addition, it has shown the steerability and equivariance of the representations are highly practical and effective in several tasks and applications. However, the exposition of the proposed method still lacks in the following aspects:\n\n1) The statement “Our method is generalizable across architectures and pre-training losses.” on page 2 has not been proven empirically, as the authors only employ ResNet and cross-entropy loss for experiments.\n\n2) The authors, on page 3, claim the proposed method is widely applicable as long as a transformation is parameterized in input space and this covers the specific transformations, i.e., rotation, that the prior work mainly concentrates on. But I am not convinced about the generalizability of the method to those specific transformations and ask for evidence.\n\n3) The proposed method applies the same augmentation for invariant and equivariant learning, and the uniformity regularizer is proposed to resolve this conflict. But I could not find any ablation studies for this regularizer and sensitivity analysis for hyperparameters \\alpha and \\beta. In addition, I want to ask why the independent views for equivariance and uniformity losses are used, making them inconsistent with Eq. (5). Please answer these points with experimental results.\n\n4) The ρ value for the equivariant model in Table 1 seems to be still large as the model is trained with exactly the same loss function as the measurement. Can you provide the curve for the numerator (equivariance metric), the denominator (invariance metric), and the value itself separately over the course of training?\n\n5) In Figure 5, the order of retrieved images for e(g(x)) and M(e(x)) are opposite to each other, which is supposed to be the same in general as in Figure 14. What is the reason for this one? In some figures, the right parenthesis is missing.\n\n6) Table 3 is not referred to in any part of the paper, and the values are not the same as that of Table 4. Please explain the difference between them. \n\n7) For OOD in Figure 6, I want to see the results further when input data is explicitly forwarded with 60 augmentations for both invariant and equivariant models to identify the upper bound though the time is consumed significantly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject