EquiMod: An Equivariance Module to Improve Visual Instance Discrimination
Authors: Alexandre DEVILLERS
Mathieu Lefort
Venue: ICLR 2023 poster (2023)
Keywords: Representation learning
Self-supervised learning
Contrastive learning
Equivariance

Abstract: Recent self-supervised visual representation methods are closing the gap with supervised learning performance. Most of these successful methods rely on maximizing the similarity between embeddings of related synthetic inputs created through data augmentations. This can be seen as a task that encourages embeddings to leave out factors modified by these augmentations, i.e. to be invariant to them. However, this only considers one side of the trade-off in the choice of the augmentations: they need to strongly modify the images to avoid simple solution shortcut learning (e.g. using only color histograms), but on the other hand, augmentations-related information may be lacking in the representations for some downstream tasks (e.g. literature shows that color is important for bird and flower classification). Few recent works proposed to mitigate this problem of using only an invariance task by exploring some form of equivariance to augmentations. This has been performed by learning additional embeddings space(s), where some augmentation(s) cause embeddings to differ, yet in a non-controlled way. In this work, we introduce EquiMod a generic equivariance module that structures the learned latent space, in the sense that our module learns to predict the displacement in the embedding space caused by the augmentations. We show that applying that module to state-of-the-art invariance models, such as BYOL and SimCLR, increases the performances on the usual CIFAR10 and ImageNet datasets. Moreover, while our model could collapse to a trivial equivariance, i.e. invariance, we observe that it instead automatically learns to keep some augmentations-related information beneficial to the representations. 

Published as a conference paper at ICLR 2023
EQUIMOD: AN EQUIVARIANCE MODULE TO IMPROVE VISUAL INSTANCE DISCRIMINATION
Alexandre Devillers & Mathieu Lefort Univ Lyon, UCBL,, CNRS, INSA Lyon LIRIS, UMR5205, F-69622 Villeurbanne, France {alexandre, wweeeeeelerrre mathieu. mathiie...eoooreeliiiii.... cnrs.
ABSTRACT
Recent self-supervised visual representation methods are closing the gap with su- pervised learning performance. Most of these successful methods rely on max- imizing the similarity between embeddings of related synthetic inputs created through data augmentations. This can be seen as a task that encourages embed- dings to leave out factors modified by these augmentations, i.e. to be invariant to them. However, this only considers one side of the trade-off the choice of the augmentations: they need to strongly modify the images to avoid simple so- lution shorteut learning (e.g. using only color histograms), iugmentations-relate. information may be lacking in the representations for some downstream tasks (e.g. literature shows that color is important for bird and flower classification). Few recent works proposed mitigate this problem of using only an invariance task by exploring some form of equivariance to augmentations. This has been performed by learning additional embeddings space(s), where some aug- mentation(s) cause embeddings to differ, yet in a non-controlled way. In this work, we introduce EquiMod generic equivariance module that structures the learned latent space, in the sense that our module learns to predict the displacement in the embedding space caused by the augmentations. We show that applying that mod- ule to state-of-the--rt invariance models, such as BOL and SimCLR, increases the performances on the usual CIFARIO 10 and ImageNet datasets. Moreover, while our model could collapse to trivial equivariance, i.e. invariance, we observe that it instead automatically learns to keep some augmennatioos--reltte information beneficial to the representations.
on the other hand,
Source code available
https://gtthhb..cm/DDer///
EquiMod
INTRODUCTION
Using relevant and general representation is central for achieving good performances on downstream tasks, for instance when learning object recognition from high-dimensional data like images. His- torically, feature engineering was the usual way of building representations, but we can currently rely on deep learning solutions to automate and improve this process of representation learning. Still, it is challenging as it requires learning structured latent space while controlling the precise amount of features to put representations: too little information will lead not interesting rep- resentations, yet too many non-pertinent features will make it harder for the model to generalize. Recent works have focused on Self-Supervised Learning (SSL), i.e. determining a supervisory sig- nal from the data with pretext task. It has the advantages of not biasing the learned representation toward a downstream goal, as well as not requiring human labeling, allowing the use of plentiful raw data, especially for domains lacking annotations. In addition, deep representation learning courages network reuse via transfer learning, allowing for better data efficiency and lowering the computational cost of training for downstream tasks compared to the usual end-to-end fashion.
The performances of recent instance discrimination approaches SSL of visual representation are progressively closing the gap with the supervised baseline (Caron et al. 2020 Chen al. 2020a b

---

Published as a conference paper at ICLR 2023
Chen & He. 202 1: Bardes et al. 2021 Grill al. 2020 He et al., 2020: Misra & Maaten 2020
Zbontar et al. 2021). They are mainly stamese networks performing an instance discrimination task. Still, they have various distinctions that make them different from each other (see Liu (2021) for review and SSzgedy et al. (2033) for a unification of existing works). Their underlying mechanism is to maximize the similarity between the embedding of related synthetic inputs, a.k.a. views, cre- ated through data augmentations that share the same concepts while using various tricks to avoid collapse towards constant solution (Jing et al., 2021; Hua al., 2021). This induces that the latent space learns an invariance to the transtormations used, which causes representations to lack augmentations-rekan related information.
Even if these models are self-supervised they rely on human expert knowledge to find these relevant invariances. For instance, as most downstream tasks in computer vision require object recognition, existing augmentations do not degrade the categories of objects in images. More precisely, the choice of the transformations was driven by some form of supervision, as it was done by experi- mentally searching for the set of augmentations giving the highest object recognition performance on the ImageNet dataset (Chen et al. 2020a). For instance, it has been found that color jitter the most efficient augmentation on ImageNet. One possible explanation is that color histograms easy-to-learn shortcut solution (Geirhos et al.j (2020). which is not removed by cropping augmen- tations (Chen et al., 2020a). Indeed, there are many objects in the categories of ImageNet, and as an object category does not change when its color does, the loss of color information is worth removing the shortcut. Still, it has been shown that color is an essential feature for some downstream tasks (Xiao et al., 2020).
Thus, for given downstream task, we can separate augmentations into two groups: the ones for which the representations benefit from insensitivity (or invariance) and the ones for which sen- sitivity (or variance) is beneficial (Dangovski et al.) (02T). Indeed, there is a trade-off in the choice of the augmentations: they require to modify significantly the images to avoid simple so- lution shortcut learning (e.g. relying just on color histograms), yet some downstream tasks may need ingmenttaiions-ellttee information in the representations. Theoretically, this trade-off limits the generalization of such representation learning methods relying on invariance. Recently, some works have explored different ways of including sensitivity to augmentations and successfully im- proved augmentations innvantttionsnnnvriiatt SSL methods on object classification by using tasks forcing sensi- tivity while keeping invariance objective in parallel. Dangovski et al. (2021) impose sensitivity to rotations, an augmentation that is not beneficial for the invariance task, while we focus in this paper on sensitivity to transformations used for invariance. Xiao et al. (2020) proposes to learn as many tasks as there are augmentations by learning multiple latent spaces, each one being invariant to all but one transformation, however, it does not control the way augmentations-relate information is conserved. One can see this as an implicit way of learning variance to each possible augmentation. Contrary to these works that do not control the way augmentations-rela here we propose to explore sensitivity by introducing an equivariance module that structures its la- tent space by learning to predict the displacement in the embedding space caused by augmentations in the pixel space.
information is conserved,
The contributions of this article are the following:
We introduce a generic equivariance module EquiMod to mitigate the invariance to aug- mentations in recent methods of visual instance discrimination;
We show that using EquiMod with state-of-the-art invariance models, such BYOL and SimCLR, boosts the classification performances on CIFAR10 and ImageNet datasets;
We study the robustness of EquiMod to architectural variations of its sub-components;
We observe that model automatically learns specific level of equivariance for each augmentation.
Sec. 2 will present our EquiMod module as well as the implementation details while in Sec. 3 we will describe the experimental setup used to study model and present the results obtained. The Sec. 4 will position our work w.r.t. related work. Finally, in Sec. we will discuss our current results and possible future works.
2

---

Published as a conference paper at ICLR 2023
2 EQUIMOD
2.1 NOTIONS OF INVARIANCE AND EQUIVARIANCE
As in Dangovski ct al. (202)), we relate the notions of augmentations sensitivity and insensitivity to the mathematical concepts of invariance and equivariance. Let T be distribution of possible transformations, and f denotes a projection from the input space to a latent space. That latent space said to be invariant to T if for any given input the Eq. i respected.
EET f(((=)) = f(x)
(I)
Misra & Maaten (020) used this definition of invariance to design pretext task for representation learning. This formulation reflects that the embedding of non-augmented input sample æ will
not change if the input transformed by any the transformations in T. However, more recent
works (Bardes et al. 2021 Chen et al.) 2020a Chen & He 2021 Grill et al., 2020; Zbontar et al.
2021 focused on another formulation of invariance defined by the following Eq.
EE T, WE ET
f(t(z)) f(t
With this definition, the embedding produced by an augmented sample æ is independent of the transformation used. Still, note that Eq. | implies Eq. 2 and that the identity function is part of T, which is the case with recent approaches, then both definitions are indeed equivalent.
While insensitivity to augmentation is reflected by invariance, sensitivity can be obtained by achiev- ing variance, i.e. replacing the equality by inequality in Eq. [r Eq. 2 Yet, this is not interesting property, as any injective function will satisfy this constraint. this paper, we propose to use equiv- ariance as a way to achieve variance to augmentations for structuring latent space. Eq. 3 gives the definition of equivariance used in the following work.
At E T, Euee f(t(x)) = (f(())))
(3)
With ue being transformation in the latent space parameterized by the transformation f. it can be seen as the counterpart of the transformation but in the embedding space. With this definition, the embeddings from different augmentations will be different and thus encode somehow information related to the augmentations. Yet, un is always the identity then this definition of equivariance becomes the same as invariance [q Indeed, one see invariance cquivariance. In the following, we only target non-trivial equivariance where u, produces some displacement in the latent space. See Fig. for visual comparison of invariance and equivariance.
trivial specific case of
2.2 METHOD
EquiMod is generic equivariance module that acts as complement existing visual instance discrimination methods performing invariance (Bardes et al. (202]; Chen et al. 2020a] Chen & Hej 2021 Grill al. 2020; Zboonar et al., 2021). The objective of this module is capture some augmentatioos--rlatee information originally suppressed by the learned invariance to improve the learned representation. The main idea relies on equivariance, in the sense that our module learns to predict the displacement in the embedding space caused by the augmentations. This way, by having non-null displacement, we ensure embeddings contain augmentations relatttt information. We first introduce a formalization for these existing methods (see Bardes et al. (202T) for an in- depth explanation of this unification), before introducing how our approach adds on top.
Let f and t' denote two augmentations sampled from the augmentations distribution T. For the given input image æ, two views are defined as v, := t(x) and v, : t'(x). Thus, for N original images, this results in batch of 2.N views, where the first N elements correspond to first view (v:) for each of the images, and the last N elements correspond to a second view (vz) for each of the images. Following previous works, from images, and 9s a projection head parameterized by d. which projects representations in an embedding space. This way, the representations are defined as h, == fo(v:) as well h2 == fo(w;),
note fe an encoder parameterized by 0 producing representations
3

---

Published as a conference paper at ICLR 2023
fo
h4g
Maximize similarity
Invariance
Equivariance
K
(-))
fa
-
ho
5/y
uel (-)
<(-)
Invariance Task
f())
The model learns similar embed- dings for an augmented view (z() and the prediction of the displacement in the embed- ding space caused by that augmentation (2)), a learned representation of the parameters of the transformation, see Sec. [ for notation details.
Figure 2:
Figure I: On the left, invariance described by Eq. | on the right, equivariance considered in this paper and described by Eq. 3
and the embeddings z; == go(h)) as well as 2, : = Then, the model learns maximize the similarity between z; and 2j. while using diverse tricks to maintain a high entropy for the embeddings, preventing collapse constant representations.
To extend those preceding works, we introduce a second latent space to learn our equivariance first define second projection head d'r parameterized by o' whose objective is to project representations in our latent space. Using this projection head we note x == Ig. (h.) and z == g'r (fz)), ), the embeddings of the views v, and v in this latent space we introduce. Moreover, the way we define equivariance in Eq 3 requires us to produce the embedding of the non-augmented image æ, thus note the representation h. fo(x)), which used to create the embedding z/= == (ho) for the given image æ. Next, as mentioned in Sec 2. to learn an equivariant latent space one needs to determine transformation u, for any given T, this can be done either by fixing or by learning it. In this work, learn the transformation uq- To this end, we define ug a projection parameterized by the learnable parameters vú, referenced later as the equivariance predictor (implementation details about how is encoded and influences us are given below in Sec. 2.3 The goal of this predictor is to produce & from a given z and t (resp. d for z' and t'). One can see & as an alternative way to obtain z using the equivariance property defined by Eq. Instead of computing the embedding of the augmented view v; t(x), we apply via us on the embedding z of the original image
task. For this purpose,
Therefore, to match this equivariance principle, we need to train d'r and us so that applying the transformation via predictor the latent space (ž)) is similar to applying the transformation the input space and then computing the embedding (z/). For this purpose, we denote (z), 2)) as positive pair (resp. (z/, (z),, )), and design our equivariance task so that our model learns to maximize the similarity between the positive pairs. Yet, one issue with this formulation is that it allows collapsed solutions, e.g. every z' being a constant. To avoid such simple solutions, we consider negative pairs (as in Chen et al.] (2020a); He et al. (2020) repulse embedding from other embedding coming from views of different images. We use the Normalized Temperature-scaled cross entropy (NT- Xent) loss learn from these positive and negative pairs, thus defining our equivariance loss for the positive pair of the invariance loss (i,j) as Eq.
exp(sim(z/, 2/) Σ Dk=1 1 (ogtnk exp (2/1 2(//7')
EquiMMdd log 4.
(4)
2N
where 7' is a temperature parameter, sim(a, b) is the cosine similarity defined as a T 1/ ([]])]]]|]|)
and I [[ is the indicator function evaluated to (0 otherwise) when k + and k: + j.
This way, we exclude from negative pairs the views of the same image, related to index and j. that considered positive pairs in the invariance methods. While we could consider these pairs as negative and still be following 3, we found that not using them as negative nor as positive leads to slightly better results. One hypothesis is that repulsing views that can be very close in the pixel
4

---

Published as a conference paper at ICLR 2023
space (e.g. if the sampled augmentations modify weakly the original image) could induce training instability. One can notice that by. and uh are learned simultaneously, thus they can influence each other during the training phase. We finally define the total loss of the model as:
== Linvrriianeee + EquisMad
with Mod being the loss Eq. 4 applied all pairs, both (i,j) and (j, of batch and L Invariance being the loss of the invariance baseline. À is hyperparameter that ponders the equiv- ariant term of the loss.
2.3 IMPLEMENTATION DETAILS
We tested our module as a complement of 3 different baselines. The first one is SimCLR (Chen al. 2020a) as represents a contrastive approach to instance discrimination and performs well on CIFAR. The second one is BYOL (Grill et al.) 2020), which offers a different kind of architec- ture (as it bootstrapping approach rather than a contrastive one) while having the highest top-1 accuracy with linear evaluation on ImageNet using ResNer50 backbone in a self-supervised fash- ion. We also tested Barlow Twins (Zbontar et al.) 2021) as it is not exactly a contrastive approach nor bootstrapping one to illustrate the generality of our approach, yet limited to CIFARIO due to computational limitation. Here are the details of each part of the architecture, including the baseline ones and our equivariance module:
Encoder: we follow existing works and use a convolutional neural network for the encoder Se. more specifically deep residual architectures from He et al. (2016).
Invariance projection head: for the projection head Jo (and potential predictor as in BYOL Grill et al. (2020),, we used the same experimental setups as the original papers, except for SimCLR where used 3 layers projection head as in Chen & He (02TT)
Equivariance projection head: the setup of our projection head d' is 3 layers Multi- Layer Perceptron (MLP), where each Fully-Connected (FC) layer is followed by a Batch Normalization (BN) and ReLU activation, except the last layer which is only followed by BN and no ReLU. Hidden layers have 2048 neurons each.
Equivariant predictor: the predictor uy is FC followed by BN. Its input is the concate- nation of a representation of and the input embedding z/-- More precisely is encoded by numerical learned representation of the parameters that fully define it. More precisely, we reduce the augmentation to a vector composed binary values related the use of trans- formations (for transformations applied with certain probability) and numerical values corresponding to some parameters (of the parameterized transformations). This vector is projected in 128d latent space with perceptron learned jointly with the rest of the model, see Sec A for details and examples of this encoding. This way, the input dimension of the predictor is the dimension of the latent space plus the dimension of the encoding of augmentations, while the output dimension is the same as the latent space.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETTINGS
In our experimentations, we tested our method on ImageNet (IN) (Deng et al, 2009) and CI- FAR10 (Krizhevsky et al., 2009). As mentioned hefore, we have used our module as a complement to SimCLR, BYOL, and Barlow Twins, 3 sfate-of-the-art invariance methods with quite different ideas, to test the genericity of our module. For these methods, we used the same experimental setup as the original papers. As in previous works, while training on ImageNet we used ResNet50 with- out the last FC, but while training on CIFARIO we used the CIFAR variant of ResNet18 (He et al. 2016). For all our experimentations and BN parameters were excluded from both weight decay and LARS adaptation as Grill et al. 2020)). Finally, we have fixed À to it led the best and more stable results.
used the LARS You et al. 2017) optimizer, yet, biases
5

---

Published as a conference paper at ICLR 2023
3.1.1 SIMCLR
The model is trained for 800 epochs with 10 warm-up epochs and cosine decay learning rate schedule. We have used a batch size of 4096 for ImageNet and 512 for CIFARIO, while using an initial learning rate of 2.4 for ImageNet (where we use 4.8 for SimCLR without EquiMod, as in the original paper) and 4.0 for CIFARIO. For the optimizer, we fix the momentum to 0.9 and the weight decay to le-6. Both the invariant and equivariant latent space dimensions have been set to 128. Finally, we use r' 0.2 for our loss, but 0.2 ImageNet 0.5 with CIFARIO for the loss of SimCLR (we refer the reader to the original paper for more information about the loss of SimCLR (Chen al. 2020a)).
3.1.2 BYOL
The model learned for 1000 epoch (800 on CIFARIO) with 10 warm-up epochs and cosine decay learning rate schedule. The batch size used is 4096 for ImageNet and 512 for CIFAR1O. We have been using an initial learning rate of 4.8 for ImageNet (where we use 3.2 for BYOL without EquiMod, as in the original paper) while using 2.0 for CIFAR10. Momentum of the optimizer is set to 0.9 and weight decay to 1.5e on ImageNet, but 1e on CIFARIO. The invariant space has 256 dimensions while we keep our equivariant latent space to 128. Last, we use r' = 0.2 for our loss, and Thase = 0.996 for the momentum encoder of BYOL with a cosine schedule as in the original paper (once again, we refer the reader to the paper for more details (Grill et al. (020
3.1.3 BARLOW TWINS
We tested our method with Barlow Twins only on CIFAR1O with the following setup: 800 epochs with 10 warm-up epochs and a cosine decay learning rate schedule, a batch size of 512, an initial learning rate of 1.2, momentum of 0.9 and weight decay of 1.5e
Both the invariant and equiv- ariant latent space has 128 dimensions, while we use 7' = 0.2 for our loss and ABrllow Twins = 0.005 for the loss of Barlow Twins (as in the original paper (Grill et al. 2020)).
3.2 RESULTS
3.2.1 LINEAR EVALUATION
After training on either ImageNet or CIFARIO, IO, we evaluate the quality of the learned representation with the linear evaluation which is usual in the literature. To this end, we train linear classifier on top of the frozen representation, using the Stochastic Gradient Descent (SGD) for 90 epochs, which is sufficient for convergence, with batch size of 256, a Nesterov momentum of 0.9, no weight decay, an initial learning rate of 0.2 and a cosine decay learning rate schedule.
Results of this linear evaluation are presented in Table while some additional results are present in supplementary material Sec. Across all baselines and datasets tested, EquiMod increases the performances of all the baselines used, except BYO while trained on 1000 epochs. Still, is worth noting that under 100 and 300 epochs training (Sec. A.3). EquiMod improves the performances of BYOL. Overall, this supports the genericity our approach, and moreover, confirms our idea that adding equivariance task helps to extract more pertinent information than just an invariance task and improves representations. On CIFARIO IO, we achieve the second-best performance after E-SSL, yet, contrary to us, they tested their model on an improved hyperparameter setting of SimCLR.
3.2.2 EQUIVARIANCE MEASUREMENT
The way our model is formulated could lead the learning of invariance rather than equivariance. Indeed, learning an invariant latent space as well as the function identity for uy is an admissible solution. Therefore, verify that our model is really learning equivariance, we define two metrics of equivariance Eq 5 and Eq 6 The first one evaluates the absolute displacement toward 2f caused by the predictor ugg. One can see this as how much applying the augmentation t to z/, in the latent space via ue makes the resulting embedding é/ more similar to z/. This way, if our model is learning invariance, we should observe an absolute displacement of O, as up would be the identity. On the
We also performed 100 and 300 epochs training, see Sec. A.3

---

Published as a conference paper at ICLR 2023
ImageNet
CIFARIO
Method
Top-1
Top-5
Top-1
Top-5
PIRL (Misra Maaten
2020
63.6
E-SimCLR (Dangovski et al.
2021
68.3
94.1
E-SimSiam Dangovski et al.
2021
68.6
94.2
SimCLR (Chen et al.
2020a)
69.3
89.0
SimSiam Chen He
2021
71.3
SWAV (w/o multi-crop) Caron et al.
2020
71.8
Barlow Twins (Zbontar et al.
2021
73.2
91.0
VICReg (Bardes al. 2021
73.2
91.1
BYOL (Grill al. 2020)
74.3
91.6
SimCLR*
71.57
90.48
90.96
99.73
SimCLR* + EquiMod
72.30
90.84
92.79
99.78
BYOL*
74.03
91.51
90.44
99.62
BYOL* + EquiMod
73.22
91.26
91.57
99.71
Barlow Twins*
86.94
99.61
Barlow Twins* EquiMod
88.87
99.71
Table Linear Evaluation; top-1 and top-5 accuracies (in %) under linear evaluation on ImageNet and CIFARIO (symbols denote our re-implementations, and denote only 100 epochs training).
contrary, if it learning equivariance, we should observe a positive value, meaning that the us plays its role in predicting the displacement in the embedding space caused by the augmentations. negative displacement means displacement in the opposite direction, in other words, means that the predictor performs worse than the identity function. Furthermore, small displacement does not mean poor equivariance, for instance, if x is very similar to z/. the room for displacement is already very small. This is why we also introduce the second metrie, which evaluates the relative displacement toward z caused by ug. It reflects by which factor applying the augmentation to 2 in the latent space via uy. makes the resulting embedding Σ less dissimilar z/. Thus, if the model is learning invariance, we should see no reduction nor augmentation of the dissimilarity, thus the factor should remain at 1 while a model achieving equivariance would exhibit a positive factor.
1 sim 2/1- 2%) sim 2 2))
sim 2/4 + sim 2/1
(5)
(6)
Fig. 3ishows the absolute equivariance measured for each augmentation. Note that this is performed on model already trained with the usual augmentation policy containing all the augmentations. If an augmentation induces a large displacement, it means the embedding is highly sensitive to the given augmentation. What we can see from Fig. is that regardless of the dataset used, the model achieves poor sensitivity to horizontal flip and grayscale. However, on ImageNet, we observe high sensitivity to color jitter as well as medium sensitivity crop and gaussian blur. On CIFAR10 we observe a strong sensitivity to crop and a medium sensitivity to color jitter. Therefore, we can conclude that our model truly learns an equivariance structure, and that the learned equivariance more sensitive to some augmentation such as crop color jitter.
3.2.3 INFLUENCE OF THE ARCHITECTURES
We study how architectural variations can influence model. More precisely, explore the impact of the architecture of the S'y- uh as well as the learned projection of t mentioned in Scc. Scc. A.1 To this end, we train models for each architectural variation on CIFARIO, and report the top- accuracies under linear evaluation, the results are reported in Table What we observe in Table 2a that the projection head of the equivariant latent space benefits from having more layers, yet this effect seems to plateau at point. These results are in line with existing works (Chen al. 2020a). While testing various architectures for the equivariant predictor Table we note only small performance variations, indicating that up is robust to architectural changes. Finally, looking
7

---

Published as a conference paper at ICLR 2023
Absolute equivariance
Relative equivariance
0.2
ImageNet I CIFARIO
15
ImageNet CIFARIO
10
0.1
Color jitter
Grayscale Blur
H-flip Color jitter
Grayscale Blur
Crop
H-flip
Crop
Figure 3: Absolute equivariance measure for each augmentation (the dashed line represents invariance).
Figure 4: Relative equivariance measure for each augmentation (the dashed line represents invariance).
Table [c we observe that removing the projection of only leads to small drop performance. On the contrary, complex architectures (two last lines) lead to bigger drop in accuracy. Furthermore, while testing different output dimensions (lines the output and for the equivariant latent space led to the highest results. Some more analysis on hyperparameter variations of our model, such as X, batch size, can be found in Sec. A.2
4), we that using the same dimension for
Layers in the projection of Top-1
None
92.50
Layers in D'r
Top-1
Layers in Ug
Top-1
I (O: 16-d)
92.57
None
88.46
1
92.79
1 (O: 128-d)
92.79
1
91.58
(H: 16-d)
92.67
I (O: 2048-d)
92.50
2
92.58
(H: 128-d)
92.59
2 (H: 16-d; O: 128-d)
92.47
3
92.79
(H: 2048-d)
92.70
2 (H: 128-d; O: 128-d)
92.13
(a) Equivariance projec- tion head
(b) Equivariant predictor
2 (H: 2048-d; O: 128-d)
92.05
(c) Augmentation projector
Table Top-1 accuracies (in %) under linear evaluation on CIFARIO for some architectural varia- tions of our module. H stands for hidden layer, O for output layer, f denotes default setup.
4 RELATED WORK
Most of the recent successful methods of SSL of visual representation lear latent space where embeddings augmentations from the same image are learned to be similar. Yet, such instance discrimination tasks admit simple constant solutions. To avoid such collapse, recent methods im- plement diverse tricks to maintain a high entropy for the embeddings. Grill et al. (2020) rely on momentum encoder as well as an architectural asymmetry, Chen He (2021) depend on stop gradient operation, Zbontar et al. (2021) rely on redundancy reduction loss term, Bardes et al. (20TT) rely on variance term well as covariance term in the loss, Chen al. (2020a) use negative pairs repulsing sample in batch. In this work, our task also admits collapse solutions, thus we make use the same negative pairs as Chen al. (2020a) avoid such collapse. The most recent methods are creating pairs of augmentations However, our addition does not rely on pairs of augmentations, and only needs a source image and augmentation. This is similar to Misra & Maaten (2020) which requires to have source image and an augmentation, however, they use these pairs to learn an invariance task while we use them to leam equivariance task.
maximize the similarity between those pairs.
Our approach is part of a line of recent works, which try to perform additional tasks of sensitivity to augmentation while learning invariance task. This is the case of E-SSL (Dangovski al. (021) which simultaneously learns to predict rotations applied to the input image while learning an invar- ance pretext task. This way, their model learns to be sensitive to the rotation transformation, usually
8

---

Published as a conference paper at ICLR 2023
not used for invariance. Where this can be considered as a form of equivariance (a rotation in input space produces a predictable displacement in the prediction space) this is far from the equivariance we explore in this paper. Indeed, E-SSL sensitivity task can be seen learning an instance- instance-invariant invariant pretext task, where for any given input, the output represents only the augmentation (rotation) used. Here, explore equivariance sensitivity both to images and to augmentations. Moreover, we only consider sensitivity to the augmentations used for invariance. In LooC (Xiao et al. (020), authors propose use as many different projection heads as there are augmentations and learn each of these projection heads to be invariant to all but one augmentation. This way the projection heads can implicitly learn to be sensitive to an augmentation. Still, they do not control how this sensitivity occurs, where we explicitly define an equivariance structure for the augmentations-rellted informa- tion. Note that work has tried to tackle the trade-off from the other side, by trying to reduce the shortcut learning occurring, instead of adding sensitivity to augmentations. |Robinson et al. (202T) shows that shortcut learning occurring in invariant SSL is partly due to the formulation of the loss function and proposes method to reduce shortcut learning in contrastive learning.
Some other works have also successfully used equivariance with representation learning. For in- stance, Dayaraman & Grauman (2015) uses the same definition of equivariance as and success- fully learns an equivariant latent space tied to ego-motion. Still, their objective is to learn embodied representations well as using the learned equivariant space, in comparison we only use equiv- ariance as a pretext task to learn representations. Moreover, we do not learn equivariance on the representations, but rather on a non-linear projection of the representations, Lene & Vedaldi (2015) learns an equivariant predictor on top of representations to measure their equivariance, however, to learn that equivariance, they require the use of strong regularizations.
5 CONCLUSION AND PERSPECTIVES
Recent successful methods for self- celf-supervised visual representation rely on learning a pretext task of invariance to augmentations. This encourages the learned embeddings to discard information related to transformations. However, this does not fully consider the underlying dilemma that occurs in the choice of the augmentations: strong modifications of images are required to remove some possible shorteut solutions, while information manipulated by the augmentation could be useful to some downstream tasks. In this paper, we have introduced EquiMod, a generic equivariance module that can complement existing invariance approaches. The goal of our module let the network learn an appropriate form of sensitivity to augmentations. It is done through equivariance via a module that predicts the displacement in the embedding space caused by the augmentations. Our method is part of research trend that performs sensitivity to augmentations. Nonetheless, compared to other existing works, we perform sensitivity to augmentations also used for invariance, therefore reducing the trade-off, while defining a structure in our latent space via equivariance.
Testing EquiMod across multiple invariance baseline models and datasets almost always showed improvement under linear evaluation. It indicates that model can capture more pertinent infor- mation than with just an invariance task. In addition, we observed a strong robustness of our model under architectural variations, which is non-negligible advantage as training such methods is com- putationally expensive, and so does the hyperparameters exploration. When exploring the sensitivity to the various augmentations, we noticed that the latent space effectively learns to be equivariant to almost all augmentations, showing that it captures most of the augmentations-relltc. information.
For future work, we plan on testing our module on more baseline models or even as a standalone. As EquiMod almost always improved the results in our tests, it suggests that EquiMod could improve performances on many more baselines and datasets. Then, since E-SSL adds sensitivity rotation, yet still does not consider sensitivity to augmentations used for invariance, it would be interesting to study if combining EquiMod and E-SSIL can improve even further the performances. Another research axis is to perform an in-depth study of the generalization and robustness capacity of our model. To this end, we want to explore its capacity for transferability (fine-tuning) and few-shot learning, both usual object recognition datasets, but also more challenging datasets containing flowers and birds as in Xiao et al. (2020). Since the trade-off theoretically limits the generalization on the learned representation, and since reduce the effect of the trade-off, we hope that EquiMod may show advanced generalization and robustness properties. On a distant horizon, the equivariant structure learned by our latent space may open some interesting perspectives related world model.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work performed using HPC resources from GENCI-IDRIS (Grant 2021-AD01 1013160 and 0022-A013101380 and GPUs donated by the NVIDIA Corporation. We gratefully acknowl- edge this support.
REFERENCES
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invvrriace-cooo weg.rreooooaaannnnneeggaaaaiiiao for self-supervised learning. arXiv preprint 1iiv:2205500906 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09880 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Che- ung, Pulkit Agrawal, and Marin Soljačic. Equivariant Contrastive Learning. arXiv preprint arXiv:2111.00899 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scalee scale hi- erarchical image database. In 2009 IEEE conference computer vision and patlern recognition, pp. 248-255. leee, 2009.
Robert Geirhos, förn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11) (11):665-673) 2020. Publisher: Nature Publishing Group
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, and Mohammad Ghesh- laghi Azar. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006.00733. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738. 2020.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In Proceedings of the IEEF/CVF International Confer- ence on Computer Vision, pp. 9598-9608, 2021.
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1413-1421, 2015.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint 1rXiv:2110.09348, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
10

---

Published as a conference paper at ICLR 2023
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999,201.
Ran Liu. Understand and Improve Contrastive Learning Methods for Visual Representation: A
Review, arXiv preprint arXiv:2106.03233 2021.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems, 34:4974-4986, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.0500 2020.
Yang You, Igor Gitman. and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint Xivi 7008 170..07888.888. 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230, 2021.
A
APPENDIX
A. ENCODING OF THE AUGMENTATIONS
We use the classical augmentations of the literature, which depend on the dataset and model used, applied in the given order:
Resized Crop: crop a subregion of the image;
Horizontal flip: flip the image with given probability;
Color jitter: jitter the image on different aspects with random order (brightness, satura- tion, contrast, and huc) and with a given probability;
Gcalescale: gray-scale the image with given probability;
Gaussian blur (not used with CIFAR10 except in BYOL): blur the image using a sampled a and with given probability:
Solarize (applied only with BYOL): solarize the image with given probability.
We refer the reader to the original papers (Chen et al. 2020a Grill et al. 2020 Zbontar et al. 2021 know how the different methods parameterize these augmentations (e.g. values of the probability, or intervals of values sampled, as factors in color jitter).
To encode these augmentations, we represent them by a numerical vector where some of the compo- nents are binary values related to the use of augmentations (for those applied with some probability) and some others are numerical values corresponding to some parameters (of the parameterized trans- formations). We only consider the corresponding augmentations w.r.t. the tested dataset and model. For each of these considered augmentations except crop, we define an element valued when the augmentation is performed and valued at 0 otherwise (since each augmentation, but the crop, applied with a given probability it may be applied or not). Then, some augmentations require additional elements. To this end, we define elements to represent these parameters using the follow- ing direct ways (note that when parametrized augmentation is not applied due to its probability of application, its numerical components are set some predefined default values):
Resized Crop (4 elements): x and y coordinates of the top-left pixel of the crop as well as width and height of the crop.
11

---

Published as a conference paper at ICLR 2023
Color Jitter (8 elements): the jitter factors for brightness, saturation, contrast, and hue (1,1,1,0 is the default encoding color jitter is not applied), as well as their order of application. More precisely, to encode the order of modification, mapping {0 brightness, with "1,3, 0" would mean that contrast jitter is first applied, then hue, contrast, and finally brightness (0, 1, 2, 3 is the default encoding if color jitter is not applied).
the following contrast, 2 saturation, 3 : hue). For instance an encoding
Gaussian Blur (1 element): the value of sigma used (0 if blur is not applied).
At this point, we have numerical vector that represents which augmentations are applied or not and what are their parameters if any, see the following Sec. A. and Sec. 1.1..AA.122foo some examples. We then normalize this vector component-we computed over many examples, and we use perceptron to project the constructed vector into 128d latent space. This perceptron is learned jointly with the rest of the model.
using experimental mean and standard deviation
A.1.1 EXAMPLE 1
Here an example of one transformation applied during the learning of BYOL on ImageNet. Let's consider the randomly generated transformation composed of the following augmentations:
Crop at coordinates x, y=(12, 9) with width, height of (120, 96);
Probabilistic horizontal flip not triggered;
Probabilistic color jitter triggered with factors and order of: hue -0.09, contrast 1, saturation 0.84, brightness 1.13;
Probabilistic gray-scale triggered;
Probabilistic blur not triggered;
Probabilistic solarize not triggered;
According to A. for the binary part representing the performed augmentations we have [0, 1, 1, O, 0] for [No H-Flip, Yes Color jitter, Yes Gray-scale, No Blur, No Solarize] (one per augmentation, except crop which is always performed), and for the parameterized transformations: [12, 9, 120, 96, 1.13, 1, 0.84, -0.09, 3, 1, 2, O, 0] for [Crop X, Crop Y, Crop Width, Crop Height, Brightness Factor, Contrast Factor, Saturation Factor, Hue Factor, Index of the First Color Modification Applied, Index of the Second Color Modification Applied, Index of the Third Color Modification Applied, Index of the Fourth Color Modification Applied, Default value for sigma (as blur is not triggered))
Finally, this gives us the 18d vector [0, 1, 1, O, 0, 12, 9, 120, 96, 1.13, 1, 0.84, -0.09, 3, 1, 2, 0, 0), which is then normalized and given a perceptron project it to 128d vector.
A.1.2 EXAMPLE 2
And here is another example this time with SimCLR on CIFAR10 (which uses different augmen- tation policy, thus solarization and blur are not considered).
Let's consider the randomly generated transformation composed of the following augmentations:
Crop at coordinates x.y=(1, 2) with width,height of (24, 27);
Probabilistic horizontal flip triggered;
Probabilistic color jitter not triggered;
Probabilistic gray-scale not triggered;
For the binary part representing the performed augmentations, we have [1, 0, 0] for [Yes H-Flip. No Color jitter, No Gray-scale]. And for the parametrized transformations [1, 2, 24, 27, 1, 1, 1, 0, 0, 1, 3] for [Crop X, Crop Y, Crop Width, Crop Height, Brightness Factor (Default), Contrast Factor (Default). Saturation Factor (Default), Hue Factor (Default), Index the First Color Modification (Default), Index of the Second Color Modification (Default), Index of the Third Color Modification (Default). Index of the Fourth Color Modification (Default)). Note the default values for all the parameters of the color jitter which is not triggered.
12

---

Published as a conference paper at ICLR 2023
This gives us the 15d vector [1, 0, 0, 1, 24, 27, 1, L, 1, 0, 1, which is then normalized and given to a perceptron to project to a 128d vector.
A.2 INFLUENCE OF HYPERPARAMETER, (A, AND BATCHSIZE)
In this section, similarly to Sec 3.2.3 study how variations of minor hyperparameters can influ- ence our model. To that purpose, we train models on CIFARIO for each hyperparameter modification and present the top-1 accuracy under linear evaluation.
We first inspect the influence of the A. the weighting factor between our equivariance loss and the invariance baseline loss. One can see Table 3a that when is small (< 1) there drop in performance. As À can be seen as weighting the importance between the equivariance and the invariance terms the loss, this confirms that our model learns better features when our equivariance addition is considered with at least the same importance as the invariance task. On the opposite, interestingly, where is set to high values such as or 10, we do not observe a clear modification of the performance. This tends to indicate that there is degradation of the representation when the equivariance prioritized.
Then study the temperature hyperparameter of the NT-Xent loss that Similarly to what is reported in Chen et al. (2020a), we find Table 3b that the optimal values to be around 0.2 and 0.5.
learn equivariance.
Finally, we explore the impact the batch size on the learned representations. This hy hyperparameter directly determines the number of negative pairs, therefore highly influences the learning dynamic. We observe Table 3 decrease in performance where the batch size is too small (< 256) or too big
(2 1024). Once again, these findings are in line with the literature (Chen al. 2020a).
A Factor
Top-1
0
90.96
Temperature 7'
Top-1
Batch size
Top-1
0.1
92.07
0.05
92.13
64
92.23
0.2
92.31
0.1
92.13
128
92.24
0.5
92.37
0.2 f
92.79
256
92.38
1
92.79
0.5
92.31
512 f
92.79
2
92.33
92.14
1024
92.23
5 92.81
10
92.66
(b) Temperature of the NT- Xent used our equivariance loss
(c) Batch size
(a) Weighting factor between equivariance and invariance losses
Table 3: Top-1 accuracies (in %) under linear evaluation on CIFAR10 for some hyperparameter variations of module. denotes default setup.
A.3 ADDITIONAL RESULTS
The shows the impact of the number of training epochs on the results of the linear evaluation of BYO with and without EquiMod.
13

---

Published as a conference paper at ICLR 2023
ImageNet
CIFARIO
Method
Top-1
Top-5
Top-1
Top-5
BYOL* (100 epochs)
62.09
84.01
BYOL* EquiMod (100 epochs)
65.55
86.74
BYOL* (300 epochs)
71.34
90.35
BYOL* + EquiMod (300 epochs)
72.03
90.77
BYOL* (1000 epochs)
74.03
91.51
90.44
99.62
BYOL* EqqiMod (1000 epochs)
73.22
91.26
91.57
99.71
Table 4: Linear Evaluation; top-1 and top-5 accuracies (in %) under linear evaluation on ImageNet and CIFARIO (symbols denote our re-implementationss)
14


Reviews:
[{"review_id": "xsx63usfXt", "reviewer": "Reviewer_9hFv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presented a new module to improve self-supervised contrastive visual representation learning. Specifically, the proposed module focused on equivariance in the leaned latent space. Experimental analysis on two public datasets showed that when applying the proposed module to existing methods (SimCLR, BYOL and Barlow Twins) the performance was improved. The main contribution of this paper is the proposed equivariance module.", "summary_of_the_review": "The main idea of this work about the equivariance in latent space is interesting and may have the potential to explore and contribute to the community. The shown experiments also suggested the effectiveness of the proposed method to some extent. But some technical design was not clearly presented without sufficient evidence to back them up. There are also several unclear claims and statements. The experiments are also insufficient to validate the claims. As a result, I would suggest the authors revise their paper accordingly for a future submission.", "strength_and_weaknesses": "**Strengths**\n\n\\+ The idea of equivariance in the feature space to help self-supervised representation learning is interesting.\n\n\\+ The proposed method was shown to be effective on linear evaluation (Table 1), by showing a performance gain when added to some prior works.\n\n\n**Weaknesses**\n\n\\- The statement \"These methods rely on maximizing the similarity between...\" in the Abstract is inaccurate. Not all self-supervised visual representation learning methods are based on such a contrastive learning scheme. Instead, most of the early methods are based on pretext tasks. Suggest revising the corresponding claims and also the title.\n\n\\- The main idea of this work is based on the assumption of the self-supervised visual learning method being contrastive-based ones. Whereas there are quite a few non-contrastive learning approaches. As a result, the main contributions of this work may be a bit limited.\n\n\\- It is unclear how the defined u_\\psi was guaranteed to represent the transformation t. From the description, t was encoded by another set of layers, which are also learnable, making the equivariance here (esp. the \"equivariant predictor\") a bit misleading.\nThe encoding of the transformations (Sec. 2.3.2) is rather confusing. In the beginning, it looks like the authors used one-hot encoding, but later some of the augmentations directly use the coordinates or real values (i.e. 1, 2, 3). The generalization for the augmentation set, as a result, is another issue. What if new augmentations were included? How would they be encoded?\n\n\\- It is unclear why the final model still needs the conventional invariance loss (i.e. the first term in the final loss function), if the proposed new equivariance-based loss is as claimed to be effective. Jointly optimizing both invariance and equivariance also seems to be confusing. How does the model actually learn in this case?\n\n\\- It is unclear why the Barlow Twins was only evaluated on the small-scale dataset CIFAR10.\n\n\\- The quality of the learned representation was only evaluated on the linear evaluation setting, which is a bit insufficient to get a clear conclusion. There are quite a few other downstream tasks that could be used as reported in the literature (e.g. fine-tuning, detection, and segmentation to name a few).\n\n\\- The proposed method was motivated by the case where \"augmentation information is useful\" as claimed by the authors, and example cases are flowers and birds (as claimed in the Introduction). But this was not validated in the experiment. There are some fine-grained datasets for such categories (e.g. the Oxford 102 flowers dataset and the Caltech-UCSD Birds-200 dataset) that should have been used to validate the claims.\n\n\\- It is unclear why not use an objective similar to Eq. (5) or (6) to constrain the equivariance model training, but instead only maximize the similarity between z_i' and \\hat{z}_i'?\n\n\\- Would an absolute value makes more sense for Eq (5)? Otherwise, what does a negative value mean, as the second case \"H-flip\" shown in Fig. 3?\n\n\\- Line below Eq. (5), \"Fig. 4\" should be \"Fig. 3\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F0O9Xc2ulq", "reviewer": "Reviewer_A3J1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an equivariance regularizer as a modification to the usual invariance-inducing self-supervised losses. This is an interesting approach to enabling equivariance as there is no need to have a special architecture as prior work. The authors are able to train multiple self-supervised losses on a standard ResNet-50 and show good linear probe improvements over invariant baselines. ", "summary_of_the_review": "Given the overall novelty and clean/simple idea which seems to work well, I recommend acceptance.", "strength_and_weaknesses": "Strengths:\n\n1. A novel approach to enforcing equivariance is presented. The regularizer and modification to regular architectures is simple and effective.\n\n2. The performance of the technique is shown on multiple losses and \n\nWeaknesses/Questions:\n\n1. In Eq. (4) why is j not shown on the numerator? \n\n2. Is the only difference between the equivariance projection head and equivariant predictor the fact that the augmentation parameters are fed into the predictor? Otherwise equivariance projection head and predictor could have just been merged? Is equivariance projection head used elsewhere?\n\n3. In Figure 3 and 4, what is the value of these measures for a regularly trained (invariant) model? This will tell us the benefit of adding the regularizer. \n\n4. What is the reason that equivariance is stronger for color compared to other augmentations for ImageNet? Any insight into this behavior?\n\n5. Did you try any finetuning experiments?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1NGo_1REYi1", "reviewer": "Reviewer_URYd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Proposes a modification of the data-augmentation approach to self-supervised learning where the parameters of the transformation used in data-augmentation can be used by the network to predict the latent variables of the un-augmented input.  This is advantageous when there exists a set of input features (such as color histogram) which would potentially cause the network to converge to bad local minima (shortcut solutions), yet could still be useful for classification.  Previous approaches would use data augmentations to enforce invariance to those features, which avoids the bad local minima but at the cost of losing the ability to use those features for classification.  Equimod therefore improves performance by combining a loss that encourages invariance to data augmentations and a loss that encourages the learning of a representation that is equivariant to data augmentations.  This is hypothesized to allow the network to avoid bad local minima without becoming totally invariant to useful features that are perturbed by data augmentations.  Experimental results show improved performance of SSL methods by modifying them to include the Equimod sub-network for predicting the effect of data augmentations on the latent space.", "summary_of_the_review": "The paper presents a very interesting method for the field of SSL, and after rebuttal, provides more clarity and insight into the proposed method.  While a better evaluation (e.g. on problems such as flowers or trees) might allow the work to have a greater impact, I  think in its present form it will be useful for SSL researchers.", "strength_and_weaknesses": "Strengths\n+ Analyzes the key ideas behind data augmentation and identifies a key shortcoming of data augmentation.\n+ Proposes an innovative, elegant, and general solution to overcome the identified problem of data augmentation.\n+ Concept is carefully explained with aid of diagrams and appropriate mathematical notation.\n+ Evaluation shows meaningful improvements achieved using method.\n+ Evaluation includes an experiment that shows that Equimod indeed learns equivariances, not just invariances.\n+ Evaluation includes ablation study of architectures.\n\nWeaknesses\n- Evaluations could be made stronger.  Training for only 90 or 100 iterations is too short; methods should be training until convergence for a fair comparison with and without Equimod. [addressed by rebuttal]\n- Intuition for Equimod loss (eq 4) not fully explained.  How does the loss differ in the case of a positive pair vs a negative pair?  And how are indices i and j sampled (e..g uniformly at random)? [addressed by rebuttal]\n- Not clear how to tune the scaling of the Equimod loss.  The paper could be improved if the choice of lambda could be explored.  How is lambda set for the experiments?  Could an ablation on lambda (setting to different values and seeing the performance) be conducted? [addressed by rebuttal]\n- Relatedly, the theory for how well Equimod might work to avoid \"shortcut solutions\" and optimally trade off is totally heuristic at the moment.  We see that Equimod works, but the paper doesn't give us a clear idea of the details of how that is possible theoretically.  Perhaps a theoretical analysis of a toy model would help shed some light on the success of the method.\n- What is the computational cost of Equimod? [addressed by rebuttal]\n- Will the code be shared? [addressed by rebuttal]", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster