Equivariant Self-Supervised Learning: Encouraging Equivariance in Representations
Authors: Rumen Dangovski
Li Jing
Charlotte Loh
Seungwook Han
Akash Srivastava
Brian Cheung
Pulkit Agrawal
Marin Soljacic
Venue: ICLR 2022 Poster (2022)
Keywords: self-supervised learning
contrastive learning
photonics science

Abstract: In state-of-the-art self-supervised learning (SSL) pre-training produces semantically good representations by encouraging them to be invariant under meaningful transformations prescribed from human knowledge. In fact, the property of invariance is a trivial instance of a broader class called equivariance, which can be intuitively understood as the property that representations transform according to the way the inputs transform. Here, we show that rather than using only invariance, pre-training that encourages non-trivial equivariance to some transformations, while maintaining invariance to other transformations, can be used to improve the semantic quality of representations. Specifically, we extend popular SSL methods to a more general framework which we name Equivariant Self-Supervised Learning (E-SSL). In E-SSL, a simple additional pre-training objective encourages equivariance by predicting the transformations applied to the input. We demonstrate E-SSL’s effectiveness empirically on several popular computer vision benchmarks, e.g. improving SimCLR to 72.5% linear probe accuracy on ImageNet. Furthermore, we demonstrate usefulness of E-SSL for applications beyond computer vision; in particular, we show its utility on regression problems in photonics science. Our code, datasets and pre-trained models are available at https://github.com/rdangovs/essl to aid further research in E-SSL.

Published as a conference paper at ICLR 2022
EQUIVARIANT CONTRASTIVE LEARNING
Rumen Dangovski MIT EECS rumenrd@mit.co edu
Li Jing
Charlotte Loh MIT EECS cloh@mit. edu
Facebook Al Research 1 1ing fb con
Seungwook Han
Akash Srivastava
Brian Cheung
MIT-IBM Watson Al Lab sh32648cclumbia edu
MIT-IBM Watson AI Lab akashsri@mit.. edu
MIT CSAIL & BCS cheungbemit.co edu
Pulkit Agrawal MIT CSAIL pulkitagémit.edt edu
Marin Soljačié
MIT Physics sol soljacicenin ceaciiemmitee.. edu
ABSTRACT
In state-of-the-aar self-supe learning (SSL) pre-training produces semanti- cally good representations by encouraging them to invariant under meaningful transformations prescribed from human knowledge. In fact, the property of in- variance is a trivial instance of broader class called equivariance, which can be intuitively understood the property that representations transform accord- ing to the way the inputs transform. Here, we show that rather than using only invariance, pre-training that encourages non-trivial equivariance some transfor- mations, while maintaining invariance to other transformations, can be used to improve the semantic quality of representations. Specifically, we extend popu- lar SSL methods to more general framework which we name Equivariant Self- Supervised Learning (E-SSL). In E-SSI, a simple additional pre-training objec- tive encourages equivariance by predicting the transformations applied to the in- put. We demonstrate E-SSL's effectiveness empirically on several popular com- puter vision benchmarks, e.g. improving SimCLR 72.5% linear probe accuracy on ImageNet. Furthermore, we demonstrate usefulness of E-SSL for applications beyond computer vision; in particular, show its utility regression problems in photonics science. Our code, datasets and pre-trained models are available at http::///ggithuu...mm//reeeeee
to aid further research in E-SSL.
INTRODUCTION
Human knowledge about what makes good representation and the abundance of unlabeled data has enabled the learning of useful representations via self-supervised learning (SSL) pretext tasks. State-of-the-art SSL methods encourage the representations not contain information about the way the inputs are transformed, i.c. to be invariant to a set of manually chosen transformations. One such method is contrastive learning, which sets up binary classification problem learn invariant features. Given set of data points (say images), different transformations of the same data point constitute positive examples, whereas transformations of other data points constitute the negatives (He et al., 2020 Chen al. 2020). Beyond contrastive learning, many SSL methods also rely on learning representations by encouraging invariance (Grill et al., 2020; Chen & 2021 Caron et al. 2021 Zbontar et al. 2021 Here, we refer to such methods Invariant-SSL (I-SSL).
The natural question I-SSL is to what transformations should the representations insensi- tive (Chen al. 2020 Tian al. 2020 Xiao et al.] 2020). Chen et al.) (2020) highlighted the importance of transformations and empirically evaluated which transformations are useful for con- trastive learning (e.g., see Figure in their paper). Some transformations, such as four-fold rotations, despite preserving semantic information, were shown to be harmful for contrastive learning. This does mean that four-fold rotations are not useful for I-SSL all. In fact, predicting four-fold

---

Published as a conference paper at ICLR 2022
rotations is good proxy task for evaluating the representations produced with contrastive learn- ing (Reed et al. 2021). p). Furthermore, instead of being insensitive to rotations (invariance), training neural network to predict them. i.c. to be sensitive to four-fold rotations, results in good image representations (Gidaris et al., 2018: 2019). These results indicate that the choice of making fea- tures sensifive insensifive a particular group of transformations can have substantial effect on the performance of downstream tasks. However, the prior work in SSL has exclusively focused on being either entirely insensitive (Grill et al. 2020 Chen & He| 2021 Caron et al. 2021 Zbontar et al. 2021) or sensitive (Agrawal et al.) 2015 Doersch et al.) 2015 Zhang et al.]
& Favaro 2016; Gidaris al., 2) to set of transformations. In particular, the I-SSL literature has proposed to simply remove transformations that hurt performance when applied as invariance.
To understand how sensitivity/ insensitivity to a particular transformation affects the resulting fea- tures, we ran series of experiments summarized in Figure I We trained and tested a simple I-SSI. baseline, SimCLR (Chen et al., 2020), on CIFAR-10 using only the random resized cropping trans- formation (solid yellow line). The test accuracy is calculated as the retrieval accuracy of a k-nearest neighbors (kNN) classifier with a memory bank consisting of the representations on the training set obtained after pre-training for 800 epochs. Next, in addition being invariant to resized cropping, we additionally encouraged the model to be either sensitive (shown in pink) or insensitive (shown in blue) second transformation. We encourage insensitivity by adding the transformation the SimCLR data augmentation, and sensitivity by predicting it (see Section 4 We varied the choice of this second transformation. We found that for some transformations, such as horizontal flips and grayscale, insenstivity results in better features, but is detrimental for transformations, such as four- fold rotations, vertical flips, 2x2 jigsaws (4! = 24 classes), four-fold Gaussian blurs (4 levels of blurring) and color inversions. When we encourage sensitivity to these transformations, the trend reversed. In summary, we observe that if invariance to a particular transformation hurts feature learning, then imposing sensitivity to the same transformation may improve performance. This leads us to conjecture that instead of choosing the features to be only invariant or only sensitive as done in prior work, it may be possible to learn better features by imposing invariance to certain transfor- mations (c.g., cropping) and sensitivity to other transformations (c.g.. four-fold transformations).
The concepts of sensitivity and insensitivity are both captured by the mathematical idea of equiv- ariance (Agrawal et al., 2015 Jayaraman & Grauman 2015) Cohen & Welling 2016). Let G be a group of transformations. For any g E G let (z()) denote the function with which g transforms an input image x. For instance, if G is the group of four-fold rotations then T₂ (x) rotates the image by multiple of 7/2. Let ʃ be the encoder network that computes feature representation, (ar). I-SŠL. encourages the property of "invariance to G," which states [TT, (2)) = f(x), i.e. the output representation, f(z). does not vary with Tg. Equivariance, a generalization of invariance, is defined as, Vx (T) = Ty (x)),, where Ty is fixed transformation (i.e., without any parameters). Intuitively, equivariance encourages the feature representation to change in a well defined manner to the transformation applied to the input. Thus, invariance is trivial instance of equivariance, where f(x). While there are many possible choices for TV (Co-) hen & Welling 2016 Broostein et al.) 202 I). I-SSL uses only the trivial choice that encourages J to be insensitive to G. In contrast, 7 is not the identity, then f will be sensitive to G and we say that the "cquivariance to G" will be non-trivial.
T, is the identity function, i.e. (fx)))
Therefore, in order to encourage potentially more useful equivariance properties, we generalize SSL to Equivariant Self-Suuervised Learning (E-SSL) framework. In our experiments standard computer vision data, such as the small-scale CIFAR-10 (Torralba et al., 2008 Krizhevsky 2009) and the large-scale ImageNet (Deng al. 2009). we show that extending I-SSL to E-SSL by also predicting four-fold rotations improves the semantic quality of the representations. We show that this approach works for other transformations too, such as vertical flips, 2x2 jigsaws, four-fold Gaussian blurs and color inversions, but focus on four-fold rotations as the most promising improvement we obtain with initial E-SSL experiments in Figure
We also note that the applications of E-SSL in this paper are task specific, meaning that the represen- tations from E-SSL may work best for particular downstream task that benefits from equivariance dictated by the available data. E-SSL can be further extended to applications in science; in particular, we focus on predictive tasks using (unlabelled and labelled) data collected via experiments or simu- lations. The downstream tasks in prediction problems in science are often fixed and can be aided by incorporating scientific insights. Here, we also explore the generality of E-SSL beyond computer vi-
2

---

Published as a conference paper at ICLR 2022
100
insensitive
accuracy (%)
sensitive
80
baseline
horizontal flips
grayscale four-fold rotations vertical flips 2x2 Jigsaws
four-fold blurs color inversions
SSL representations should be encouraged to be either insensitive or sensitive transfor- mations. The baseline is SimCLR with random resized cropping only. Each transformation on the horizontal axis combined with random resized cropping. The dataset is CIFAR-10 and the kNN accuracy is on the test set. More experimental details can be found in Section
Figure 1:
sion, on a different application: regression problems in photonics science and demonstrate examples where E-SSL effective over I-SSL.
Our contributions can be summarized as follows:
We introduce E-SSL, a generalization of popular SSL methods that highlights the comple- mentary nature of invariance and equivariance. To our knowledge, we are the first to create a method that benefits from such complementarity.
We improve state-of-the-art SSL. methods CIFAR-10 and ImageNet by encouraging equivariance to four-fold rotations. We also show that E-SSL is more general and works for many other transformations, previously unexplored related works.
We demonstrate the usefulness of F-SSL. beyond computer vision with experiments on regression problems photonics science. We also show that our method works both for finite and infinite groups.
The rest of the paper organized as follows. In Section we elaborate on related work. In Section introduce experimental method for E-SSL. In Section we present main experiments in computer vision. In Section provide a discussion around our work that extends our study beond computer vision. Beginning from Appendix [A] we provide more details behind our findings and discuss several potential avenues of future work.
RELATED WORK
To encourage non-trivial equivariance, we observe that a simple task that predicts synthetic transformation applied to the imput, works well and improves I-SSL already; some prediction tasks create representations that can be transferred to other tasks interest, such as classification, object
detection and segmentation. While prediction tasks alone have been realized successfully before in
SSL (Agrawal et al.) 2013) Doersch et al.] 2015 Zhang et al.] 2016), Misra et al.] 2016); Noroozi Lee al. 2017; Mundhenk
2016 Zhang et al. 2019; Zhang 2020), to our knowledge we are the first to combine simple predictive objectives of synthetic transformations with I-SSL, and successfully improve the semantic quality of representations. We found that the notion of equivariance captures the generality of our method.
& Favaro 2016 Zamir
2018 Gidaris et al.) 2018
To improve representations with pretext tasks, Gidaris al. (2018) use four-fold rotations predic- tion as pretext task for learning useful visual representations via new model named RotNet. Feng (2019) learn decoupled representations: one part trained with four-fold rotations prediction and another with naramermetric instance discrimination (Wu et al., 2018) and invariance to four-fol. ro- tations. Yamaguchi et al. (202I) use a joint training objective between four-fold rotations prediction and image enhancement prediction. Xiao et al. 22200) propose to learn representations follows: for each atomic augmentation from the contrastive learning's augmentation policy, they leave out and project to space which I-SSL encourages invariance to all augmentations, but the left- out one. The resulting representation could either be a concatenation of all projected left-out views' representations, the representation in the shared space, before the individual projections. Our
3

---

Published as a conference paper at ICLR 2022
framework
MoCo (He
Egomation (Agrowal et
SimCUR (Chen et 2020)
Context (Doersch et 2015)
BYOL Krill 2020)
Colorization (Zheng et 2016)
(zbontar et, 2021)
Jigsaw (Ncorezi Favaro, 2016)
insensitive
sensitive
DINO (Caron et 2021)
RotNet (Gidari al, 2018)
etc.
etc.
E-5SL (ours)
tranffrrration
Tlps
grayscale
methods. Egomotion, Context, Colorization and Jigsaw use other transformations than rotations, but their patterns looks like that of RotNet's. Likewise, for E-SSI. can use transformations different from rotation.
Figure 2: E-SSL framework. Left: framework. Right:
method differs from the above contributions in that E-SSL is the only hybrid framework that en- courages both insensitive representations for some transformations and sensitive representations for others and does not require representations to be sensitive and insensitive to a particular transforma- tion the same time. Thus, what distinguishes our work is the complementary nature of invariance and equivariance for multiple transformations, including finite and infinite groups.
To obtain performance gains from transformations, Tian et al. (2020) study which transformations are the best for contrastive learning through the lens of mutual information. RRed et al. (2021) use four-fold rotations prediction an evaluation measure to tune optimal augmentations for con- trastive learning. Wang & Qi (202T) use strong augmentations to improve contrastive learning by matching the distributions of strongly and weakly augmented views' representation similarities to a memory bank. |Wang et al. (2021) provide an effective way to bridge transformation innefformationnisessstiii insensitive and transformatioo-sseniitnn
approaches self-superived learning methods via residual relaxation. A growing body of work encourages invariance to domain agnostic transformations (Tamkin et al., Lee et al. 2021 Verma et al. 202I) or strengthens invariance with regularization (Foster et al, 2021 Our framework is different from the above works, because work with transformations that encourage equivariance beyond invariance.
2021
To understand and improve equivariant properties of neural networks, Lenc & Vedaldi (0055) study emerging equivariant properties of neural networks and (Cohen & Welling 2016; Bronstein et al. 2021) construct equivariant neural networks. In contrast, our work does not entoree strict equiv- ariance, but only encourages equivariant properties for the encoder network through the choice of the loss function. While striet equivariance is concerned with groups, some of the transformations, such as random resized cropping and Gaussian blurs, may not even be groups, but they could still be analyzed in the E-SSL framework. Thus, ours is flexible framework, which allows us consider variety of transformations and how the encoder might exhibit equivariant properties to them.
METHOD
Our method is designed to test our primary conjecture that a hybrid approach of sensitive and in- sensitive representations learns better features. Surprisingly, this hybrid approach is not yet present in SSL, as Figure illustrate, In this figure, we can view transformations in SSL as "levers." Each downstream task has an optimal configuration of the levers, which should be tuned the SSL objec- tive: left for insensitive and right for sensitive representations. E.g., make representations insensitive horizontal flips and grayscale and sensitive to four-fold rotations, vertical flips, 2x2 jigsaws, Gaus- sian blurs or color inversions. Formally, insensitive and sensitive features correspond to trivial and regular group representations, respectively. Here, we present an effective method achieve this control.
Let f(:: Oy) with trainable parameters 0, be a backbone encoder. Analogously, let (i(:; O,,) be projector network for the I-SSL loss. There might be extra prediction head and parameters, depending on the objective, which we omit for simiplicity. Let (=:60ppp) be the predictor network for encouraging sensitivity, which we will call "predictor for equivariance." We share the backbone encoder f jointly for I-SSL and the objective of predicting the transformations from the backbone representations. Let SSL. be the I-SSL. loss and SSL. be the added E-SSL. loss that encourages
4

---

Published as a conference paper at ICLR 2022
Invorincce
equisatienee
projector
predictor
predictor
predictor
predictor
backbone
backbone
backbone
backbone
backbonnt
bacibone
prediction view
wew
view ?
Figure 3: Sketch of E-SSL with four-fold rotations prediction, resulting in backhone that is sensi- tive to rotations and insensitive to flips and blurring. ImageNet example n01534433:169.
sensitivity particular transformation. Let the parameter À be the strength of the E-SSL loss. The optimization objective for an image a with views {æ*} in the batch is given as follows
arg min (P: Op))))))) AEgea [PredictionLossig P2 (f(TT 0y); 0p:)] (1)
where fESSSL (the expectation in the second summand) can take either one or all of the views, we take only one for simplicity. The goal of &,SSI. is to predict g from the representation P2([(T, (f); Op2), 0p.) which encourages equivariance to the group of transformations G. The PredictionLos could be either cross entropy loss finite groups or LI/ MSE loss infinite groups. In practice replace the expectation with an unbiased estimate. Most of our experiments in this paper focus on finite groups, but we show one example for an infinite group in Appendix
E-SSL can be constructed for any semantically meaningful transformation (see for example, Fig- ure []. From Figure we choose four-fold rotations as the most promising transformation and we also present empirical results about
fix it for the upcoming section. As minor motivation, the similarities between four-fold rotations prediction and I-SSL in Appendix C In particular, both tasks benefit from the same data augmentation. Figure sketches how our construction works for predicting four-fold rotations. In particular, we sample each of the 4 possible rotations uniformly and use the cross entropy loss for the PredictionLoss Equation !
What transformations could work for E-SSL? A common property of the successful transfor- mations have studied up this point that they form groups in the mathematical sense, i.e. (i) each transformation is invertible, (ii) composition of two transformations is part of the set of transformations and (iii) compositions are associative.
In this paper, we encourage equivariance to group of transformation by predicting them. This does not guarantee that the encoder we learn will be strictly equivariant to the group. practice we observe that invariance and equivariance is well encouraged by the training objectives we use (see Appendix D.3 for detailed analysis). In fact, even strict equivariance is possible, i.c. there exists an encoder that is non-trivially equivariant, under reasonable assumption which is formulated as follows. Let X the set of all images. Let G be a group whose elements E transform X G, X} be the set of all transformed We write
via the function T, X X. Let X' [Ty (x)
images. Let f(; 0): X' + be an encoder network that we learn with parameters f(.) = f(: 9) for simplicity. Finally, let S = {f(=') (e))) af X) be the set of all representations of the images in X'. The following is our statement.
Proposition 1 (Non-trivial Equivariance). Given Tg: X' X' for the group G, there exists encoder f: X' - S that is non-trivially equivariant to the group G under the assumption that if [(Tg(=)) = f(Tg (æ!)) then g and x' for all g, g E G and x x' X.
5

---

Published as a conference paper at ICLR 2022
We defer the proof to Appendix [ The significance of this proof is that it explicitly constructs a non- trivially equivariant encoder network for groups G if the assumption satisfied. The intuition of the assumption is that if the representations of two transformed inputs are the same, the inputs should coincide, and likewise the transformations. More formally, this assumption reflects the condition when the dataset contains only one element of each group orbit. We speculate that satisfying this assumption is reasonable for the datasets in this work, since we observe a natural setting of the data, e.g. horizontal mirror symmetry in Gpm, and we consider transformations that disturb this natural setting. In Appendix G we also show that E-SSL is crucial for the Flowers- 102 dataset for which this assumption might be less clear. In Appendix H we also present a natural modification of E-SSL for scenarios, where that assumption is violated.
Could other transformations still help? To motivate our work, in Figure | we observed additional transformations that could be useful, such as vertical flips, 2x2 jigsaws, four-fold Gaussian blurs and color inversions. All of these transformations are groups, except for four-fold Gaussian blurs. Each element of Gaussian blurs is invertible (de-blurring), but the inverse is not transformation in the set. Interestingly, we observe that four-fold Gaussian blurs still improve the baseline, which means the success of E-SSL may not be limited to groups.
We might also consider combining the prediction of multiple transformations to encourage sensi- tivity all of them. However, the gains saw in Figure may not add up when combine transformations, because they may not be independent. The gains may also depend on the trans- formations that we choose for I-SSL. While we combinations of transformations as promising future work, we focus on a single transformation to make a clear presentation of E-SSL.
4 EXPERIMENTS
4.1 SETUPS
CIFAR-10 setup. We use the CIFAR-10 experimental setup from (Chen & He 2021 We con- sider two simple I-SSL methods: SimCLR (with InfoNCE loss (Oord et al.) 2018) and temperature 0.5) and SimSiam (Chen & He| 2021). We were able to obtain baseline results close to those in (Chen) & He 2021 The predictor for equivariance takes a smaller crop with size 16x16. We report perfor- mance on the standard linear probe. We tune 0.4 both for SimCLR and SimSiam (full tuning in
Table in Appendix D). Remaining experimental details can be found in Appendix D
ImageNet setup. We use the original augmentation setting for each method. The predictor for equivariance takes a smaller crop with size 96x96. We use a ResNet-50 (He et al. (201)) backbone for each method. In terms of optimizer and batch size settings, we follow the standard training recipe for each method. For our SimCLR experiments we use slightly more optimal implementation that uses BYOL's augmentations (i.c. it includes solarization). initializes the ResNet with zero BatchNorm weights and uses the InfoNCE loss with temperature 0.2.
Photonic-crystals setup. Photonic crystals (PhC) are periodically-struutured structured materials engineered for wide ranging applications by manipulating light waves (Yablonovitch 1987 Joannopoulos al.) 2008). The density-of-statee of these crystals and thus here, we consider the regression task of predicting the DOS of PhCs. Examples of this dataset are depicted in Section 5 and further details can be found in Appendix I The use of symmetry or invariance knowledge is common in scientific problems; here, the DOS labels are invariant to several physical transformations of the unit cell, namely, rolling translations (due to its periodicity), operations arising from the symmetry group (C4o) of the square lattice, i.e rotations and mirror flips, and refractive scaling. We construct an encoder network comprising of simple convolutional and fully-connected layers (see Appendix FF) and create various synthetic datasets to investigate encouragement of equivariance. After SSL/ E-SSL, we fine-tune the network with L1 loss; for better interpretability of prediction accuracies, we use relative error metric (Liu ct al. 2018 Loh et al. 2021) for evaluation, given by foos = (w.. DOS DOS)) DOS), reported in (%). We deter the results Section 5 because of the novelty of the experimental setup.
(DOS) is often used as a design metric to engineer the desired properties
The predictor P2 for E-SSL. The predictor a 2 layer MLP for CIFAR- 10 and Photonic-crystals, and layer MLP for ImageNet, followed by linear head that produces the logits for the an n-way

---

Published as a conference paper at ICLR 2022
Algorithm PyTorch-style pseudocode for E-SSL, predicting four-fold rotations.
# E: backbone encoder network
pl: projector natwork for I-SSL
p2: predictor network for E-SSL
# sal loss: loss function for I-SSL
lambda: weight the E-SSL
for in loader:
large views
V V_large augment (x, small_cropaat crop_paaee)))) f 1ist of views
v_small augment (x, small_ crop-fruo)) change: with size=96 and scale= (0.05, 0.14)
loss
loss invariance = ssl loss (V (V [___aage))
labels [evv[[[[vall])) small]
4Nx1
v y_cat cat ([v
dim=0) $ 4Nx3x96x96
v. eqquvvariance rot90 (v_ cat, labels) constructing the rotated views
logits p2 (v equivariance)) 4Nx4
loss equivariance loss loss invariance 1ambda loss equivariance
(logits, labels) rotation prediction
optimization step loss. backwaack wtt.mmize.steppl
Table Linear probe accuracy (%) on CIFAR-10. Models are pre-trained for 800 epochs. Baseline
results are from Appendix D in (Chen & He 2021). Standard deviations are from 5 different random initializations for the linear head. Deviations are small because the linear probe robust to the seed.
SimCLR
SimSiam
Method
(Chen et al. 2020)
(Chen & He 2021
Baseline (Chen & He| (2021)
91.1
91.8
Baseline (our reproduction)
92.0#
91.6#
E-SSL (ours)
94.1
94.2
Ablating E-SSL
Single random rotation Linear predictor for equivariance No SSL augmentation in equivariance views
93.4 (L 0.7)
92.6 (4 1.6)
93.3 00 (1 0.8)
93.4 0o (1 0.8)
92.7 (4 1.4)
92.0+ (.2)
Alternatives to E-SSL
Disentangled representations Insensitive instead of sensitive
91.3 oo (L 2.7) 86.3 (4 7.8)
91.1+ au 3.1).) 16.1 (4 8.1)
classification (for example four-fold rotations is 4-way classification), single node for the tinuous group experiment. The predictor's 's hidden dimension is shared across all layers and equals 2048 for CIFAR-10 10 and ImageNet and 512 for PhC. After each linear layer, there a Layer Nor- malization Ba et al. 2016) followed by ReLU. We experimented with Batch Normalization (Toffe Szegedy 2015) (with trainable affine parameters) instead of Layer Normalization, but did not observe any significant changes. For some experiments, we discovered that removing the last ReLU from the MLP improves the results slightly. particular, for SimSiam on CIFAR-10 and for all models on ImageNet we omit the last ReI.U.
Finally, Algorithm I presents pscudocode for E-SSL with four-fold rotations on ImageNet. In our implementation, we use smaller resolution for the rotated images, so that can fit all views on the same batch and have minimal overhead for pre-training (additional details in Table in Appendix
4.2
MAIN RESULTS
CIFAR-10 results. To highlight the benefits of our method, Table demonstrates the improvement we obtain by using E-SSL on top of SimCLR and SimSiam and then shows different ablations and alternative methods. We label the E-SSL extensions as E-SimCLA RR and B-SimSiam respectively. We

---

Published as a conference paper at ICLR 2022
SimCUR seml-supervised
SimCLR remove data aug.
SimSiam semi-supervised
SimSiam remove data aug.
94
92
80
top- linear probe
90
60
88
70
40
Baseline
86
E-SSL (ours)
20
No Augment.
Crop only
Crop Fiip only
No Grayscale
Full Augment,
No Augment:
Crop only
Crop Füip only
No Grayscale
Full Augment.
fraction of training data
fraction of training data
Figure 4: Reducing the labels for training and the data augmentation for pre-training on CIFAR-10. Error bars for 5 different training data splits.
Table 2: Linear probe accuracy (%) on ImageNet. Each model is pre-trained for 100 epochs. Base- line results are from Table B.1 (Chen et al. 2020) from Table (Chen He 202I). l). Numbers marked with use less optimal setting than reproduction for SimCLR (see ImageNet setup).
SimCLR
SimSiam
Barlow Twins
Method
Chen 2020
Chen & He 2021)
Zhontar et al. 2021
Baseline (Chen al. 2020
64.7* *
Baseline (Chen & He 2021
66.5*
68.1
Baseline (our reproduction)
67.3
68.
66.9
E-SSL (ours)
68.3
68.6
68.2
observe that we can increase a tuned baseline accuracy by about 3%. When ablating E-SSL, we see that each component of E-SSL is important. Most useful is the SSL augmentation applied on top of the rotated views. We also study alternatives to E-SSL. With "Disentangled representations" investigate whether "middle ground" is optimal for E-SSL: half of the representation to be insensitive to a transformation and the other half to be sensitive to the same transformation. This results in degradation of performance, which reflects be either insensitive or sensitive. We conducted this experiment by using four-fold rotations in I- SSL for half of the representation and E-SSL for the other half. Finally, making the representations "Insensitive instead of sensitive" to four-fold-rotations hurts the performance significantly, as it is also observed in Figure ! and in (Chen et al. 2020 Xiao et al. 2020
hypothesis that the representations should
Figure 4 reveals that E-SSL is more robust to removing transformations for I-SSL or reducing the labels for training. For example, E-SimCLR and E-SimSiam with only random resized cropping obtain 83.5% and 84.6% accuracies. Encouraging sensitivity to one transformation, namely four- fold-rotations, can reduce the need for selecting many transformations for I-SSL and with only 1 1% of the training data, E-SimCLR and E-SimSiam achieve 90.0#t 1.0% and 88.6.t 1.0% respectively.
ImageNet results. Table demonstrates our main results on the linear probe on ImageNet after pre-training with various state-of-the-art I-SSL methods and their E-SSL versions. By only sweeping À and slightly reducing the original learning rate for SimSiam we obtain consistent 1% 0.5% 1.3% improvements for SimCLR/ SimSiam/ Barlow Twins respectively. Additionally, in Table 3 we observe consis- tent benefits of using E-SSL with longer pre- training. Finally. after 800 epochs of pre- training E-SimCLR achieves 72.5% which is 0.6% better than SimCLR's Rss 71.9% baseline.
Table 3: Linear probe accuracy (%) on ImageNet with longer pre-training. "BT" is short for "Bar- low Twins."
pre-training epochs
Method
100
200
300
SimCLR (repro)
67.3
69.7
70.6
E- E-SimLLR (ours)
68.3
70.5
71.5
BT (repro)
66.9
70.0
71.1
E-BT (ours)
68.2
71.0
71.9
8

---

Published as a conference paper at ICLR 2022
Input unit cells
Label
four fold translations
crystal
G
DOSS: corbitran yumtts
Blos
four-fold rotations
6pm
muvor
KM>
Frequency unbttrayyuntt
Figure 5: PhC datasets with transformations for sensitivity. The regression task is to predict the DOS labels (an example of a label in IR IR:00 is shown on the right) from 2D square periodic unit cells (examples of the inputs in R32x32 are shown the left). We consider two types of input unit cells; at the top is the Blob dataset where the feature variation is always centered; at the bottom is the Group pm (Gpm) dataset where inputs have horizontal mirror symmetry.
Table 4: Fine-tuning the backbone on PhC datasets using 3000/ 2000 labelled train/ test samples. Relative error (%) is fpos 12w, DOSpred DOS / DOS). Lower is better. SimCLR for Blob includes C4 (rotations and flips); SimCLR for Gpm includes rolling translations and mirrors. E-SimCLR encourages the features to be sensitive to the selected transformation explained in the text (four-fold translations for Blob and four-fold rotations for Gpm). "+ Transform" means adding this transformation to SimCLR. Error bars are for 3 different training data splits.
PhC Dataset
Supervised
SimCLR
SimCLR + Transform E-SimCLR (ours)
Blob
1.068 oms
0.987 0.005
0.999 0.005
0.974: + 0.099
Gpm
3.212 oon 3.122 0.002
3.139 + 0.005
3.091 + 0.006
DISCUSSION
To show that other domains benefit from E-SSL in a qualitatively similar way the applications in the previous section, here we introduce two datasets in photonics science. Figure 5 depicts the datasets, i.e. input-label pairs consisting of 21) square periodic unit cells of PhCs and their associated DOS. The physics of the problem dictates that the DOS is invariant to (rolling) translations, scaling of all pixels by a fixed positive factor, and operations of the C4v symmetry group, i.c. rotations and mirror flips. In choosing the transformations that E-SSL should encourage sensitivity to, we observe that the transformations that have worked for CIFAR-10 10 and ImageNet disturb the natural setting of the data (e.g. rotations disturb the natural upright setting of images). Thus, encourage sensitivity to transformations that fit this observation, and insensitivity to the rest of the transfornations.
In Figure 5 the top dataset is "Blob" dataset where the shape variation in each image is centered. We encourage sensititivy to the group of four-fold translations, given by G = {e, h, v, hw), where and u are 1/2-unit cell translations in the horizontal and vertical axis, respectively, is the unit element (no transformation) and fvo is the composition of h and v. In the bottom dataset of Figure the PhC unit cells are generated have a horizontal mirror symmetry, (or crystallographic plane) group pm. We encourage sensitivity to the group of four-fold rotations (the same group we used for CIFAR- and ImageNet), since rotating any of the images disturbs the (horizontal) mirror symmetry. More accurately. since only + 7/2 rotations disturb the symmetry, we separate them in two classes, (T/2, - 2/2} and (0, r). and perform binary prediction in E-SSI..
use the 2D wallpaper
Table shows the results of fine-tuning the backbone and an additional DOS DOe-predictor head (see Appendix F with 3000 labelled samples for this regression task. We observe that encouraging sensitivity to the selected transformations (via E-SimCLR) leads to the largest reduction in the enror. On the contrary, including these transformations to SimCLR (indicated by "+ Transform") increases the error. Furthermore, to infinite groups (see Appendix F). This supports our observations about the usefulness E-SSL over I-SSL. and demonstrates E-SSL's generality beyond computer vision.
explore scaling transformations and show that E-SSL can be generalized

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank Shurong Lin, Kristian Georgiev, Alexander Atanasov, Peter Lu and the anonymous re- viewers for fruitful conversations and support. R.D. dedicates this work the memory of Boyko Dangovski.
The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center (Reuther et al. 2018) for providing HPC and consultation resources that have contributed to the research results reported within this paper/report.
Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number 14-750-19-2-10000 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or im- plied, of the United States Air Force or the U.S. Government, The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
This material is also based in part upon work supported by the Air Force Office of Scientific Re- search under the award number FA9550-21-1-0333 and the U. S. Army Research Office through the Institute for Soldier Nanotechnologies at MIT, under Collaborative Agreement Number W911NF- 18-2-0048. This work is also supported in part by the the National Science Foundation under Coop- erative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi (ttgg//ffffrrr//))
REPRODUCIBILITY STATEMENT
Algorithm [ the original public code for each of the I-SSL methods we use in the paper and the experimental setups in Section 4.1 and in Appendices D and can be used for reproducibility. Our code is available at httpp:: // /ttgtggtttb/ttt///////t/g/t com/ rdangovs /essl
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In Proceedings of the IEEE International Conference on Computer Vision, pp. 37 37-45, 2015.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. In NeurIPS Deep Learning Symposium, 2016.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning:
Grids, groups, graphs, geodesics, and gauges, 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2/04. 14294, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Thomas Christensen, Charlotte Loh, Stjepan Picek, Domagoj Jakobović, Li Jing, Sophie Fisher, Vladimir Ceperic, John D. Joannopoulos, and Marin Soljačić. Predictive and generative ma- chine learning models for photonic crystals. Nanophotonics, (113):413341999 October 2020. ISSN 2192-8614 44 doi: 10. 20s511//m/oohh222---1
URL httss:/ www degruyter. com/ document /oieee///////ii 151..1115.1 http://e.........1...../apa00000
Thomas Christensen, Hoi Chun Po, John D. Joannopoulos, and Marin Soljačić. Location and topol-
ogy of the fundamental gap in photonic crystals, 2021.
10

---

Published as a conference paper at ICLR 2022
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- PMLR, 2016.
ence on machine learning. pp.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp.
Icee, 2009.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference computer vision, pp- 1422-1430, 2015.
Zeyu Feng, Chang Xu, and Dacheng Tao. Self-supervised representation learning by rotation fea- ture decoupling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10364-10374, 2019.
Adam Foster, Rattana Pukdee, and Tom Rainforth. Improving transformation invariance in con- trastive representation learning. In International Conference Learning Representations, 2021. URL
net www.:/uuneeeeeemeeeeam id NomEDg
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with self-supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8059-8068, 2019.
Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
D. Joannopoulos, S. G. Johnson, N. Winn, and R. Meade, Photonic Crystals: Molding the Flow of Light. Princeton University Press, 2 edition, 2008. URL http::/tt//bbb initio mit. ledu/bookk
Steven G. Johnson and J. D. Joannopoulos. Block-iterative frequency-domain methods for Maxwell's equations in a planewave basis. Optics Express, 8(3): January 2001. ISSN 1094-4087. doi: 10. 10.1336/OE.8 00.6///.....0113.. 8.00333..
abst rastract..o cfm wtffmurriooee.... 8-
Samuel Kim, Peter Y. Lu, Charlotte Loh, Jamie Smith, Jasper Snoek, and Marin Soljačić. Scalable and Flexible Deep Bayesian Optimization with Auxiliary Information for Scientific Problems. arXiv:2104. 1667, April 2021. URL [tRlitttt///arxttt http:/axxvvv..g/gaa/// hbs//bbs/10000 httpaab//..........
Alex Krizhevsky. Learning multiple layers of features from tiny images. technical report, 2009.
Hsin Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming Ming-Hsuan Yang. Unsupervised represen- tation learning by sorting sequences. In Proceedings of the IEEE International Conference on Computer Vision pp. 667-676. 2017.
11

---

Published as a conference paper at ICLR 2022
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999, 2015.
Boyuan Liu, Steven G. Johnson, John D. Joannopoulos, and Ling Lu. Generalized Gilat- Raubenheimer method for oensity-of-states calculation in photonic crystals. Journal of Optics, 20(4):044005, April 2018. ISSN 2040-8978, 2040-8986. doi: 10.
URL
http: wwwp://gg/gg.a.o../////// .77993
Charlotte Loh, Thomas Christensen, Rumen Dangovski, Samuel Kim, and Marin Soljacic. Surrogate- and invariance-booosted contrastive learning for data-scarce applications in science, 2021. URL http:::/awxw..ogggg///oo//////// abs
08406
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision, pp. 527-544. Springer, 2016.
T. Nathan Mundhenk, Daniel Ho, and Barry Y. Chen. Improvements to context based self-supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008.
Mchdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv:1807.03744 2018.
Colorado Reed, Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer. Evaluating self- supervised pretraining without using labels. In CVPR, 2021.
Albert Reuther, Jeremy Kepner, Chansup Byun, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Vijay Gadepally, Michael Houle, Matthew Hubbell, et al. Interactive supercom- puting on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2018
Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks:
Learning views for unsuper- vised representation learning. In International Conference on Learning Representations, 2021. URL httpihttts:///peeeeeee wtt.:rrwwweeeeeeeeoooemmmmmmmm
n wor for.mmihddeeooooW.sst
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint arXiv:2005. 10243, 2020.
Antonio Torralba, Rob Fergus, and William T. Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11):1958 19181)))988819999 1970, 2008 doi: 10.1 109/T0AMM...00...I.
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards domain-agnosti contrastive learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10530-10541. PMLR, 18-24 Jul 2021. URL https::/ppoceeednggggg/ooo press/v139/40 ve rma html
Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. arXiv preprint arXiv:2104.07713, 2021.
12

---

Published as a conference paper at ICLR 2022
Yifei Wang, Zhengyang Geng, Feng Jiang. Chuming Li, Yisen Wang. Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. Advances in Neu- ral Information Processing Systems, 34, 2021.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via parametric instance-level discrimination. In CVPR, 2018.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.05653 2020.
Eli Yablonovitch. Inhibited Spontameous Emission in Solid-State Physics and Electronics. Phys- ical Review Letters, (8(20))2059-2062 May 1987. doi: 10. 1 10.1100 ett.58..222 2059. URL
https: /opp:://iike aps http:/iw.pp......./......... 1103/ www.////hwyeeeeet.eeeeeeee
2059
Shin'y ya Yamaguchi, Sekitoshi Kanai, Tetsuya Shioda, and Shoichiro Takeda. Image enhanced ro- tation prediction for self-supervised learning. In 2021 IEEE International Conference Image Processing (ICIP), pp. 489-493, IEEE, 2021.
Amir Zamir, Tilman Wekel, Pulkit Agrawal, Colin Wei, Jitendra Malik, and Silvio Savarese. Generic 3d representation via pose estimation and matching. In European Conference on Com- puter Vision, pp. 535-553. Springer, 2016.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint vrXiv22103.0323: 2021.
Liheng Zhang. Equivariance and invariance for robust unsupervised and semi-supervised learning. 2020.
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jiebo Luo. Aet vs. aed: Unsupervised representation learning by auto-encoding transformations rather than data. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 2547-2555, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
A SUMMARY OF MAIN TEXT AND LAYOUT OF APPENDIX
In this paper we motivated the generalization of state-of-the-arr methods self-supervised learning to the more general framework of equivariant self-supervised learning (E-SSL). In E-SSL rather than using only invariance as trivial case equivariance, we encouraged non-trivial equivariance and improved state-of-the-art methods common computer vision benchmarks and regressions tasks in photonics science. We also discussed that there are many types of equivariance we can consider for E-SSL. We observed that most of the successful transformations for E-SSL that we explored form groups, but forsee that potentially many more transformations could be explored.
For future work could learn transformations that equivariances, instead of setting them man- ually. Thus, the concept of E-SSL could potentially be extended to natural language processing or other science domains, whose transformations for SSL are less well-understood. To facilitate further research in E-SSL, below we provide additional details and analysis of the experiments in the main text. We also discuss interesting avenues for future work.
B PROOF OF PROPOSITION 1
Proof. To construct non-trivially equivariant f, we first need to show that both X' and S G-sets, iee. that there is a group action T, of G on X', which is given by the statement of the proposition, and another (non-trivial) group action Ty of G on S, which will construct. Then, need show that f commutes with the group action, i.e. that (TT))')) (f')))
13

---

Published as a conference paper at ICLR 2022
Group actions. Note that by the setup of the problem, input X', i.e. To is known. For example, if G is the group of four-fold rotations, then Tg is the rotation of the imput by multiple of t/2. We proceed to construct the non-trivial group action T of G on S.
already given how G acts the
(TT,((T,(a'))),, Define the function T': G + as -(T, that is well-defined, because gg' E G by the closure of the group and 8 is uniquely written as 8 = f[Tg (ar') )). To see why s is uniquely written, it suffices show that if ((Tz(('')) then both y = =g and x' = x", which follows directly from our assumption in the statement.
where s f (xx')) Note
f(T (x"))
Now, prove that T" is group action, it suffices show properties.
Identity: se. s for f(g'(x')) and is the unit element of the group. To show that, note that by definition (e = (TT.(TT (ar))) = J(Tg' (æ))), because eg' g'.
Compositionality: (g, (h, ((TT(
(gh, f(Tg f(TTr(x'))) To show this, pand the LHS and use the definition of T" to obtain as follows (g, (h, [[Tg)(()) (g (TT, (Tgr(a"")) (gh, fTy (x'))), [)), because the group operation is associative.
((Tg(TTh' (ax))) [(Tghg/ (x²)) J(T gt (Tg (x'
Hence, 7' is a group action, and thus S is G-set, and we can write (' T.
Commuting with the group action. To see this property, note that (f(x')) (TT)((())))) f(T, (Tz(2))) f(Tz as desired. Note that Ty is non-trivial.
Therefore, we can conclude that f. which satisfies the constructed group action y. is not-trivially equivariant to the group G.
C ROTATION PREDICTION AND I-SSL BENEFIT FROM SIMILAR DATA AUGMENTATION.
Recently, rotation prediction with linear head from the frozen backbone representations proved to be useful for validating the augmentation policies of contrastive learning (Reed et al. 2021). This shows that the two tasks of classification of ground truth classes and syntheti rotation classes from frozen backbone representations benefit from similar augmentation policies. We took this experi- ment step further, and performed rotation prediction with the augmentation policies, typically used contrastive learning.
The result is in Table 5) Interestingly, RotNet benefits from augmentations, typically used in con- trastive learning, and the RotNet training shares the same sweet spot (Tian et al. (020) as kNN classification. There are several takeaways from this experiment: (i) we can find good augmenta- tions for contrastive learning by doing RotNet alone, i.e. without doing any contrastive learning: (ii) RotNet benefits from augmentations needed in contrastive learning; (iii) may be able combine four-fold rotations prediction and contrastive learning.
D CIFAR-10 EXPERIMENTS
D.1 EXPERIMENTAL SETUP
Our experiments use the following architectural choices: ResNet- 18 backbone (the CIFAR-10 ver- sion has kernel size 3, stride 1, padding 1 and there is no max pooling afterwards): 512 batch size (only our baseline SimSiam model uses batch size 1024); 0.03 base learning rate for the baseline SimCLR and SimSiam and 0.06 base learning rate for E-SimCLR and E-SimSiam; 800 pre-training epochs; standard cosine decayed learning rate; 10 epochs for the linear warmup; two layer projector with hidden dimension 2048 and output dimension 2048; for SimSiam a two layer (bottleneck) pre- dictor with hidden dimension 512 whose learning rate is not decayed; the last batch normalization for the projector does not have learnable affine parameters; 0.0005 weight decay value; SGD with momentum 0.9 optimizer. The augmentation is Random Resized Cropping with scale (0.2, 1.0),
14

---

Published as a conference paper at ICLR 2022
Table 5: RotNet's augmentation sweet spot. kNN and Rotation Prediction have the sweep spot (Level 4) which gives best accuracy in both columns. RotNet is trained on CIFAR-10 10 for 100 epochs with the same optimization setup as our I-SSL experiments. Accuracies are on the split. (4 marks the deviation from the sweet spot. Every new level adds a new augmentation to the previous level incrementally.
Level
Added Augmentation
Supervised KNN Acc. (%)
Rotation Prediction Acc. (%)
0
44.8 ( 19.8)
90.2 (4 4.8)
random resized cropping
59.2 (4 5.4)
93.7 () 1.3)
2
horizontal flips w.p. 0.5
59.4 (4 5.2)
94.5 () 0.5)
3
color jitter w.p. 0.8
64.3 (4 0.3)
94.9 (4 0.1)
4
grayscale w.p. 0.2
64.6
95.0
5
Gaussian blur w.p. 0.2
64.1 (L 0.5)
94.5 (1 0.5)
random rotation (+*/6)
59.4 (4 5.2)
93.1 (4 1.9)
7
vertical flip w.p. 0.5
51.9 (1 12.7)
90.6 (4 4.4)
aspect ratio (3/4, 4/3) and size 32x32, Random horizontal Flips with probability 0.5, Color Jittering (0.4, 0.4, 0.4, 0.1) with probability 0.8 and Grayscale with probability 0.2. Some of our evaluations KNN-classifed with 200 neighbors, cosine similarity and Gaussian kernel with temperature 0.1. This evaluation correlates well with the standard linear probe, but is more efficient to calculate. We report the kNN accuracy in at the end of the 800 epochs of training. For main results, report a linear probe accuracy from training linear classifier for 100 epochs on top of the frozen representations with SGD with momentum 0.9 and cosine decay of the learning rate, batch size 256 and initial laerning rate of 30. For linear probe experiments we try different initalizations of the linear head and report mean and standard deviations. The deviations are negligible because the linear probe is robust to the random seed. All parameters are reported in a Pytorch-like style.
For Figure we use resolution of 32x32 for the transformations studied. The 4 levels of the Gaussian blur are for kernel sizes 0, 5, 9 and 15 in the default Gaussian blur torchvision implementation. The prediction of the transformations follows the experimental setup in Section 3 When we apply the transformations in I-SSL, add them in the beginning of the augmentation policy with probability 1. The same setup is used for "Disentangled representations" and "Insensitive instead sensitive" in T
D.2 ADDITIONAL EXPERIMENTS
Explored hyperparameters. Both for SimCLR and SimSiam we ran a grid search over the fol- lowing hyperparameters: base learning rate: {0.01, 0.03, 0.06), batch size: {512, 1024), (for E-SSL): {0.0, 0.2, 0.4, 0.6. 0.8, 1.0}, predictor's MLP depth: {2. 3, 4}, predictor's normalization: (None, BatchNorm, LayerNorm). nonlinearity the last MLP layer of the predictor: (True, False).
Tuning Table d shows tuning of the CIFAR-10 results. We observe noticeable improvements the SSL baselines by using E-SSL instead.
Table 6: Tuning the À parameter for CIFAR-I0.
Baseline
E-SSL.
Method
0.0
0.2
0.4
0.6
0.8
1.0
SimCLR
92.0+ oe
93.6+
94.1 ae
94.0+ ao
94.1 as
93.5+ ao
SimSiam
91.1+
00
94.2+
93.7 uo
93
93.3+
Sensitivity to transformations for I-SSL. Table 7 demonstrates that E-SSL can produce good representation with as few SSL transformations for T-SSL as possible. We observe that E-SSL is less sensitive than SSL to the choice of data augmentation.
15

---

Published as a conference paper at ICLR 2022
Table 7: Comparing the augmentation sensitivity for CIFAR-10. Levels: 0 is transformations; adds random resized cropping; 2 adds horizontal flips; 3 adds color jitter: 4 adds grayscale.
Augmentation Level
Method
SimCLR
28.0= 0.1
76.0 a.1
77.0= 0.
87.4 0.
92.0 00
E-SimCLR
69.5=
83.5 n.
84.6=
91.6=
94.1
SimSiam
17.6 0.1
71.5 0.0
72.2 0.0
88.
91.1
E-SimSiam
67.5= 0.1
84.6# n.
85.7 0.1
92.9 94.2
The importance of complete invariance or sensitivity. Table studies whether a middle ground for the representations exist, i.e. whether it is possible to have part of the representation invariant and the other part sensitive to the transformation. If we apply the E-SSL loss only to half of the rep- resentation, then there very small drop in the performance. Furthermore, we observe that having a disjoint mix between insensitivity and sensitivity in the representation is noticeably harmful.
Table 8: Studying the effect of disjoint representations CIFAR-10. Split Representation means that we encourage similarity only on one half the backbone representation. Disentangled Repre- sentation means that one half of the representation is trained to be insensitive to four-fold rotations and the other half is sensitive four-fold rotations. Linear probe accuracy (%) after 800 epochs.
Method
Baseline
Split Representation Disentangled Representation
E-SimCLR
94.1
94.1 + 0o (4 0.0) 93 (4 0.4)
91.3 00 (1 2.7) 91.1 00 (4 3.1)
E-SimSiam
94.2#
Fully connected backbone. We perform simple experiment with fully connected backbone, instead of a ResNet-18. The hidden dimensions of the backhone are listed in order as (3x32x32, 2048, 2048, 512] with Batch Normalization and ReLUs in between. The rest of the experimental setup is exactly the same. On the linear probe (%), we obtain 70.5+0.00 for SimCLR and 70. for E-SimCLR, and 70.910.0 for SimSiam and 73.5+0.11 for E-SimSiam, highlighting noticeable gains from using E-SSL.
CIFAR-100 experiments. We test our CIFAR-IO experimental setup directly on CIFAR-100, On the linear probe (%), we obtain 65.8 for SimCLR and 69.5 for E-SimCLR, and 65.8 for SimSiam and 69.3+ for E-SimSiam, highlighting sizable gains from using E-SSL.
Large crop study. We study whether using a large crop with a single rotation on CIFAR-10 can be just as good as small crop. We obtain 93.9+10.0 on the linear probe using E-SimCLR, which is only 0.2 absolute points below our best result of 94.1 14.1100 +0.0 using four small crops.
D.3 NORM-DIFFERENCES ANALYSIS
In Figure 6 we present analysis that shows our training objectives encourage invariance and equiv- ariance to transformations. We take our best performing E-SimCLR and E-SimSiam methods on 10. During training we keep track of two measures that can capture how invariant/ equivari- ant the backbone representations are.
The "invariance measure" computes the negative cosine similarity between two views of the back- bone representations. The lower this measure is, the higher the similarity between the two views, and thus the more invariant the backbone representations serve that during training high similarity between the two views is maintained (roughly between 0.8 and 0.9), which indicates that invariance is encouraged in the backbone representations, desired.
the transformations in I-SSL. We ob-
Likewise, the "equivariance measure" computes the average cosine similarity of the backbone rep- resenations, between all six pairs of the four rotated views. The lower this measure is, the lower
16

---

Published as a conference paper at ICLR 2022
the similarity between the four views, and thus the non-trivially equivariant the backbone rep- resentations are to the transformations for equivariance. We observe that the measure decays to about 0.3 during training, which indicates that the backbone representations are encouraged to be equivariant to four-fold rotations, as desired.
E-SimCLR
E-SimSiam
0.0
0.0
invariance measure
-0.4
-0.6
-0.6
00.8
0.5
20000
40000
60000
80000
20000
40000
60000
80000
1.0
1.0
equivariance measure
0.5
0.5
0.2
0.2
0.0
20000
8.0
training steps
Figure 6: Demonstration of the evolution of the invariance (top) and equivariance (bottom) measures during training. Left is E-SimCLR and right is E-SimSiam.
E IMAGENET EXPERIMENTS
We had limited computational resources, we kept the learning rates the same in the original methods. Only for SimSiam we found that choosing a smaller learning rate 0.08 leads to better results for E-SimSiam. We only swept the parameter, where for SimCLR and SimSiam the sweep was between 0 and and for Barlow Twins it was between 0 and 100. The optimal À is 0,4 for SimCLR, 0.08 for SimSiam, for Barlow Twins. We use (0.05, 0. 14) scale range for 100 pre- training epochs. For more pre-training epochs we use (0.05, 0. 04) for SimCLR and (0.08, 1.0) for Barlow Twins.
Table lists the overhead from using rotation prediction in our experiments.
Table 9: Overhead in doing rotation prediction. Reported GPU hours for an experiment on 100 epochs.
SimCLR
SimSiam
Barlow Twins
Baseline
256
295
246
E-SSL (ours)
307
364
294
Overhead
20%
23%
19%
17

---

Published as a conference paper at ICLR 2022
F PHC EXPERIMENTS
Dataset generation. 2D Photonic erystals (PhCs) are characterized by a periodically varying per- mitivitty e(x, y); here, for simplicity we consider a "two-tone" permitivitty profile i.e. E E (2), with E; E [1, 20j discretized to resolution of 32 x 32. To generate the unit cells in the "blob" dataset, we follow the proceedure in Christensen al. (222)). For the Gpm dataset, the unit cells are de- fined using a level set of a 2D Fourier sum function like in Kim et al. 2:21)): Loh et al. (202T), , with additional constraints applied
the lattice create the mirror symmetry adopted from the method in Christensen et al. (202T). We then follow the procedure in Loh et al. (2021) to compute, and sub- sequently process, the density-of-stan Bands (MPB) software (Johnson & Joannopoulos] 2001) and the Generalized BllatRRaubeneimmrr method in an implementation from Liu ct al. (2018).
(DOS) of each unit cell, specifically, via the MIT Photonics
Network architecture. We use encoder network composing of simple convolutional (CNN) and fully-connected (FC) layers for the backbone; specifically, our backhone begins with 3 CNN layers, all with a kernel size of and channel dimensions given by [64, 256, 256]. The output is flattened and fed into 2 FC layers each with 1024 nodes (i.c. the representations have dimension 1024). We include BatchNorm (Toffe & Szegedy, 2013), ReLU and MaxPooling for the CNNs, and ReLU only for the first FC layer. The projector and predictor networks, P1 and Pa are 2-layer MLPs with hidden dimension 512, with BatchNorm and ReLU between each layer except the last and the projection dimension for Pı is 256. Additionally, since this is regression task and the label space is much larger than in image classification tasks, we include a dense DOS- 4OS-predictor head after the representations, which is fine-tuned with 3000 labelled samples after SSL or E-SSL. The DOS- predictor has 4 FC layers, with number of nodes given by [1024, 1024, 512, 400). We explore two fine-tuning protocols of the DOS-predicteo freezing the backbone (discussed later in the Appendix) or fine-tuning the backbone (discussed in the main text).
Hyperparameters. For SSL and E-SSL, we performed 250 pre-training epochs using the SGD optimizer with a standard cosine decayed learning rate; the batch size was fixed to 512. The pre- trained model was saved at various epochs {20, 50, 100, 180, 250) for further fine-tuning. Fine- tuning was performed for 100 epochs using Adam optimizer and a fixed batch size of 64. No transformations were applied to the input during fine-tuning for both freezing or fine-tuning the backbone. We ran grid search over the following hyperparameters; for pre-training, base learning rate: {10 3 10 10 33. À (for E-SSL): {0.2, 1.0, 2.0, 5.0, 10.0), and for fine-tuning: a learning rate in {10 3 10
10 53.
Frozen backbone experiment. In Table 10 we present our results from freezing the backbone encoder while fine-tuning the DOS-predictor head. We observe similar trends as in Table 4 where we allowed fine-tuning of the backbone. Relative error is reported in % and the lower the error is, the better. SimCLR for Blob includes Cav (rotations and flips) and SimCLR for Gpm includes rolling translations and flips. E-SimCLR encourages the features to be sensitive to the selected transformation (four-fold translations for Blob and four-fold rotations for Gpm), which improves the performance of SimCI.R. On the contrary, adding the selected transformation to SimCLR, as indicated by "+ Transform"., increases the error of SimCI.. Error hars are reported for different choices of training data. Supervised (frozen) refers to the impractical situation of freezing random backbone and fine-tuning the DOS- SoS-predictor.
Table 10: Frozen backbone experiment on PhC datasets for 3000/ 2000 labelled train/ test samples.
PhC Dataset
Supervised (frozen)
SimCLR
SimCLR Transform E-SimLLR (ours)
Blob Gpm
1.686 0014
1.237 a: 0.05
1.242 0.1
1.165 & 0,020
5.450 omm
3.2144 0os
3.313 0.00
3.187 = 0.000
Continuous group experiment. In all experiments shown so far, we dealt with finite groups of transformations. To show that E-SSL generalizes beyond the finite group setting, we also explore transformations from a continuous group. An example is the scaling transformation where every pixel of the input unit cell is scaled by the same positive factor. More specifically, this set of positive
18

---

Published as a conference paper at ICLR 2022
scaling transformations (()) sæ defines a continuous group G {g(s)]s E R+ which leaves the DOS labels invariant due to the physics of the problem and normalization applied when pre- processing the dataset (Loh et al., 2021). In our experiment, we uniformly sample s E (1, Smarl and apply the inverse with probability 0.5 (i.e. we cap the scaling factor to a maximum of Smez {5, 10} for numerical stability during training and we apply up-scaling and down-scaling with equal probability). To encourage equivariance to this group, we simply predict the scale factor applied to the input using LI loss (i.e. the final layer of the predictor p2 is single node). In Table II we show results after fine-tuning the backbone and the DOS predictor network with 3000 labelled samples. We observe similar trends Table 4 encouraging sensitivity to scaling produces the lowest error and including scaling to SimCLR increases the error. To isolate the effect of scaling transformation, the remaining physics-governed invariances excluding scaling (translations, rotations and mirrors) are used in SimCLR and the invariance part of E-SimCLLR for both datasets.
Table 11: Fine-tuning the backbone on PhC datasets using 3000/ 2000 labelled train/ test samples.
Relative error (%) is Dos = (Ew, DOS DOS DOS). Lower is better. E-SimCLR encourages the features be sensitive to scaling. "+ Scaling" means adding scaling to SimCLR. Error bars are for 3 different training data splits.
PhC Dataset
Supervised
SimCLR
SimCLR + Scaling
E-SimCLR (ours)
Blob (Smax = 10)
1.068 = 0.11
0.988 oom
1.005 0.06
0.974 0.000
Blob (Smaz = 5)
1,068 0.015
0.988 0.01
1.000
0.987
Gpm (Smax = =10) 3.212: oon
3.073 0.003
3.1121001 3.082+0001
3.062 :: 0.005
Gpm (Smaz -5)
3.212+ oo1
3.073+ 0.003 0.03
3.058 0.008
G FLOWERS- 102 EXPERIMENTS
In order to study the importance of the assumption in Proposition]] we perform an experiment with dataset that might not be amenable to such an assumption at first sight. We choose the Flowers- 102 dataset (Nilsback & Zisserman 2008), because at first sight the dataset might not benefit from four-fold rotations in E-SSL.. Therefore, this dataset complements the experiments we performed for CIFAR-10 10 and ImageNet.
Experimental setup. We train SimCLR and E-SimCLR. We use the same optimization hyperpa- rameters from the experimental setup for our CIFAR-10 experiments. We downsize the images to 96x96 resolution and use the standard ResNet- instead of its modified version for CIFAR-10. For the data augmentation in I-SSI., we use the same RandomResizedCropppe size 96 of the crops being the only difference), the same Color Jittering and Random horizontal flips as in the CIFAR-10 experiment. We report the kNN accuracy in (%) on the validation set. We study both four-fold rotations and four-fold translations as transformations for invariance/ equivariance. Four-fold rotations are chosen following the hypothesis that most of the data points should be in- variant to rotation. Four-fold translations are chosen, because of our observation that most of the data points are centered, just like in the Blob PhC dataset. The for predicting four-fold translations is 0.01 and for four-fold rotations is 0.5 (chosen from grid search among {0.001, 0.01, 0.1, 0.5, 1.0, 2.0)).
as in CIFAR-10 (with
Results. Following our observations in Figure we observe that encouraging insensitivity to four- fold rotations and translations, by adding the transformations to the SimCLR data augmentation, worsens the SimCLR baseline. In contrast, using these transformations for E-SSL improves the baselines and further shows the utility of E-SSL for real-world data. We even observe benefit from encouraging equivariance to four-fold rotations, which is against the intuition that rotations should be invariant. This is probably due to the fact that some images in the dataset are not truly rotationally invariant (see examples of the data points in Figure 8)
19

---

Published as a conference paper at ICLR 2022
Translation Experiment
Rotation Experiment
KNN accuracy (%)
50
40
SimCLR (baseline)
SimCLR (baseline)
E-SimCLR Translation (ours)
E-SimCLR w/ Rotation (ours)
SimCLR w/ Translation
SimCLR w/ Rotation
300
400
406
500
epochs
Figure 7: E-SimCLR gives sizable improvements for the Flowers- 102 SSL pre-training. kNN accu- racy (%) is on the validation set.
Figure 8: The Flowers- 102 is not completely invariant rotation. The top shows data points which are roughly invariant to four-fold translations. The bottom row shows counterexamples to that hypothesis.
H RELATIVE ORIENTATION PREDICTION
In our experiments we demonstrate that a shared biased between the train and test sets exists, should exploit it via the E-SSL training objective. However, in some scenarios, the class label of the downstream tasks depends on the orientation (c.g., classifying road signs) and the current E-SSL method may not be very useful, because both and Ty (a)) exist in the data. This situation
20

---

Published as a conference paper at ICLR 2022
invites natural generalization of our method in the spirit of (Agrawal et al. 2015). If æ and (x) exist in the data then we can modify E-SSL minimally to form a useful objective. particular, we can set the objective for P2 to predict the relative orientation between two data points, i... given æ, we form (xx(2) by sampling Ty from G uniformly, and then predict g from (2; p2)) where [f(x), [ (Tg-(x))) is the concatenation of the two representations. This modification requires
minimal change to our framework.
To test the usefulness of the modified method when a: and (2) exist in the data, we artificially modify CIFAR-I0 so that any rotation of an image can appear in the dataset. We consider the downstream task of predicting the rotation orientation of an image, which clearly depends on the orientation of the image. Our hypothesis is that the modified E-SimCLR will be better than SimCLR, which what we observe in our results below. Intuitively, SimCLR not able to capture the orientation of the image, while the modified E-SimCLR is able to, because the latter predicts relative orientation.
The experimental setting is follows: we pretrain for 100 epochs. The predictor for equivariance's input dimension is doubled, because we concatenate two representations. We set À to 0.4. All other hyperparameters are the same as the rest of the CIFAR-10 experiments. The downstream task is 4- way rotation orientation classification. Using pre-training with SimCLR on the standard CIFAR- 10 dataset, we obtain baseline linear probe (%) accuracy 67.1=0.1. Using our modification of E-SSL on the same experimental setting, we obtain 71.2 linear probe accuracy, which is a sizable gain and points to the promise of the relative orientation prediction future work.
The relative orientation prediction scenario is also highly relevant other domains such as in pho- tonic crystals. For example, can modify the PhC setup and remove the "orientation bias" of the datasets while predicting a different property, the band structures. Unlike the DOS, the PhC band structures are invariant rotations and we can consider the group of four-fold rotations. This setup would fit the scenario described above since 1) both æ and (x)( exist in the data due to the lack of bias and 2) the downstream task is sensitive to rotation. We will explore the above framework on this setup in future work.
21


Reviews:
[{"review_id": "xsafyFXWoJe", "reviewer": "Reviewer_7QWR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is simple and straightforward, making it easy to understand. The idea makes sense to me and the experiments show good results.\n\nIt is not clear how the authors did in the introduction to encourage the model to be sensitive or not sensitive to a specific transformation. I would imagine that for encouraging insensitive, the authors did not do anything specific, and for encouraging it to be sensitive, the authors used an additional branch to predict just like the main framework. Still, it would be great if the authors could make it more clear.\n\nI do not really buy the equivariance story. The story and the title are somewhat overclaiming. The authors bring the definition of equivariance in the introduction; however, can the authors be more specific what is the $T'_g$ in their framework? I do not see any post transformation on top of the model output on the newly added branch but just labels. The model did not really learn equivariance but used some auxiliary pretext tasks to help learn better representations.\n\nIt would be great to see how the model performs with longer training iteration on ImageNet-1K, to confirm the gain will not be diminished.", "summary_of_the_paper": "This paper presents a simple solution to improve the existing self-supervised representation learning by adding an additional branch to predict the rotations. Experiments are conducted on CIFAR-10, ImageNet, and two PhC datasets and the results show that the model with the additional branch can outperform the corresponding baselines.", "summary_of_the_review": "Overall I think this paper has its own contribution. It finds that using an additional pretext header can improve the performance. However, I don't think the story itself is convincing enough and believe that might mislead readers. How the model performs with longer training iteration on ImageNet-1K also remains unclear. These prevent me from giving a higher rating.\n\n--post rebuttal\n\nAfter reading the authors response I raise my rating to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sjNoWVONSr3", "reviewer": "Reviewer_p4FW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# A. Strengths\n- 1.\tThe proposed approach achieves good performance gain over baselines on CIFAR10 and ImageNet.\n- 2.\tThe proposed approach generalizes the existing invariant SSL approach to also include equivariance; the idea is intuitive to follow and reasonable.\n- 3.\tAuthor promised to release code and the supplemental materials reports the hyperparameters sweeps and results.\n\n# B. Weaknesses\n- 1.\tNovelty & Contribution: The approach can be viewed as combining “predicting image rotation” (Gidaris et al. 2018) with existing invariant SSL. The authors should more clearly explain the differences in the related works section. Also, the work proposes a “general framework”, I find the writing which only focuses on four-fold rotation not adequate. I would recommend writing out the generalized formulation in more detail, e.g. using equivariance, and not just a very abstract Eq. 1.\n- 2.\tLimitation in approach: The proposed approach is limited to finite groups as it is based on classification. Has the author thought about further generalization? \n- 3.\tOrganization: The analysis in Sec. 5.2 seems out of place. Why not include it in the approach section? Also, personally, I feel the related works before the discussion does not help with the organization. I would recommend placing it after the introduction. \n- 4.\tClarity: The link between equivariance and the proposed approach is not clearly explained in the paper. As the authors have introduced the equivariance definition and notation, it might be beneficial to use those in their approach. Next, the drawing of Figure. 3 should be improved. For example, I cannot understand why for the equivariance there is only one backbone f; for the invariance there are two?. Also, having a single arrow going into “equivariance” is very confusing.  \n- 5.\tAblations in Table 1: Can the author explain “Linear enhancing predictor” and “No SSL augmentation in the enhanced views” in the ablation table. Those lack a clear description in the text and it's not included in Algorithm 1. \n- 6.\tAdditional ablation studies. \n  - a)\tThe authors should investigate whether the learned representation is indeed equivariant/invariant. For example, computing norm-differences between the features. This would verify, empirically, whether the desired features are learned.\n  - b)\tI would like to see an ablation on the “crop with size and scale” for the different prediction views. I understand that it is used to save memory; however, I wonder if it has an impact on performance. For example, in Fig. 3 the crop nicely centers on the object and removes background. Please conduct an experiment without the crop/resize, to make it feasible for the memory, just uniformly sample the rotation during training, i.e., an unbiased estimate of Eq. 2.\n\n# C. Misc. \n- Maybe move Alg. 1 and Table 3 to the top of the page; I think this would make the paper neater.\n- Might consider citing the following related works:\n  - Misra, Ishan, C. Lawrence Zitnick, and Martial Hebert. \"Shuffle and learn: unsupervised learning using temporal order verification.\" Proc. ECCV. 2016\n  - Lee, Hsin-Ying, et al. \"Unsupervised representation learning by sorting sequences.\" Proc. ICCV. 2017 \n  - Mundhenk, T. Nathan, Daniel Ho, and Barry Y. Chen. \"Improvements to context based self-supervised learning.\" Proc. CVPR, 2018.\n", "summary_of_the_paper": "This paper proposes a framework which generalizes self-supervised learning (SSL) to also learn equivariance behavior. A family of SSL (the authors named invariant SSL) encourages representations which learn features that are invariant to certain transformations, e.g., horizontal flips. As invariance is a special case of equivariance, this paper explores whether encouraging equivariant representation is beneficial. Specifically, they focused on four-fold rotations and showed that combined with existing SSL methods lead to improvement in classification performance on CIFAR10 and ImageNet. The author also showcases application to photonics science.", "summary_of_the_review": "My main concern is with the writing and novelty. The paper proposes a general framework. However, it is written as a combination of invariant SSL + four-fold rotation; see additional suggestions in weaknesses section. The authors should better highlight their novelty and differences from existing work. Hence, I recommend a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dYO3K_HQ9D", "reviewer": "Reviewer_tezr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Paper is well written and easy to follow\n* The proposed method is simple (adding an additional loss) and shows some encouraging results on different methods (SimCLR, SimSiam, and Barlow Twins) on two image datasets CIFAR-10 and ImageNet\n- I find the results on ImageNet a little weak though, because it shows only results after 100 epoch of training. Longer training can change results (see https://arxiv.org/pdf/2011.10566.pdf - Table-4).\n- The main weakness of the paper is that although it claims E-SSL as a more general method (which I would assume would replace I-SSL), the actual method boils down to adding an additional loss (which is not novel per se). Basically, the claim is stronger than the actual method/implementation.\n", "summary_of_the_paper": "The paper advocates Equivariance Self-Supervised Learning (E-SSL) as a more general framework than the Invariance SSL (I-SSL), which is common in SoTA vision SSL methods, such as SimCLR and Barlow Twins. The paper starts with empirical evidence (Fig-1) that adding equivariance objectives (e.g. 4-fold rotation and vertical flips) to SimCLR can improve performance. The proposed E-SSL framework boils down to adding an additional equivariance objective (mainly 4-fold rotations) to popular I-SSL methods (Eqn. 1 & 2). The empirical results show encouraging results in CIFAR-10 and ImageNet. Finally, the paper applies the proposed method in two datasets of photonic data (regression task from 2D square periodic unit cells).", "summary_of_the_review": "The paper claims proposing a general E-SSL framework, but ends up adding an additional loss from prior work (four-fold rotation prediction) to popular SSL methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V6vZDIcdpuh", "reviewer": "Reviewer_9Qn8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n-\tI am familiar with idea of learning invariant representations and have seen it work in practice.  The idea to learn sensitivity using equivariance seems like logical and useful extension to me.   Moreover, I think the introduction is very nicely written and clearly explains the intuition of the idea.  \n-\tThe experiments in figure 1 lays out a clear roadmap for others to follow.  One may explore which transformations it may be beneficial to be insensitive or sensitive too.  There is a clear logical inverse correlation in which if it is hurts performance for features to be invariant to a transformation, it is likely to be beneficial to be sensitive to them.\n-\tThe further experiments on CIFAR-10, ImageNet, and Photonic Crystals show a clear benefit to the method.  In particular, I like the combination of standard benchmark datasets and good baselines as well as interesting application domains where the proposed method may be truly be useful.\n  \n### Weakness/Limitations/Questions\n\n-\tThe biggest question and issue for me is the following assumption: the distribution/dataset is assumed to contain only one element of each group orbit.  One can see this assumption is implicit in eqn. (2).  To the authors credit, it is spelled out explicitly in Proposition 1, but I think it is down-played a bit, when it should probably be explored more.  Taking the example of 90 degree rotations, the assumption is that every image has a correctly inferable orientation.  This is, in fact, what the loss of eqn. (2) is learning.  So if both a datapoint x and a datapoint rot(x) exist in the dataset, the model cannot learn eqn (2) correctly for both.   \nThis assumption may hold for Imagenet which contains natural photos which have a right side up, but many types of computer vision tasks do not have such features: satellite photos, xrays, microscope images, astronomy images, etc.  In fact, domains in which orientation is arbitrary are exactly the domains in which equivariance has been most relevant in DL.  I believe the current implementation of eqn. (2) makes such domains inaccessible for the current method.   \n\n-\tIn particular, it seems the PhC dataset is of this kind (though I may not have understood correctly).  The authors say DOS is rotation invariant. Thus intuitively it seems an invariant representation would be better.  Thus it’s surprising to me a rotation sensitive representation works better.  \nIt may be sensitivity to rotation only helps not due to the domain or task, but due to a bias in the creation of the dataset in which some implicit “correct orientation” is enforced.  Perhaps if the dataset contains many rotated views of each sample this issue would go away and some other form of representation learning would work better?  I’d welcome more explanation here.   \n  \n-\tThe authors cite several related works which also learn features which are sensitive to translations.  How close is their method to these.  Do they also invoke equivariance? That is, I’m not sure whether equivariant semi-supervised learning (with faithful representations) is new to this paper or whether the contribution of this paper is the combination of equivariance to some transformations and invariance to others.  Please clarify if possible.\n\n-\tIn table 3, it’s a bit hard for me to interpret the numbers.  Due to the normalization, values like 3.0 or 1.0 seem like very large errors and the difference between the methods seems rather small.   In that case, these results here are not that strong.  Do you have qualitative results?  I want to know it’s not the case that all models are simply failing at the task. \n\n\n### Small Notes/Further Questions:\n-\tMapping Sensative/Insensative onto the regular represention/trivial representation makes for an extreme dichotomy.  One might also consider representations between the regular and trivial rep.  In fact, this is exactly what the authors do in considering a quotient rep C_4/C_2 for the cyclic group in the phototonic-crystal experiment.  \n-\tCan you clarify the ablated E-SSL models: “Linear enhancing predictor” and “No SSL augmentation in the enhanced views.”  I couldn’t find the description in the text. \n", "summary_of_the_paper": "This work presents a framework for learning representations with invariances (insensitivity) to some transformations and sensitivities to others.  Previous work has only considered representations with insensitivity or sensitivity but not a combination.  The authors use the concept of invariance or equivariance to symmetry group transformations to learn such representations.  The advantage of the proposed method is demonstrated relative to SimCLR and SimSiam on Cifar-10, ImageNet, and a new scientific application domain: learning frequency responses of photonic crystals.  The authors also prove a theoretical characterization of when their methods works.", "summary_of_the_review": "I quite like the idea and experiments, but I have some reservations related to my questions.  Depending on the response, I’d be happy to raise my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an extra loss to add on top of the contrastive learning. The contrastive learning seek representations invariant to transformation, while the extra loss the authors proposed encourage representations to be equivariant to the transformation (i.e. retain information about transformation in later representations). While reviewers and I agree this is a sensible motivation, and acknowledge good results that authors have obtained, the fact that most, if not all, improvement is combing from the 4-way rotation transformation is a bit unsatisfactory. Furthermore, this additional loss was proposed before and is actually quite well known, so the actual novelty in the proposed technique is somewhat limited. Nevertheless, this paper provides a comprehensive evaluation, obtaining a reasonable improvement, and makes a good case for using an equivariant seeking loss. The authors are strongly encouraged to release their code (including training details for reproducing ImageNet results) as the improvements they present are central to the acceptance.
Decision: Accept (Poster)