Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a simple solution to improve the existing self-supervised representation learning by adding an additional branch to predict the rotations. Experiments are conducted on CIFAR-10, ImageNet, and two PhC datasets and the results show that the model with the additional branch can outperform the corresponding baselines.
main_review: The idea is simple and straightforward, making it easy to understand. The idea makes sense to me and the experiments show good results.

It is not clear how the authors did in the introduction to encourage the model to be sensitive or not sensitive to a specific transformation. I would imagine that for encouraging insensitive, the authors did not do anything specific, and for encouraging it to be sensitive, the authors used an additional branch to predict just like the main framework. Still, it would be great if the authors could make it more clear.

I do not really buy the equivariance story. The story and the title are somewhat overclaiming. The authors bring the definition of equivariance in the introduction; however, can the authors be more specific what is the $T'_g$ in their framework? I do not see any post transformation on top of the model output on the newly added branch but just labels. The model did not really learn equivariance but used some auxiliary pretext tasks to help learn better representations.

It would be great to see how the model performs with longer training iteration on ImageNet-1K, to confirm the gain will not be diminished.
summary_of_the_review: Overall I think this paper has its own contribution. It finds that using an additional pretext header can improve the performance. However, I don't think the story itself is convincing enough and believe that might mislead readers. How the model performs with longer training iteration on ImageNet-1K also remains unclear. These prevent me from giving a higher rating.

--post rebuttal

After reading the authors response I raise my rating to 6.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work presents a framework for learning representations with invariances (insensitivity) to some transformations and sensitivities to others.  Previous work has only considered representations with insensitivity or sensitivity but not a combination.  The authors use the concept of invariance or equivariance to symmetry group transformations to learn such representations.  The advantage of the proposed method is demonstrated relative to SimCLR and SimSiam on Cifar-10, ImageNet, and a new scientific application domain: learning frequency responses of photonic crystals.  The authors also prove a theoretical characterization of when their methods works.
main_review: ### Strengths:
-	I am familiar with idea of learning invariant representations and have seen it work in practice.  The idea to learn sensitivity using equivariance seems like logical and useful extension to me.   Moreover, I think the introduction is very nicely written and clearly explains the intuition of the idea.  
-	The experiments in figure 1 lays out a clear roadmap for others to follow.  One may explore which transformations it may be beneficial to be insensitive or sensitive too.  There is a clear logical inverse correlation in which if it is hurts performance for features to be invariant to a transformation, it is likely to be beneficial to be sensitive to them.
-	The further experiments on CIFAR-10, ImageNet, and Photonic Crystals show a clear benefit to the method.  In particular, I like the combination of standard benchmark datasets and good baselines as well as interesting application domains where the proposed method may be truly be useful.
  
### Weakness/Limitations/Questions

-	The biggest question and issue for me is the following assumption: the distribution/dataset is assumed to contain only one element of each group orbit.  One can see this assumption is implicit in eqn. (2).  To the authors credit, it is spelled out explicitly in Proposition 1, but I think it is down-played a bit, when it should probably be explored more.  Taking the example of 90 degree rotations, the assumption is that every image has a correctly inferable orientation.  This is, in fact, what the loss of eqn. (2) is learning.  So if both a datapoint x and a datapoint rot(x) exist in the dataset, the model cannot learn eqn (2) correctly for both.   
This assumption may hold for Imagenet which contains natural photos which have a right side up, but many types of computer vision tasks do not have such features: satellite photos, xrays, microscope images, astronomy images, etc.  In fact, domains in which orientation is arbitrary are exactly the domains in which equivariance has been most relevant in DL.  I believe the current implementation of eqn. (2) makes such domains inaccessible for the current method.   

-	In particular, it seems the PhC dataset is of this kind (though I may not have understood correctly).  The authors say DOS is rotation invariant. Thus intuitively it seems an invariant representation would be better.  Thus it’s surprising to me a rotation sensitive representation works better.  
It may be sensitivity to rotation only helps not due to the domain or task, but due to a bias in the creation of the dataset in which some implicit “correct orientation” is enforced.  Perhaps if the dataset contains many rotated views of each sample this issue would go away and some other form of representation learning would work better?  I’d welcome more explanation here.   
  
-	The authors cite several related works which also learn features which are sensitive to translations.  How close is their method to these.  Do they also invoke equivariance? That is, I’m not sure whether equivariant semi-supervised learning (with faithful representations) is new to this paper or whether the contribution of this paper is the combination of equivariance to some transformations and invariance to others.  Please clarify if possible.

-	In table 3, it’s a bit hard for me to interpret the numbers.  Due to the normalization, values like 3.0 or 1.0 seem like very large errors and the difference between the methods seems rather small.   In that case, these results here are not that strong.  Do you have qualitative results?  I want to know it’s not the case that all models are simply failing at the task. 


### Small Notes/Further Questions:
-	Mapping Sensative/Insensative onto the regular represention/trivial representation makes for an extreme dichotomy.  One might also consider representations between the regular and trivial rep.  In fact, this is exactly what the authors do in considering a quotient rep C_4/C_2 for the cyclic group in the phototonic-crystal experiment.  
-	Can you clarify the ablated E-SSL models: “Linear enhancing predictor” and “No SSL augmentation in the enhanced views.”  I couldn’t find the description in the text. 

summary_of_the_review: I quite like the idea and experiments, but I have some reservations related to my questions.  Depending on the response, I’d be happy to raise my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper advocates Equivariance Self-Supervised Learning (E-SSL) as a more general framework than the Invariance SSL (I-SSL), which is common in SoTA vision SSL methods, such as SimCLR and Barlow Twins. The paper starts with empirical evidence (Fig-1) that adding equivariance objectives (e.g. 4-fold rotation and vertical flips) to SimCLR can improve performance. The proposed E-SSL framework boils down to adding an additional equivariance objective (mainly 4-fold rotations) to popular I-SSL methods (Eqn. 1 & 2). The empirical results show encouraging results in CIFAR-10 and ImageNet. Finally, the paper applies the proposed method in two datasets of photonic data (regression task from 2D square periodic unit cells).
main_review: * Paper is well written and easy to follow
* The proposed method is simple (adding an additional loss) and shows some encouraging results on different methods (SimCLR, SimSiam, and Barlow Twins) on two image datasets CIFAR-10 and ImageNet
- I find the results on ImageNet a little weak though, because it shows only results after 100 epoch of training. Longer training can change results (see https://arxiv.org/pdf/2011.10566.pdf - Table-4).
- The main weakness of the paper is that although it claims E-SSL as a more general method (which I would assume would replace I-SSL), the actual method boils down to adding an additional loss (which is not novel per se). Basically, the claim is stronger than the actual method/implementation.

summary_of_the_review: The paper claims proposing a general E-SSL framework, but ends up adding an additional loss from prior work (four-fold rotation prediction) to popular SSL methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework which generalizes self-supervised learning (SSL) to also learn equivariance behavior. A family of SSL (the authors named invariant SSL) encourages representations which learn features that are invariant to certain transformations, e.g., horizontal flips. As invariance is a special case of equivariance, this paper explores whether encouraging equivariant representation is beneficial. Specifically, they focused on four-fold rotations and showed that combined with existing SSL methods lead to improvement in classification performance on CIFAR10 and ImageNet. The author also showcases application to photonics science.
main_review: # A. Strengths
- 1.	The proposed approach achieves good performance gain over baselines on CIFAR10 and ImageNet.
- 2.	The proposed approach generalizes the existing invariant SSL approach to also include equivariance; the idea is intuitive to follow and reasonable.
- 3.	Author promised to release code and the supplemental materials reports the hyperparameters sweeps and results.

# B. Weaknesses
- 1.	Novelty & Contribution: The approach can be viewed as combining “predicting image rotation” (Gidaris et al. 2018) with existing invariant SSL. The authors should more clearly explain the differences in the related works section. Also, the work proposes a “general framework”, I find the writing which only focuses on four-fold rotation not adequate. I would recommend writing out the generalized formulation in more detail, e.g. using equivariance, and not just a very abstract Eq. 1.
- 2.	Limitation in approach: The proposed approach is limited to finite groups as it is based on classification. Has the author thought about further generalization? 
- 3.	Organization: The analysis in Sec. 5.2 seems out of place. Why not include it in the approach section? Also, personally, I feel the related works before the discussion does not help with the organization. I would recommend placing it after the introduction. 
- 4.	Clarity: The link between equivariance and the proposed approach is not clearly explained in the paper. As the authors have introduced the equivariance definition and notation, it might be beneficial to use those in their approach. Next, the drawing of Figure. 3 should be improved. For example, I cannot understand why for the equivariance there is only one backbone f; for the invariance there are two?. Also, having a single arrow going into “equivariance” is very confusing.  
- 5.	Ablations in Table 1: Can the author explain “Linear enhancing predictor” and “No SSL augmentation in the enhanced views” in the ablation table. Those lack a clear description in the text and it's not included in Algorithm 1. 
- 6.	Additional ablation studies. 
  - a)	The authors should investigate whether the learned representation is indeed equivariant/invariant. For example, computing norm-differences between the features. This would verify, empirically, whether the desired features are learned.
  - b)	I would like to see an ablation on the “crop with size and scale” for the different prediction views. I understand that it is used to save memory; however, I wonder if it has an impact on performance. For example, in Fig. 3 the crop nicely centers on the object and removes background. Please conduct an experiment without the crop/resize, to make it feasible for the memory, just uniformly sample the rotation during training, i.e., an unbiased estimate of Eq. 2.

# C. Misc. 
- Maybe move Alg. 1 and Table 3 to the top of the page; I think this would make the paper neater.
- Might consider citing the following related works:
  - Misra, Ishan, C. Lawrence Zitnick, and Martial Hebert. "Shuffle and learn: unsupervised learning using temporal order verification." Proc. ECCV. 2016
  - Lee, Hsin-Ying, et al. "Unsupervised representation learning by sorting sequences." Proc. ICCV. 2017 
  - Mundhenk, T. Nathan, Daniel Ho, and Barry Y. Chen. "Improvements to context based self-supervised learning." Proc. CVPR, 2018.

summary_of_the_review: My main concern is with the writing and novelty. The paper proposes a general framework. However, it is written as a combination of invariant SSL + four-fold rotation; see additional suggestions in weaknesses section. The authors should better highlight their novelty and differences from existing work. Hence, I recommend a weak reject.
review_title: null

Meta-review: This paper proposes an extra loss to add on top of the contrastive learning. The contrastive learning seek representations invariant to transformation, while the extra loss the authors proposed encourage representations to be equivariant to the transformation (i.e. retain information about transformation in later representations). While reviewers and I agree this is a sensible motivation, and acknowledge good results that authors have obtained, the fact that most, if not all, improvement is combing from the 4-way rotation transformation is a bit unsatisfactory. Furthermore, this additional loss was proposed before and is actually quite well known, so the actual novelty in the proposed technique is somewhat limited. Nevertheless, this paper provides a comprehensive evaluation, obtaining a reasonable improvement, and makes a good case for using an equivariant seeking loss. The authors are strongly encouraged to release their code (including training details for reproducing ImageNet results) as the improvements they present are central to the acceptance.
Decision: Accept (Poster)