Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an equivariance regularizer as a modification to the usual invariance-inducing self-supervised losses. This is an interesting approach to enabling equivariance as there is no need to have a special architecture as prior work. The authors are able to train multiple self-supervised losses on a standard ResNet-50 and show good linear probe improvements over invariant baselines. 
strength_and_weaknesses: Strengths:

1. A novel approach to enforcing equivariance is presented. The regularizer and modification to regular architectures is simple and effective.

2. The performance of the technique is shown on multiple losses and 

Weaknesses/Questions:

1. In Eq. (4) why is j not shown on the numerator? 

2. Is the only difference between the equivariance projection head and equivariant predictor the fact that the augmentation parameters are fed into the predictor? Otherwise equivariance projection head and predictor could have just been merged? Is equivariance projection head used elsewhere?

3. In Figure 3 and 4, what is the value of these measures for a regularly trained (invariant) model? This will tell us the benefit of adding the regularizer. 

4. What is the reason that equivariance is stronger for color compared to other augmentations for ImageNet? Any insight into this behavior?

5. Did you try any finetuning experiments?
clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow, clearly written, and the method is novel. I imagine that it will be easy to reproduce given the details in the paper.
summary_of_the_review: Given the overall novelty and clean/simple idea which seems to work well, I recommend acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presented a new module to improve self-supervised contrastive visual representation learning. Specifically, the proposed module focused on equivariance in the leaned latent space. Experimental analysis on two public datasets showed that when applying the proposed module to existing methods (SimCLR, BYOL and Barlow Twins) the performance was improved. The main contribution of this paper is the proposed equivariance module.
strength_and_weaknesses: **Strengths**

\+ The idea of equivariance in the feature space to help self-supervised representation learning is interesting.

\+ The proposed method was shown to be effective on linear evaluation (Table 1), by showing a performance gain when added to some prior works.


**Weaknesses**

\- The statement "These methods rely on maximizing the similarity between..." in the Abstract is inaccurate. Not all self-supervised visual representation learning methods are based on such a contrastive learning scheme. Instead, most of the early methods are based on pretext tasks. Suggest revising the corresponding claims and also the title.

\- The main idea of this work is based on the assumption of the self-supervised visual learning method being contrastive-based ones. Whereas there are quite a few non-contrastive learning approaches. As a result, the main contributions of this work may be a bit limited.

\- It is unclear how the defined u_\psi was guaranteed to represent the transformation t. From the description, t was encoded by another set of layers, which are also learnable, making the equivariance here (esp. the "equivariant predictor") a bit misleading.
The encoding of the transformations (Sec. 2.3.2) is rather confusing. In the beginning, it looks like the authors used one-hot encoding, but later some of the augmentations directly use the coordinates or real values (i.e. 1, 2, 3). The generalization for the augmentation set, as a result, is another issue. What if new augmentations were included? How would they be encoded?

\- It is unclear why the final model still needs the conventional invariance loss (i.e. the first term in the final loss function), if the proposed new equivariance-based loss is as claimed to be effective. Jointly optimizing both invariance and equivariance also seems to be confusing. How does the model actually learn in this case?

\- It is unclear why the Barlow Twins was only evaluated on the small-scale dataset CIFAR10.

\- The quality of the learned representation was only evaluated on the linear evaluation setting, which is a bit insufficient to get a clear conclusion. There are quite a few other downstream tasks that could be used as reported in the literature (e.g. fine-tuning, detection, and segmentation to name a few).

\- The proposed method was motivated by the case where "augmentation information is useful" as claimed by the authors, and example cases are flowers and birds (as claimed in the Introduction). But this was not validated in the experiment. There are some fine-grained datasets for such categories (e.g. the Oxford 102 flowers dataset and the Caltech-UCSD Birds-200 dataset) that should have been used to validate the claims.

\- It is unclear why not use an objective similar to Eq. (5) or (6) to constrain the equivariance model training, but instead only maximize the similarity between z_i' and \hat{z}_i'?

\- Would an absolute value makes more sense for Eq (5)? Otherwise, what does a negative value mean, as the second case "H-flip" shown in Fig. 3?

\- Line below Eq. (5), "Fig. 4" should be "Fig. 3".
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of the writing are generally okay, but need improvement. The idea is relatively new to self-supervised learning, but some key components were not well clarified with sufficient details and justification. The general idea is not hard to reproduce, but it may not be easy to reproduce the exact same architecture/results due to the lack of some technical details (e.g. the encoding scheme and the detailed network architectures).
summary_of_the_review: The main idea of this work about the equivariance in latent space is interesting and may have the potential to explore and contribute to the community. The shown experiments also suggested the effectiveness of the proposed method to some extent. But some technical design was not clearly presented without sufficient evidence to back them up. There are also several unclear claims and statements. The experiments are also insufficient to validate the claims. As a result, I would suggest the authors revise their paper accordingly for a future submission.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Proposes a modification of the data-augmentation approach to self-supervised learning where the parameters of the transformation used in data-augmentation can be used by the network to predict the latent variables of the un-augmented input.  This is advantageous when there exists a set of input features (such as color histogram) which would potentially cause the network to converge to bad local minima (shortcut solutions), yet could still be useful for classification.  Previous approaches would use data augmentations to enforce invariance to those features, which avoids the bad local minima but at the cost of losing the ability to use those features for classification.  Equimod therefore improves performance by combining a loss that encourages invariance to data augmentations and a loss that encourages the learning of a representation that is equivariant to data augmentations.  This is hypothesized to allow the network to avoid bad local minima without becoming totally invariant to useful features that are perturbed by data augmentations.  Experimental results show improved performance of SSL methods by modifying them to include the Equimod sub-network for predicting the effect of data augmentations on the latent space.
strength_and_weaknesses: Strengths
+ Analyzes the key ideas behind data augmentation and identifies a key shortcoming of data augmentation.
+ Proposes an innovative, elegant, and general solution to overcome the identified problem of data augmentation.
+ Concept is carefully explained with aid of diagrams and appropriate mathematical notation.
+ Evaluation shows meaningful improvements achieved using method.
+ Evaluation includes an experiment that shows that Equimod indeed learns equivariances, not just invariances.
+ Evaluation includes ablation study of architectures.

Weaknesses
- Evaluations could be made stronger.  Training for only 90 or 100 iterations is too short; methods should be training until convergence for a fair comparison with and without Equimod. [addressed by rebuttal]
- Intuition for Equimod loss (eq 4) not fully explained.  How does the loss differ in the case of a positive pair vs a negative pair?  And how are indices i and j sampled (e..g uniformly at random)? [addressed by rebuttal]
- Not clear how to tune the scaling of the Equimod loss.  The paper could be improved if the choice of lambda could be explored.  How is lambda set for the experiments?  Could an ablation on lambda (setting to different values and seeing the performance) be conducted? [addressed by rebuttal]
- Relatedly, the theory for how well Equimod might work to avoid "shortcut solutions" and optimally trade off is totally heuristic at the moment.  We see that Equimod works, but the paper doesn't give us a clear idea of the details of how that is possible theoretically.  Perhaps a theoretical analysis of a toy model would help shed some light on the success of the method.
- What is the computational cost of Equimod? [addressed by rebuttal]
- Will the code be shared? [addressed by rebuttal]
clarity,_quality,_novelty_and_reproducibility: Quality

I am very impressed by the ideas and the writing of this paper.  The method is simple and well-motivated, and successfully makes use of the concept of equivariance within the context of SSL.  The evaluations address many aspects of the method (influence of architecture, which transformations are actually learned) that a potential use would find relevant.

Clarity

The motivation and details of the method are clearly presented.  The conceptual figures 1 and 2 also aid in grasping the method quickly.


Originality

The ideas of the paper are strikingly original.  As noted, some other works have made use of equivariance before, but the elegance of this approach surpasses them.
summary_of_the_review: The paper presents a very interesting method for the field of SSL, and after rebuttal, provides more clarity and insight into the proposed method.  While a better evaluation (e.g. on problems such as flowers or trees) might allow the work to have a greater impact, I  think in its present form it will be useful for SSL researchers.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The authors introduce a generic equivariance module that structures the learned latent space and show that is is directly applicable to state-of-the-art invariance models, such as SimCLR and BYOL, and that it increases performance on the CIFAR10 and ImageNet datasets. 
The paper is also well written and easy to follow.
Although one reviewer scoring low there has been a very good response, in my opinion, from the authors and no response or acknowledgment thereof from the reviewer which is interpreted as a positive response in this case.
I therefore am pleased to accept this work for publication.

justification_for_why_not_higher_score: There is still the outlier reviewer and I am not certain about it being at spotlight level, however happy to reconsider.
Decision: Accept: poster