Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to learn equivariant representations that are steerable by a learned function in representation space. They empirically demonstrate the method as an improvement to ImageNet classification, where the pre-trained model performs better in transfer, retrieval, corruption, OOD detection settings.
strength_and_weaknesses: Strength

1. The idea is interesting and intuitive. Experiments are convincing.
2. The retrieval visualization is compelling. It clearly shows the property of the feature space learned by the proposed method compared with invariant learning.
3. The test-time augmentation is a useful application.

Weakness
1. The scope is limited to supervised ImageNet classification, while the self-supervised setting can be more appealing. As the paper claims, the approach “is general and easily extends to self-supervised settings”, it is relatively straightforward to try this in a contrastive learning setting (e.g., just remove the supervised loss)? If there was an issue with the contrastive learning setting, could the authors discuss it?
2. How does this method compare with the inverse-prediction style method as in Xiao et al., 2021, Dangovski et al., 2022? Which is more effective? I feel their methods can also be easily extended to the supervised setting -- just attach a loss with an inverse predictor.
3. How to determine alpha and beta in the loss? I have earlier noticed that increasing uniformity alone (alpha=0, beta>0) can sometimes improve transfer and robustness.

Minor: The citation should use \citet instead of \citep in some places.

clarity,_quality,_novelty_and_reproducibility: I think the writing is very clear, clarity and quality are good. The originality is modest. I don’t think there is any reproducibility issue.
summary_of_the_review: The paper demonstrates an effective way to improve equivariance of (classification) models using an augmentation-parameterized predictor. The method is intuitive and promising, but there are a few issues to be resolved before acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for learning equivariant representations with respect to photometric/geometric transformations, which has received little attention in the equivariant learning literature. The representations are also designed to be steerable so that the transformations can be directly applied in the latent space. The implementation of the method is simple and intuitive in that a mapping between two embeddings is introduced given an augmentation and regularized to be equivariant by a loss function. Since the augmentations are performed in the latent space through the mapping function, it brings significant speedup (nearly 50x) for test-time augmentations, and the equivariance of the mapping facilitates an interesting application for image retrieval, which works with image augmentations. At last, the paper demonstrates the method is beneficial to transfer learning and out-of-detection tasks.
strength_and_weaknesses: This paper has strength in novelty since it might be the first work incorporating transformations other than matrix groups for equivariant learning. In addition, it has shown the steerability and equivariance of the representations are highly practical and effective in several tasks and applications. However, the exposition of the proposed method still lacks in the following aspects:

1) The statement “Our method is generalizable across architectures and pre-training losses.” on page 2 has not been proven empirically, as the authors only employ ResNet and cross-entropy loss for experiments.

2) The authors, on page 3, claim the proposed method is widely applicable as long as a transformation is parameterized in input space and this covers the specific transformations, i.e., rotation, that the prior work mainly concentrates on. But I am not convinced about the generalizability of the method to those specific transformations and ask for evidence.

3) The proposed method applies the same augmentation for invariant and equivariant learning, and the uniformity regularizer is proposed to resolve this conflict. But I could not find any ablation studies for this regularizer and sensitivity analysis for hyperparameters \alpha and \beta. In addition, I want to ask why the independent views for equivariance and uniformity losses are used, making them inconsistent with Eq. (5). Please answer these points with experimental results.

4) The ρ value for the equivariant model in Table 1 seems to be still large as the model is trained with exactly the same loss function as the measurement. Can you provide the curve for the numerator (equivariance metric), the denominator (invariance metric), and the value itself separately over the course of training?

5) In Figure 5, the order of retrieved images for e(g(x)) and M(e(x)) are opposite to each other, which is supposed to be the same in general as in Figure 14. What is the reason for this one? In some figures, the right parenthesis is missing.

6) Table 3 is not referred to in any part of the paper, and the values are not the same as that of Table 4. Please explain the difference between them. 

7) For OOD in Figure 6, I want to see the results further when input data is explicitly forwarded with 60 augmentations for both invariant and equivariant models to identify the upper bound though the time is consumed significantly.

clarity,_quality,_novelty_and_reproducibility: The writing is clear, but I would recommend making a compact description for notations (explanation for Eq. (1)) and equations (does not seem to need two separate equations in Eq. (5)). Both the novelty and reproducibility are good as the proposed method provides a new direction for equivariant learning and is also simple to implement.
summary_of_the_review: This work is novel in its idea and direction, but it significantly lacks investigations on the method itself, as I commented on the weaknesses. 
Since the authors' response addresses some of my concerns, I raise my rating, leaning toward weak accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes steerable equivariant embedding for representation learning. Unlike invariant embedding typically achieved through learning with data augmentation, the model jointly learns a feature embedding and a (NN-parametrized) latent space transformation. Equivariant embedding is achieved by minimizing the cross-entropy loss regularized by the equivariance-promoting regularizer and the uniformity regularizer.
strength_and_weaknesses: **Strength**
1. The paper is generally well-written.
2. Extensive experiments have been conducted.

**Weakness**
1. Equation (2) is misleading. Instead of $g_a(x, \theta_a)$ and $M_a(e(x), \theta_a)$, the notations $g(x, \theta_a)$ and $M(e(x), \theta_a)$ should be used, since, if I understand correctly, the mapping $M$ and $g$ depends on $a$ only through $\theta_a$.
2. Instead of using $L_U$ and $L_E$ as two regularization terms, why not use $\rho_a$ directly? The authors mentioned that pushing $e(x)$ and $e(g_a(x))$ apart destabilizes training. However, isn't pushing them apart what the authors aim to achieve?
3. Equation (4) and Equation (5) are a bit confusing. What are $x_i$ and $x_j$ in the definition of $L_U(x)$? Is $x$ a batch of data containing $x_i$ and $x_j$?
4. What happens if the number of transformations $a$ is large in equation (5)? For example, consider the cyclical group $C_8$; does that mean the loss $L_E$ will be the sum of eight terms?
5. What is SimCLR at the bottom of page 5?
6. The comparison between the invariant and the proposed equivariant embedding (especially in Section 4.1, table 1) seems unfair and somewhat pointless. After all, the invariant embedding is trained (through augmentation) to be invariant, and thus freezing the encoder while training $M$ with equation (3) will essentially learn an identity map. This surely leads to a large $\rho_a$ defined in equation (2). What I am trying to say is that invariant embedding is indeed doing its job correctly, and it is pointless to compare it to the proposed model by forcing it to learn an $M$.
7. In section 4.2, how is $w_n$ decided? Again, the invariant model is doing its job correctly. From a different point of view, one might say that the invariant model in Figure 2 is robustly retrieving the same figure after input transformation, whereas the figures retrieved by the proposed embedding actually seem arbitrary.
8 What is $r_i$ in MRR? How is it calculated?
9. More explanation is needed for Figure 2, 3, 4, 5. For the first row $e(x)$, are they the retrieved image from the same query? Does it mean the authors have trained multiple embedding models?
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written. Learning an equivariant embedding is interesting, and jointly learning $M$ and $e$ is original.
summary_of_the_review: Even though the paper tries to address an interesting question, there are many issued identified in the *weakness* part of this review. There is also no theoretical contribution in the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to train representations equivariant to data augmentations by optimizing two additional regularizers together with the standard cross-entropy loss. Specifically, equivariance is promoted by minimizing the L2 loss between the embedding transformed by $M_a$ and embedding obtained by feeding augmented data, and uniformity is forced to reduce invariance. Experimental results show the effectiveness of the proposed method.
strength_and_weaknesses: Strengths

+ The idea of promoting equivariance to data augmentations seems interesting and useful.

Weaknesses

- Steerability is not inspired/explained well. Throughout the description on the method in Section 3, "steer" is never used, except when combining loss to argue that the final loss is to train a steerable model. What is steerability in deep representations, how does the proposed loss promote steerability, and how are steerability and equivariance related? It seems the authors do not distinguish steerability and equivariance well.

- There are prior works about training equivariant networks. How do the authors compare them with this work? There are many properties that models can be equivariant, and it seems this work is limited to learning to be equivariant to data augmentations only.

- How does the embedding map M look like? I couldn't find the actually experimented version throughout the paper.

- Experimental comparisons are limited to vanilla ResNet-50 training vs. proposed method. For example, I wonder how the proposed method is compared to existing methods for enhancing robustness/detecting out-of-distribution detection in Table 3. Given that the proposed method is orthogonal to them, it would be interesting to see if the combined method can improve the performance further.

- Experimental setting is limited to ImageNet-C. Experiments on standard OOD detection benchmarks might also be interesting.

- I couldn't find ablation studies on the choice/combination of hyperparameters/losses.

- Typo: 2nd and 3rd rows of Figure 1 have the same metric.

- Typo: $e(g_a(x)$ at the end of Section 3.3.
clarity,_quality,_novelty_and_reproducibility: Writing is clear and well-written in general, except for confusion between steerability and equivariance. Equivariant and/or steerable deep learning is not novel. The code is not shared but they claimed that the code will be released.
summary_of_the_review: The idea is interesting, but the experimental setting is quite limited to show the effectiveness of the proposed method. Also, I recommend authors distinguish steerability and equivariance in a better way. Please answer my concerns above.

**post-rebuttal**

The authors partially addressed my concerns, so my initial rating is unchanged. Below are additional comments:

- It is not clear if the mapping on the embedding space $M$ behaves as expected. How do you guarantee that nonlinear and complex data augmentations on the data space can always be simplified to be linear transformations on the embedding space (given that the actual implementation of $M$ is essentially the summation of the linearly transformed embedding vector and augmentation parameter vector)? I think more experimental analyses other than a few number of image retrieval results are required to make the results more convincing.

- Scalability on the size of model and dataset is still questionable; author response did not address this.

- Together with scalability, it would be interesting to see if hyperparameters ($\alpha$ and $\beta$) are sensitive to the choice on the model and dataset.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Paper Summary - The paper extends widely-used invariant representation learning to more general equivariant learning that is also steerable or controllable by a learned function in the representation space. This is achieved by optimizing additional loss functions (the equivariance-promoting regularizer and the uniformity regularizer) together with the standard cross-entropy loss, without the need of changing the model architecture. Equivariance is promoted by minimizing the L2 loss between the transformed embedding and the embedding obtained by feeding augmented data. The equivariance-promoting regularizer is claimed to be general, and mainly integrated into supervised pre-training. Photometric and geometric transformations are the main focus of the paper. Experimental results demonstrate improved representations for tasks such as image retrieval, robustness, transfer learning, and out-of-distribution detection.


Strengths and Weaknesses - While the reviewers recognize that the paper presents an interesting idea for equivariant representation learning with a simple, intuitive implementation, they also have various concerns regarding the original submission and the revised paper. First, a major issue is that the paper is claimed to be very general across architectures and pre-training losses in the original submission, but the actual empirical evaluation is of limited scope. The revision changes the tone a little bit. But the reviewers feel that the experiments are still unconvincing. Importantly, as the reviewers point out, while invariant representation learning is a central concept in self-supervised learning, this paper focuses on integrating the proposed equivariance-promoting loss in the supervised setting. It would be of broader impact if such a loss had been investigated in the self-supervised setting as well. Also, only the ResNet-50 architecture, cross-entropy loss, and specific transformations have been studied. Second, some important baselines are missing, such as the inverse-prediction style method (Xiao et al., 2021, Dangovski et al., 2022) as pointed out by the reviewers, making the comparisons unfair. Third, more in-depth analysis about the model behavior and scalability is missing.
justification_for_why_not_higher_score: The authors’ rebuttal clarified some confusion like the imprecise definition of steerability and hyper-parameter sensitivity, but failed to address the major concerns as mentioned above. So I cannot recommend acceptance based on the current version.
justification_for_why_not_lower_score: N/A.
Decision: Reject