Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

- Proposes new method to improve transformation invariance in contrastive representation learning and demonstrates utility on downstream tasks
- Proposes using feature averaging from multiple transformations at test time leading to further improvements 
- Introduces Spirograph dataset to explore the importance of learning feature invariances in the context of contrastive learning

Clarity
- Paper very well written and easy to follow. Figures supplement the text well
- Experiments include error bars to show statistical significance of results
- Supplementary material clarifies experimental setup and very comprehensive
- Perhaps consider changing “Self-supervized” -> “self-supervised”? 

Novelty/Significance
- New contrastive objective with gradient regularizer term to encourage transformation invariance and the Spirograph dataset are well motivated 
- Results over contrastive baselines suggest the contributions are important improvements to the contrastive training recipe 

Questions/Comments/Clarifications
- “Unfortunately, directly changing the similarity measure hampers the algorithm.” - Please add a citation to validate this claim. Is it coming from the experiments in SimCLRv1 - Chen et al, 202

- “However, there are many ways to maximize the InfoNCE objective without encouraging strong invariance in the encoder.” - Please add a citation to validate this claim

- Eq 9 and 10:  (Fe(α, β, x) − Fe(α’ , t, x))^2. Should t not be β?

- “It may be beneficial, however, to aggregate information from differently transformed versions of inputs to enforce invariance more directly” -> it is unclear why invariances need to enforced at test time if the learned representation is already invariant

- The authors point this out but the gradient regularization term is unfortunately encouraging invariance only to differentiable 
 transforms and this is a key limitation

- Worth pointing out for certain tasks likely near OOD detection, you may want to be transformation covariant rather than invariant. Would be interesting to see results of the proposed method on OOD detection benchmarks following https://arxiv.org/abs/2007.05566
- One major limitation is lack of baselines beyond vanilla contrastive training. For example, it would have been good to compare test time feature averaging with test time augmentation ensembling. Similarly instead of gradient regularization, the model could directly predict augmentation parameters and have the gradients of that loss penalized/ reversed. Adding such additional baselines, would solidify the improvements as best in class.
- It is also unclear how much train/test time compute model adds.

Overall, this is a very nicely written paper and very solid contribution. If the authors address my concerns, would be happy to increase my score.
review_title: a very well done paper proposing methods to improve transformation invariance in contrastive learning with strong empirical results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: 1. Summary
Given one image, the paper first generates different views which are controlled by differentiable parameter \alpha, and then minimizes the additional "conditional variance" term~(expectation of these views' squared differences). Therefore, the paper encourages representations of the same image remain similar under the augmentation. A testing strategy is further proposed by voting features with different augmentations. Results demonstrate the effectiveness.

2. Strength
    * The proposed "conditional variance" and its first-order approximation which is driven by a learnable augmentation control factor \alpha is a clear contribution，and makes it efficient to minimize the representations of different views of the same image. 
    * Results in Cifar and the new proposed dataset validate the effectiveness of the methods.

3. Weakness
    * Correct me if I were wrong. It seems more than half of the improvements come from the testing strategies. The claimed contribution of "feature averaging" during testing looks more like a testing time augmentation for me, although the paper has some differences in the augmentation details. I also think that the claim of SOTA is not rigorous if it has benefited from testing strategies.
    * It seems the proposed methods can be plugged into many recent studies. Thus more comprehensive experiments are needed to validate its generalization, e.g., combining with other baselines and conducting on ImageNet.

---------------------------
After reading the authors’ feedback and comments from other reviewers, I raised my score from 5 to 6.  I agree the feature averaging is another testing strategy.
review_title: Reasonable idea, while some of the contributions are not novel enough against existing works.

final_score: rating: 7: Good paper, accept
review_content: The paper focuses on a subtle different yet seemingly novel approach to learn invariance, i.e. instead of forcing the representations of two transformed inputs to be similar, it minimizes the gradient of the representation w.r.t to the transformation. This is indeed closer if not exactly the definition of tolerance or invariance to transformations is.

The problem is certainly an important one. The paper uses trick to to use a different representation of the conditional variance and first order approximation to arrive at a more computationally viable formulation. The formulation overall is interesting and described in sufficient detail. The paper does provide some relations to adversarial robustness as well, indeed it is quite intuitive to find connections between robustness and invariance.

The paper also provides validation experiments to show that indeed the conditional variance is reduced using this regularization.  However, there seems to be missing plots depicting this difference as a function of lambda. Is the method highly sensitive to lambda? or does a large range provide similar effects? If its the latter case, why so? what is preventing a large decrease in conditional variation given this loss augmentation?

Averaging across test samples does seem to help significantly. This is an important property in practice when there might be multiple transformations available that one needs to optimally utilize. This is one solution for such a case.

How would these ideas apply to more complicated nuisance transformations such as object pose?



review_title: More direct enforcement of invariance, with an effective test time strategy

Meta-review: The paper recieved three consistently positive reviews. While I agree with most of them, I have two major concerns regarding the novelty of the paper, which the authors are strongly recommended to address in the final version.

1. Taking derivative with respect to the parameters of transformation isn't novel. The standard tangent prop algorithm has been around for over a decade:

P. Simard, Y. LeCun, J. S. Denker, and B. Victorri. Transformation invariance in pattern recognition-tangent distance and tangent propagation. In Neural Networks: Tricks of the Trade. 1996.
(see Eq 26 in https://halshs.archives-ouvertes.fr/halshs-00009505/document)

Salah Rifai, Yann N. Dauphin, Pascal Vincent, Yoshua Bengio, Xavier Muller. The Manifold Tangent Classifier. NIPS 2011.
(see Eq 6 therein)

I understand there is some normalization in Eq 5, sampling of \alpha, \alpha', and the direction, and using the expectation to approximate the norm of the gradient. But such novelty is really incremental, or at least some empirical comparison will be necessary. It will also be necessary to cite the tangent distance/prop literature.

2. The new gradient based regularizer in Eq 11 and 12 appears completely decoupled with contrastive learning. It can be applied to any representation learning where f_\theta is an encoder. It does not use any substantial element from contrastive learning, although it might be "inspired" by contrastive learning. One may argue that such generality is an advantage, but 1) there is really no need to take such a big detour into contrastive learning just in order to derive the invariance regularizer in Eq 12, and 2) writing in this way can be quite confusing and/or misleading.
Decision: Accept (Poster)