Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a simple regularization for self-supervised/supervised representation learning, which aims to benefit downstream tasks. Main idea is to make the network to be variant to different augmentation. Extensive experimental results are reported to show the effectiveness.
main_review: 1. The idea is simple and seems to provide good performance gains. However, I am curious what's the main learning objective of the main paper. The main loss tries to learn augmentation-invariant representation while the AugSelf tries to learn augmentation-variant representation. These two losses are contradicting. What do we want the network to learn? To be invariant or variant to augmentation. If some augmentations are bad for downstream tasks, such as color jittering and Gaussian blurring, can we just remove these negative augmentation in the main loss? Maybe just removing these augmentation is sufficient to bring large performance gains. Adding the regularization will make things complex. So I suggest to add ablation studies on this: main loss without color/blurring, and more importantly provide more insights about why your regularized method could work.

2. How can you determine λ for each augmentation? Can you provide the sensitivity analysis to these hyperparamters.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a self-supervision method with learning the difference of augmentation parameters as the pretext task to improve the transferability of learned representations. 
main_review: Strength:
+ The paper proposes a new self-supervision method to improve the transferability of learned representations. The method is simple and clear. 
+ The proposed method consistently improves the unsupervised learning methods. 

Weakness:
- There are no other sota representation learning methods, especially other self-supervision related methods with different design of pretext tasks for comparison. 


Question:
Does the augmentation help with the robustness of the learned representation under perturbations? 
limitations_and_societal_impact: I don't see any potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents AugSelf, an auxiliary self-supervised task in contrastive learning to improves the transferability of learned representations. The task predicts the difference of augmentation parameters between two randomly augmented samples, and therefore forces the representation to preserve the information which could be potentially discarded by augmentations. Extensive experiments demonstrate the effectiveness of the proposed method.

main_review: Originality: The observation that “learning representations with an augmentation-invariant objective might hurt its performance in downstream tasks” and the idea of preserving the information that is removed by the augmentations to improve transferability is already presented in [1]. However, the idea of adding an auxiliary task to preserve the information is novel and interesting.

Quality: 
The proposed method of predicting the augmentations to preserve information is novel and interesting. The method is simple and can be easily adapted to different contrastive learning framework and is applicable to a wide range of common quantifiable augmentations. The experiments and ablation studies are also extensive.

Minor points: 
1.	the observation that “learning representations with an augmentation-invariant objective might hurt its performance in downstream tasks” should not be claimed as part of the contribution since [1] already observed it.
2.	The checklist is not well formatted. The instructions are not deleted and there are still placeholders. Also, the error bars and standard deviation is missing, which could be >0.5 as shown in [1].

Clarity: The paper is well written and easy to follow.

Significance: the paper proposes a simple method to preserve information that could be potentially removed by the strong augmentations of contrastive learning. Although the idea of preserving the information removed by augmentations to improve transferability is not new, the method is simple and effective to preserve the information and its performance on multiple datasets are consistently good.

[1] Xiao, Tete, et al. "What should not be contrastive in contrastive learning." arXiv preprint arXiv:2008.05659 (2020).

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims at improving the transferability of representations, especially self-supervised pre-trained representations. Motivated by the observation that augmentation-invariance may be harmful to downstream tasks, e.g., location- or color-sensitive ones, they proposed an auxiliary augmentation-aware self-supervision loss named AugSelf. AugSelf learns the difference of augmentation parameters (e.g., cropping positions, color adjustment intensities) between two randomly augmented samples. Experiments show that AugSelf helps for transfer learning tasks with two pre-trained models and a supervised learning model, as well as few-shot learning setups.

main_review: Strongness:
+ This paper is well-written and easy to follow, making it enjoyable to read.
+ The proposed method is very simple, even somewhat intuitive, by introducing an auxiliary augmentation-aware self-supervision loss.
+ Experiments on several benchmarks demonstrate the effectiveness of the proposed method.

Weakness:
- My major concern is about the claim on improving the transferability of representations. It is reasonable to achieve better performance by introducing an auxiliary augmentation-aware self-supervision loss for location- or color-sensitive tasks, but why will AugSelf work for those tasks that are not augmentation-sensitive? If not, the proposed method is not a general method to improve transferability, instead, it captures dataset-specific information.
- Another major concern is the tradeoff between preserving augmentation-aware information in learned representations with the loss of augmentation-invariant information. If the model exhausts itself to capture augmentation-aware information, will it be harmful to augmentation-invariant information? Meanwhile, how to decide the value of the hyperparameter of $\lambda$ between these two losses.
- There are two protocols to evaluate the transferability of representations: linear evaluation and fine-tuning. It seems that many fine-tuning baselines focusing on improving transferability are missing, e.g. L2SP [1], DELTA [2], BSS [3], DELTA [4].

[1] Li, X., Grandvalet, Y., and Davoine, F. Explicit inductive bias for transfer learning with convolutional networks. In ICML, 2018.
[2] Li, X., Xiong, H., Wang, H., Rao, Y., Liu, L., and Huan, J. Delta: Deep learning transfer using feature map with attention for convolutional networks. In ICLR, 2019.
[3] Chen, X., Wang, S., Fu, B., Long, M., and Wang, J. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In NeurIPS, 2019.
[4] You, K., Kou, Z., Long, M., and Wang, J. Co-tuning for transfer learning. In NeurIPS, 2020.

- The question "when is learning invariance to a given set of augmentations harmful to representation learning?" mentioned in the Introduction section is unsolved. 
- Actually, the augmentation-aware loss is defined on the same specific augmentation, instead of the whole augmentation space. Will it be possible that the proposed method be extended into different types of augmentations? If so, which kind of different information will be learned?
- Why the improvement of the proposed methods on some datasets such as STL10 in Table 9 is minor? 
-Typos: Line 83 "In these methods, if h optimal, then..."



limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper aims to improve the transferability of representations via augmentation-aware self-supervision, by preserving both augmentation-invariant and augmentation-aware information. The main finding is that augmentation-variant information may be relevant to downstream tasks, which is interesting and inspiring for the self-supervised learning community. The rebuttal is informative and relevant, and after the discussion all reviewers unanimously recommended acceptance. Reviewers also pointed out some important directions for improvement, such as a larger set of augmentations for pre-training, more sophisticated fine-tuning methods, and a clear elaboration on the relationship between augmentation invariance and awareness. Authors shall guarantee to include the rebuttal material into their future draft.
Decision: Accept (Poster)