Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a residual relaxation method to encode the semantic shift for the multi-view self-supervised learning. The method relaxes the exact alignment for the latent vector representation from different views. The experimental results show the method can improve the performance of the data augmentation.
main_review: 1. Originality: The work is a novel combination of the multi-view representation learning and the predictive learning, and proposes residual relaxation to improve the performance.

2. Quality: The work is technically sound. But, from the ablation study, the predictive learning term contributes more than the proposed residual relaxation technique.

3. Clarity: The paper is well written and organized. 

4. Significance: The result has referential significance. The other researchers could build on the idea of residual relaxation for other applications. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper focus on the representation learning using a relaxed alignment based on the fact that some strong data augmentations may hurt the performance. The authors introduced an adaptive residual vector and designed several novel objectives to make the residual vector useful and meaningful. Experiments shows a significant improvement over various backbones. 
main_review: Strengths:
- The paper is well organized and easy to follow. The proposed prelax method is well motivated and general. Prelax can be used for various data augmentations even there is some semantic shift, which is very important for real-world applications. 
- Prelax is also a unified approach to bridge the pretext-invariant features and pretext-aware features for representation learning. 
- The authors also provided the theoretical analysis based on the information-theoretic framework, which shows that Prelax provably enjoys better downstream performance.

Weakness:
- Prelax has three versions, Prelax-std, Prelax-rot and Prelax-all, then which one should be used when facing with an application?
- How to choose the hyperparameters alpha and gama?
- The difference between R2S and R3S is not clear.
limitations_and_societal_impact: No significant social risk
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the authors propose a new approach, named Pretext-aware Residual Relaxation (Prelex) to learning the representations from multiple views. The method mainly employs the residual relaxed similarity loss to improve the alignment between positive samples. 
main_review: Majors.
1 Motivation. Fig. 1 (a) shows the main motivation that rotation augmentation causes severe accuracy drop in a standard supervised model. In line 34, the authors claim the rotation augmentation carries meaningful semantics while being harmful to existing multi-view methods due to a large semantic shift. Thus, they propose a generic approach to address this dilemma. I have two questions. (1) I am curious whether the setting in Fig. 1 (a) is right or suitable to conclude the rotation is harmful. The setting in Fig. 1 (a) is that they train the model using two data augmentations (random crop and random horizontal flip in line 69 of the supple), then evaluate the model on images under four data augmentations, including horizontal flip, grayscale, color jitter and rotation. The drop of accuracy on rotated images seems reasonable since the trained model doesn’t ‘see’ such data? Maybe, these experimental results cannot support their motivation. (2) If the setting in Fig.1 (a) is right, the authors should try to indicate the effectiveness of their proposed method to deal with the dilemma on the same setting in Fig. 1 (a). I don’t find any results about it. 

2 Fig. 2 shows the network. But it needs more correlation with the equations in the manuscript and indications, the meaning of the symbols. 

3 Comparison with existing work [5] and [10]. The authors need to provide more comparisons to list the similarities and differences in a table format, including the key loss items, such as the residual relaxed similarity loss, predictive learning loss, and constraint on the similarity. The comparison is used to highlight the novel part of Prelax. After reading the whole manuscript, at the first time, I am confused that the proposed Prelax is an individual loss combination (in Section 3) or some additional parts used in SOTA [5][10] (such as the items, ‘SimSiam + Prelax’ or ‘BYOL+Prelax’ in Table 1) 

4 In section 4, they design three variants of Prelax. I want to know whether they investigate the dual order in Prelax-std, rot, all. For example, in eq (14), the order is $r_{12}$. How about the $r_{21}$. Also for eq (15,15,17,18,19,20).  

5 Experiment. The datasets used in this work include CIFAR-10, CIFAR-100, Tiny-ImageNet-200 and ImageNette, which might not support a wide range of benchmark datasets. The authors need to report the results on more downstream tasks as the table 5 in [5], such as detection and segmentation.  

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel Pretext-aware Residual Relaxation (Prelax) approach to self-supervised learning that does not require exact alignment of the two different views, which can be harmful for views created from some types of augmentations, but introduces a residual vector that encodes the semantic shift caused by the augmentation. To ensure pretext-awareness, a loss is introduced to predict the pretext task from the residual vector. A theoretical analysis is provided to show Prelax's advantage over unsupervised approaches that either only consider multi-view learning or only predictive learning. Experiments indicate that adding Prelax to common self-supervised methods such as SimSiam and BYOL improves performance.
main_review: Originality: 
The proposed method bridges the gap between the contrastive/similarity-based methods, which learn pretext-invariant features, and the predictive learning frameworks that learn pretext-aware features. While previous work have attempted to do this in an ad-hoc manner by directly combining multiple losses or by using separate feature spaces, the authors propose a well-motivated approach of encoding the pretext-awareness into a residual alignment vector. Building on the theoretical framework developed in [1], they further provide an information theoretic perspective, illustrating the benefits of the proposed approach. 

Quality: 
The paper is technically sound and claims are well-motivated. However, an analysis of the interdependence and effects of the multiple losses would have been interesting. For instance by having a closer look at the gradients and monitor the alignment/mismatch of the various terms (or comparing them to the gradient of a supervised loss). Further, do the authors observe negligible improvements or performance degradation when increasing the number of views further beyond three?

Clarity: 
The paper is overall easy to follow and the material is presented in a manner that should allow reproducibility.
- One statement, that could have been expanded a bit is the discussion of the results in the out-of-domain linear evaluation (2b). Do the authors have a hypothesis on why the Prelax-std/Prelax-rot perform slightly better than Prelax-all?
- The authors definition of multi-view seems a bit limited. Although multi-view representation learning includes the self-supervised techniques where additional views are created through augmentations, it commonly also includes settings where data is actually obtained from multiple views (different camera angles, or even different modalities).
- The claim that rotation causes severe accuracy drops is a bit strong and depends on the application domain. While it is a valid assumption for natural images, this is not the case in other domains. The authors might want to more explicitly include that this claim/work is addressing the case of natural images (although the method in itself is more general). 

Significance:
The proposed approach is a relative generic framework that can be combined with most existing methods and should motivate additional work on trying to develop methods that exploit both pretext-invariant and pretext-aware features explicitly. Results illustrate a sufficient boost over the baseline approaches that only focus on similarity-based methods or predictive learning.
limitations_and_societal_impact: The authors do not discuss the limitations of their proposed approach. While the reviewer agrees that the negative social impact should be limited due to its general methodology focus, there are certain limitations, such as the reliance on manually choosing the augmentations, which will be domain and dataset dependent.
review_title: null

Meta-review: This paper relaxes the alignment objective in multiview self-supervised learning when the data augmentation causes semantic shifts in different views. The proposed pretext-aware residual relaxation method allows an adaptive residual vector between different views. The proposed method can benefit from stronger image augmentations like rotation and outperform the existing methods. Based on a recent theoretical framework on self-supervised learning, the authors provide theoretical guarantees of the proposed method. The paper is well motivated, well written and has practical impact on downstream image classification tasks. Reviewers all agreed that the proposed solution is interesting and the improvement is significant. I am recommending acceptance of this paper.  The authors need to make sure they add the new results in the rebuttal to the final version.

Decision: Accept (Poster)