Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Two contributions of this work are (1) an identifiability result for self-supervised learning (SSL), (2) a new synthetic dataset and empirical results evaluating the theoretical results and it’s comparison to standard SSL techniques with data augmentation. 

Theoretically, authors prove that using
(a) “anchor” samples $x$ from a generative model which generates data using (potentially dependent) content $c$ and style $s$ latent random factors through an invertible mapping $x=f(c,s)$, and 
(b) positive samples generated by the same content signal $c$ and an augmented style signal $\tilde{s}$ which is independent of $c$ given $s$,
one can learn a perfect encoder $\hat{c} = g(x)$ using a regularized SSL loss or by generative modeling, such that $\hat{c}$ contains all the information to perfectly predict $c$ in a statistical sense. This means that under some regularity conditions, SSL approaches using data augmentation can learn to isolate content factors (invariant to augmentation) from style factors (which can vary under augmentations). This identifiability result follows the recently popular line of similar results in non-linear ICA and representation learning thoery. However, different from previous results this result does not need independence of latent (content or style) random factors.

Next authors provide a new synthetic dataset to quantify the SSL representation learning. They quantify the performance of SimCLR contrastive (different from non-contrastive theory) SSL representation learning method on this dataset under their generative model and standard data augmentation methods. This showcase both the validity and the limitations of their theory and assumption.
main_review: ### Strengths
1. This is the first such identifiability result in SSL, which is able to explain the empirical success of SSL methods. This could lead to better framework to understand SSL methods which are currently poorly understood.
2. Although I couldn’t check the math very carefully, part of the analysis seems novel. These could be useful in future results.
3. Novel Benchmark dataset
4. Fair empirical study to validate their theory and assumption. Fair discussion of limitations of the work.
5. Good discussion to past relevant works.
6. Very clear exposition of the contributions. Provides most of the details of empirical study (see below).
7. Discussion of open theory questions.
8. Provides guidelines for selecting data augmentation. Also compares the usage of pre-final layer as downstream embedding (usage of poorly-understood “projection” MLP).

### Questions
1. No empirical comparison to state of the art contrastive methods like MoCo v2 and non-contrastive methods like SimSiam and Barlow Twins.
2. Although the theory is for non-contrastive SLL loss with regularization, the experiments use constrastive SimCLR loss. It not clear how (31) approximately match (5).
3. Table 1: What is “latent transformation (LT)”. It is not clear how this is done and what distribution was used for $P_A$ and $P(\tilde{s}|s)$?
4. line 313: What does "using CL to estimate the entropy term in (5)" mean?

### Other
1. Typo line 982: “Object hue is dependent on object class, 983 background hue, & object hue, see Tab. 4”
2. Might be good to use the color formatting used main text to the Appendix tables?

## Update
The author response and additional experiments reasonably address most of my concerns. As mentioned by reviewers ``8wZr`` and ``Lc8M``, one drawback for the current theory is that these guarantees do not provide any upper-bound for the complexity of the relation between content factors $c$ and the estimator content factors $\hat{c}$. Additionally, I would also like to note that these are not finite sample guarantees. However, one may overlook because of the paper’s novelty and the interesting insights it gives. I am keeping my score.
limitations_and_societal_impact: Good discussion of limitations of the work and some discussion of the negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper provides some soft theory towards explaining the effect of data augmentation on self-supervised learning. Specifically, data observations are generated from content information that is invariant to data augmentation and style variables that will be modified. The theoretical results stem from ICA and show that with a squared alignment loss or regularization one can identify the content variables.


main_review: 
It is very hard for me to identify the novelty of this paper. 


First, I think the overall message of the paper, that data augmentation only modifies style information, is intuitive and believable. However, it is not entirely new. 
The authors should have properly discussed and compared with a whole line of prior theoretical work on SSL (e.g. [1-6] and many more I have not listed), many of which share very similar conceptual messages as the paper. Whether it is named as 'content' as in this paper or latent variables in related papers are just a matter of writing style. 

~~The theoretical part is not that satisfactory. Unlike existing work, the authors only establish the conditions to identify the learned representation, but it doesn't necessarily explain why downstream tasks can be learned with simple functions (such as linear functions) from the representations. This is more essential for SSL. Otherwise, it's not clear to me why the theoretical results are specifically for explaining the success of SSL and what its distinction to existing ICA work is. 
Identifying an invariant representation has been extensively studied in prior work, as discussed by the authors, and I don't see how the theoretical results specifically apply to SSL.~~
==================
Post rebuttal: My original concern is that even if the paper proves that with infoNCE one can identify the content features, the relation between downstream task and the content feature still remains unclear and the sample complexity might still be high. This is why I thought that proving the identifiability alone was not sufficient to show the advantage of SSL. After reading the rebuttal and other reviews I realized this is not the focus of this paper. Proving that the style features being removed is quite meaningful itself, and it can be theoretical support on why the features learned by SSL will be useful for transfer learning. 
==================

The dataset is quite interesting, but I guess it might be more suitable for some causal relation work. I guess it is not the main contribution of the paper anyway, since it's not newly collected but comes from a preprocessing and combination of different existing datasets. 

To summarize, I think this paper did a lot of work but no result is good enough for the level of NeurIPS. Plus, significant prior work has been missing. 


[1] Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning, 2019

[2] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals
topic posterior information to linear models, 2020

[3] JD Lee, Q Lei, N Saunshi, J Zhuo. Predicting what you already know helps: Provable self-supervised learning, 2020

[4] Tosh, Christopher, Akshay Krishnamurthy, and Daniel Hsu. "Contrastive learning, multi-view redundancy, and linear models, 2020

[5] Tsai, Yao-Hung Hubert, et al. "Self-supervised learning from a multi-view perspective".

[6] Tian, Yuandong, Xinlei Chen, and Surya Ganguli. "Understanding self-supervised learning dynamics without contrastive pairs."
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies identifiability of the latent representation when samples are augmented in a way that preserves the "content" component of the latent variable while changing the "style" component. By focusing on an approximate (asymptotic) form of the InfoNCE cost function, the paper provides theoretical results on identifiability and experimental results on synthetic datasets, where style variables depend on content variables.
main_review: The paper has some novelty and the theoretical development appears correct. However, I think the issues listed below overweigh its advantages.

Major:
- There is somewhat of a disconnect between the theory and experiments, some of which are discussed at the end of the paper. I would like to point out an aspect that was not discussed and that appears to be responsible for some of the discrepancy between the theory and the experimental results: the theory considers the content as a continuous variable, but the proposed dataset and the associated experiment has discrete object classes.

- Baseline comparisons are missing, especially in the second experiment. The cited literature, where the latent factors may be assumed to not depend on each other, would provide meaningful baselines. 

- It would be very useful to include a non-synthetic dataset. Some common choices in the cited literature include MNIST, FashionMNIST, etc.

Minor: 
- line 289: f^-1 acts on x, why is the domain Z?
- Would the authors please explain why the negative L2 is used in one experiment and cosine similarity in the other experiment? I wonder if there is insight to be gained through these choices.
- The paper focuses on images as input data, which is fine, but there are components of the development that are agnostic to the underlying data. It could be useful, perhaps in the introduction only, to consider unstructured datasets as well.

--- Originality: The theoretical development is useful and novel.

--- Quality: The paper is well written.

--- Clarity: The expressions and explanations are clear. Proofs are detailed enough.

--- Significance: I think the disconnect mentioned above and the already discussed caveats in the experimental results significantly decrease the significance of the paper.

------------------------- UPDATE -------------------------
I decided to increase my score from 5 to 6, mostly in consideration of the authors’ positive and satisfying response on discrete vs. continuous content latents. Please see the discussion with the authors.

------------------------- UPDATE 2 -------------------------
I decided to further increase my score from 6 to 7, after the latest improvements the authors suggested and a re-evaluation of the paper overall.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the effect of data augmentation in self-supervised representation learning. They take a latent variable perspective where there is a content component and style component of the latent variable, and consider settings where the content is invariant to the augmentation whereas the style component changes a lot under the data augmentation. In the latent variable model, arbitrary dependencies between the content and style latent variables are allowed. 
 
The theoretical contribution of this paper is to show that, under certain assumptions on how the augmentation influences the latent variables, 1) a generative model for the data would perfectly identify the true content variables (up to an invertible function) and 2) minimizing the (entropy-regularized) squared difference between encoder outputs on two different augmentations of the data would also give an encoder which perfectly recovers the content variables (also up to an invertible function). The assumptions for these results are that the augmentations don't change the content latent variable and only the style latent variable, and that the augmentations can map the coordinates of the style latent variables to any value. 

This paper also provides experiments for verifying aspects of the theory, first on synthetic Gaussian data, and also in a semi-synthetic setting of rendered images of 3D objects where different latents control different aspects of the image. They compare the effect of using InfoNCE objective on pairs of examples obtained via data augmentation against the effect of using InfoNCE objective on pairs of examples obtained via directly modifying the style latent variables. They show that training InfoNCE with modifying the latent variables does indeed preserve a lot of content information while discarding style information. Data augmentation is observed to behave in a similar way in some instances, though the equivalence is not exact. 
main_review: This paper is generally well-written, and the perspective of analyzing data augmentation through the influence on the latent variables is novel and quite interesting. However, there are shortcomings of the results: 1) the theoretical results are for a quite idealized setting with strong assumptions, and appear to be somewhat brittle and may not translate well outside of the idealized scenarios and 2) the empirical results are not very convincing in terms of mapping the data augmentations to the latent variable transformations. More detailed comments below.

Originality: the perspective of using data augmentation to disentangle content and style is quite interesting and novel, to my best knowledge.  

Quality: The theorems are technically sound, but require strong assumptions to hold, and do not necessarily provide a complete picture on why data augmentation is useful in self-supervised learning. First, they require the augmentation to align perfectly with the style variables. In particular, Condition (iv) of Theorem 4.2 seems concerning because it requires the distribution on style change induced by the augmentation to be supported almost everywhere. This is likely unrealistic for data augmentations in practice. Second, in this result, there is no quantifiable bound on the complexity of the mapping between $\hat{z}$, the latent recovered in the identifiability result, and $z$, the true latent. 

Furthermore, Theorem 4.3 and 4.4 require knowing the exact dimension of the content variable to set the dimension of the encoder -- this seems quite restrictive. Do the authors have a sense of what happens if there is a dimension mismatch here? 

There also appears to be large mismatches between the theoretical model of data augmentation and what actually happens in practice in Table 1, though this is already addressed by the authors in Section 6. 

Clarity: The paper is generally well-written, and the theoretical results are clear and easy to follow. 

Significance: The idea of reasoning about data augmentation based on their effect on the latent space is potentially impactful and something for future work to build on. 

========================================================================================

Update post rebuttal: I'd like to thank the authors for the clarifications and answers to my questions. My overall opinion of the paper is still the same, so I'm keeping my score at 6.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The reviewers have reached a consensus that the paper is a good addition to the neurips. Please see the reviewers' discussion on the pros and cons for the paper. 
Decision: Accept (Poster)