Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary 

This paper takes a causal viewpoint on self-supervised contrastive representation learning. The data is modeled as being generated from two independent latent factors: style and content, where content captures all information necessary for downstream tasks, and style captures everything that is affected by training augmentations. The main contribution is a specific regularizer for self-supervised contrastive learning, motivated by the assumptions about the data generation. The learned representations are evaluated in terms of robustness, classification, and generalization performance on ImageNet and in terms of performance on the RL Atari benchmark. The proposed approach is shown to outperform even very recent competing approaches.

## Pros & Cons

+ The explicit expression of the assumptions behind the data generation process as a (causal) network. It explains the assumptions made for augmentations and proxy tasks to make sense; it also nicely subsumes previous formulations.
+ Visualization of the resulting feature space
+ Performance appears great, the evaluation seems sufficient enough.

- Proof of Theorem 1 has problems, fixing it, meant to change the causal graph. Although, this might be fixable (see questions below).
- There is in fact no causal language needed for this paper. There is no causal discovery or anything happening. It is really just: Here are my assumptions about the mechanisms of data generation and everything else follows just from statistical independence. For instance, the proposed invariance criterion can also be formulated as the distribution $p(Y_t \mid C)$ being invariant to distributional shifts of $p(S)$; and trying to gain robustness against covariate shifts is nothing new per se. Being explicit about the dependency between downstream tasks is important, but was not correctly stated in the paper. 

## Questions and concerns
- for intervening in a causal graph on a root note, I don't need the "do" notation. There is no need to cut the graph. So the statements are all trivial from this perspective.

- The proof of theorem 1 has problems, I think. Namely, after the first equal, you write $p^{do(s_i)}(Y_t \mid Y^R)$, which should be $p^{do(s_i)}(Y_t \mid Y^R, f(X))$ unless $Y_t \perp f(X) \mid Y^R$. In this case, you can not progress further:

$$p^{do(s_i)}(Y_t \mid Y^R, f(X)) = p(Y_t \mid Y^R, f(X), S=s_i) \neq p(Y_t \mid Y^R, f(X)) \neq p(Y_t \mid Y^R, f(X), S=s_j) = p^{do(s_j)}(Y_t \mid Y^R, f(X))$$

This is because $Y_t \not\perp S \mid f(X), Y^R$, so you can not drop the conditioning on $S$. 

However, if $Y_t \perp f(X) \mid Y^R$ holds, the proof would work again. This would be the case when $C \rightarrow Y^R \rightarrow Y_t$, i.e. the refinement task "causes" the downstream task. From the causal graph of Figure 1a, this is not the case. Although this would make sense intuitively, as the instance discrimination task needs more information than the downstream tasks, and you even state something in this direction in footnote 3. So I would ask the authors to clarify the intended causal connections between $C$, $Y^R$, and $Y_t$. 

- Again on the proof, it looks like the assumption that $Y^R$ is a refinement for all tasks in $Y$ is not even needed?
- Can you phrase the concept of "refinements" in terms of causality? What does it mean for task $Y^R$ to be a refinement of $Y_t$?. Although you state that you use the "causal concept of refinement", a causal explanation is not given as far as I can see.

- You change the definition of the $p^{do(a) }$ to one with two interventions $p^{do(a_{ik})}$. This suddenly appears below Eq 2. and is then used in what follows. To me, this looks like one of the most important contributions. Clearly, it does not follow from the causal graph or so directly. However, it is one way of defining the regularizer that is consistent with the framework and it seems to be important. So please improve the presentation and introduce it properly.

- How would you place downstream tasks like object detection or segmentation in your framework? I would imagine that learning for instance discrimination does not keep all the information necessary to solve these kinds of tasks.
 
## Suggestions and Comments
- I like that you make the modeling assumptions explicit
- Get rid of most of the causality jargon, and do not try to oversell your paper into the hyped field. Your paper is not doing causal inference or anything of the kind.
- Spend much more time on the regularizer and its definition with the 4 interventions that just fall from the sky. This is actually your contribution: defining the regularizer in the way you do it. A comparison to a simple baseline with just the X-entropy between two interventions would be good (or is that one of the baseline methods?).
- your style factors are also called nuisance factors in the literature
- consider making it more prominent in the paper that you solve the instance discrimination task
- the causal graph presented (in figure 1a: the meaning of arrows is not clearly defined)
- if the arrows are causal than it seems to contradict the refinement idea 
- Fig 1b: make the 4 interventions more explicitly visible. Add the Bear picture to the top and bottom row

- also, it would be beneficial to see the experiments that support and visualize theorem 1 (e.g. to show that the KL for other tasks is also decreasing when we decrease the KL for instance discrimination task).

- Typos: signal -> signals (line 2 abstract), 
- Theorem 1: probl also a quantifier for $t$ is needed

- Page 5: "and so the left hand side of 4" <- this should probably be "the right hand side of 4"

Overall: the paper has great results and can be a valuable contribution, but it has problems in the formulation and is overselling on being causal. 

Update: Thanks for fixing the statement of the assumptions such that Theorem 1 can hold.   Update 4 -> 5. Some of my concerns are still not addressed in the revised version.

review_title: Strong results, but problems in the formulation

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a framework for self-supervised representation learning using causality. The proposed model is formulated by assuming that the Data generation schema is composed of two independent mechanisms (ie., Style and Content) and only content is relevant for learning the underlying task. Thus, the Content is a good representation of the data and the goal of representation learning could be cast as a content estimation. Then, the authors use interventions on the Style (i.e., data augmentation in their formulation) to learn invariant representation under data augmentation (Style variable). To achieve this invariant prediction they propose a new constructive objective (ReLIC). 

The paper is well written and easy to follow. My only concern is that the whole proposal relies on the assumption that Data generation is composed of two independent mechanisms (S and C) and the authors utilize various data augmentations as interventions on the Style variable S as they don’t have access to S. However, no details are given for the impact of the used data augmentation techniques on the learning better representations. 

review_title: Interesting ideas,  good problem formulation and results

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a new understanding of self-supervised learning from a causal perspective. Specifically, a causal graph with style and content is assumed for the generating process of the inputs, such as images. Another assumption is that the down-stream tasks only rely on the content variable. By making use of the independent causal mechanism, the authors propose a new invariance regularization term, which is achieves good performance on several real datasets. Also, a new understanding of contrastive learning is provided.

Strength

The understanding of self-supervised learning from a causal perspective is novel. The causal generative model assumed in this paper seems to be reasonable in many real scenarios. Especially, the image data were generated from content and style and usually the downstream tasks such as object recognition depends on the content.

The experimental results on Imagenet and Atari demonstrate the effectiveness of the method.

Weakness

The idea of independence mechanisms was originally proposed in [1] and has deep connections to the modularity of a causal system and the concept of exogeneity in economics (Pearl, 2009). In specific, given two variables C and E, we say C is exogenous if P(E|C) remains invariant to changes in the process that generates C. In [1], the independence mechanism is defined as follows:
 “We finally assume that the mechanism is “independent” of the distribution of the cause in the sense that P(E|C) contains no information about P(C) and vice versa; in particular, if P(E|C) changes at some point in time, there is no reason to believe that P(C) changes at the same time.” When P(c) and P(E|C) both change, they change independently of each other [2]. It would be better if the authors could explore the literature a little bit more and add corresponding discussions. 
[1] Schölkopf, Bernhard, et al. "On causal and anticausal learning." Proceedings of the 29th International Coference on International Conference on Machine Learning. 2012.
[2] Huang, Biwei, et al. "Causal discovery from heterogeneous/nonstationary data." Journal of Machine Learning Research 21.89 (2020): 1-53.

The proposed invariance regularization is closely related to the consistency regularization [3] in semi-supervised learning. The relation and difference to consistency regularization needs to be discussed.
[3] Sajjadi, Mehdi, Mehran Javanmardi, and Tolga Tasdizen. "Regularization with stochastic transformations and perturbations for deep semi-supervised learning." Advances in neural information processing systems. 2016.

The refinement seems counterintuitive to me. The authors define a refinement of one problem as another more fine-grained problem. For a downstream task, when could it be easier to get the constructed labels for a refinement than obtain the labels of the downstream task?

The independence of content and style seems to be a strong assumption. In computer vision, one extensively studied problem is how to make use of context, e.g., background to help object recognition. For example, a monitor will have a high probability to stay on a desk. A car has a high probability to be on the road. Could the authors give some scenarios where the context can be independent of content?

review_title: New perspective on self-supervised learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
## General Summary:

The authors propose a causal interpretation of the self-supervised representation learning problem. They introduce a method (ReLIC) that employs invariance constraints on the proxy objectives to enforce better generalization. 

The invariance is enforced through an additional constraint across interventions on the style generating factor, more concretely, these interventions manifest themselves in form of simple data augmentation strategies (rotation, translation, scaling, etc).

The authors interpret contrastive learning through the concept of refinements.

To the extent of my knowledge, all relevant related work has been mentioned (Invariant Causal Prediction, Invariant Risk Minimization). It would be nice if the authors would make the separation in the main text more clear, the main difference being the regularization and the way they choose the interventions.

## Overall Recommendation

I find the method proposed in the paper novel and well explained and put in the causal framework. Overall, I think it's an accept but to be a stronger one the authors should address the points in the comments.

## Writeup

The paper is clearly written, with a few improvements that can be made (a bit of notation and some typos, unfinished sentences).


## Pros
The causal framework is well-motivated, i like the separation in content and style factors. The interpretation of self-supervised learning as invariant prediction with refinements is valid. The results seem significant, since ReLIC outperforms other approaches  in an RL (Atari) and classification setting (ImageNet).

I like the motivating sentence in the paper that real-world meta-data is abundant and can be used to construct refinements more efficiently, which speaks for the relevance of the contribution.

To the extent of my knowledge the contribution is novel, I am not aware of any other work that connected contrastive learning with the causal framework.

## Cons

The problem of choosing the intervention on style factors is a bit understated, the interventions mentioned need not be content-preserving.

I find the evaluation on the Atari benchmark a bit misguided, although the method shows superior performance in some environments, RL is exactly the setting where it is difficult to make good interventions for better generalization, i.e. it is difficult to determine in a data-driven way what separates the content from the style. 

## Comments

In Section 2 would be useful to formally define $Y_t$ as a set of labels. I am not sure about the multi-environment setup, since the distribution p(X) changes also in switching environments (domain shift), which would require different formalism.

p.4 par.1 where you separate style from content. Shouldn't style be also what causes the fine-grained instance separation. For example, if I have the color of eyes of dogs. In the instance classification task, this would be part of content vs. in the cats and dogs task, this would be part of style. Which would imply that the causal graph in fig. 1  should also look differently, since color of eyes doesn't cause a dog to be a dog.

p.4 ReLIC objective. I realize the shorthand notation, but the outer expectation should be over $x ~ p(X)$?

p.4 par. 3 it feels a bit off to me to name content C a representation, whereby it's a latent causal factor that we do not observe. When talking abut representations, we mostly mean f(X)? But sure, if we would use C, we would get an invariant predictor by definition of the causal graph.

p.4. last paragraph - fine-grained problems that->than $Y_t$

p.4. last paragraph - at this point, it is very tricky to state what is a content-preserving data augmentation, this is very much tailored to the problem at hand. A very basic example, random cropping is not content preserving if it doesn't show the dog. This is going back to my critique of the work that the problem of choosing intervention is oversimplified. But I understand, given that we know how to do interventions on the style variable, we will be able to extract an invariant predictor.

p.6 par. 2  Unfinished sentence  Unlike...

p. 7. I think that the experiment on ImageNet shows exactly the point that it is necessary to find good interventions, since the methods employing strong augmentation outperform ReLIC. It would be interesting to see what is the performance of ReLIC with the same set of strong augmentations.

p. 7 table 2, why aren't the strong augmentation baselines mentioned in this table? I am not familiar with the type of augmentations that were done in those baselines.

p. 8 reinforcement learning evaluation. Again, I think this is a bit misguided. It is clear that in the general sense we need invariant representations for reinforcement learning,  but choosing the right interventions is difficult.

review_title: Interpretation is nice, but seems to oversimplify the problem. 

Meta-review: The manuscript proposes a causal interpretation of the self-supervised representation learning problem. The data is modeled as being generated from two independent latent factors: style and content, where content captures all information necessary for downstream tasks, and style captures everything that is affected by data augmentations (e.g. rotation, grayscaling, translation, cropping). The main contribution is a specific regularizer for self-supervised contrastive learning, motivated by the assumptions about the data generation. 

Reviewers agreed that the manuscript is oversold on the causal jargon, as was noted, the manuscript does not perform any causal inference. Nevertheless, they think that there is an interesting interpretation of self-supervised learning and the results are noteworthy.

Decision: Accept (Poster)