Representation Learning via Invariant Causal Mechanisms
Authors: Jovana Mitrovic
Brian McWilliams
Jacob C Walker
Lars Holger Buesing
Charles Blundell
Venue: ICLR 2021 Poster (2021)
Keywords: Representation Learning
Self-supervised Learning
Contrastive Methods
Causality

Abstract: Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signal by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks with data augmentations and have achieved significant success, but our theoretical understanding of this success remains limited. In this paper we analyze self-supervised representation learning using a causal framework.  We show how data augmentations can be more effectively utilized through explicit invariance constraints on the proxy classifiers employed during pretraining. Based on this, we propose a novel self-supervised objective, Representation Learning via Invariant Causal Mechanisms (ReLIC), that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. Further, using causality we generalize contrastive learning, a particular kind of self-supervised method,  and provide an alternative theoretical explanation for the  success  of  these  methods. Empirically, ReLIC significantly outperforms competing methods in terms of robustness and out-of-distribution generalization on ImageNet, while also significantly outperforming these methods on Atari achieving above human-level performance on 51 out of 57 games.

Published as a conference paper at ICLR 2021
REPRESENTATION LEARNING VIA INVARIANT CAUSAL MECHANISMS
Jovana Mitrovic Brian McWilliams DeepMind London, UK
Jacob Walker
Lars Buesing Charles Blundell
(mitrovic, bmew, jewalker, lbuesing, cblundell googglll..oooog.e.oooooooo com
ABSTRACT
Self-supervised learning has emerged as a strategy to reduce the reliance on costly supervised signals by pretraining representations only using unlabeled data. These methods combine heuristic proxy classification tasks with data augmentations and have achieved significant success, but our theoretical understanding of this success remains limited. In this paper we analyze self-supervised representation learning using causal framework. We show how data augmentations can be more effectively utilized through explicit invariance constraints on the proxy classifiers employed during pretraining. Based on this, we propose novel self- supervised objective, Representation Learning via Invariant Causal Mechanisms (RELIC), that enforces invariant prediction of proxy targets across augmentations through an invariance regularizer which yields improved generalization guarantees. Further, using causality we generalize contrastive learning. a particular kind of self-supervised method, and provide an alternative theoretical explanation for the success these methods. Empirically, RELIC significantly outperforms competing methods in terms of robustness and while also significantly outperforming these methods on Atari achieving above human-level performance on 51 out of 57 games.
generalization on ImageNet,
INTRODUCTION
Training deep networks often relies heavily on large amounts of useful supervisory signal, such as labels for supervised learning or rewards for reinforcement learning. These training signals can be costly or otherwise impractical to acquire. On the other hand, unsupervised data is often abundantly available. Therefore, pretraining representations for unknown downstream tasks without the need for labels extrinsic reward holds great promise for reducing the cost of applying machine learning models. To pretrain representations, self-supervised learning makes use proxy tasks defined on unsupervised data. Recently, self sulfvsuper methods using contrastive objectives have emerged as one of the most successful strategies for unsupervised representation learning (Oord et al. 2018 Hjelm al. 2018 Chen al. 2020a). These methods learn representation by classifying every datapoint against all others datapoints (negative examples). Under assumptions on how the negative examples are sampled, minimizing the resulting contrastive loss has been justified maximizing lower bound on the mutual information (MI) between representations (Poole et al. 2019). However, (Tschannen et al. 2019) has shown that performance on downstream tasks may be more tightly correlated with the choice of encoder architecture than the achieved MI bound, highlighting issues with the MI theory of contrastive learning. Further, contrastive approaches compare different views of the data (usually under different data augmentations) to calculate similarity scores. This approach to computing scores has been empirically observed as key success factor of contrastive methods, but has yet be theoretically justified. This lack of solid theoretical explanation for the effectiveness of contrastive methods hinders their further development.
To remedy the theoretical shortcomings, we analyze the problem of self-supervised representation learning through a causal lens. We formalize intuitions about the data generating process using causal graph and leverage causal tools to derive properties of the optimal representation. We show that a representation should be an invariant predictor of proxy targets under interventions on features that are only correlated, but not causally related to the downstream targets of interest.

---

Published as a conference paper at ICLR 2021
Since neither causally nor purely correlationally related features observed and thus performing actual interventions on them is not feasible, for learning representation with this property we use data augmentations to simulate subset of possible interventions. Based propose a regularizer which enforces that the prediction of the proxy targets is invariant across data augmentations. We propose a novel objective for seffssuper representation learning called RErre- sentation Learning with Invariant Causal mechanisms (RELIC), We show how this explicit invariance regularization leverages augmentations more effectively than previous self-supervised methods and that representations learned using RELIC are guaranteed to generalize well to downstream tasks under weaker assumptions than those required by previous work (Saunshi et al. 2019).
causal interpretation,
Next we generalize contrastive learning and provide an alternative theoretical explanation to MI for the success of these methods. We generalize the proxy task of instance discrimination commonly used in contrastive learning using the causal concept of refinements (Chalupka et al. (014). Intuitively, refinement of task can be understood as a more fine-gr rne-grained variant of the original problem. For example, refinement for classifying cats against dogs would be the task of classifying individual cat and dog breeds. The instance discrimination task results from the most fine-grained refinement, e.g. discriminating individual cats and dogs from one another. We show that using refinements as proxy tasks enables us to learn useful representations for downstream tasks. Specifically, using causal tools, show that learning representation on refinements such that is invariant predictor of proxy targets across augmentations is a sufficient condition for these representations to generalize to downstream tasks (cf. Theorem [ In summary, we provide theoretical support both for the general form of the contrastive objective as well as for the use of data augmentations. Thus, we provide an alternative explanation to mutual information for the success of recent contrastive approaches namely that of causal refinments of downstream tasks.
We test RELIC on variety of prediction and reinforcement learning problems. First, we evaluate the quality of representations pretrained on ImageNet with special focus on robustness and out-of- distribution generalization. RELICC performs competitively with current state-of-the-ar methods on ImageNet, while significantly outperforming competing methods on robustness and -ut-of-distiibution generalization of the learned representations when tested on corrupted ImageNet (ImageNet-C (Hendrycks & Dietterich 2019h) and a version of ImageNet that consist of differrntt renditions of the same classes (ImageNet-R (Hendrycks et al. )) In terms of robustness, RELIC also significantly outperforms the supervised baseline with absolute reduction of 4.9% in error. Unlike much prior work that specifically focuses on computer vision tasks, we test RELIC for representation learning in the context of reinforcement learning on the Atari suite (Bellemare et al., 2013). There we find that RELIC significantly outperforms competing methods and achieves above human-level performance on 51 out of 57 games.
Contributions.
We formalize problem of self-supervised representation learning using causality and propose to more effectively leverage data augmentations through invariant prediction.
We propose a new supervised objective, REpresentation Learning with Invariance Causal mechanisms (RELIC), that enforces invariant prediction through explicit regularizer and show improved generalization guarantees.
We generalize contrastive learning using refinements and show that learning on refinements is a sufficient condition for learning useful representations; this provides an alternative explanation to MI for the success of contrastive methods.
2 REPRESENTATION LEARNING VIA INVARIANT CAUSAL MECHANISMS
Problem setting. Let X denote the unlabelled observed data and y = (Y:)7 be set of unknown tasks with Y, denoting the targets for task l. The tasks {Yr)== can represent both a multi-envirrnment as well as multi-task setup. Our goal is to pretrain with unsupervised data representation f(X) that will be useful for solving the downstream tasks y
Causal interpretation. To effectively leverage common assumptions and intuitions about data generation of the unknown downstream tasks for the learning algorithm, propose to formalize
2

---

Published as a conference paper at ICLR 2021
Data generation
Yi
S
C
x-entropy
Yr
KL
X
x-entropy
f(X)
Y
Representation Learning
Figure 1: (a) Causal graph formalizing assumptions about content and style of the data and the relationship between targets and proxy tasks. The dashed arrows are not causal and represent learning. while the dashdotted lines denote that y is refinement Yi's. All other arrows are causal. (b) RELIC objective. KL refers the Kullback-Leibles divergence, while x-entropy denotes cross entropy.
them using causal graph. We start from the following assumptions: a) the data is generated from content and style variables, with b) only content (and not style) being relevant for the unknown downstream tasks and c) content and style are independent, i.e. style changes are content-prees preserving. For example, when classifying dogs against giraffes from images, different parts of the animals constitute content, while style could be, for example, background, lighting conditions and camera lens characteristics. By assumption, content a good representation of the data for downstream tasks and we therefore cast the goal of representation learning as estimating content. In the following, compactly formalize these assumptions with a causal graph see Figure la
Let C and S be the latent variables describing content and style. In Figure la the directed arrows from C' and S to the observed data (e.g. images) indicate that X is generated based on content and style. The directed arrow from C to the target Y, (e.g. class labels) encodes the assumption that content directly influences the target tasks, while the absence of any directed arrow from S to Y; indicates that style does not. Thus, content C. has all the necessary information predict Y, The absence of any directed path between C and S in Figure la encodes the intuition that these variables independent, i.c. C' IL S.
Using the independence of mechanisms (Peters et al. 2017). we can conclude that under this causal model performing interventions S does not change the conditional distribution CC), i.c. manipulating the value of S does not influence this conditional distribution. Thus, P(Y|(C)) is invariant under changes in style We call an invariant representation for under S, i.e.
p do((= (Y; C) ))((S=+)) (Y) Y 84, E S,
(1)
where y da(('=s) denotes the distribution arising from assigning S the value with S the domain of S (Pearl 2009). Specifically, using C as a representation allows for us predict targets stably across perturbations, i.e. content both a useful and robust representation for tasks y
Since the targets Yt are unknown, we will construct a proxy task Y in order to learn representations from unlabeled data only. order learn useful representations for Y,, we will construct proxy tasks that represents more fine-grained problems than Y;: for more formal treatment of proxy tasks please refer to Section 3 Further, to learn invariant representations, such as C, we enforce Equation [ which requires to observe data under different style interventions, i.c. need data that describes the same content under varying style. Since we do not have access to S, to simulate style variability content-preservinn data augmentations (e.g. rotation, grayscaling, translation, cropping for images). Specifically, we utilize data augmentations as interventions on the style variable S, i.e.
See (Peters 2017 for review of causal graphs and causality
3

---

Published as a conference paper at ICLR 2021
applying data augmentation o; corresponds to intervening S and setting it to Sa;- Although we are not able to generate all possible styles using a fixed set of data augmentations, we will use augmentations that generate large sets of diverse styles as this allows to learn better representations. Note that the heuristic of estimating similarity based on different views from contrastive learning can be interpreted an implicit invariance constraint.
RELIC objective. Equation ! provides general scheme to estimate content (c.f. Figure Tap. We
operationalize this by proposing to learn representations such that prediction of proxy targets from the representation is invariant under data augmentations. The representation s(X) must fulfill the following invariant prediction criteria
(Invariant prediction)
p plo(a)) (Y (f(()) p ) (Y (x()) Va,, aj E A.
(2)
A = fai...., am J is the set data augmentations which simulate interventions on the style variables and p do(a)) denotes p sa))S
To achieve invariant prediction, propose explicitly enforce invariance under augmentations through a regularizer. This gives rise to an objective for self-supervised learning we call Representa- tion Learning via Invariant Causal Mechanisms (RELIC). We write this objective
min X) ~AXA
E Bik.
Σ Cs(YR, f(x))
BE (ac
s.t. KL (pdolaw) (Y f(X)), do(ow) (y f(X))) Sp
where is the proxy task loss and KL is the Kullback Leibler (KL) divergence. Note that any distance measure distributions can be used in place of the KL divergence. We explain the remaining terms in detail below.
Concretely, as proxy task we associate every datapoint x; the label y = This corresponds the instance discrimination task, commonly used in contrastive learning (Hadsell et al. (206)). We take pairs of points (z,, ..y) compute similarity scores and use pairs of augmentations am (a, as) e A x A perform style intervention. Given batch of samples /. D, we use
p (oloo))( = fxx)) & exp
with xt data augmented with a and T softmax temperature parameter. We encode f using neural network and choose h to be related to f, e.g. h f or as a network with an exponential moving average of the weights of f (e.g. target networks similar to (Grill et. 2020)). To compare representations we use the function (xx(),, h(x;)) = (g(f(x;)). g((()) where g is a fully- connected neural network often called the critic.
Combining these pieces, we learn representations by minimizing the following objective over the full set of data H, E 'D and augmentations au E x A
exp (o(f(x;)) h(x? ))/T) M== exp (#([(*"') h(xatt()))))))))))) /)/r)
min ΣΣ log 41A
Σ alc.
KL(ndolana) pdolase)
(3)
with M the number of points we use to construct the contrast set and the weighting of the invariance penalty. We used the shorthand do(()) for do(a) (y = jl f(x)). With appropriate choices for di g. and h above, Equation recovers many recent state-of-the-ar methods (c.f. Table in Section A). Figure lb presents a schematic of the RELIC objective.
The explicit invariance penalty encourages the within-class distances (for a downstream task of inter- est) of the representations learned by RELIC to be tightly concentrated. We show this empirically in Figure [ggureeaand and theoretically in Appendix B In the following section provide theoretical justification for using an instance discrimination- based contrastive loss using a causal perspective. We also show (cf. Theorem below) that minimizing the contrastive loss alone (i.e. 0) does not guarantee generalization. Instead, invariance across augmentations must be explicitly enforced.
I
SSince neither content style are priori known, choosing set of augmentations implicitly defines which aspects of the data considered style and which content.
4

---

Published as a conference paper at ICLR 2021
3 GENERALIZING CONTRASTIVE LEARNING
Learning with refinements. In contrastive learning, the task of instance discrimination, i.c. classi- fying the dataset {(x), vfth 22 E D). is used as the proxy task. To better understand contrastive learning and motivate this proxy task, we generalize instance discrimination using the causal concept of refinements (Chalupka et al. 2014). Intuitively, refinement of one problem another more fine-grained problem. If task Y, Is to classify cats against dogs. then a refinement of Y, is the task of classifying cats and dogs into their individual breeds. See Figure for a further visual example. For any set of tasks, there exist many different refinements. However, the most fine-grained refinement corresponds exactly to classifying the datasct {(z,, y = 12 E D). Thus, the instance discrimi- nation task used in contrastive learning is specific type of refinement. For definition and formal treatment of refinements please refer to Appendix D
Let Y be targets of proxy task that is refinement for all tasks in V. Leveraging causal tools, we connect learning on refinements to learning on downstream tasks. Specifically, we provide a theoretical justification for exchanging unknown downstream tasks with these specially constructed proxy tasks. We show that if f(X) is an invariant representation for YR under changes in style S, then f(X) is also an invariant representation for tasks in under changes in style S. Thus by enforcing invariance under style interventions on a refinement, we learn representations that generalize to downstream tasks This is summarized in the following theorem.
Theorem 1. Let (Yi)?= be a family of downstream tasks. Let Y be a refinement for all tasks in y. If f(X) is an invariant representation for y" under style interventions S, then f(X) is an invariant representation for all tasks y under style interventions S, i.e.
p dols)) (a [f(X)) = -ooo))) ( ((x)) = p pools) (Ye ((()) = dolos)) (Y))((x)) (4)
for all 83 E S with pdotes) pdo(S=s.), Thus, f(X) is a representation that generalizes to y
Theorem states that if Y is refinement of y then learning a representation on Y is sufficient condition for this representation to be useful on V. For a formal exposition of these points and accompanying proofs, please refer to Appendix D Recall that the instance discrimination proxy task is the most fine-grained refinement, and so the right hand side of 4 is satisfied for any downstream task satisfying the stated assumptions of the theorem.
We generalize contrastive learning through refinements and connect representations learned on refinements and downstream tasks in Theorem [ Thus, using causality we provide an alternative explanation to mutual information for the success of contrastive learning. Note that our methodology of refinements is not limited to instance discrimination tasks and is thus more general than currently used contrastive losses. Real world data often includes rich sources of metadata which can be used to guide the construction of refinements by grouping the data according to any available meta-data. Note that the coarser we can create refinement, the more data efficient we can expect to be when learning representations for downstream tasks. Further, we can also expect to require less supervised data to finetune the representation.
4 RELATED WORK
Contrastive objectives and mutual information maximization. Many recent approaches to self- supervised learning are rooted in the well-established idea of maximizing mutual information (MI), e.g. Contrastive Predictive Coding (CPC) (Oord al., 2018 Hénaff et al. 2019), Deep InfoMax
(DIM) (Hjelm et al., (018)) and Augmented Multiscale DIM (AMDIM) (Bachman et al. 2019). These methods are based on noise contrastive estimation (NCE) (Gutmann Hyvärimen 2010) which,
under specific conditions, can be viewed as a bound on MI (Poole et 2019). The resulting objective functions are commonly referred to as InfoNCE.
The precise role played by mutual information maximization in self- supervsee learning is subject to some debate. (Txchannen et al. 2019) argue that the performance on downstream tasks is
Note that since refinements are more fine -raingr that the original task, if a representation captures refinement then it also captures the downstream tasks as strictly more information is needed to solve the refinement.
5

---

Published as a conference paper at ICLR 2021
not correlated with the achieved bound on MI, but may be more tightly correlated with encoder architecture and capacity. Importantly, InfoNCE objectives require custom architectures to ensure the network does not converge to non-informative solutions thus precluding the use of standard architectures. Recently, several works (He et al. 2019 Chen et al. 2020a) successfully combined contrastive estimation with standard ResNet-50 architecture. In particular, SimCLR (Chen et al. 2020a) relies on a set of strong augmentation while (He ct al. 2019) uses memory bank. Inspired by target networks in reinforcement learning, (Grill et al. 2020) proposed BYOL: an algorithm for self-supervised learning which remarkably does not use a contrastive objective. Although theoretical explanation for the good performance of BOLL is presently missing, interestingly the objective, an & distance between two different embeddings of the input data resembles the & form of our regularizer proposed in Equation
Appendix
B
Recently, Saunshi et al. 2019) proposed learning theoretic frame- work to analyze the performance of contrastive objectives. However, without strong assumptions on intra-class concentration they note that contrastive objectives are fundamentally limited in the representations they are able to learn. RELIC explicitly enforces intra-class concen- tration via the invariance regularizer, ensuring that it generalizes under weaker assumptions. Unlike (Saunshi et al., 2019) who do not dis- cuss augmentations, we incorporate augmentations into our theoretical explanation of contrastive methods.
ReLIC
AMDIM
clistance 162
LiearChetnrrar Roda
The reasons for the improvement in performance from AMDIM through to SimCLR and BYOL are not easily explained by either the MI max- imization or the learning theoretic viewpoint. Further, it is not clear why relatively minor architectural differences between the methods result in significant differences in performance nor is it obvious how current state-of -f-th---tt- can be improved. In contrast to prior art, the performance of RELIC is explained by connections to causal theory. As such it gives a clear path for improving results by devising problem appropriate refinements, interventions and invariance penalties. Furthermore, the use of invariance penalties in RELIC as dictated by causal theory yields significantly more robust representations that generalize better than those learned with SimCLR or BYOL.
Distribution of the linear discriminant ra- tio (FLDA. see text) of for RELIC, SimCLR and AMDIM (y-axis clipped to aid visualization).
Figure 2:
Causality and invariance. Recently, the notion of invariant prediction has emerged as an important operational concept in causal inference (Peters et al. 2016). This idea has been used to learn classifiers which are robust against domain shifts (Gong et al. 2016). Notably, (Heinze-Demi & Meinshausen 2017) propose to use group structure to delineate between different environments where the aim is to minimize the classification loss while also ensuring that the conditional variance of the prediction function within each group remains small. Unlike (Heinze-Deml & Meinshausen, 2017) who use supervised data and rely on having a grouping in the training data, our approach does not rely on ground-trum truth targets and can flexibly create groupings of the training data if none are present. Further, we enforce invariant prediction within the group by constraining the distance between distributions resulting from contrasting data across groups
The relationship between causal inference and semi-supervised learning has been explored in (Schölkopf et al.] 2012). In particular, in order for unlabeled data to be helpful for learning, the relationship between the predictors and targets must be anti-causal. Our setting is somewhat different as we assume a latent decomposition of the observed variables and learn representations based on proxy targets which are refinements of the true targets.
Our invariance penalty is similar in practise to consistency regularizers which have recently gained popularity in semi-supervised learning (Sajjadi et al. 2016 Xie et al. 2019: Sohn et al.) 2020). These typically supplement the supervised training signal with a term which enforces similarity
between the function's output on two views of data. Although recently proposed consistency methods are heuristic, they have deeper roots in the more rigorous co-training approach to semi-supervised learning from multiple views (Blum & Mitchell 1998 Kakade & Foster 2007 Sridharan & Kakade
2008, Mc Williams & Montana 2012
Mc Williams et al. 2013). 3). The main conceptual difference
"The of augmentations includes Gaussian blurring, various colour distortions, flips and random cropping.

---

Published as a conference paper at ICLR 2021
between our approach and consistency-base methods is that do not have access supervised training signal and that the RELIC penalty is motivated from the underlying causal graph.
EXPERIMENTS
We first visualize the influence of the explicit invariance constraint RELIC on the linear separability of the learned representations. We then evaluate RELIC number of prediction and reinforcement learning tasks for usefulness and robustness. For the prediction tasks, we test RELIC after pretraining the representation in self-supervised way on the training set of the ImageNet ILSVRR--201 dataset (Russakovsky et al., 2015). We evaluate RELIC in the linear evaluation setup on ImageNet and test its robustness and out-of-distribution generalization on datasets related to ImageNet. Unlike much prior work in contrastive learning which focuses specifically on computer vision tasks, we test RELIC also in the context of learning representations for reinforcement learning. Specifically, test RELIC on the suite of Atari games (Bellemare et al. 2013) which consists of 57 diverse games of varying difficulty.
Linear evaluation. In order to understand how representations learned by RELIC differ from other methods, we compare it against those learned by AMDIM and SimCLR in terms of Fischer's linear discriminant ratio (Friedman et al. 2009; :: FLDA = μ| p4 II ExJec, llf(z;) fzz)) where
H4 IC Liec, fax) is the mean of the representations of class k and C. the index of that class. larger PIDA implies that classes are more easily separated with linear classifier. This can be achieved by either increasing distances between classes (numerator) or shrinking within-class variance (denominator).
Figure 2 shows the distribution of FLDA for RELIC, SimCLR and AMDIM after training measured on the (downsampled) ImageNet validation set. The distance between medians of RELIC and SimCLR is 162. AMDIM is tightly concentrated close to 20. The invariance penalty ensures that even though labels are a-priori unknown- for RELIC within-class variability of f is concentrated leading better linear separability between classes in the downstream task of interest. This is reflected in the rightward shift of the distribution of FLDAA in Figure 2 for RELIC compared with SimCLR and AMDIM which do not impose such a constraint.
Next we evaluate RELIC's representa- tion by training linear classifier of top of the fixed encoder following the pro- cedure in (Kolesnikov et al., 2019 Chen eet al. 2020a) and Appendix E.4 In Ta- ble we report top-1 and top-5 accu- racy on the ImageNet test set. Methods denoted with * use SimCLR augmenta- tions (Chen et al.) 2020a). while meth- ods denoted i use custom, stronger aug- mentations. Comparing methods which use SimCLR augmentations, RELIC outperforms competing approaches on both ResNet-50 and ResNet-50 with target network. For completeness. we report results for SwAV (Caron et al.) 2020) and InfoMin (Tian et al.) (2020). but note that these methods use stronger augmentations which alone have been shown to boost performance by over 5%. fair comparison between different ob- jectives can only be achieved under the same architecture and the same set of augmentations.
Table 1: Accuracy (in %) under linear evaluation Ima- geNet for different self-supervised representation learning methods. Methods with use SimCLR augmentations. Methods with use custom, stronger augmentations.
Method
Top-1
Top-5
Re architecture PIRL
63.6
CPC v2
63.8
85.3
CMC
66.2
87.0
SimCLR Chen al.
2020a)
69.3
89.0
SWAV Caron et al.
2020
70.1
RELIC (ours)
70.3
89.5
InfoMin Aug. Tian et al
2020
73.0
91.1
SWAV (Caron et al 2020
75.3
ResNer-50 with targel network
MoCo v2 [Chen et al. 2020b
71.1
BYOL Grill et all 2020
74.3
91.6
RELIC (ours)
74.8
92.2
7

---

Published as a conference paper at ICLR 2021
Robustness and generalization. We evaluate robustness and -utto--diitribbution generalization of RELIC's representation on datasets Imagenet-C (Hendrycks Dietterich] 2019) and ImageNet-R (Hendrycks et al. 2020), respectively. To evaluate RELIC's representation, we train linear classifier on top of the frozen representation following the procedure described in (Chen et al.) 2020a) and appendix |E.322 For Imagener-C
report the mean Corruption Error (mCH) and Corruption Errors for Noise corruptions in Table 3 RELIC has significantly lower mCE than both the supervised ResNet-50 baseline and the unsupervised methods SimCLR and BYOL. Also, has the lowest Corruption Error on 14 out of 15 corruptions when compared to SimCLR and BYOL. Thus, we see that RELIC learns the most robust representation. RELIC also outperforms SimCLR and BYOL on ImageNet-R showing its superior out-of of-distribution generalization ability; see Table 2 details and results please consult
For further
Table 2: Top-1 error rates for different self-supervised representation learning methods on ImageNet- R. All models are trained only on clean ImageNet images and RELICr refers to RELIC using a ResNet-50 with target network as in BYOL. (Grill et al., 2020).
Method
Supervised 63.9
SimCLR 81.7
RELIC
BYOL
RELICT 76.2
Top-1 Error (%)
77.4
77.0
Table 3: Mean Corruption Error (mCE), mean relative Corruption Error (mrCE) and Corruption
Errors for the "Noise" class of corruptions (Gaussian, Shot, Impulse) on ImageNet-C. The mCE value is the average across 75 different corruptions. Methods are trained only clean ImageNet images.
Method
mCE
mrCE
Gaussian
Shot
Impulse
Supervised
76.7
105.0
80.0
82.0
83.0
ResNet-50 architecture:
SimCLR
87.5
111.9
79.4
81.9
89.6
ReLIC
76.4
87.7
67.8
70.7
77.0
ResNet-50 with target network:
BYOL
72.3
90.0
65.9
68.4
73.7
ReLIC
70.8
88.4
63.6
65.7
69.2
Reinforcement Learning. Much prior work in contrastive learning has focused specifically on computer vision tasks. In order to compare these approaches in a different domain, we investigate representation learning in the context of reinforcement learning. We compare RELIC as an auxiliary loss against other state of the art self-supervised losses human normalized scores as a metric, we use the original architecture and hyperparameters of the R2D2 agent (Kapturowski et al. 2019) and supplement with second encoder trained with a given representation learning loss. When auxiliary losses are present, the Q-Network takes the output of the second encoder as input. The Q-Network and the encoder are trained with separate optimizers. For the augmentation baseline, the Q-Network takes two identical encoders trained end-to-end. Table 4 shows a comparison between RELIC, SimCLR, BYOL. CURL (Srinivas et al. 20200), and feeding augmented observations directly to the agent (Kostrikov et al. (020). We find that RELIC has significant advantage over competing self-supervised methods, performing best in 25 out of 57 games. The next best performing method, CURL, performs best in 11 games. Note that none of these methods outperform R2D2 (Kapturowski et al. 2019) which achieves superhuman performance in 52 games. Although previously published work shows the auxiliary tasks help Atari, this is in the very low-data regime. Here we show that ReLIC is able to close the gap in the more common Atari set up. We hypothesize that the performance penalty resulting from adding auxiliary self-supervised losses stems from the naive combining of the outputs of the two encoders; we will explore alternative options for combining encoder outputs in future. Full details are presented in Section E.6
agent trained on 57 Atari games. Using
6 CONCLUSION
In this work we have analyzed self-supervised learning using a causal framework. Using a causal graph, we have formalized the problem of self-supervised representation learning and derived
8

---

Published as a conference paper at ICLR 2021
Table 4: Human Normalized Scores of Auxiliary Methods over 57 Atari Games.
Atari Performance
RELIC
SimCLR
CURL
BYOL
Augmentation
Capped mean
91.46
88.76
90.72
89,43
80.60
Number of superhuman games
51
49
49
49
34
Mean
3003.73
2086.16
2413.12
1769.43
503.15
Median
832.50
592.83
819.56
483.39
132.17
40%
Percentile
356.27
266.07
409.46
224.80
94.35
30%
Percentile
202.49
174.19
190.96
150.21
80.04
20% Percentile
133.93
120.84
126.10
118.36
57.95
10% Percentile
83.79
37.19
59.09
44.14
32.74
5% Percentile
20.87
12.74
20.56
7.75
2.85
properties of the optimal representation. We have shown that representations need to be invariant predictors of proxy targets under interventions on features that are only correlated, but not causally related to the downstream tasks. We have leveraged data augmentations simulate these interventions and have proposed to explicitly enforce this invariance constraint, Based on this, we have proposed new self-supervised objective, Representation Learning via Invariant Causal Mechanisms (RELIC), that enforces invariant prediction of proxy targets across augmentations using an invariance regularizer. Further, we have generalized contrastive methods using the concept of refinements and have shown that learning a representation on refinements using the principle of invariant prediction is a sufficient condition for these representations to generalize to downstream tasks. With this, we have provided an alternative explanation mutual information for the success of contrastive methods. Empirically we have compared RELIC against recent self-supervised methods on a variety of prediction and reinforcement learning tasks. Specifically, RELIC significantly outperforms competing methods in terms robustness and dittr--diitiitution generalization of the representations it learns on ImageNet. RELIC also significantly outperforms related self-supervised methods the Atari suite achieving superhuman performance on 51 out of 57 games. We aim to investigate the construction of more Saars--grained refinements and the empirical evaluation of different kinds of refinements in future work.
Acknowledgements. We thank David Balduzzi, Melanie Rey, Christina Heinze-Deml, Ilja Kuzborskij, Ali Eslami, Jeffrey de Fauw and Josip Djolonga for invaluable discussions.
REFERENCES
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15509-15519, 2019.
Mare G. Bellemare, Yavar Naddaf, Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents (extended abstract). J. Artif. Intell. Res., 47:253-279 2013.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training, In Proceed- ings of the eleventh annual conference on Computational learning theory, pp. 92-100, 1998.
M. Caron, I. Misra, Mairal, Priya Goyal, P. Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. ArXiv, abs/2006.09882, 2020.
Krzysztof Chalupka, Pietro Perona, and Frederick Eberhardt. Visual causal feature learning. arXiv preprint arXiv:1412.22 2014.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.057000 2020a.
Xinlei Chen, Haoqi Fan, Ross B. Girshick. and Kaiming He. Improved baselines with momentum contrastive learning. ArXiv, abs/2003.0429) 042997 2020b.

---

Published as a conference paper at ICLR 2021
Paul Erdós and Alfréd Rényi. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 5(1):17-60, 1960.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning, volume 2. Springer series in statistics New York, 2009.
Alan Frieze and Michal Karoński. Introduction to random graphs. Cambridge University Press, 2016.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International conference machine learning, pp. 2839-28449 2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (6)), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.
Christina Heinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. arXiv preprint 1rX0: 1469, 2017.
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: 1905.0922,, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006. 16241, 2020.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.0666 2018.
loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv, abs/1 502.03367, 2015.
Sham M Kakade and Dean P Foster. Multi-view regression via canonical correlation analysis. In International Conference Computational Learning Theory, pp. 82-96. Springer, 2007.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement learning. Iclr, 2019.
A. Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self self-supervised visual representation learning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1920-1929, 2019.
10

---

Published as a conference paper at ICLR 2021
Ily Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts, In ICLR, 2017.
Brian McWilliams and Giovanni Montana. Multi-view predictive partitioning in high dimensions. Statistical Analysis and Data Mining: The ASA Data Science Journal, 5(4):304-321, 2012.
Brian McWilliams, David Balduzzi, and Joachim M Buhmann. Correlated random features for fast semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 440-448, 2013.
V. Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv:/807.03748. 2018.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant
prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):47-11122, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithmss. MIT press, 2017.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference Machine Learning, pp. 5171-5180, 2019.
Olga Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Zhiheng Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal Computer Vision, 11:2211-222, 2015.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29: 19:1163-117, 2016.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar, theoretical analysis of contrastive unsupervised representation learning. In International Confer- ence on Machine Learning, pp. 5628-5637, 2019.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv: wwrXiv:/006.4477 /206.647,, 2012 2012.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.06685, 2020.
Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. 2008.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, arXiv preprint arXiv:2004.000 041300 2020.
Yonglong Tian, C. Sun, Ben Poole, Dilip Krishnan, C. Schmid, and Phillip Isola. What makes for good views for contrastive learning. ArXiv, abs/2005, 10243, 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint arXiv: /907. 13625, 2019.
Haohan Wang, Songwei Ge, E. Xing, and Zachary Chase Lipton. Learning robust global representa- tions by penalizing local predictive power. ArXiv, Ahs/1905.1n 13549, 2019.
11

---

Published as a conference paper at ICLR 2021
Qizhe Xie, Zihang Dai, Eduard Hovy. Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation. arXiv preprint 1rXiv:190..12848, 2019.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arviv: 1708. 03888, 2017.
12


Reviews:
[{"review_id": "contffMfion", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework for self-supervised representation learning using causality. The proposed model is formulated by assuming that the Data generation schema is composed of two independent mechanisms (ie., Style and Content) and only content is relevant for learning the underlying task. Thus, the Content is a good representation of the data and the goal of representation learning could be cast as a content estimation. Then, the authors use interventions on the Style (i.e., data augmentation in their formulation) to learn invariant representation under data augmentation (Style variable). To achieve this invariant prediction they propose a new constructive objective (ReLIC). \n\nThe paper is well written and easy to follow. My only concern is that the whole proposal relies on the assumption that Data generation is composed of two independent mechanisms (S and C) and the authors utilize various data augmentations as interventions on the Style variable S as they don’t have access to S. However, no details are given for the impact of the used data augmentation techniques on the learning better representations. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZavgwHCK5yU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## General Summary:\n\nThe authors propose a causal interpretation of the self-supervised representation learning problem. They introduce a method (ReLIC) that employs invariance constraints on the proxy objectives to enforce better generalization. \n\nThe invariance is enforced through an additional constraint across interventions on the style generating factor, more concretely, these interventions manifest themselves in form of simple data augmentation strategies (rotation, translation, scaling, etc).\n\nThe authors interpret contrastive learning through the concept of refinements.\n\nTo the extent of my knowledge, all relevant related work has been mentioned (Invariant Causal Prediction, Invariant Risk Minimization). It would be nice if the authors would make the separation in the main text more clear, the main difference being the regularization and the way they choose the interventions.\n\n## Overall Recommendation\n\nI find the method proposed in the paper novel and well explained and put in the causal framework. Overall, I think it's an accept but to be a stronger one the authors should address the points in the comments.\n\n## Writeup\n\nThe paper is clearly written, with a few improvements that can be made (a bit of notation and some typos, unfinished sentences).\n\n\n## Pros\nThe causal framework is well-motivated, i like the separation in content and style factors. The interpretation of self-supervised learning as invariant prediction with refinements is valid. The results seem significant, since ReLIC outperforms other approaches  in an RL (Atari) and classification setting (ImageNet).\n\nI like the motivating sentence in the paper that real-world meta-data is abundant and can be used to construct refinements more efficiently, which speaks for the relevance of the contribution.\n\nTo the extent of my knowledge the contribution is novel, I am not aware of any other work that connected contrastive learning with the causal framework.\n\n## Cons\n\nThe problem of choosing the intervention on style factors is a bit understated, the interventions mentioned need not be content-preserving.\n\nI find the evaluation on the Atari benchmark a bit misguided, although the method shows superior performance in some environments, RL is exactly the setting where it is difficult to make good interventions for better generalization, i.e. it is difficult to determine in a data-driven way what separates the content from the style. \n\n## Comments\n\nIn Section 2 would be useful to formally define $Y_t$ as a set of labels. I am not sure about the multi-environment setup, since the distribution p(X) changes also in switching environments (domain shift), which would require different formalism.\n\np.4 par.1 where you separate style from content. Shouldn't style be also what causes the fine-grained instance separation. For example, if I have the color of eyes of dogs. In the instance classification task, this would be part of content vs. in the cats and dogs task, this would be part of style. Which would imply that the causal graph in fig. 1  should also look differently, since color of eyes doesn't cause a dog to be a dog.\n\np.4 ReLIC objective. I realize the shorthand notation, but the outer expectation should be over $x ~ p(X)$?\n\np.4 par. 3 it feels a bit off to me to name content C a representation, whereby it's a latent causal factor that we do not observe. When talking abut representations, we mostly mean f(X)? But sure, if we would use C, we would get an invariant predictor by definition of the causal graph.\n\np.4. last paragraph - fine-grained problems that->than $Y_t$\n\np.4. last paragraph - at this point, it is very tricky to state what is a content-preserving data augmentation, this is very much tailored to the problem at hand. A very basic example, random cropping is not content preserving if it doesn't show the dog. This is going back to my critique of the work that the problem of choosing intervention is oversimplified. But I understand, given that we know how to do interventions on the style variable, we will be able to extract an invariant predictor.\n\np.6 par. 2  Unfinished sentence  Unlike...\n\np. 7. I think that the experiment on ImageNet shows exactly the point that it is necessary to find good interventions, since the methods employing strong augmentation outperform ReLIC. It would be interesting to see what is the performance of ReLIC with the same set of strong augmentations.\n\np. 7 table 2, why aren't the strong augmentation baselines mentioned in this table? I am not familiar with the type of augmentations that were done in those baselines.\n\np. 8 reinforcement learning evaluation. Again, I think this is a bit misguided. It is clear that in the general sense we need invariant representations for reinforcement learning,  but choosing the right interventions is difficult.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ory7SMthdnw", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary \n\nThis paper takes a causal viewpoint on self-supervised contrastive representation learning. The data is modeled as being generated from two independent latent factors: style and content, where content captures all information necessary for downstream tasks, and style captures everything that is affected by training augmentations. The main contribution is a specific regularizer for self-supervised contrastive learning, motivated by the assumptions about the data generation. The learned representations are evaluated in terms of robustness, classification, and generalization performance on ImageNet and in terms of performance on the RL Atari benchmark. The proposed approach is shown to outperform even very recent competing approaches.\n\n## Pros & Cons\n\n+ The explicit expression of the assumptions behind the data generation process as a (causal) network. It explains the assumptions made for augmentations and proxy tasks to make sense; it also nicely subsumes previous formulations.\n+ Visualization of the resulting feature space\n+ Performance appears great, the evaluation seems sufficient enough.\n\n- Proof of Theorem 1 has problems, fixing it, meant to change the causal graph. Although, this might be fixable (see questions below).\n- There is in fact no causal language needed for this paper. There is no causal discovery or anything happening. It is really just: Here are my assumptions about the mechanisms of data generation and everything else follows just from statistical independence. For instance, the proposed invariance criterion can also be formulated as the distribution $p(Y_t \\mid C)$ being invariant to distributional shifts of $p(S)$; and trying to gain robustness against covariate shifts is nothing new per se. Being explicit about the dependency between downstream tasks is important, but was not correctly stated in the paper. \n\n## Questions and concerns\n- for intervening in a causal graph on a root note, I don't need the \"do\" notation. There is no need to cut the graph. So the statements are all trivial from this perspective.\n\n- The proof of theorem 1 has problems, I think. Namely, after the first equal, you write $p^{do(s_i)}(Y_t \\mid Y^R)$, which should be $p^{do(s_i)}(Y_t \\mid Y^R, f(X))$ unless $Y_t \\perp f(X) \\mid Y^R$. In this case, you can not progress further:\n\n$$p^{do(s_i)}(Y_t \\mid Y^R, f(X)) = p(Y_t \\mid Y^R, f(X), S=s_i) \\neq p(Y_t \\mid Y^R, f(X)) \\neq p(Y_t \\mid Y^R, f(X), S=s_j) = p^{do(s_j)}(Y_t \\mid Y^R, f(X))$$\n\nThis is because $Y_t \\not\\perp S \\mid f(X), Y^R$, so you can not drop the conditioning on $S$. \n\nHowever, if $Y_t \\perp f(X) \\mid Y^R$ holds, the proof would work again. This would be the case when $C \\rightarrow Y^R \\rightarrow Y_t$, i.e. the refinement task \"causes\" the downstream task. From the causal graph of Figure 1a, this is not the case. Although this would make sense intuitively, as the instance discrimination task needs more information than the downstream tasks, and you even state something in this direction in footnote 3. So I would ask the authors to clarify the intended causal connections between $C$, $Y^R$, and $Y_t$. \n\n- Again on the proof, it looks like the assumption that $Y^R$ is a refinement for all tasks in $Y$ is not even needed?\n- Can you phrase the concept of \"refinements\" in terms of causality? What does it mean for task $Y^R$ to be a refinement of $Y_t$?. Although you state that you use the \"causal concept of refinement\", a causal explanation is not given as far as I can see.\n\n- You change the definition of the $p^{do(a) }$ to one with two interventions $p^{do(a_{ik})}$. This suddenly appears below Eq 2. and is then used in what follows. To me, this looks like one of the most important contributions. Clearly, it does not follow from the causal graph or so directly. However, it is one way of defining the regularizer that is consistent with the framework and it seems to be important. So please improve the presentation and introduce it properly.\n\n- How would you place downstream tasks like object detection or segmentation in your framework? I would imagine that learning for instance discrimination does not keep all the information necessary to solve these kinds of tasks.\n \n## Suggestions and Comments\n- I like that you make the modeling assumptions explicit\n- Get rid of most of the causality jargon, and do not try to oversell your paper into the hyped field. Your paper is not doing causal inference or anything of the kind.\n- Spend much more time on the regularizer and its definition with the 4 interventions that just fall from the sky. This is actually your contribution: defining the regularizer in the way you do it. A comparison to a simple baseline with just the X-entropy between two interventions would be good (or is that one of the baseline methods?).\n- your style factors are also called nuisance factors in the literature\n- consider making it more prominent in the paper that you solve the instance discrimination task\n- the causal graph presented (in figure 1a: the meaning of arrows is not clearly defined)\n- if the arrows are causal than it seems to contradict the refinement idea \n- Fig 1b: make the 4 interventions more explicitly visible. Add the Bear picture to the top and bottom row\n\n- also, it would be beneficial to see the experiments that support and visualize theorem 1 (e.g. to show that the KL for other tasks is also decreasing when we decrease the KL for instance discrimination task).\n\n- Typos: signal -> signals (line 2 abstract), \n- Theorem 1: probl also a quantifier for $t$ is needed\n\n- Page 5: \"and so the left hand side of 4\" <- this should probably be \"the right hand side of 4\"\n\nOverall: the paper has great results and can be a valuable contribution, but it has problems in the formulation and is overselling on being causal. \n\nUpdate: Thanks for fixing the statement of the assumptions such that Theorem 1 can hold.   Update 4 -> 5. Some of my concerns are still not addressed in the revised version.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "25sFpI8cQW6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new understanding of self-supervised learning from a causal perspective. Specifically, a causal graph with style and content is assumed for the generating process of the inputs, such as images. Another assumption is that the down-stream tasks only rely on the content variable. By making use of the independent causal mechanism, the authors propose a new invariance regularization term, which is achieves good performance on several real datasets. Also, a new understanding of contrastive learning is provided.\n\nStrength\n\nThe understanding of self-supervised learning from a causal perspective is novel. The causal generative model assumed in this paper seems to be reasonable in many real scenarios. Especially, the image data were generated from content and style and usually the downstream tasks such as object recognition depends on the content.\n\nThe experimental results on Imagenet and Atari demonstrate the effectiveness of the method.\n\nWeakness\n\nThe idea of independence mechanisms was originally proposed in [1] and has deep connections to the modularity of a causal system and the concept of exogeneity in economics (Pearl, 2009). In specific, given two variables C and E, we say C is exogenous if P(E|C) remains invariant to changes in the process that generates C. In [1], the independence mechanism is defined as follows:\n “We finally assume that the mechanism is “independent” of the distribution of the cause in the sense that P(E|C) contains no information about P(C) and vice versa; in particular, if P(E|C) changes at some point in time, there is no reason to believe that P(C) changes at the same time.” When P(c) and P(E|C) both change, they change independently of each other [2]. It would be better if the authors could explore the literature a little bit more and add corresponding discussions. \n[1] Schölkopf, Bernhard, et al. \"On causal and anticausal learning.\" Proceedings of the 29th International Coference on International Conference on Machine Learning. 2012.\n[2] Huang, Biwei, et al. \"Causal discovery from heterogeneous/nonstationary data.\" Journal of Machine Learning Research 21.89 (2020): 1-53.\n\nThe proposed invariance regularization is closely related to the consistency regularization [3] in semi-supervised learning. The relation and difference to consistency regularization needs to be discussed.\n[3] Sajjadi, Mehdi, Mehran Javanmardi, and Tolga Tasdizen. \"Regularization with stochastic transformations and perturbations for deep semi-supervised learning.\" Advances in neural information processing systems. 2016.\n\nThe refinement seems counterintuitive to me. The authors define a refinement of one problem as another more fine-grained problem. For a downstream task, when could it be easier to get the constructed labels for a refinement than obtain the labels of the downstream task?\n\nThe independence of content and style seems to be a strong assumption. In computer vision, one extensively studied problem is how to make use of context, e.g., background to help object recognition. For example, a monitor will have a high probability to stay on a desk. A car has a high probability to be on the road. Could the authors give some scenarios where the context can be independent of content?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript proposes a causal interpretation of the self-supervised representation learning problem. The data is modeled as being generated from two independent latent factors: style and content, where content captures all information necessary for downstream tasks, and style captures everything that is affected by data augmentations (e.g. rotation, grayscaling, translation, cropping). The main contribution is a specific regularizer for self-supervised contrastive learning, motivated by the assumptions about the data generation. 

Reviewers agreed that the manuscript is oversold on the causal jargon, as was noted, the manuscript does not perform any causal inference. Nevertheless, they think that there is an interesting interpretation of self-supervised learning and the results are noteworthy.

Decision: Accept (Poster)