Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
Authors: Julius Von Kügelgen
Yash Sharma
Luigi Gresele
Wieland Brendel
Bernhard Schölkopf
Michel Besserve
Francesco Locatello
Venue: NeurIPS 2021 Poster (2021)
Keywords: self-supervised learning
data augmentation
identifiability
nonlinear ICA
content style separation
causal representation learning
multi-view learning

Abstract: Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.

Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
Julius von Küggelgen*1. +1
Yash Sharma
Luigi Gresele*1
Wieland Brendel
Bernhard Schölkopff
Michel Besserve!!
Francesco Locatello
Max Planck Institute for Intelligent Systems Tübingen
University of Cambridge
Tübingen AI Center, University of Tübingen
IMPRS for Intelligent Systems
"Amazon
Abstract
Self-supervised representation learning has shown remarkable success in number of domains. common practice perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant, We seek to understand the empirical success of this approach from theoretical perspective. We formulate the augmentation process as latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to invertible mapping in both generative and dis- criminative settings. We find numerical simulations with dependent latent variables are consistent with theory. Lastly, we introduce Dideat dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.
Introduction
Learning good representations of riggh-dimessional observations from large amounts of unlabelled data is widely recognised as important step for more capable and data-efficient learning systems [10, 72]. Over the last decade, self- supf-supe learning (SSL) has emerged as the dominant paradigm for such unsupervised representation learning [1, 20, 21, 34, 41, 47, 48, 90, 91, 115, 122, 125, 126]. The main idea behind SSL is to extract supervisory signal from unlabelled observations by leveraging known structure of the data, which allows for the application of supervised learning techniques. A common approach is directly predict some part of the observation from another part (e.g.. future from past, or original from corruption), thus forcing the model to learn meaningful representation in the process. While this technique has shown remarkable success in natural language processing [13, 23, 30, 81, 84, 86, 95, 99] and speech recognition [5, 6, 100, 104], where finite dictionary allows one to output distribution over the missing part, such predictive SSL methods are not easily applied to continuous or high-dimensional domains such as vision. Here, a common approach is to learn a joint embedding of similar observations views such that their representation is close [7, 12, 22, 44]. Different views can come, for example, from different modalities (text & speech; video & audio) time points. As still images lack such multi-modality temporal structure, recent advances in representation learning have relied on generating similar views by means of data augmentation.
Joint first author. Joint senior author. Correspondence to: jvkêtue centtue..pp.cc. m Code available at: https://www. https:/www.ggllub.oom///.// ggilub.com//ssarmmall 126/ssl identifiability
35th Conference on Neural Information Processing Systems (NeurIPS 2021))

---

In order to be useful, data augmentation is thought to require the transformations applied to generate additional views to be generally chosen to preserve the semantic characteristics of an observation, while changing other "nuisance" aspects. While this intuitively makes sense and has shown remark- able empirical results, the success of data augmentation techniques in practice is still not very well understood from a theoretical perspective-despee
some efforts [17, 19, 28]. In the present work, we seek to better understand the empirical success of SSL with data augmentation by formulating the generative process as a latent variable model (LVM) and studying identifiability of the representation, i.e., under which conditions the ground truth latent factors can provably be inferred from the data [77].
Related work and its relation the current. Prior work on unsupervised representation learning from an LVM perspective often postulates mutually independent latent factors: this independence assumption is, for example, at the heart of independent component analysis (ICA) [24, 56] and disentanglement [10, 14, 18, 49, 65, 71]. Since it is impossible to identify the true latent factors without any supervisory signal in the general nonlinear case [57. 82], recent work has turned to weakly- or self-supervised approaches which leverage additional information in the form of multiple views [39, 83, 108, 129], auxiliary variables [58, 63), or temporal structure [45, 54, 55, 69]. To identify or disentangle the individual independent latent factors, it is typically assumed that there is a chance that each factor changes across views, environments, time points.
Our work-being being directly motivated by common practices in SSL with data augmentation-diife the following two key aspects (see Fig. for an overview). First, we do not assume independence and instead allow for both nontrivial statistical and causal relations between latent variables. This is in line with a recently proposed [105] shift towards causal representation learning [40, 76, 85, 87, 106, 107, 112, 123, 127], motivated by the fact that many underly- ing variables of interest may not be independent but causally related to each other.' Second, instead of a scenario wherein result of augmentation,
style change
from these works in
S
X
augmented
original
all latent factors may change we assume a partition of the latent space into two blocks: a content block which is shared or invariant across different augmented views, and a style block that may change. This is aligned with the notion that augmentations leave certain semantic aspects (i.e., content) intact and only affect style, and is thus more appropriate assumption for studying SSL. In line with earlier work [39, 54, 57, 58, 63, 69, 82, 83, 129], we focus on the setting of continuous ground-truth latents, though we believe our results to hold more broadly.
Figure 1: Overview our problem for- mulation. We partition the latent variable z into content с and style s, and allow for statistical and causal dependence of style on content. We assume that only style changes between the original view and the augmented view x, i.c., they are ob- tained by applying the same deterministic function to z = (c, s) and 2 = (c, 5).
Structure and contributions. Following a review of SSL with data augmentation and identifiability theory ( 2), we formalise the process of data generation and augmentation as LVM with content and style variables ($ 3). We then establish identifiability results of the invariant content partition ($ 4), validate our theoretical insights experimentally ($ 5), and discuss our findings and their limitations in the broader context of SSL with data augmentation ($ 6). We highlight the following contributions:
we prove that SSL with data augmentations identifies the invariant content partition of the rep- resentation in generative (Thm. 4.2) and discriminative learning with invertible (Thm. 4.3) and non-invertible encoders with entropy regularisation (Thm. 4.4); in particular, Thm. 4.4 provides a theoretical justification for the empirically observed effectiveness of contrastive SSL methods that use data augmentation and InfoNCE [91] as an objective, such as SimCLR [20]:
we show that our theory is consistent with results in simulating statistical dependencies within blocks of content and style variables, as well as with style causally dependent content ($ 5.1);
we introduce Dadsa,, dataset of 3D objects which allows for the study of identifiability in causal representation learning setting, and use it to perform a systematic study of data augmentations used in practice, yielding novel insights on what particular data augmentations are truly isolating as invariant content and discarding as varying style when applied ($ 5.2).
E.g., [69]. Fig. 11 where dependence between latents was demonstrated for multiple natural video data sets.

---

2 Preliminaries and background
Self-supervised representation learning with data augmentation. Given an unlabelled dataset of observations (e.g., images) x, data augmentation techniques proceed as follows. First, a set of observation-level transformations t E T are specified together with a distribution Pt over 'T. Both T and Pt are typically designed using human intelligence and domain knowledge with the intention of not changing the semantic characteristics of the data (which arguably constitutes a form of weak supervision). For images, for example, a common choice for T are combinations of random crops [1 13] horizontal or vertical flips, blurring, colour distortion [52, 113], or cutouts [31]; and Pe is a distribution over the parameterisation of these transformations, e.g., the centre and size of crop (20, 31J. For each observation x, a pair of transformations t, ~ Pt is sampled and applied separately to x to generate a pair of augmented views (&, x') = (t(x),t'(x))
The joint-emhedding approach to SSL. then uses a pair of encoder functions (g. gg). i.e. deep nets, to map the pair (X, x)) to a typically lower-dimensional representation (2, 2') = (g(*), g' xx()). )- Often, the two encoders are either identical, g g', or directly related (e.g., via shared parameters or asynchronous updates). Then, the encoder(s) (g. g') are trained such that the representations (ž, ž') are "close", i.e., such that sim(Z, x') is large for some similarity metric sim(:), e.g., the cosine similarity [20, 129], or negative L2 norm [129]. The advantage of directly optimising for similarity in representation space over generative alternatives is that reconstruction can be very challenging for high-dimensional data. The disadvantage is the problem of collapsed representations: 3 To avoid collapsed representations and force the encoder(s) to learn meaningful representation, two main families of approaches have been used: (i) contrastive learning (CL) [20, 47, 48, 91, 115, 126]; and (ii) engularisation-baeed SSL [21, 41, 128].
The idea behind CL is to not only learn similar representations for augmented views (x;, x() of the same x;, or positive pairs, but to also use other observations xj (j * i) to contrast with, i.e., to enforce dissimilar representation actoss negative pairs (x,, x,). In other words, CL pulls representations of positive pairs together, and pushes those of negative pairs apart. Since both aims cannot be achieved simultancously with a constant representation, collapse is avoided. A popular CL objective function (used, e.g., in SimCLR [20]) is InfoNCE [91] (based on noise-contrastive estimation [42, 43]):
LIMONNC (g: T, K) = E (x. rrx Ex log 5
exp/simm K/TT)
exp [sim (2, &()/T
where Ex (x)) - temperature, and K- is the number of negative pairs. InfoNCE (1) has an interpretation as multi-class logistic regression, and lower bounds the mutual information across similar views (ž, )') a common representation learning objective [4, 9, 15, 50, 75, 79, 80, 97, 120]. Moreover, (1) can be interpreted as alignment (numerator) and uniformity (denominator) terms, the latter constituting a nonparametric entropy estimator of the representation as K - oo [124]. CL with InfoNCE can thus be seen as alignment of positive pairs with (approximate) entropy regularisation.
Instead of using negative pairs, in CL, a set of recent SSL methods only optimise for alignment and avoid collapsed representations through different forms of regularisation. For example, BYOL [41] and SimSiam [21] rely on "architectural regularisation" in the form of moving-average updates for a separate "target" net g' (BYOL only) or stop-gradien operation (both). BarlowTwins [128], on the other hand, optimises the cross correlation between (š, x¹) to be close to the identity matrix, thus enforcing redundancy reduction (zero off-diagonals) in addition to alignment (ones on the diagonal).
Identifiability of learned representations. In this work, we address the question of whether SSL with data augmentation can reveal or uncover properties of the underlying data generating process. Whether a representation learned from observations can be expected to match the true underlying latent factors-up to acceptable ambiguities and subject to suitable assumptions the generative process and inference model-is captured by the notion of identifiability [77].
Within representation learning, identifiability has mainly been studied in the framework of (nonlinear) ICA which assumes model of the form =f(z) and aims to recover the independent latents, or sources, z, typically up to permutation or element-wise transformation. A crucial negative result states that, with i.i.d. data and without further assumptions, this is fundamentally impossible [57). However, recent breakthroughs have shown that identifiability can be achieved an auxiliary variable (e.g..
NNot that recent work has investigated automatically discovering good augmentations [26, 27].
'If the only goal is to make representations of augmented views similar, degenerate solution which simply maps any observation the origin trivially achieves this goal.
3

---

a time stamp or environment index) renders the sources conditionally independent [45, 54, 55, 58]. These methods rely on constructing positive and negative pairs using the auxiliary variable and learning representation with CL. This development has sparked a renewed interest in identifiability in the context of deep representation learning [63, 64, 69, 83, 102, 108, 109, 129].
Most closely related to SSL with data augmentation are works which study identifiability when given second view X of an observation x, resulting from modified version 2 of the underlying latents or sources z [39, 69, 83, 101, 108, 129]. Here, is either an element-wise corruption of z [39, 69, 101, 129] or may share random subset of its components [83, 108]. Crucially, all previously mentioned works assume that any of the independent latents (are allowed to) change, and aim to identify the individual factors. However, in the context of SSL with data augmentation, where the semantic (content) part of the representation is intended to be shared between views, this assumption does not hold.
3 Problem formulation
We specify our problem setting by formalising the processes of data generation and augmentation, We take latent-variable model perspective and assume that observations x (e.g., images) are generated by a mixing function f which takes a latent code z as input. Importantly, we describe the augmentation process through changes in this latent space as captured by a conditional distribution Pa as opposed to traditionally describing the transformations t as acting directly at the observation level.
Formally, let z be continuous v.v. taking values in open, simply-connected n-dim. representation space 2 S R with associated probability density Pz. Moreover, let f 2 + X be smooth and invertible mapping to an observation space X' S Rd and let x be the continuous r.v. defined as x x= f(z). The generative process for the dataset of original observations of is thus given by:
Z ~ Pu:
x x= f())..
Next, we formalise the data augmentation process, As stated above, we take a eupresentationncentiee view, i.e. we assume that an augmentation x of the original x is obtained by applying the same mixing or rendering function the original representation z of x. Specifying the effect of data augmentation thus corresponds to specifying conditional distribution Pajz which captures the relation between z and 2.
modified representation z which is (stochastically) related to
In terms of the transformation-centri view presented in $ 2. we can view the modified representation 2 E Z as obtained by applying f-1 to a transformed observation x = t(x) E X where ~ Pt, i.e., 2 = (aff--(() (x). The conditional distribution ps|z in the representation space can thus be viewed as being induced by the distribution Pt over transformations applicd at the observation level.5
We now encode the notion that the set of transformations T used for augmentation is typically chosen such that any transformation t E T leaves certain aspects of the data invariant. To this end, we assume that the representation z can be uniquely partitioned into two disjoint parts:
(i) an invariant part e which will always be shared across (z, Z). and which refer to content; iii varying part s which may change across (z, Z), and which we refer to as style.
We assume that e and take values in content and style subspaces C S 1' and S S R" respectively, i.e. n = ne + n2 and Z = C x S. W.l.o.g., we let с corresponds to the first n. dimensions of z: z = (c, s),
of
c: Zl:ne
We formalise the process of data augmentation with content- preserving transformations by defining the conditional Pz, such that only a (random) subset of the style variables change time. Assumption 3.1 (Content-invariance). The conditional density P2|z over 2 takes the form
Pziz ( = 6(ē - ())sss(((s))
for some continuous density Pai on S S, where 5(-) is the Dirac delta function, i.e., č = C a.e. Assumption 3.2 (Style changes). Let A be the set of subsets of style variables A C {1,..., Na) and let pA be a distribution on A. Then, the style conditional Pala is obtained via
Pals. (5|s, A) = 8( S )F &. (sA)))) where lsAl is a continuous density on SA x SA, SA C S denotes the subspace of changing style variables specified by A, and A = (1,..., } \ A denotes the complement of A.
PA.
4 'hhile x may be high-dimensional n & d, invertibility of implics that X' is an n-dim. sub-manifold Rd. We investigate this correspondence between changes in observation and latent space empirically in 5.

---

Note that Assumption 3.2 is less restrictive than assuming that all style variables need to change. since it also allows for only a (possibly different) subset of style variables to change for any given observation. This is in line with the intuition that not all transformations affect all changeable (i.c.. style) properties of the data: e.g., a colour distortion should not affect positional information, and, in the same vein, (horizontal or vertical) flip should not affect the colour spectrum.
The generative process of an augmentation or transformed observation x is thus given by
A ~ PA-
Z A ~ Paiz,A>
x = f(2).
(3)
Our setting for modelling data augmentation differs from that commonly assumed in (multi-view) disentanglement and ICA in that we do not assume that the laten! factors z = (c,s) are mutually (or conditionally) independent, i.c., we allow for arbitrary (non-factorised) marginals Pz in (2).
Causal interpretation: data augmentation as counterfactuals under soft style intervention. We now provide a causal account of the above data generating process by describing the (allowed) causal dependencies among latent variables using structural causal model (SCM) [94]. As we will see, this leads to an interpretation of data augmentations as counterfactuals in the underlying latent SCM. The assumption that stays invariant as s changes is consistent with the view that content may causally influence style, + s, but not vice versa, see Fig. 1. We therefore formalise their relation as: (u()),
= (c,
(ue: Ug) Pa, X Pu.
where Uc; are independent exogenous variables, and fe, f, are deterministic functions. The latent causal variables (c,s) are subsequently decoded into observations x f(c,s) Given factual obser- vation x = f(cc ) which resulted from (uy, ug), we may ask the counterfactual question: "what would have happened if the style variables had been (randomly) perturbed, all else being equal?" Consider, e.g., a soft intervention [35] on s, i.e., an intervention that changes the mechanism fa to
do(s == (c, Ug UA)).
where a is an additional source of stochasticity accounting for the randomness of the augmentation process (p Psls.4). The resulting distribution over counterfactual observations x f(c², SFF) can be computed from the modified SCM by fixing the exogenous variables to their factual values and performing the soft intervention:
CF c=
S f= ug UA):
n Pus.
This aligns with intuition and assumed problem setting of data augmentations as style corruptions. We note that the notion of augmentation as (hard) style interventions is also at the heart ReLIC [87]. recently proposed,
SSL regularisation term for eettacc-dixerrmmnatte However, ReLIC assumes independence between content and style and does not address identifiability. For another causal perspective on data augmentation in the context of domain generalisation, c.f. [59].
[44, 126].
Theory: block-identifiabilie of the invariant content partition
Our goal is to prove that we can identify the invariant content partition under distinct, weaker set of assumptions, compared to existing results in disentanglement and nonlinear ICA [39, 69, 83, 108, 129]. We stress again that our primary interest is not to identify or disentangle individual (and independent) latent factors 2j. but instead to separate content from style, such that the content variables can be subsequently used for downstream tasks. We first define this distinct notion of block-identiffabiliiy
Definition 4.1 bookkiidetiffaabiitty... We say that the true content partition c = (x) is Mlock-ddentified by a function g X' + Z if the inferred content partition è = g(x)1:n, contains all and only information about c, i.e., there exists an invertible function R - R s.t. è h(c).
Defn. 4. is related to independent subspace analysis [16, 53, 73, 1 114], which also aims to iden- tify blocks of random variables opposed to individual factors, though under an independence assumption across blocks, and typically not within multi-view setting as studied in the present work.
4.1 Generative self-supervised representation learning
First, we consider generative SSL, i.e., fitting generative model to pairs (x, x) of original and augmented views.? We show that under our specified data generation and augmentation process ($ 3),
Thee recently proposed Independently Modulated Component Analysis (IMCA) [64] extension of ICA notable exception, but only allows for trivial dependencies across in the form of shared base measure.
For notational simplicity. we present our theory for pairs (x, X) rather than for two augmented views (x, x'). typically used in practice but also holds for the latter, see for further discussion.
5

---

as well as suitable additional assumptions (stated and discussed in more detail below), it is possible isolate (i.e., block block-identin identify) the invariant content partition. Full proofs are included in Appendix A.
Theorem 4.2 (Identifying content with a generative model), Consider the data generating process described in 3, i.e., the pairs (x, x) of original and augmented views are generated according to (2) and (3) with Pa/z as defined in Assumptions 3.1 and 3.2. Assume further that
(i) f 2 + X is smooth and invertible with smooth inverse (i.e., a diffeomorphism) (i) Pu is a smooth, continuous density on 2 with pxl Pa(z) > 0 almost everywhere;
(mi) for any I (1,-.. Ths) 3A E {1, nat S.I. 1 E A; (A > 0; Р&a is smooth w.r.l. both SA and Sai and for any SA PSAlAA (+Is)) > 0 in some open, non-empty subset containing SA.
If. for a given as (1 S n2 < n). a generative model (fm: PA, P. f) assumes the same generative process ($ 3), satisfies the above assumptions (i)-(iti), and matches the data likelihood,
(x,x(x.)) (x,x(x,x)) x)
v(x, X x X,
then it block-identifies the true content variables via g f-1 in the sense of Defin. 4.1.
Proof sketch. First, show (using (i) and the matching likelihoods) that the representation à g(x) extracted by g is related to the true z by a smooth invertible mapping h = gof such that é = h(z) invariant across (z. 2) almost surely w.r.t. Ps.a. Second, show by contradiction (using (ii), (ii)) that h(.)1m, can, in fact, only depend on the true content e and not on style for otherwise the invariance from step would be violated in a region of the style (sub)space of measure greater than zero
Intuition. Thm. 4.2 assumes that the number of content (n₂) and style (n₂) variables is known, and that there is a positive probability that each style variable may change, though not necessarily on its own, according to (iti). In this case, training a generative model of the form specified in $ 3 (i.c., with invariant content partition and subsets of changing style variables) by maximum likelihood on pairs (x, x) will asymptotically (in the limit of infinite data) recover the true invariant content partition up to an invertible function, i.e., it isolates, or unmixes, content from style.
Discussion.
The identifiability result of Thm. 4.2 for generative SSL is of potential relevance for existing variational autoencoder (VAE) [68] variants such as the GroupVAE [51]," or its adaptive version AdaGVAE [83]. Since, contrary to existing results, Thm. 4.2 does not assume independent latents, it may also provide a principled basis for generative causal representation learning algo- rithms [76, 107, 127]. However, important limitation to its practical applicability is that generative modelling does not tend to scale very well to complex high nigh-dimensional observations, such as images.
4.2 Discriminative self-supervised representation learning
We therefore next turn to discriminative approach, i.c., directly learning encoder function g which leads to a similar embedding across (x, x). As discussed in $ 2, this is much more common for SSL with data augmentations. First, we show that if an invertible encoder g is used, then learning a representation which is aligned in the first n. dimensions is sufficient to block-identify content.
Theorem 4.3 (Identifying content with an invertible encoder). Assume the same data generating process ($ 3) and conditions (i)-(iv) as in Thm. 4.2. Let g X + Z be any smooth and invertible function which minimises the following functional:
Aalign (g) = E(xx))~Px.
g(x) tim
L:ne
(4)
Then g block-identifies the true content variables in the sense of Desinition 4.1.
Proof sketch. First, we show that the global minimum of (4) is reached by the smooth invertible function f-1 Thus, any other minimiser g must satisfy the same invariance across (x, x) used in step 1 of the proof of Thm. 4.2. The second step uses the same argument by contradiction as in Thm. 4.2.
Intuition. Thm. 4.3 states that if under the same assumptions on the generative process as in Thm. 4.2 4.2-we directly learn representation with an invertible encoder, then enforcing align- ment between the first ne latents sufficient to isolate the invariant content partition. Intuitively, invertibility guarantees that all information is preserved, thus avoiding collapsed representation.
*This step is partially inspired by [83]: the technique used to prove the second main step is entirely novel. 9 "wiich also uses fixed content-style partition for multi-view data, but assumes that all latent factors are mutually independent, and that all style variables change between views, independent of the original style:

---

Discussion. According to Thm. 4.3, content can be isolated if, e.g., flow-based architecture [32. 33, 67, 92. 93] is used, or invertibility is enforced otherwise during training [8, 60). However, the applicability of this approach is limited as it places strong constraints on the encoder architecture which makes it hard to scale these methods up to nigh--dimensioal settings. As discussed in sate-oo-theeanl SSL methods such as SimCLR [20]. BYOL [41], SimSi am [21]. or BarlowTwins [128] do not invertible encoders, but instead avoid collapsed representations which would result from naively optimising (4) for arbitrary, non-invertible s-uin different forms of regularisation.
To close this gap between theory and practice, finally, we investigate how to block-identity content without assuming an invertible encoder. We show that, if we add a regularisation term to (4) that encourages maximum entropy of the learnt representation, the invertibility assumption can be dropped. Theorem 4.4 (Identifying content with discriminative learning and non-invertible encoder). Assume the same data generating process ($ 3) and conditions (i)-(iv) as in Thm. 4.2. Let g : X - 10, be any smooth function which minimises the following functional:
L Ean (g) = (x,x))
[(x) - 8(()
H (g(x))
(5)
2]
where H-) denotes the differential entropy of the random variable g(x) taking values in (0, 1) Then g block-identifies the true content variables in the sense of Defn. 4.1.
Proof sketch. First, use the Darmois construction [29, 57] to build function d : C + (0, mapping = (x)()) I:me to uniform random variable. Then g' d o f attains the global minimum of (5) because is invariant across (x, x) and the uniform distribution is the maximum entropy distribution on (0, 1). Thus, any other minimiser g of (5) must satisfy invariance across (x,x) and map to uniform r.v. Then, use the same step 2 as in Thms. 4.2 and 4.3 to show that - Z + (0, 1)th cannot depend on style, i.e., is a function from C (0, 1)me. Finally, we show that h must be invertible since it maps Pe to a uniform distribution, using result from [129].
Intuition. Thm. 4.4 states that if we do not explicitly enforce invertibility of g in Thm. 4.3, additionally maximising the entropy of the learnt representation (i.e., optimising alignment and uni- formity (124)) avoids collapsed representation and recovers the invariant content block. Intuitively, this is because any function that only depends on e will be invariant across (x, x), so it is beneficial to preserve all content information to maximise entropy.
Discussion. Of our theoretical results, Thm. 4.4 requires the weakest set of assumptions, and is most closely aligned with common SSL practice. As discussed in $ 2. contrastive SSL with negative samples using InfoNCE (1)
objective can asymptotically be understood as alignment with entropy regularisation [124], i.e., objective (5). Thm. 4.4 thus provides a theoretical justification for the empirically observed effectiveness of CL with InfoNCE: subject to assumptions, CL with InfoNCE asymptotically isolates content, i.e., the part of the representation that is always left invariant by augmentation. For example, the strong image classification performance based on representations learned by SimCLR [20]. which uses color distortion and random crops as augmentations, can be explained in that object class is a content variable in this case. We extensively evaluate the effect of various augmentation techniques on different ground -tuth latent factors in our experiments in $ 5. There is also an interesting connection between Thm. 4.4 and Barlow owrlowswins [128], which only uses positive pairs and combines alignment with a redundancy reduction regulariser that enforces decorrelation between the inferred latents. Intuitively, redundancy reduction is related to increased entropy: g" constructed in the proof of Thm. 4.4-ann thus also any other minimiser of (5) (5)-attains the global optimum of the BarlowTwing objective, though the reverse implication may not hold.
5 Experiments
We perform two main experiments. First, we numerically test our main result, Thm. 4.4, in a fully- controlled, finite sample setting ($ 5.1), using CL to estimate the entropy term in (5). Second, we seek to better understand the effect of data augmentations used in practice (S 5.2). To this end, we introduce new dataset of 3D objects with dependencies between number of known ground-truth factors, and use it to evaluate the effect of different augmentation techniques on what is identified as content. Additional experiments are summarised in $ 5.3 and described in more detail in Appendix C.
5.1 Numerical data
Experimental setup. We generate synthetic data as described in $ 3. We consider n. = n, = 5. with content and style latents distributed as e ~ N E.) and s/c ~ N( + Bc, E.), thus allowing
7

---

posspp spi
class
hueng
hueepp
pb)
rofabj
huc.es
Figure 2: (Left) Causal graph for the Causal3Dident dataset. (Right) Two samples from each object class.
for statistical dependence within the two blocks (via De and 2n,) and causal dependence between content and style (via B3. For f. we use 3-layer MLP with LeakyReLU activation functions." 10 The distribution PA over subsets of changing style variables is obtained by independently flipping the same biased coin for each 8;- The conditional style distribution is taken as Paalea = (SA, EA). We train an encoder g on pairs (x, x) with InfoNCE using the negative L2 loss as the similarity measure, ie., approximate (5) using empirical averages and negative samples. For evaluation, use kernel ridge regression [88] to predict the ground truth e and s from the learnt representation report the R² coefficient of determination. For more detailed account, we refer to Appendix D.
g(x) and
Results. In the inset table, we report mean + std. dev. over 3 random seeds across four generative pro-
Generative process
R² (nonlinear)
cesses of increasing complexity covered by Thm. 4.4:
p(chg.)
Stat.
Cau.
Content
Style
"p(chg.)". "Stat.", and "Cau." denote respectively the change probability for each 8,, statistical dependence 0.75
1,0
x
x
1,00 + 0.00
0.07 1 0.00
0.75
x
x
1.00 0.00
0.06 0.05
0,98 0.03
0.37 0.05
within blocks (E. * I * Σ.), and causal depen- dence of style content (B 0). An R² close one indicates that almost all variation is explained by è, i.c., that there is a 1-1 mapping, as required by Defn. 4.1. As can be seen, across all setlings, content is block-identifed. Regarding style, we observe an increased score with the introduction of dependencies, which we explain in an extended discussion in Appendix C.I. Finally, we show in Appendix C. that a high R2 score can be obtained even if we use linear regression to predict from è (R2 0.98 0.01, for the last row).
0.75
/
0.99 + 0.01
0,80 + 0.08
5.2 dingh-dimensional images: Causal3Dident
Causal3DIdent dataset. 3Dldent [129] is benchmark for evaluating identifiability with rendered 224 x 224 images which contains hallmarks of natural environments (e.g. shadows, different lighting conditions, 3D object). For influence of the latent factors on the renderings, Fig. 2 of [129]. In 3DIdent, there is a single object class (Teapot [89]), and all 10 latents are sampled independently. For Didsat3DIdent, we introduce six additional classes: Hare [121]. Dragon [110], Cow [62], Armadillo [70], Horse [98], and Head [111]; and impose a causal graph over the latent variables, see Fig. 2. While object class and all environment variables (spotlight position & hue, background hue) are sampled independently, all object latents are dependent," see Appendix for details. !2
Experimental setup. For g. we train a convolutional encoder composed of a R8 [46] and additional fully-connected layer, with LeakyRel.U activation. As in SimCLR [20], we use InfoNCE with cosine similarity, and train on pairs of augmented examples (x, X'). As n, is unknown and variable depending the augmentation, we fix dim(c) = 8 throughout. Note that we find the results to be, for the most part, robust to the choice of dim(c), see inset figure. We consider the following data augmentations (DA): crop, resize & flip; colour distortion (jitter & drop); and rotation E (90', 180", 270"). For comparison, we also consider directly imposing a content-style
DA: crop (large) color distortion
cless
0.3
pan_apl
huc bg
rel obj
02-
8
30
12
dlmce)
"chsen to lead to invertibility almost surely by following the settings used by previous work [54, 55] "e.g.. our causal graph entails hares blend into environment (object hue centered about background spotlight hue), form active camouflage observed in Alaskan [78], Arctic [2], Snowshoe hares.
We made the Dldeat dataset publicly available this URL.
8

---

Table 1: Causal3Dident results: I2 mean 1 std. dev. over random seeds. DA: data augmentation. LT: latent transformation, bold: R 2 0.5, red: R² < 0.25, Results for individual axes of object position & rotation are aggregated, see Appendix C for the full table.
Views genernted by
Class
Positions
Hues
Rotations
object
spotlight
object
spollight
background
DA: colour distortion
0-42 + 0.01
0.61 +: 0.10
0.17 = 0.00
0.10 0.01
0:01 0.00
0.01 + 0.00
0.33 = 0.02
LT: change
1.00
0.33
0.91 0.00
0.30
0.00
0.30
DA: crop (large)
0.28 1.3::00000 0.04
0.09 1: 0.08
0.21 0.13
0.87 = 0.00
0.09 0.00
1.00 = 0.00
0.02 = 0.02
DA: crop (small)
0.14 1 0.00
0.00 0.01
0.00 4: 0.01
0.00 4: 0.00
0.00 4 0.00
1,00 0.00
0.00 :: 0.00
change positions
1,00 0.00
+ 0:23
0.00 + 0.01
0.46 + 0.02
0.00 0.00
0.00
0:28 = 0.01
DA: crop (large) colour distortion
0.97 + 0.00
0.59 + 0.07
0.59 + 0.05
0.28 + 0.00
0.01 0.01
0.01 + 0.00
0.74 + 0.03
crop (small) + colour distection
1.00
0,93::0.00
0.30
0.00
0.08
0.56
LT: change positions hues
1.00 =: 0.00
0.22 1: 0.22
0.07 4: 0.08
0.32 4: 0.02
0.00 0.01
0.02 1: 0.03
0.34 =: 0.06
DA: rotation
0.33 + 0.06
0.17 0.08
0.23 0.12
0.83 0:01
0:30
0.99 D:00
0.05
LT: change rotations
1,00 = 0.00
0.53 :: 0.33
0,90 +: 0.00
0.11 = 0.00
0.00 -: 0.00
0.97 0.00
0.28 = 0.00
DA: rotation colour distortion
0.59 0.01
0.58 0.06
0.21 = 0.01
0.12 + 0.02
0:01 0.00
0.01 + 0.00
0.33 0.01
LT: change rotations hues
1.00
1 0.34
1: 0.00
0.30
0.00 0.00
0.00
0.28
partition by performing a latent transformation (LT) to generate views. For evaluation, we use linear logistic regression to predict object class, and kernel ridge to predict the other latents from é."
Results. The results are presented in Tab. 1. Overall, our main findings can be summarised as:
(i) it can be difficult to design image-level augmentations that leave specific latent factors invariant;
i augmentations & latent transformations generally have similar effect on groups of latents; (iii) augmentations that yield good classification performance induce variation in all other latents.
We observe that, similar directly varying the hue latents, colour distortion leads discarding of hue information as style, and a preservation of (object) position as content. Crops, similar to varying the position latents, lead to discarding of position as style, and preservation of background and object hue as content, the latter assuming crops are sufficiently large. In contrast, image-level rotation affects both the object rotation and position, and thus deviates from only varying the rotation latents.
Whereas class is always preserved as content when generating views with latent transformations, when using data augmentations, we can only reliably decode class when crops & colour distortion are used in conjunction a result which mirrors evaluation on ImageNet [20]. As can be seen by our evaluation of crops & colour distortion in isolation, while colour distortion leads to a discarding of hues as style, crops lead to discarding of position & rotation as style. Thus, when used in conjunction, class is isolated as the sole content variable. See Appendix C.2 for additional analysis.
5.3 Additional experiments and ablations
We also perform an ablation on dim(O (é) for the synthetic setting from $ 5.1, see Appendix C.1 for details. Generally, we find that if dim(c) < The. there is insufficient capacity to encode all content, so a lower-dimensional mixture of content is learnt. Conversely, if dim(é) > ne, the excess capacity is used encode some style information (as that increases entropy). Further, we repeat our analysis from $ 5.2 using BarlowTwinee also loosely related to Thm. 4.4. The results mostly mirror those obtained for SimCLR and presented in Tab. 1. see Appendix C.2 for details. Finally, we ran the same experimental setup as in $ 5.2 also on the MPI3D-reat dataset [38] containing >> 1 million real images with ground-truie annotations of 3D objects being moved by robotic arm. Subject to some caveats, the results show a similar trend as those on Dadea,3 see Appendix C.3 for details.
[128] (instead of SimCLR) which, as discussed at the end of Š 4.2, is
Discussion
Theory vs practice. We have made an effort to tailor our problem formulation ($ 3) to the setting of data augmentation with pretenngnreserving transformations. However, some of our more technical assumptions, which are necessary to prove block-identifiabiiityy of the invariant content partition, may not hold exactly in practice. This is apparent, e.g., from our second experiment ($ 5.2), where we observe that that-while class should, in principle, always be invariant across views (i.e., content)-when when
'eeee Appendix C.2 for results with linear regression, as well as evaluation using a higher dimensoaim intermediate layer by considering projection head [20].

---

using only crops, colour distortion, or rotation, g appears to encode shortcuts [37, 96]. Data augmentation, unlike latent transformations, generates views x which are not restricted the 1 1-dim. image manifold X' corresponding to the generative process of Causal3Dident, but may introduce additional variation: c.g,, colour distortion leads to rich combination of colours, typically a 3-dim. feature, whereas Causal3DIdent only contains one degree of freedom (hue). With additional factors, any introduced invariances may be encoded content in place of class. Image-level augmentations also tend to change multiple latent factors in a correlated way, which may violate assumption (ii)) of our theorems, i.e., that psals is fully-supporter locally. We also assume that is continuous, even though Causal3Dldent and most disentanglement datasets also contain discrete latents. This is a very common assumption in the related literature [39, 54, 57, 58, 63, 69, 82, 83, 129] that may be relaxed in future work. Moreover, our theory holds asymptotically and at the global optimum, whereas in practice solve non-convex optimisation problem with finite sample and need to approximate the entropy term in (5), e.g., using a finite number of negative pairs. The resulting challenges for optimisation may be further accentuated by the higher dimensionality of X' induced by image-level augmentations. Finally, we remark that while, for simplicity, we have presented our theory for pairs (x, x) of original and augmented examples, in practice, using pairs (x, x') of two augmented views typically yields better performance. All of our assumptions (content invariance, changing style, etc) and theoretical results still apply to the latter case. We believe that using two augmented views helps because it leads to increased variability across the pair: for if and x differ from x in style subsets A and A', respectively, then X') differ from each other (a.s.) in the union A U A'.
Beyond entropy regularisation. We have shown a clear link between identifiable maximum entropy approach to SSL (Thm. 4.4) and SimCLR [20] based on the analysis of [124], and have discussed an intuitive connection to the notion of redundancy reduction used in BarlowTwin [128]. Whether other types of regularisation such as the architectural approach pursued in BYOL [41] and SimSiam [21] can also be linked to entropy maximisation, remains an open question. Deriving similar results to Thm. 4.4 with other regularisers is a promising direction for future research, c.f. [116).
The choice of augmentation technique implicitly defines content and style. As we have defined content the part of the representation which is always left invariant across views, the choice of augmentation implicitly determines the content-style partition. This is particularly important to keep in mind when applying SSL with data augmentation to safety-critical domains, such medical imaging. We also advise caution when using data augmentation to identify specific latent properties, since, observed in $ 5.2, image-level transformations may affect the underlying ground-trutt factors in unanticipated ways. Also note that, for a given downstream task, we may not want to discard all style information since style variables may still be correlated with the task of interest and may thus help improve predictive performance. For arbitrary downstream tasks, however, where style may change in an adversarial way, it can be shown that only using content is optimal [103].
What how information is encoded. We focus on what information is learnt by SSL with data aug- mentations by specifying a generative process and studying identifiability of the latent representation. Orthogonal to this, a different line of work instead studies how information is encoded by analysing the sample complexity needed to solve a given downstream task using a linear predictor [3, 74, 116- 119]. Provided that downstream tasks only involve content, can draw some comparisons. Whereas our results recover content only up to arbitrary invertible nonlinear functions (see Defn. 4.1), our problem setting is more general: [3, 74] assume (approximate) independence of views (x, x) given the task (content), while [118, 119] assume (approximate) independence between one view and the task (content) given the other view, neither of which hold in our setting.
Existing represennation learning approaches typically assume mutually independent latents, though dependencies clearly exist in nature [106]. We demonstrate that in a non-i.i.d scenario, e.g.. by constructing multiple views of the same example with data augmentation, we can learn useful representations in the presence of this neglected phenomenon. More specifically, the present work contributes, to the best of our knowledge, the first: (i) identifiability result under arbitrary dependence between latents; and (ii) empirical study that evaluates the effect of data augmentations not only on classification, but also on other continuous ground-truth latents. Unlike existing identifiability results which rely on change as learning signal, our approach aims to identify what is always shared across views, i.e., also using invariance as learning signal. We hope that this change in perspective will be helpful for applications such as optimal style transfer or disentangling shape from pose vision, and inspire other types of counterfactual training to recover more fine-grained causal representation.
Conclusion.
"lass is distinguished by shape, a feature commonly unused in downstream tasks on natural images [36]
10

---

Acknowledgements
We thank: the anonymous reviewers for several helpful suggestions that triggered improvements in theory and additional experiments; Cian Eastwood, Ilyes Khemakem, Michael Lohaus, Osama Makansi, Ricardo Pio Monti, Roland Zimmermann, Weiyang Liu, and the MPI Tübingen causality group for helpful discussions and comments; Hugo Yèche for pointing earlier version of the manuscript; github user TangTangFei for catching a bug in the implementation of the experiments from $ 5.1 (that has been corrected in this version); and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting YS.
mistake in $2 of an
Funding Transparency Statement
WB acknowledges support via his Emmy Noether Research Group funded by the German Science Foundation (DFG) under grant no. BR 6382/1-1 as well as support by Open Philantropy and the Good Ventures Foundation. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S 18039A, 011S18039B, and by the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645.
References
[1] Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In Proceedings of the IEEE International Conference Computer Vision, pages 37-45, 2015.
[2] Aretic Wildlife. Churchill Polar Bears
2021.
[3] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. theoretical analysis of contrastive unsupervised representation learning. In 36th International Conference on Machine Learning, pages 9904-9923. International Machine Learning Society (IMLS), 2019.
[4] Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances Neural Information Processing Systems 32, pages 15509-15519, 2019.
[5] Alexei Baevski. Steffen Schneider, and Michael Auli. vq-wav2vee: Self-supervised learning discrete speech representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[6] Alexci Bacvski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems 33, 2020.
[7] Suzanna Becker and Geoffrey Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):
1992.
[8] Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning, pages 573-582. PMIR,, 2019.
[9] Anthony J Bell and Terrence Sejnowski. An dnformation-maximizatiii approach to blind separation and blind dcconvolution. Neural computation, (10):1129-1153 1995.
[10] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1828, 2013.
[11] Blender Online Community. Blender a 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam, 2021.
[12] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger. and Roopak Shah. Signature verification using a "siamese" time delay neural network. Advances in Neural Information Processing Systems, 737-744, 1993.
[13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot leamners. Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
[14] Christopher Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. arXiv preprint arXiv:1804.03599, 2018.
[15] J-F Cardoso. Infomax and maximum likelihood for blind source separation. IEEE Signal processing letters, 4(4): 124, 1997.
11

---

[16] Michael A Casey and Alex Westner. Separation of mixed audio sources by independent subspace analysis. In ICMC, pages 154-161, 2000.
[17] Olivier Chapelle and Bernhard Schölkopf. Incorporating invariances in nonlinear SVMs. In Advances in Neural Information Processing Systems 14, pages 609 616, Cambridge, MA, USA, 2002. MIT Press.
[18] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in vaes. In Advances in Neural Information Procexsing Systems, pages 2615-2625, 2018.
[19] Shuxiao Chen, Edgar Dobriban, and Jane H Lec. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21245):1-77, 2020.
[20] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages
1597-1607. PMLR, 2020.
[21] Xinlei Chen and Kaiming He. Exploring simple siamese representation leamning In IEFE Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[22] S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. In IEEE Conference Computer Vision and Pattern Recognition, volume 1, pages 539-546, 2005.
[233] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2461- 2505, 2011.
[24] Pierre Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.
[25] Thomas M Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
[26] Ekin D Cubuk, Barret Zoph. Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. IEFE Conference on Computer Vision and Pattern Recognition, pages 13-1223, 2019.
[27] Ekin D Cubuk, Barret Zoph, Jonathon Shiens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In IEEE Conference on Computer Vision and Puttern Recognition Workshops, pages 702-703, 2020.
[28] Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré. A kernel theory of modern data augmentation. In International Conference on Machine Learning, pages 1528-1537. PMLR, 2019.
[29] G Darmois. Analysc des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, page 231, 1951.
[30] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4171-4186, 2019.
[31] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
[32] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. arXiv preprint 11Xiv:1410.8516. 2014.
[33] Laurent Dinh, Jascha Dicl-Dickstein, and Samy Bengio. Density estimation using real NVP. Sth
International Conference on Learning Representations, 2017.
[34] Carl Doersch, Abhinav Gupta, and Alexci A Efros. Unsupervised visual representation learning by context prediction. In IEEE International Conference on Computer Vision, pages 1422-1430, 2015.
[35] Frederick Eberhardt and Richard Scheines. Interventions and causal inference. Philosophy of science, 74 (5):981 9919999, 2007.
[36] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trainee cnns are biased towards texture; increasing shape bias improves accuracy and rohustness. International Conference on Learning Representations (ICLR), 2019.
[37] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthis Bethge, and Felix A. Wichmann. Shortcut Icarning in deep neural networks. Nature Machine Intelligence, 2(11):6656673, Nov 2020.
[38] Muhammad Walced Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkow, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: new disentanglement dataset. Advances in Neural Information Processing Systems, 32:15740- 1111511777511111111 15751,
12

---

[39] Luigi Gresele. Paul K. Rubenstein, Arash Mehrjou. Francesco Locatello, and Bernhard Schölkopf. The Incomplete Rosetta Stone problem: Identifiability results for multi-view nonlinear ICA. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI, 2019.
[40] Luigi Gresele, Julius von Kügelgen, Vincent Stimper, Bernhard Schölkopf. and Michel Besserve. Inde- pendent mechanism analysis, new concept? In Advances in Neural Information Processing Systems. 2021.
[41] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bi- lal Piot, Koray Kavukcuogiu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems 33, 2020.
[42] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304, 2010.
[43] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The Journal of Machine Learning Research, 13:307-361, 2012.
[44] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning invariant mapping. In IEEE Conference on Computer Vision and Pattern Recognition, volume 2. pages 1735-1742, 2006.
[45] Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pages 939-948. PMLR, 2020.
[46] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[47] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference Computer Vision and Pattern Recognition, pages 9726 9735, 2020.
[48] Olivier J. Hénaff. Data-efficient image recognition with contrastive predictive coding, In Proceedings the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4182-4192, PMI.R, 2020..
[49] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In Sth International Conference on Learning Representations, 2017.
[50] R. Devon Hjelm, Alex Fedorov, Samuel Maroic-Marcchloon. Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, 2019.
[51] Haruo Hosoya. Group-based learning of disentangled representations with generalizability for novel contents. In IJCAI, pages 2506-2513, 2019.
[52] Andrew G. Howard. Some improvements on deep convolutional neural network based image classification, 2013.
[53] Aapo Hyvärinen and Patrik Hoyer. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation, 1257): 2000.
[54] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. In Advances in Neural Information Processing Systems, pages 3765-3773, 2016.
[55] Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. In Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.
[56] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411 14400, 2000.
[57] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and unique- ness results. Neural Networks. 12(3):422-433 1999.
[58] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Staristies, pages 859- 868. PMLR, 2019.
[59] Maximilian Issc, Jakub M Tomezak, and Patrick Forré. Selecting data augmentation for simulating interventions. In International Conference on Machine Learning, pages 4555-4562. PMLR, 2021.
[60] Jörn Henrik Jacobsen, Aroold Smeulders, and Edouard Oyallon. i-RevNet: Deep invertible networks. In International Conference Learning Representations, 2018
13

---

[61] Edwin T Jaynes. On the rationale of maximum-entropp methods. Proceedings of the IEEE, 70(9):
939-952 1982.
[62] Keenan's 3D
Model
Repository.
Keenan's 3D
Model
Repository
https://www.cs.cmuu wdw./d///wuuuu/ucccc.c.....w//// smerace/Poject//Medeepppppiiii
2021.
[63] Ilyes Khemakhem, Diederik Kingma. Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoen- coders and nonlinear ICA: unifying framework. In The 23nd International Conference on Artificial Intelligence and Statistics, AISTATS, volume 108, pages 2207-2217, 2020.
[64] Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, and Aapo Hyvärinen. lce-beem: Identifiable conditional energy-based deep models based on nonlinear ICA. Advances in Neural Information Processing Systems 33, 2020.
[65] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pages (649-2658,, PMLR, 2018,
[66] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference Learning Representations, 2015.
[67] Diederik P Kingma and Prafulla Dhariwal. Glow: generative flow with invertible 1x 1 convolutions. In Advances in Neural Information Processing Systems, pages 10236-10245, 2018.
[68] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations (ICLR), 2014.
[69] David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. International Conference on Learning Representations (ICLR). 2021.
[70] Venkat Krishnamurthy and Marc Levoy. Fitting smooth surfaces to dense polygon meshes. In John Fujii, editor. Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, New Orleans, LA, USA, August 4-9, 1996, pages 313-324, ACM, 1996.
[71] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
[72] Brenden M Lake, Tomer D Ullman, B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[73] Quac Le, Will Zou, Serena Yeung, and Andrew Y Ng. Learning hierarchical invariant spatio- temporal features for action recognition with independent subspace analysis. In CVPR 2011, pages 3361-3368. IEEE, 2011.
[74] Jason D Lee, Qi Lei, Nikunj Saunshi. and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. arXiv preprint arXiv:2008.0106: 2020.
[75] Te- Ne-Won Lee, Mark Girolami, and Terrence J Sejnowski. Independent component analysis using an extended infomax algorithm for mixed subgaussian and supergaussian sources. Neural computation, 11 (2):417-441, 1999.
[76] Felix Leeb, Yashas Annadani, Stefan Bauer, and Bernhard Schölkopf. Structural autoencoders improve representations for generation and transfer. arXiv preprint arXiv:2006.077 077996 2020.
[77] Erich L Lehmann and George Casella. Theory of point extimation. Springer Science & Business Media, 2006.
[78] Lepus Othus. Animal Diversity Web. http://aiinalliiessssrrrccuucuus othus/, 2021.
[79] Ralph Linsker. Self-organization in perceptual network. Computer, 105-:1111117,, 1988.
[80]
Ralph Linsker. An application of the principle maximum information preservation to linear systems. In Advances in Neural Information Processing Systems, pages 186-194, 1989.
[81] Yinhan Liu, Myle Ou,, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeulemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[82] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf. and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 41 4124, 2019.
[83] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervisedd disentanglement without compromises. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 6348 6359, PMLR, 2020.
[84] Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations. In 6th International Conference Learning Representations, 2018
14

---

[85] Chaochao Lu,, Yuhuai Wu, Jose Miguel Hernández-Lobato, and Bernhard Schölkopf. Nonlinear invariant risk minimization: A causal approach. arXiv preprint 10X1:22102.12353 2021.
[86] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionaliyy. In Advances in Neural Information Processing Systems, pages 3111-311111 1111119 2013.
[87] Jovana Mitrovic, Brian MeWilliams, Jacob C. Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. 9th International Conference on Learning Representations, 2021.
[88] Kevin P Murphy. Machine learning: probabilisticc perspective. MIT Press, Cambridge, MA, 2012.
[89] Martin Edward Newell. The Utilization of Procedure Models in Digital Image Synthesis. PhD thesis, The University of Utah, 1975. AA17529894.
[90] Mehdi Norozzi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 19-84. Springer, 2016.
[91] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation lcarning with contrastive predictive coding, arXiv preprini /
2018
[92] George Papamakarios. Theo Pavlakou, and lain Murray, Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pages 2338 2347, 2017.
[93] George Papamakarios. Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1-64 2021.
[94] Judea Pearl. Causality. Cambridge university press, 2009.
[95] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[96] Mohammad Pezeshki, Sékou-Oumat Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: learning proclivity in neural networks. arXiv preprint 1:Xiv:2011.09465 2020.
[97] Ben Poolc, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
[98] Emil Praun, Adam Finkelstein, and Hugues Hoppe. Lapped textures. In Judith R. Brown and Kurt Akeley, editors, Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, New Orleans, LA, USA, July 23-28, 2000, pages 465-470. ACM, 2000.
[99] Alec Radford, Karthik Narasimhan, Tim Salimans, and llya Sutskever. Improving language understanding by generative pre-training. Technical report, OpenAI, 2018
[100] Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-takk self-supervised learning for robust speech recognition. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 9989-6993, 2020.
[101] H. Richard, L. Gresele, A. Hyvarinen. B. Thirion, A. Gramfort, and P. Ablin. Modeling shared responses in neuroimaging studies through multiview ica. In Advances in Neural Information Processing Systems 33, pages 19149-19162. Curran Associates, Inc., December 2020.
[102] Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning, pages 9030 20009009..000000 PMLR, 2021.
[103] Matco Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1399-134 2018.
[104] Steffen Schneider, Alexei Baevski, Ronan Collobert. and Michael Auli. wav2vec: Unsupervised pre- training for speech recognition. In Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, pages 3465 3469, 2019.
[105] Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv:1911. 1.10500, 2019.
[106] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Kc, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 2021.
[107] Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Disentangled generative causal representation learning. arXiv preprint arXiv: 20X::20100026::: 2020.
[108] Rui Shu, Yining Chen, Abhishck Kumar, Stefano Ermon, and Ben Poolc. Weakly supervised disentangle- ment with guarantees. In 8th International Conference on Learning Representations, 2020.
[109] Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ica with general incompressible-foor flow networks (gin). In International Conference on Learning Representations, 2020.
15

---

[110] Stanford Scanning Repository.
The
Stanford
3D
Scanning
Repository.
Swanfoord.duu/aattt/DDssceeee 2021.
[111] Suggestive Contour Gallery.
Suggestive
Contour
Gallery
2021.
[112] Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, pages 6056 6065. PMLR, 2019.
[113] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguclov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. Computer Vision and Pattern Recognition (CVPR). 2015.
[114] Fabian Theis. Towards general independent subspace analysis. Advances in Neural Information Processing Systems, 19:1361-1366 2006.
[115] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision ECCV 2020, volume 12356, pages 776-794. Springer, 2020.
[116] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In Marina Mcila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning,, volume 139, pages 10268-10278, 2021.
[117] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals topic posterior information to linear models. arXiv preprint arXiv:2003.0222 2020.
[118] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithumic Learning Theory, pages 1179-1206, PMLR, 2021.
[119] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-PPillipe Morency. Self-supervised learning from multi-view perspective. In International Conference on Learning Representations, 2020.
[120] Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[121] Greg Turk and Mare Levoy. Zippered polygon meshes from range images. In Dino Schweitzer, Andrew S. Glassner, and Mike Keeler, editors, Proceedings the 21th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1994, Orlando, FL, USA, July 24-29, 1994, pages 311-318. ACM, 1994.
[122] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and compos- ing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, pages 1096- 1103, 2008.
[123] Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. ICLR Workshop on "Causal Learning for Decision Making". 2020.
[124] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9929 9939. PMLR, 2020.
[125] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEFE International Conference on Computer Vision, pages 2794-2802, 2015.
[126] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Conference Computer Vision and Pattern Recognition, CVPR, pages 3733-3742. IEEE Computer Society, 2018
[127] Mengyue Yang. Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Structured causal disentanglement in variational autoencnder. arXiv preprint arXiv:2004.0869) 2020.
[128] Jure 7bootar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 12310-12322 2021.
[129] Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 12979-12990, 2021.
16

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the "Theory vs practice" paragraph, as well as lines 409-4 109-411 in $ 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] While our work is theoretical in nature, we discuss some potential negative impacts in lines 405-409 where we caution against a naive use of data- fata--ugmentation in safety-critien domains without extensive prior study.
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Our problem formulation formally described in detail in $ 3. All our theoretical results are accompanied by clear and complete account of all necessary assumptions.
(b) Did you include complete proofs of all theoretical results? [Yes] Due to space constraints, and since our proofs introduce novel techniques and nontrivial, we only sketch the high-level ideas behind our proofs in the main paper. The full proofs with detailed explanations are included in Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL)? [Yes] The Causal3Dldent dataset is included as a public URL, see the hyperlink in footnote 12. Code to reproduce our experiments is publicly available on github.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Provided Appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] All results presented in $ 5 report the mean and standard deviation over 3 random seeds also stated in the main paper.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Provided in Appendix
4. If you are using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] The 3DIdent dataset, as well as the 3D models used to extend are cited in 5.2.
(b) Did you mention the license of the assets? [Yes] Provided in Appendix B
(c) Did you include any assets either in the supplemental material as URL? [Yes] See question 3 (a): the newly-introduced Causal3Dident dataset has been publicly relcased. (d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes] We did not use/curate any data from other people, see Appendix B for discussion of this point.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The data does not contain any personally identifiable information or offensive content, see Appendix B for discussion of this point.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica- ble? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board
(IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "u7wIAaAqIOJ", "reviewer": "Reviewer_Pvr4", "summary": "This paper studies identifiability of the latent representation when samples are augmented in a way that preserves the \"content\" component of the latent variable while changing the \"style\" component. By focusing on an approximate (asymptotic) form of the InfoNCE cost function, the paper provides theoretical results on identifiability and experimental results on synthetic datasets, where style variables depend on content variables.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has some novelty and the theoretical development appears correct. However, I think the issues listed below overweigh its advantages.\n\nMajor:\n- There is somewhat of a disconnect between the theory and experiments, some of which are discussed at the end of the paper. I would like to point out an aspect that was not discussed and that appears to be responsible for some of the discrepancy between the theory and the experimental results: the theory considers the content as a continuous variable, but the proposed dataset and the associated experiment has discrete object classes.\n\n- Baseline comparisons are missing, especially in the second experiment. The cited literature, where the latent factors may be assumed to not depend on each other, would provide meaningful baselines. \n\n- It would be very useful to include a non-synthetic dataset. Some common choices in the cited literature include MNIST, FashionMNIST, etc.\n\nMinor: \n- line 289: f^-1 acts on x, why is the domain Z?\n- Would the authors please explain why the negative L2 is used in one experiment and cosine similarity in the other experiment? I wonder if there is insight to be gained through these choices.\n- The paper focuses on images as input data, which is fine, but there are components of the development that are agnostic to the underlying data. It could be useful, perhaps in the introduction only, to consider unstructured datasets as well.\n\n--- Originality: The theoretical development is useful and novel.\n\n--- Quality: The paper is well written.\n\n--- Clarity: The expressions and explanations are clear. Proofs are detailed enough.\n\n--- Significance: I think the disconnect mentioned above and the already discussed caveats in the experimental results significantly decrease the significance of the paper.\n\n------------------------- UPDATE -------------------------\nI decided to increase my score from 5 to 6, mostly in consideration of the authors’ positive and satisfying response on discrete vs. continuous content latents. Please see the discussion with the authors.\n\n------------------------- UPDATE 2 -------------------------\nI decided to further increase my score from 6 to 7, after the latest improvements the authors suggested and a re-evaluation of the paper overall.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~7 hours"}, "extra_text": {}}, {"review_id": "p3QWsTE_Gxn", "reviewer": "Reviewer_Lc8M", "summary": "This paper studies the effect of data augmentation in self-supervised representation learning. They take a latent variable perspective where there is a content component and style component of the latent variable, and consider settings where the content is invariant to the augmentation whereas the style component changes a lot under the data augmentation. In the latent variable model, arbitrary dependencies between the content and style latent variables are allowed. \n \nThe theoretical contribution of this paper is to show that, under certain assumptions on how the augmentation influences the latent variables, 1) a generative model for the data would perfectly identify the true content variables (up to an invertible function) and 2) minimizing the (entropy-regularized) squared difference between encoder outputs on two different augmentations of the data would also give an encoder which perfectly recovers the content variables (also up to an invertible function). The assumptions for these results are that the augmentations don't change the content latent variable and only the style latent variable, and that the augmentations can map the coordinates of the style latent variables to any value. \n\nThis paper also provides experiments for verifying aspects of the theory, first on synthetic Gaussian data, and also in a semi-synthetic setting of rendered images of 3D objects where different latents control different aspects of the image. They compare the effect of using InfoNCE objective on pairs of examples obtained via data augmentation against the effect of using InfoNCE objective on pairs of examples obtained via directly modifying the style latent variables. They show that training InfoNCE with modifying the latent variables does indeed preserve a lot of content information while discarding style information. Data augmentation is observed to behave in a similar way in some instances, though the equivalence is not exact. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is generally well-written, and the perspective of analyzing data augmentation through the influence on the latent variables is novel and quite interesting. However, there are shortcomings of the results: 1) the theoretical results are for a quite idealized setting with strong assumptions, and appear to be somewhat brittle and may not translate well outside of the idealized scenarios and 2) the empirical results are not very convincing in terms of mapping the data augmentations to the latent variable transformations. More detailed comments below.\n\nOriginality: the perspective of using data augmentation to disentangle content and style is quite interesting and novel, to my best knowledge.  \n\nQuality: The theorems are technically sound, but require strong assumptions to hold, and do not necessarily provide a complete picture on why data augmentation is useful in self-supervised learning. First, they require the augmentation to align perfectly with the style variables. In particular, Condition (iv) of Theorem 4.2 seems concerning because it requires the distribution on style change induced by the augmentation to be supported almost everywhere. This is likely unrealistic for data augmentations in practice. Second, in this result, there is no quantifiable bound on the complexity of the mapping between $\\hat{z}$, the latent recovered in the identifiability result, and $z$, the true latent. \n\nFurthermore, Theorem 4.3 and 4.4 require knowing the exact dimension of the content variable to set the dimension of the encoder -- this seems quite restrictive. Do the authors have a sense of what happens if there is a dimension mismatch here? \n\nThere also appears to be large mismatches between the theoretical model of data augmentation and what actually happens in practice in Table 1, though this is already addressed by the authors in Section 6. \n\nClarity: The paper is generally well-written, and the theoretical results are clear and easy to follow. \n\nSignificance: The idea of reasoning about data augmentation based on their effect on the latent space is potentially impactful and something for future work to build on. \n\n========================================================================================\n\nUpdate post rebuttal: I'd like to thank the authors for the clarifications and answers to my questions. My overall opinion of the paper is still the same, so I'm keeping my score at 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "oATDQ_XV8jj", "reviewer": "Reviewer_D9kF", "summary": "Two contributions of this work are (1) an identifiability result for self-supervised learning (SSL), (2) a new synthetic dataset and empirical results evaluating the theoretical results and it’s comparison to standard SSL techniques with data augmentation. \n\nTheoretically, authors prove that using\n(a) “anchor” samples $x$ from a generative model which generates data using (potentially dependent) content $c$ and style $s$ latent random factors through an invertible mapping $x=f(c,s)$, and \n(b) positive samples generated by the same content signal $c$ and an augmented style signal $\\tilde{s}$ which is independent of $c$ given $s$,\none can learn a perfect encoder $\\hat{c} = g(x)$ using a regularized SSL loss or by generative modeling, such that $\\hat{c}$ contains all the information to perfectly predict $c$ in a statistical sense. This means that under some regularity conditions, SSL approaches using data augmentation can learn to isolate content factors (invariant to augmentation) from style factors (which can vary under augmentations). This identifiability result follows the recently popular line of similar results in non-linear ICA and representation learning thoery. However, different from previous results this result does not need independence of latent (content or style) random factors.\n\nNext authors provide a new synthetic dataset to quantify the SSL representation learning. They quantify the performance of SimCLR contrastive (different from non-contrastive theory) SSL representation learning method on this dataset under their generative model and standard data augmentation methods. This showcase both the validity and the limitations of their theory and assumption.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n1. This is the first such identifiability result in SSL, which is able to explain the empirical success of SSL methods. This could lead to better framework to understand SSL methods which are currently poorly understood.\n2. Although I couldn’t check the math very carefully, part of the analysis seems novel. These could be useful in future results.\n3. Novel Benchmark dataset\n4. Fair empirical study to validate their theory and assumption. Fair discussion of limitations of the work.\n5. Good discussion to past relevant works.\n6. Very clear exposition of the contributions. Provides most of the details of empirical study (see below).\n7. Discussion of open theory questions.\n8. Provides guidelines for selecting data augmentation. Also compares the usage of pre-final layer as downstream embedding (usage of poorly-understood “projection” MLP).\n\n### Questions\n1. No empirical comparison to state of the art contrastive methods like MoCo v2 and non-contrastive methods like SimSiam and Barlow Twins.\n2. Although the theory is for non-contrastive SLL loss with regularization, the experiments use constrastive SimCLR loss. It not clear how (31) approximately match (5).\n3. Table 1: What is “latent transformation (LT)”. It is not clear how this is done and what distribution was used for $P_A$ and $P(\\tilde{s}|s)$?\n4. line 313: What does \"using CL to estimate the entropy term in (5)\" mean?\n\n### Other\n1. Typo line 982: “Object hue is dependent on object class, 983 background hue, & object hue, see Tab. 4”\n2. Might be good to use the color formatting used main text to the Appendix tables?\n\n## Update\nThe author response and additional experiments reasonably address most of my concerns. As mentioned by reviewers ``8wZr`` and ``Lc8M``, one drawback for the current theory is that these guarantees do not provide any upper-bound for the complexity of the relation between content factors $c$ and the estimator content factors $\\hat{c}$. Additionally, I would also like to note that these are not finite sample guarantees. However, one may overlook because of the paper’s novelty and the interesting insights it gives. I am keeping my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Good discussion of limitations of the work and some discussion of the negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "dGyDWMfFLwi", "reviewer": "Reviewer_8wZr", "summary": "This paper provides some soft theory towards explaining the effect of data augmentation on self-supervised learning. Specifically, data observations are generated from content information that is invariant to data augmentation and style variables that will be modified. The theoretical results stem from ICA and show that with a squared alignment loss or regularization one can identify the content variables.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nIt is very hard for me to identify the novelty of this paper. \n\n\nFirst, I think the overall message of the paper, that data augmentation only modifies style information, is intuitive and believable. However, it is not entirely new. \nThe authors should have properly discussed and compared with a whole line of prior theoretical work on SSL (e.g. [1-6] and many more I have not listed), many of which share very similar conceptual messages as the paper. Whether it is named as 'content' as in this paper or latent variables in related papers are just a matter of writing style. \n\n~~The theoretical part is not that satisfactory. Unlike existing work, the authors only establish the conditions to identify the learned representation, but it doesn't necessarily explain why downstream tasks can be learned with simple functions (such as linear functions) from the representations. This is more essential for SSL. Otherwise, it's not clear to me why the theoretical results are specifically for explaining the success of SSL and what its distinction to existing ICA work is. \nIdentifying an invariant representation has been extensively studied in prior work, as discussed by the authors, and I don't see how the theoretical results specifically apply to SSL.~~\n==================\nPost rebuttal: My original concern is that even if the paper proves that with infoNCE one can identify the content features, the relation between downstream task and the content feature still remains unclear and the sample complexity might still be high. This is why I thought that proving the identifiability alone was not sufficient to show the advantage of SSL. After reading the rebuttal and other reviews I realized this is not the focus of this paper. Proving that the style features being removed is quite meaningful itself, and it can be theoretical support on why the features learned by SSL will be useful for transfer learning. \n==================\n\nThe dataset is quite interesting, but I guess it might be more suitable for some causal relation work. I guess it is not the main contribution of the paper anyway, since it's not newly collected but comes from a preprocessing and combination of different existing datasets. \n\nTo summarize, I think this paper did a lot of work but no result is good enough for the level of NeurIPS. Plus, significant prior work has been missing. \n\n\n[1] Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning, 2019\n\n[2] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive estimation reveals\ntopic posterior information to linear models, 2020\n\n[3] JD Lee, Q Lei, N Saunshi, J Zhuo. Predicting what you already know helps: Provable self-supervised learning, 2020\n\n[4] Tosh, Christopher, Akshay Krishnamurthy, and Daniel Hsu. \"Contrastive learning, multi-view redundancy, and linear models, 2020\n\n[5] Tsai, Yao-Hung Hubert, et al. \"Self-supervised learning from a multi-view perspective\".\n\n[6] Tian, Yuandong, Xinlei Chen, and Surya Ganguli. \"Understanding self-supervised learning dynamics without contrastive pairs.\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have reached a consensus that the paper is a good addition to the neurips. Please see the reviewers' discussion on the pros and cons for the paper. 
Decision: Accept (Poster)