Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
Year: 2022

Abstract:
Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.

heading: 1 Introduction
text: Learning good representations of high-dimensional observations from large amounts of unlabelled data is widely recognised as an important step for more capable and data-efficient learning systems [10, 72]. Over the last decade, self-supervised learning (SSL) has emerged as the dominant paradigm for such unsupervised representation learning [1, 20, 21, 34, 41, 47, 48, 90, 91, 115, 122, 125, 126]. The main idea behind SSL is to extract a supervisory signal from unlabelled observations by leveraging known structure of the data, which allows for the application of supervised learning techniques. A common approach is to directly predict some part of the observation from another part (e.g., future from past, or original from corruption), thus forcing the model to learn a meaningful representation in the process. While this technique has shown remarkable success in natural language processing [13, 23, 30, 81, 84, 86, 95, 99] and speech recognition [5, 6, 100, 104], where a finite dictionary allows one to output a distribution over the missing part, such predictive SSL methods are not easily applied to continuous or high-dimensional domains such as vision. Here, a common approach is to learn a joint embedding of similar observations or views such that their representation is close [7, 12, 22, 44]. Different views can come, for example, from different modalities (text & speech; video & audio) or time points. As still images lack such multi-modality or temporal structure, recent advances in representation learning have relied on generating similar views by means of data augmentation.
⇤Joint first author. †Joint senior author. Correspondence to: jvk@tue.mpg.de Code available at: https://www.github.com/ysharma1126/ssl_identifiability
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In order to be useful, data augmentation is thought to require the transformations applied to generate additional views to be generally chosen to preserve the semantic characteristics of an observation, while changing other “nuisance” aspects. While this intuitively makes sense and has shown remarkable empirical results, the success of data augmentation techniques in practice is still not very well understood from a theoretical perspective—despite some efforts [17, 19, 28]. In the present work, we seek to better understand the empirical success of SSL with data augmentation by formulating the generative process as a latent variable model (LVM) and studying identifiability of the representation, i.e., under which conditions the ground truth latent factors can provably be inferred from the data [77].
Related work and its relation to the current. Prior work on unsupervised representation learning from an LVM perspective often postulates mutually independent latent factors: this independence assumption is, for example, at the heart of independent component analysis (ICA) [24, 56] and disentanglement [10, 14, 18, 49, 65, 71]. Since it is impossible to identify the true latent factors without any supervisory signal in the general nonlinear case [57, 82], recent work has turned to weakly- or self-supervised approaches which leverage additional information in the form of multiple views [39, 83, 108, 129], auxiliary variables [58, 63], or temporal structure [45, 54, 55, 69]. To identify or disentangle the individual independent latent factors, it is typically assumed that there is a chance that each factor changes across views, environments, or time points.
Our work—being directly motivated by common practices in SSL with data augmentation—differs from these works in the following two key aspects (see Fig. 1 for an overview). First, we do not assume independence and instead allow for both nontrivial statistical and causal relations between latent variables. This is in line with a recently proposed [105] shift towards causal representation learning [40, 76, 85, 87, 106, 107, 112, 123, 127], motivated by the fact that many underlying variables of interest may not be independent but causally related to each other.1 Second, instead of a scenario wherein all latent factors may change as a result of augmentation, we assume a partition of the latent space into two blocks: a content block which is shared or invariant across different augmented views, and a style block that may change. This is aligned with the notion that augmentations leave certain semantic aspects (i.e., content) intact and only affect style, and is thus a more appropriate assumption for studying SSL. In line with earlier work [39, 54, 57, 58, 63, 69, 82, 83, 129], we focus on the setting of continuous ground-truth latents, though we believe our results to hold more broadly.
Structure and contributions. Following a review of SSL with data augmentation and identifiability theory (§ 2), we formalise the process of data generation and augmentation as an LVM with content and style variables (§ 3). We then establish identifiability results of the invariant content partition (§ 4), validate our theoretical insights experimentally (§ 5), and discuss our findings and their limitations in the broader context of SSL with data augmentation (§ 6). We highlight the following contributions:
• we prove that SSL with data augmentations identifies the invariant content partition of the representation in generative (Thm. 4.2) and discriminative learning with invertible (Thm. 4.3) and non-invertible encoders with entropy regularisation (Thm. 4.4); in particular, Thm. 4.4 provides a theoretical justification for the empirically observed effectiveness of contrastive SSL methods that use data augmentation and InfoNCE [91] as an objective, such as SimCLR [20];
• we show that our theory is consistent with results in simulating statistical dependencies within blocks of content and style variables, as well as with style causally dependent on content (§ 5.1);
• we introduce Causal3DIdent, a dataset of 3D objects which allows for the study of identifiability in a causal representation learning setting, and use it to perform a systematic study of data augmentations used in practice, yielding novel insights on what particular data augmentations are truly isolating as invariant content and discarding as varying style when applied (§ 5.2).
1E.g., [69], Fig. 11 where dependence between latents was demonstrated for multiple natural video data sets.

heading: 2 Preliminaries and background
text: Self-supervised representation learning with data augmentation. Given an unlabelled dataset of observations (e.g., images) x, data augmentation techniques proceed as follows. First, a set of observation-level transformations t 2 T are specified together with a distribution pt over T . Both T and pt are typically designed using human intelligence and domain knowledge with the intention of not changing the semantic characteristics of the data (which arguably constitutes a form of weak supervision).2 For images, for example, a common choice for T are combinations of random crops [113], horizontal or vertical flips, blurring, colour distortion [52, 113], or cutouts [31]; and pt is a distribution over the parameterisation of these transformations, e.g., the centre and size of a crop [20, 31]. For each observation x, a pair of transformations t, t0 ⇠ pt is sampled and applied separately to x to generate a pair of augmented views (x̃, x̃0) = (t(x), t0(x)).
The joint-embedding approach to SSL then uses a pair of encoder functions (g,g0), i.e. deep nets, to map the pair (x̃, x̃0) to a typically lower-dimensional representation (z̃, z̃0) = (g(x̃),g0(x̃0)). Often, the two encoders are either identical, g = g0, or directly related (e.g., via shared parameters or asynchronous updates). Then, the encoder(s) (g,g0) are trained such that the representations (z̃, z̃0) are “close”, i.e., such that sim(z̃, z̃0) is large for some similarity metric sim(·), e.g., the cosine similarity [20, 129], or negative L2 norm [129]. The advantage of directly optimising for similarity in representation space over generative alternatives is that reconstruction can be very challenging for high-dimensional data. The disadvantage is the problem of collapsed representations.3 To avoid collapsed representations and force the encoder(s) to learn a meaningful representation, two main families of approaches have been used: (i) contrastive learning (CL) [20, 47, 48, 91, 115, 126]; and (ii) regularisation-based SSL [21, 41, 128].
The idea behind CL is to not only learn similar representations for augmented views (x̃i, x̃0i) of the same xi, or positive pairs, but to also use other observations xj (j 6= i) to contrast with, i.e., to enforce a dissimilar representation across negative pairs (x̃i, x̃0j). In other words, CL pulls representations of positive pairs together, and pushes those of negative pairs apart. Since both aims cannot be achieved simultaneously with a constant representation, collapse is avoided. A popular CL objective function (used, e.g., in SimCLR [20]) is InfoNCE [91] (based on noise-contrastive estimation [42, 43]):
LInfoNCE(g; ⌧,K) = E{xi}Ki=1⇠px h PK i=1 log exp{sim(z̃i,z̃0i)/⌧}PK j=1 exp{sim(z̃i,z̃0j)/⌧} i (1)
where z̃ = Et⇠pt [g(t(x))], ⌧ is a temperature, and K 1 is the number of negative pairs. InfoNCE (1) has an interpretation as multi-class logistic regression, and lower bounds the mutual information across similar views (z̃, z̃0)—a common representation learning objective [4, 9, 15, 50, 75, 79, 80, 97, 120]. Moreover, (1) can be interpreted as alignment (numerator) and uniformity (denominator) terms, the latter constituting a nonparametric entropy estimator of the representation as K ! 1 [124]. CL with InfoNCE can thus be seen as alignment of positive pairs with (approximate) entropy regularisation.
Instead of using negative pairs, as in CL, a set of recent SSL methods only optimise for alignment and avoid collapsed representations through different forms of regularisation. For example, BYOL [41] and SimSiam [21] rely on “architectural regularisation” in the form of moving-average updates for a separate “target” net g0 (BYOL only) or a stop-gradient operation (both). BarlowTwins [128], on the other hand, optimises the cross correlation between (z̃, z̃0) to be close to the identity matrix, thus enforcing redundancy reduction (zero off-diagonals) in addition to alignment (ones on the diagonal).
Identifiability of learned representations. In this work, we address the question of whether SSL with data augmentation can reveal or uncover properties of the underlying data generating process. Whether a representation learned from observations can be expected to match the true underlying latent factors—up to acceptable ambiguities and subject to suitable assumptions on the generative process and inference model—is captured by the notion of identifiability [77].
Within representation learning, identifiability has mainly been studied in the framework of (nonlinear) ICA which assumes a model of the form x = f(z) and aims to recover the independent latents, or sources, z, typically up to permutation or element-wise transformation. A crucial negative result states that, with i.i.d. data and without further assumptions, this is fundamentally impossible [57]. However, recent breakthroughs have shown that identifiability can be achieved if an auxiliary variable (e.g.,
2Note that recent work has investigated automatically discovering good augmentations [26, 27]. 3If the only goal is to make representations of augmented views similar, a degenerate solution which simply
maps any observation to the origin trivially achieves this goal.
a time stamp or environment index) renders the sources conditionally independent [45, 54, 55, 58]. These methods rely on constructing positive and negative pairs using the auxiliary variable and learning a representation with CL. This development has sparked a renewed interest in identifiability in the context of deep representation learning [63, 64, 69, 83, 102, 108, 109, 129].
Most closely related to SSL with data augmentation are works which study identifiability when given a second view x̃ of an observation x, resulting from a modified version z̃ of the underlying latents or sources z [39, 69, 83, 101, 108, 129]. Here, z̃ is either an element-wise corruption of z [39, 69, 101, 129] or may share a random subset of its components [83, 108]. Crucially, all previously mentioned works assume that any of the independent latents (are allowed to) change, and aim to identify the individual factors. However, in the context of SSL with data augmentation, where the semantic (content) part of the representation is intended to be shared between views, this assumption does not hold.

heading: 3 Problem formulation
text: We specify our problem setting by formalising the processes of data generation and augmentation. We take a latent-variable model perspective and assume that observations x (e.g., images) are generated by a mixing function f which takes a latent code z as input. Importantly, we describe the augmentation process through changes in this latent space as captured by a conditional distribution pz̃|z, as opposed to traditionally describing the transformations t as acting directly at the observation level.
Formally, let z be a continuous r.v. taking values in an open, simply-connected n-dim. representation space Z ✓ Rn with associated probability density pz. Moreover, let f : Z ! X be a smooth and invertible mapping to an observation space X ✓ Rd and let x be the continuous r.v. defined as x = f(z).4 The generative process for the dataset of original observations of x is thus given by:
z ⇠ pz, x = f(z). (2) Next, we formalise the data augmentation process. As stated above, we take a representation-centric view, i.e., we assume that an augmentation x̃ of the original x is obtained by applying the same mixing or rendering function f to a modified representation z̃ which is (stochastically) related to the original representation z of x. Specifying the effect of data augmentation thus corresponds to specifying a conditional distribution pz̃|z which captures the relation between z and z̃.
In terms of the transformation-centric view presented in § 2, we can view the modified representation z̃ 2 Z as obtained by applying f 1 to a transformed observation x̃ = t(x) 2 X where t ⇠ pt, i.e., z̃ = f 1(x̃). The conditional distribution pz̃|z in the representation space can thus be viewed as being induced by the distribution pt over transformations applied at the observation level.5
We now encode the notion that the set of transformations T used for augmentation is typically chosen such that any transformation t 2 T leaves certain aspects of the data invariant. To this end, we assume that the representation z can be uniquely partitioned into two disjoint parts:
(i) an invariant part c which will always be shared across (z, z̃), and which we refer to as content; (ii) a varying part s which may change across (z, z̃), and which we refer to as style.
We assume that c and s take values in content and style subspaces C ✓ Rnc and S ✓ Rns , respectively, i.e., n = nc + ns and Z = C ⇥ S . W.l.o.g., we let c corresponds to the first nc dimensions of z:
z = (c, s), c := z1:nc , s := z(nc+1):n,
We formalise the process of data augmentation with content-preserving transformations by defining the conditional pz̃|z such that only a (random) subset of the style variables change at a time. Assumption 3.1 (Content-invariance). The conditional density pz̃|z over Z ⇥ Z takes the form
pz̃|z(z̃|z) = (c̃ c)ps̃|s(s̃|s)
for some continuous density ps̃|s on S ⇥ S , where (·) is the Dirac delta function, i.e., c̃ = c a.e. Assumption 3.2 (Style changes). Let A be the set of subsets of style variables A ✓ {1, ..., ns} and let pA be a distribution on A. Then, the style conditional ps̃|s is obtained via
A ⇠ pA, ps̃|s,A(s̃|s, A) = (s̃Ac sAc)ps̃A|sA(s̃A|sA) ,
where ps̃A|sA is a continuous density on SA ⇥ SA, SA ✓ S denotes the subspace of changing style variables specified by A, and Ac = {1, ..., ns} \A denotes the complement of A.
4While x may be high-dimensional n ⌧ d, invertibility of f implies that X is an n-dim. sub-manifold of Rd. 5We investigate this correspondence between changes in observation and latent space empirically in § 5.
Note that Assumption 3.2 is less restrictive than assuming that all style variables need to change, since it also allows for only a (possibly different) subset of style variables to change for any given observation. This is in line with the intuition that not all transformations affect all changeable (i.e., style) properties of the data: e.g., a colour distortion should not affect positional information, and, in the same vein, a (horizontal or vertical) flip should not affect the colour spectrum.
The generative process of an augmentation or transformed observation x̃ is thus given by A ⇠ pA, z̃|z, A ⇠ pz̃|z,A, x̃ = f(z̃). (3)
Our setting for modelling data augmentation differs from that commonly assumed in (multi-view) disentanglement and ICA in that we do not assume that the latent factors z = (c, s) are mutually (or conditionally) independent, i.e., we allow for arbitrary (non-factorised) marginals pz in (2).6
Causal interpretation: data augmentation as counterfactuals under soft style intervention. We now provide a causal account of the above data generating process by describing the (allowed) causal dependencies among latent variables using a structural causal model (SCM) [94]. As we will see, this leads to an interpretation of data augmentations as counterfactuals in the underlying latent SCM. The assumption that c stays invariant as s changes is consistent with the view that content may causally influence style, c ! s, but not vice versa, see Fig. 1. We therefore formalise their relation as:
c := fc(uc), s := fs(c,us), (uc,us) ⇠ puc ⇥ pus where uc,us are independent exogenous variables, and fc, fs are deterministic functions. The latent causal variables (c, s) are subsequently decoded into observations x = f(c, s). Given a factual observation xF = f(cF, sF) which resulted from (uFc,uFs), we may ask the counterfactual question: “what would have happened if the style variables had been (randomly) perturbed, all else being equal?”. Consider, e.g., a soft intervention [35] on s, i.e., an intervention that changes the mechanism fs to
do(s := f̃s(c,us,uA)),
where uA is an additional source of stochasticity accounting for the randomness of the augmentation process (pA ⇥ ps̃|s,A). The resulting distribution over counterfactual observations xCF = f(cF, sCF) can be computed from the modified SCM by fixing the exogenous variables to their factual values and performing the soft intervention:
cCF := cF, sCF := f̃s(cF,uFs,uA), uA ⇠ puA .
This aligns with our intuition and assumed problem setting of data augmentations as style corruptions. We note that the notion of augmentation as (hard) style interventions is also at the heart of ReLIC [87], a recently proposed, causally-inspired SSL regularisation term for instance-discrimination [44, 126]. However, ReLIC assumes independence between content and style and does not address identifiability. For another causal perspective on data augmentation in the context of domain generalisation, c.f. [59].

heading: 4 Theory: block-identifiability of the invariant content partition
text: Our goal is to prove that we can identify the invariant content partition c under a distinct, weaker set of assumptions, compared to existing results in disentanglement and nonlinear ICA [39, 69, 83, 108, 129]. We stress again that our primary interest is not to identify or disentangle individual (and independent) latent factors zj , but instead to separate content from style, such that the content variables can be subsequently used for downstream tasks. We first define this distinct notion of block-identifiability. Definition 4.1 (Block-identifiability). We say that the true content partition c = f 1(x)1:nc is block-identified by a function g : X ! Z if the inferred content partition ĉ = g(x)1:nc contains all and only information about c, i.e., if there exists an invertible function h : Rnc ! Rnc s.t. ĉ = h(c).
Defn. 4.1 is related to independent subspace analysis [16, 53, 73, 114], which also aims to identify blocks of random variables as opposed to individual factors, though under an independence assumption across blocks, and typically not within a multi-view setting as studied in the present work.

heading: 4.1 Generative self-supervised representation learning
text: First, we consider generative SSL, i.e., fitting a generative model to pairs (x, x̃) of original and augmented views.7 We show that under our specified data generation and augmentation process (§ 3),
6The recently proposed Independently Modulated Component Analysis (IMCA) [64] extension of ICA is a notable exception, but only allows for trivial dependencies across z in the form of a shared base measure.
7For notational simplicity, we present our theory for pairs (x, x̃) rather than for two augmented views (x̃, x̃0), as typically used in practice but it also holds for the latter, see § 6 for further discussion.
as well as suitable additional assumptions (stated and discussed in more detail below), it is possible to isolate (i.e., block-identify) the invariant content partition. Full proofs are included in Appendix A. Theorem 4.2 (Identifying content with a generative model). Consider the data generating process described in § 3, i.e., the pairs (x, x̃) of original and augmented views are generated according to (2) and (3) with pz̃|z as defined in Assumptions 3.1 and 3.2. Assume further that
(i) f : Z ! X is smooth and invertible with smooth inverse (i.e., a diffeomorphism);
(ii) pz is a smooth, continuous density on Z with pz(z) > 0 almost everywhere;
(iii) for any l 2 {1, ..., ns}, 9A ✓ {1, ..., ns} s.t. l 2 A; pA(A) > 0; ps̃A|sA is smooth w.r.t. both sA and s̃A; and for any sA, ps̃A|sA(·|sA) > 0 in some open, non-empty subset containing sA.
If, for a given ns (1  ns < n), a generative model (p̂z, p̂A, p̂s̃|s,A, f̂) assumes the same generative process (§ 3), satisfies the above assumptions (i)-(iii), and matches the data likelihood,
px,x̃(x, x̃) = p̂x,x̃(x, x̃) 8(x, x̃) 2 X ⇥ X ,
then it block-identifies the true content variables via g = f̂ 1 in the sense of Defn. 4.1.
Proof sketch. First, show (using (i) and the matching likelihoods) that the representation ẑ = g(x) extracted by g is related to the true z by a smooth invertible mapping h = g f such that ĉ = h(z)1:nc is invariant across (z, z̃) almost surely w.r.t. pz,z̃.8 Second, show by contradiction (using (ii), (iii)) that h(·)1:nc can, in fact, only depend on the true content c and not on style s, for otherwise the invariance from step 1 would be violated in a region of the style (sub)space of measure greater than zero.
Intuition. Thm. 4.2 assumes that the number of content (nc) and style (ns) variables is known, and that there is a positive probability that each style variable may change, though not necessarily on its own, according to (iii). In this case, training a generative model of the form specified in § 3 (i.e., with an invariant content partition and subsets of changing style variables) by maximum likelihood on pairs (x, x̃) will asymptotically (in the limit of infinite data) recover the true invariant content partition up to an invertible function, i.e., it isolates, or unmixes, content from style.
Discussion. The identifiability result of Thm. 4.2 for generative SSL is of potential relevance for existing variational autoencoder (VAE) [68] variants such as the GroupVAE [51],9 or its adaptive version AdaGVAE [83]. Since, contrary to existing results, Thm. 4.2 does not assume independent latents, it may also provide a principled basis for generative causal representation learning algorithms [76, 107, 127]. However, an important limitation to its practical applicability is that generative modelling does not tend to scale very well to complex high-dimensional observations, such as images.

heading: 4.2 Discriminative self-supervised representation learning
text: We therefore next turn to a discriminative approach, i.e., directly learning an encoder function g which leads to a similar embedding across (x, x̃). As discussed in § 2, this is much more common for SSL with data augmentations. First, we show that if an invertible encoder g is used, then learning a representation which is aligned in the first nc dimensions is sufficient to block-identify content. Theorem 4.3 (Identifying content with an invertible encoder). Assume the same data generating process (§ 3) and conditions (i)-(iv) as in Thm. 4.2. Let g : X ! Z be any smooth and invertible function which minimises the following functional:
LAlign(g) := E(x,x̃)⇠px,x̃
"
g(x)1:nc g(x̃)1:nc
2
2
# (4)
Then g block-identifies the true content variables in the sense of Definition 4.1.
Proof sketch. First, we show that the global minimum of (4) is reached by the smooth invertible function f 1. Thus, any other minimiser g must satisfy the same invariance across (x, x̃) used in step 1 of the proof of Thm. 4.2. The second step uses the same argument by contradiction as in Thm. 4.2.
Intuition. Thm. 4.3 states that if—under the same assumptions on the generative process as in Thm. 4.2—we directly learn a representation with an invertible encoder, then enforcing alignment between the first nc latents is sufficient to isolate the invariant content partition. Intuitively, invertibility guarantees that all information is preserved, thus avoiding a collapsed representation.
8This step is partially inspired by [83]; the technique used to prove the second main step is entirely novel. 9which also uses a fixed content-style partition for multi-view data, but assumes that all latent factors are
mutually independent, and that all style variables change between views, independent of the original style;
Discussion. According to Thm. 4.3, content can be isolated if, e.g., a flow-based architecture [32, 33, 67, 92, 93] is used, or invertibility is enforced otherwise during training [8, 60]. However, the applicability of this approach is limited as it places strong constraints on the encoder architecture which makes it hard to scale these methods up to high-dimensional settings. As discussed in § 2, state-of-the-art SSL methods such as SimCLR [20], BYOL [41], SimSiam [21], or BarlowTwins [128] do not use invertible encoders, but instead avoid collapsed representations—which would result from naively optimising (4) for arbitrary, non-invertible g—using different forms of regularisation.
To close this gap between theory and practice, finally, we investigate how to block-identify content without assuming an invertible encoder. We show that, if we add a regularisation term to (4) that encourages maximum entropy of the learnt representation, the invertibility assumption can be dropped. Theorem 4.4 (Identifying content with discriminative learning and a non-invertible encoder). Assume the same data generating process (§ 3) and conditions (i)-(iv) as in Thm. 4.2. Let g : X ! (0, 1)nc be any smooth function which minimises the following functional:
LAlignMaxEnt(g) := E(x,x̃)⇠px,x̃
"
g(x) g(x̃)
2
2
# H (g(x)) (5)
where H(·) denotes the differential entropy of the random variable g(x) taking values in (0, 1)nc . Then g block-identifies the true content variables in the sense of Defn. 4.1.
Proof sketch. First, use the Darmois construction [29, 57] to build a function d : C ! (0, 1)nc mapping c = f 1(x)1:nc to a uniform random variable. Then g? = d f 1 1:nc attains the global minimum of (5) because c is invariant across (x, x̃) and the uniform distribution is the maximum entropy distribution on (0, 1)nc . Thus, any other minimiser g of (5) must satisfy invariance across (x, x̃) and map to a uniform r.v. Then, use the same step 2 as in Thms. 4.2 and 4.3 to show that h = g f : Z ! (0, 1)nc cannot depend on style, i.e., it is a function from C to (0, 1)nc . Finally, we show that h must be invertible since it maps pc to a uniform distribution, using a result from [129].
Intuition. Thm. 4.4 states that if we do not explicitly enforce invertibility of g as in Thm. 4.3, additionally maximising the entropy of the learnt representation (i.e., optimising alignment and uniformity [124]) avoids a collapsed representation and recovers the invariant content block. Intuitively, this is because any function that only depends on c will be invariant across (x, x̃), so it is beneficial to preserve all content information to maximise entropy.
Discussion. Of our theoretical results, Thm. 4.4 requires the weakest set of assumptions, and is most closely aligned with common SSL practice. As discussed in § 2, contrastive SSL with negative samples using InfoNCE (1) as an objective can asymptotically be understood as alignment with entropy regularisation [124], i.e., objective (5). Thm. 4.4 thus provides a theoretical justification for the empirically observed effectiveness of CL with InfoNCE: subject to our assumptions, CL with InfoNCE asymptotically isolates content, i.e., the part of the representation that is always left invariant by augmentation. For example, the strong image classification performance based on representations learned by SimCLR [20], which uses color distortion and random crops as augmentations, can be explained in that object class is a content variable in this case. We extensively evaluate the effect of various augmentation techniques on different ground-truth latent factors in our experiments in § 5. There is also an interesting connection between Thm. 4.4 and BarlowTwins [128], which only uses positive pairs and combines alignment with a redundancy reduction regulariser that enforces decorrelation between the inferred latents. Intuitively, redundancy reduction is related to increased entropy: g? constructed in the proof of Thm. 4.4—and thus also any other minimiser of (5)—attains the global optimum of the BarlowTwins objective, though the reverse implication may not hold.

heading: 5 Experiments
text: We perform two main experiments. First, we numerically test our main result, Thm. 4.4, in a fullycontrolled, finite sample setting (§ 5.1), using CL to estimate the entropy term in (5). Second, we seek to better understand the effect of data augmentations used in practice (§ 5.2). To this end, we introduce a new dataset of 3D objects with dependencies between a number of known ground-truth factors, and use it to evaluate the effect of different augmentation techniques on what is identified as content. Additional experiments are summarised in § 5.3 and described in more detail in Appendix C.

heading: 5.1 Numerical data
text: Experimental setup. We generate synthetic data as described in § 3. We consider nc = ns = 5, with content and style latents distributed as c ⇠ N (0,⌃c) and s|c ⇠ N (a+Bc,⌃s), thus allowing
for statistical dependence within the two blocks (via ⌃c and ⌃s) and causal dependence between content and style (via B). For f , we use a 3-layer MLP with LeakyReLU activation functions.10 The distribution pA over subsets of changing style variables is obtained by independently flipping the same biased coin for each si. The conditional style distribution is taken as ps̃A|sA = N (sA,⌃A). We train an encoder g on pairs (x, x̃) with InfoNCE using the negative L2 loss as the similarity measure, i.e., we approximate (5) using empirical averages and negative samples. For evaluation, we use kernel ridge regression [88] to predict the ground truth c and s from the learnt representation ĉ = g(x) and report the R2 coefficient of determination. For a more detailed account, we refer to Appendix D.
Generative process R2 (nonlinear) p(chg.) Stat. Cau. Content c Style s
1.0 7 7 1.00 ± 0.00 0.07± 0.00 0.75 7 7 1.00 ± 0.00 0.06± 0.05 0.75 3 7 0.98 ± 0.03 0.37± 0.05 0.75 3 3 0.99 ± 0.01 0.80 ± 0.08 Results. In the inset table, we report mean ± std. dev. over 3 random seeds across four generative processes of increasing complexity covered by Thm. 4.4: “p(chg.)”, “Stat.”, and “Cau.” denote respectively the change probability for each si, statistical dependence within blocks (⌃c 6= I 6= ⌃s), and causal dependence of style on content (B 6= 0). An R2 close to one indicates that almost all variation is explained by ĉ, i.e., that there is a 1-1 mapping, as required by Defn. 4.1. As can be seen, across all settings, content is block-identified. Regarding style, we observe an increased score with the introduction of dependencies, which we explain in an extended discussion in Appendix C.1. Finally, we show in Appendix C.1 that a high R2 score can be obtained even if we use linear regression to predict c from ĉ (R2 = 0.98± 0.01, for the last row).
5.2 High-dimensional images: Causal3DIdent
Causal3DIdent dataset. 3DIdent [129] is a benchmark for evaluating identifiability with rendered 224⇥ 224 images which contains hallmarks of natural environments (e.g. shadows, different lighting conditions, a 3D object). For influence of the latent factors on the renderings, see Fig. 2 of [129]. In 3DIdent, there is a single object class (Teapot [89]), and all 10 latents are sampled independently. For Causal3DIdent, we introduce six additional classes: Hare [121], Dragon [110], Cow [62], Armadillo [70], Horse [98], and Head [111]; and impose a causal graph over the latent variables, see Fig. 2. While object class and all environment variables (spotlight position & hue, background hue) are sampled independently, all object latents are dependent,11 see Appendix B for details.12
Experimental setup. For g, we train a convolutional encoder composed of a ResNet18 [46] and an additional fully-connected layer, with LeakyReLU activation. As in SimCLR [20], we use InfoNCE with cosine similarity, and train on pairs of augmented examples (x̃, x̃0). As nc is unknown and variable depending on the augmentation, we fix dim(ĉ) = 8 throughout. Note that we find the results to be, for the most part, robust to the choice of dim(ĉ), see inset figure. We consider the following data augmentations (DA): crop, resize & flip; colour distortion (jitter & drop); and rotation 2 {90°, 180°, 270°}. For comparison, we also consider directly imposing a content-style
10chosen to lead to invertibility almost surely by following the settings used by previous work [54, 55] 11e.g., our causal graph entails hares blend into the environment (object hue centered about background &
spotlight hue), a form of active camouflage observed in Alaskan [78], Arctic [2], & Snowshoe hares. 12We made the Causal3DIdent dataset publicly available at this URL.
partition by performing a latent transformation (LT) to generate views. For evaluation, we use linear logistic regression to predict object class, and kernel ridge to predict the other latents from ĉ.13
Results. The results are presented in Tab. 1. Overall, our main findings can be summarised as: (i) it can be difficult to design image-level augmentations that leave specific latent factors invariant;
(ii) augmentations & latent transformations generally have a similar effect on groups of latents; (iii) augmentations that yield good classification performance induce variation in all other latents.
We observe that, similar to directly varying the hue latents, colour distortion leads to a discarding of hue information as style, and a preservation of (object) position as content. Crops, similar to varying the position latents, lead to a discarding of position as style, and a preservation of background and object hue as content, the latter assuming crops are sufficiently large. In contrast, image-level rotation affects both the object rotation and position, and thus deviates from only varying the rotation latents.
Whereas class is always preserved as content when generating views with latent transformations, when using data augmentations, we can only reliably decode class when crops & colour distortion are used in conjunction—a result which mirrors evaluation on ImageNet [20]. As can be seen by our evaluation of crops & colour distortion in isolation, while colour distortion leads to a discarding of hues as style, crops lead to a discarding of position & rotation as style. Thus, when used in conjunction, class is isolated as the sole content variable. See Appendix C.2 for additional analysis.

heading: 5.3 Additional experiments and ablations
text: We also perform an ablation on dim(ĉ) for the synthetic setting from § 5.1, see Appendix C.1 for details. Generally, we find that if dim(ĉ) < nc, there is insufficient capacity to encode all content, so a lower-dimensional mixture of content is learnt. Conversely, if dim(ĉ) > nc, the excess capacity is used to encode some style information (as that increases entropy). Further, we repeat our analysis from § 5.2 using BarlowTwins [128] (instead of SimCLR) which, as discussed at the end of § 4.2, is also loosely related to Thm. 4.4. The results mostly mirror those obtained for SimCLR and presented in Tab. 1, see Appendix C.2 for details. Finally, we ran the same experimental setup as in § 5.2 also on the MPI3D-real dataset [38] containing > 1 million real images with ground-truth annotations of 3D objects being moved by a robotic arm. Subject to some caveats, the results show a similar trend as those on Causal3DIdent, see Appendix C.3 for details.

heading: 6 Discussion
text: Theory vs practice. We have made an effort to tailor our problem formulation (§ 3) to the setting of data augmentation with content-preserving transformations. However, some of our more technical assumptions, which are necessary to prove block-identifiability of the invariant content partition, may not hold exactly in practice. This is apparent, e.g., from our second experiment (§ 5.2), where we observe that—while class should, in principle, always be invariant across views (i.e., content)—when
13See Appendix C.2 for results with linear regression, as well as evaluation using a higher-dimensional intermediate layer by considering a projection head [20].
using only crops, colour distortion, or rotation, g appears to encode shortcuts [37, 96].14 Data augmentation, unlike latent transformations, generates views x̃ which are not restricted to the 11-dim. image manifold X corresponding to the generative process of Causal3DIdent, but may introduce additional variation: e.g., colour distortion leads to a rich combination of colours, typically a 3-dim. feature, whereas Causal3DIdent only contains one degree of freedom (hue). With additional factors, any introduced invariances may be encoded as content in place of class. Image-level augmentations also tend to change multiple latent factors in a correlated way, which may violate assumption (iii) of our theorems, i.e., that ps̃A|sA is fully-supported locally. We also assume that z is continuous, even though Causal3DIdent and most disentanglement datasets also contain discrete latents. This is a very common assumption in the related literature [39, 54, 57, 58, 63, 69, 82, 83, 129] that may be relaxed in future work. Moreover, our theory holds asymptotically and at the global optimum, whereas in practice we solve a non-convex optimisation problem with a finite sample and need to approximate the entropy term in (5), e.g., using a finite number of negative pairs. The resulting challenges for optimisation may be further accentuated by the higher dimensionality of X induced by image-level augmentations. Finally, we remark that while, for simplicity, we have presented our theory for pairs (x, x̃) of original and augmented examples, in practice, using pairs (x̃, x̃0) of two augmented views typically yields better performance. All of our assumptions (content invariance, changing style, etc) and theoretical results still apply to the latter case. We believe that using two augmented views helps because it leads to increased variability across the pair: for if x̃ and x̃0 differ from x in style subsets A and A0, respectively, then (x̃, x̃0) differ from each other (a.s.) in the union A [A0.
Beyond entropy regularisation. We have shown a clear link between an identifiable maximum entropy approach to SSL (Thm. 4.4) and SimCLR [20] based on the analysis of [124], and have discussed an intuitive connection to the notion of redundancy reduction used in BarlowTwins [128]. Whether other types of regularisation such as the architectural approach pursued in BYOL [41] and SimSiam [21] can also be linked to entropy maximisation, remains an open question. Deriving similar results to Thm. 4.4 with other regularisers is a promising direction for future research, c.f. [116].
The choice of augmentation technique implicitly defines content and style. As we have defined content as the part of the representation which is always left invariant across views, the choice of augmentation implicitly determines the content-style partition. This is particularly important to keep in mind when applying SSL with data augmentation to safety-critical domains, such as medical imaging. We also advise caution when using data augmentation to identify specific latent properties, since, as observed in § 5.2, image-level transformations may affect the underlying ground-truth factors in unanticipated ways. Also note that, for a given downstream task, we may not want to discard all style information since style variables may still be correlated with the task of interest and may thus help improve predictive performance. For arbitrary downstream tasks, however, where style may change in an adversarial way, it can be shown that only using content is optimal [103].
What vs how information is encoded. We focus on what information is learnt by SSL with data augmentations by specifying a generative process and studying identifiability of the latent representation. Orthogonal to this, a different line of work instead studies how information is encoded by analysing the sample complexity needed to solve a given downstream task using a linear predictor [3, 74, 116– 119]. Provided that downstream tasks only involve content, we can draw some comparisons. Whereas our results recover content only up to arbitrary invertible nonlinear functions (see Defn. 4.1), our problem setting is more general: [3, 74] assume (approximate) independence of views (x, x̃) given the task (content), while [118, 119] assume (approximate) independence between one view and the task (content) given the other view, neither of which hold in our setting.
Conclusion. Existing representation learning approaches typically assume mutually independent latents, though dependencies clearly exist in nature [106]. We demonstrate that in a non-i.i.d. scenario, e.g., by constructing multiple views of the same example with data augmentation, we can learn useful representations in the presence of this neglected phenomenon. More specifically, the present work contributes, to the best of our knowledge, the first: (i) identifiability result under arbitrary dependence between latents; and (ii) empirical study that evaluates the effect of data augmentations not only on classification, but also on other continuous ground-truth latents. Unlike existing identifiability results which rely on change as a learning signal, our approach aims to identify what is always shared across views, i.e., also using invariance as a learning signal. We hope that this change in perspective will be helpful for applications such as optimal style transfer or disentangling shape from pose in vision, and inspire other types of counterfactual training to recover a more fine-grained causal representation.
14class is distinguished by shape, a feature commonly unused in downstream tasks on natural images [36]

heading: Acknowledgements
text: We thank: the anonymous reviewers for several helpful suggestions that triggered improvements in theory and additional experiments; Cian Eastwood, Ilyes Khemakem, Michael Lohaus, Osama Makansi, Ricardo Pio Monti, Roland Zimmermann, Weiyang Liu, and the MPI Tübingen causality group for helpful discussions and comments; Hugo Yèche for pointing out a mistake in §2 of an earlier version of the manuscript; github user TangTangFei for catching a bug in the implementation of the experiments from § 5.1 (that has been corrected in this version); and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting YS.
Funding Transparency Statement WB acknowledges support via his Emmy Noether Research Group funded by the German Science Foundation (DFG) under grant no. BR 6382/1-1 as well as support by Open Philantropy and the Good Ventures Foundation. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A, 01IS18039B; and by the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645.

Review summaries:
This paper studies the effect of data augmentation in self-supervised representation learning. They take a latent variable perspective where there is a content component and style component of the latent variable, and consider settings where the content is invariant to the augmentation whereas the style component changes a lot under the data augmentation. In the latent variable model, arbitrary dependencies between the content and style latent variables are allowed.    The theoretical contribution of this paper is to show that, under certain assumptions on how the augmentation influences the latent variables, 1) a generative model for the data would perfectly identify the true content variables (up to an invertible function) and 2) minimizing the (entropy-regularized) squared difference between encoder outputs on two different augmentations of the data would also give an encoder which perfectly recovers the content variables (also up to an invertible function). The assumptions for these results are that the augmentations don't change the content latent variable and only the style latent variable, and that the augmentations can map the coordinates of the style latent variables to any value.   This paper also provides experiments for verifying aspects of the theory, first on synthetic Gaussian data, and also in a semi-synthetic setting of rendered images of 3D objects where different latents control different aspects of the image. They compare the effect of using InfoNCE objective on pairs of examples obtained via data augmentation against the effect of using InfoNCE objective on pairs of examples obtained via directly modifying the style latent variables. They show that training InfoNCE with modifying the latent variables does indeed preserve a lot of content information while discarding style information. Data augmentation is observed to behave in a similar way in some instances, though the equivalence is not exact. 

Two contributions of this work are (1) an identifiability result for self-supervised learning (SSL), (2) a new synthetic dataset and empirical results evaluating the theoretical results and it’s comparison to standard SSL techniques with data augmentation.   Theoretically, authors prove that using (a) “anchor” samples $x$ from a generative model which generates data using (potentially dependent) content $c$ and style $s$ latent random factors through an invertible mapping $x=f(c,s)$, and  (b) positive samples generated by the same content signal $c$ and an augmented style signal $\tilde{s}$ which is independent of $c$ given $s$, one can learn a perfect encoder $\hat{c} = g(x)$ using a regularized SSL loss or by generative modeling, such that $\hat{c}$ contains all the information to perfectly predict $c$ in a statistical sense. This means that under some regularity conditions, SSL approaches using data augmentation can learn to isolate content factors (invariant to augmentation) from style factors (which can vary under augmentations). This identifiability result follows the recently popular line of similar results in non-linear ICA and representation learning thoery. However, different from previous results this result does not need independence of latent (content or style) random factors.  Next authors provide a new synthetic dataset to quantify the SSL representation learning. They quantify the performance of SimCLR contrastive (different from non-contrastive theory) SSL representation learning method on this dataset under their generative model and standard data augmentation methods. This showcase both the validity and the limitations of their theory and assumption.

This paper provides some soft theory towards explaining the effect of data augmentation on self-supervised learning. Specifically, data observations are generated from content information that is invariant to data augmentation and style variables that will be modified. The theoretical results stem from ICA and show that with a squared alignment loss or regularization one can identify the content variables.