Separating common from salient patterns with Contrastive Representation Learning
Authors: Robin Louiset
Edouard Duchesnay
Antoine Grigis
Pietro Gori
Venue: ICLR 2024 poster (2024)
Keywords: Contrastive Learning
Mutual Information
Contrastive Analysis
Disentanglement
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We use Contrastive Learning when performing Contrastive Analysis (i.e: separating salient factors of variation - that only exist in the target dataset  in contrast with common factors of variation between target and background datasets).

Abstract: Contrastive Analysis is a sub-field of Representation Learning that aims at separating 1) salient factors of variation - that only exist in the target dataset (i.e., diseased subjects) in contrast with 2) common factors of variation between target and background (i.e., healthy subjects) datasets. Despite their relevance, current models based on Variational Auto-Encoders have shown poor performance in learning semantically-expressive representations. On the other hand, Contrastive Representation Learning has shown tremendous performance leaps in various applications (classification, clustering, etc.). In this work, we propose to leverage the ability of Contrastive Learning to learn semantically expressive representations when performing Contrastive Analysis. Namely, we reformulate Contrastive Analysis under the lens of the InfoMax Principle and identify two Mutual Information terms to maximize and one to minimize. We decompose the two first terms into an Alignment and a Uniformity term, as commonly done in Contrastive Learning. Then, we motivate a novel Mutual Information minimization strategy to prevent information leakage between common and salient distributions. We validate our method on datasets designed to assess the pattern separation capability in Contrastive Analysis, including MNIST superimposed on CIFAR10, CelebA accessories, dSprites item superimposed on a digit grid, and three medical datasets.

Published as a conference paper at ICLR 2024
SEPARATING COMMON FROM SALIENT PATTERNS WITH CONTRASTIVE REPRESENTATION LEARNING
Robin Louiset
Edouard Duchesnay'
Antoine Grigis
Pietro Gori
ABSTRACT
Contrastive Analysis is a sub-field of Representation Learning that aims at sep- arating common factors of variation between two datasets, background (i.e.. healthy subjects) and target (i.e., diseased subjects), from the salient factors of variation, only present in the target dataset. Despite their relevance, current mod- els based on Variational Auto-Encoded have shown poor performance in learning semantically-expressivv representations. On the other hand, Contrastive Repre- sentation Learning has shown tremendous performance leaps in various applica- tions (classification, clustering, etc.). In this work, we propose to leverage the abil- ity of Contrastive Learning learn semantically expressive representations well adapted for Contrastive Analysis. We reformulate it under the lens of the InfoMax Principle and identify two Mutual Information terms to maximize and one to mini- mize. We decompose the first two terms into an Alignment and Uniformity term, as commonly done in Contrastive Learning. Then, we motivate novel Mutual Information minimization strategy to prevent information leakage between com- mon and salient distributions. We validate our method, called SepCLR, on three visual datasets and three medical datasets, specifically conceived to assess the pattern separation capability in Contrastive Analysis. Code available at httpp:: 7/github. com wwtp://///eooooiiinnoocoooo
INTRODUCTION
In Representation Learning, practitioners estimate parametric models tailored to learn meaningful and compact representations from high-dimensional data. The objective is to capture relevant fea- tures to facilitate downstream tasks such classification, clustering, segmentation, generation. Contrastive Representation Learning (CL) has made remarkable progress in learning representations that encode high-level semantic information about inputs such as images (Zbontar al. (2021): Weil
et al. (2020); [Bachman et al. (2019); He et al. ); Goyal et al. (2021) Dufumier et al. (2023): k
Barbano al. (2023a)) and sequential data (Oord et (2019): Kong et al. (2019); Tian
(020)
Schneider et al. (2023); Sun et al.) (2019). With distinct perspective, Contrastive Analysis (CA)
approaches aim discover the underlying generative factors that distinguish a target dataset from background dataset (i.e., salient factors) and that 2) are shared between them (i.e., common factors). is usually assumed that target samples comprise additional (or modified) patterns com- pared to background samples (Abid et al. (2018); Zou et al. (2013 2022); Severson al. (2018); Ruiz et al. (2019); ; Ge & Zou (2016); et al. (2021); Zou et al. (2023)). ). The ability to distinguish and separate common from salient generative factors is erucial various domains. For instance, in medical imaging. researchers seek to identify pathological patterns in a population of patients (target) compared to healthy controls (background) Antelmi et al. (2019): Aglinskas al. (202)).
Contrastive Analysis also concerns other domains like drug research (medicated vs. placebo popu- lations), surgery (pre-intervention vs. post-intervention groups), time series (signal vs. signal-free samples), biology and genetics (control vs. characteristic-tran population, Jones et al. (2021) l)) ).
Jonees
Current Contrastive Analysis methods are based on VAEs (Variational Auto-Encoders Kingma & Welling 2014). This choice is particularly suitable for generation and image-level manipulations. However, as shown in |Phuong et al.) (2018). VAE can fail learn meaningful latent representations, or even learn trivial representations when the decoder is too powerful Chen et al. (2017). Conversely, Contrastive Learning (CL) methods have demonstrated outstanding results many domains, such
Corresponding author: robin n. ouu set com
NeuroSpin, University Paris Saclay, France." *LTCI, Télécom Paris, IPParis, France

---

Published as a conference paper at ICLR 2024
as unsupervised learning Chen al. 2020), deep clustering Li et al. (2020), content style identifi- cation yon Kügelgen et al. (2021), D) background debiasing Wang et al. (2022) : Ding et al. (2022). and multi-modality Yuan et al. (202I. This performance gap might be explained by the following rea- sons. CL. methods produce representations invariant to a set of user-defined image transformations (translation, zoom, color jittering, etc.), whereas VAFs are highly sensitive these uninteresting variability factors. Furthermore, VAEs maximize the log-likelihood, which is only a function of the marginal distribution of the input data and not of the latent representations. Differently, CL methods based on the InfoNCE loss implicitly maximize the Mutual Information (MI) between input data and latent feature From a representation learning point of view, this makes much sense since the MI depends on the joint distribution between input data and representation Zhao et al. (2019). Inspired by these works, we propose to reformulate the Contrastive Analysis problem under the lens of the well-known InfoMax principle BBell & Sejnowski ( (1995); Hjelm et al. (2019) and leverage the representation power of Contrastive Learning (CL) to estimate the MI terms of our newly proposed Contrastive Analysis setting. We seek to separate the salient patterns of the target dataset from the shared (common) patterns with the background dataset. Common factors e should be representative of both target and background datasets (respectively y and r). Thus, we propose maximize the MI between x (resp. y) and c. We compute the Entropy and Alignment to estimate the MI, as in Wang & Isola 2020) and Moddiguez--Gälver et al. (022)). Since salient factors should only describe patterns typical of the target data y. we propose to maximize the MI between s and only y. Further- more, we also add the constraint that background samples' representations should always be equal to an informationless vector s' in the salient space. This objective is close to other recent CA ideas, such as in Contrastive PCA Abid & Zou 2019) and CA-VAEs, but also to Supervised Anomaly Detection intuitions, such as in DeepSAD Ruff et al. (2019), where the entropy of anomalies (eq. target) is maximized, whereas normal samples (eq. backgrounds) are set to a constant vector. We propose an extension of this salient term when fine-grained target attributes are available and pro- pose disentangling these attributes within the salient space in a supervised manner. Moreover, to avoid information leakage between the common and salient space s, we constrain the MI to be (exactly) equal to O. This choice avoids undesirable results since minimizing the MI may bring to trivial solution where e and/or would contain information. Instead, we propose method to estimate and maximize their joint entropy H(c, s) without requiring any assumptions about the form of its pdf nor a neural network-based approximation. Our contributions are summarized below:
1. We introduce SepCLR, a novel theoretical framework for Contrastive Analysis based on the InfoMax principle. We identify three Mutual Information terms: common space term, a salient space term, and a common-salient independence term.
2. We leverage Contrastive Learning to estimate the common and salient terms. We show how usual contrastive losses such as InfoNCE and SupCLR can be retrieved from the In- foMax Principle. Likewise, we derive a novel contrastive method to capture target-specific variability while canceling background variability in the salient space.
3. To reduce the information leakage between the common and salient spaces, suggest strategy that overcomes the pitfalls of usual Mutual Information Minimization methods.
2 RELATED WORKS
Our work relates to contrastive learning. mutual information, and contrastive analysis. Contrastive Learning and the InfoMax Principle. Contrastive Learning (CL) hinges an intu- ition that dates back to Becker & Hinton (1992). Given an input sample x (image or sequence) and two different views (i.e., transformations) e and v+ of that potentially overlap (spatially sequen- tially), CL is based on the assumption that e and et should share a similar information content. A parametric encoder soo is then estimated by maximizing their "agreement" in the representation space so that their similarity/dependenne is preserved in the embeddings fo(w) and fo(u+). A commonly used measure of agreement the Mutual Information between the two views embeddings that is )), where the choice of fe imposes some structural con-
maximized: 0- + arg max L (f)( fo (x straints (i.e., inductive bias). As shown in Tschannen et al. 2019), this objective can be seen as a lower bound on the InfoMax principle max [x: fo(z)) (Linsker 1988 Bell & Sejnowski [9955]
as shown in Sec 3.1 the MI between the latent representation of two views, maximized in many recent methods, is a lower bound of the MI hetween input data and latent representations.
2

---

Published as a conference paper at ICLR 2024
max ((:: y) s.t (ku(ss)(((s))==
Salient Space S (color)
Targets Y (shope color)
Views t(y)
fos
((c:s)
Backgrounds X (shupe)
Views t(x)
fec
Common Space (shape)
max l(c:x) + I(c: y)
Figure 1: SepCLR is trained identify and separate the salient patterns (color variations) of the target dataset Y from the common patterns (shape) shared between background X and target dataset Y. Views (transformations t(-)) of both datasets are fed to two different encoders, one for the salient space (fa,) and one for the common space (fe.). In the hyperspherical common space, C, embeddings of views of the same image (from both X and Y) are aligned, while embeddings from different images are repelled (max I(c: x)++ +[(c y)). This enforces C to represent the shared patterss (shape). In the salient space S. which is a Euclidean space, in order not to capture background variability (i.e: shape), background embeddings (1 [1(())) = 0). Furthermore, embeddings of views of the same image (only from Y) are aligned while embeddings from different images are pushed away from each other, and they repelled from s' (max (ss; v)). This enforces S to capture only the salient patterns of Y (color). To limit the information leakage between C' and their MI is constrained to be null, i.e: [(c; s = 0).
aligned onto an information-less null vector s'
all
Many approaches KKng et al. (2019); Tian et al. (2020); Bachman et al. (011); : Oord et al. (2019); Tsai et al. (2020); Barbano al. (2023b)) propose to maximize [[fo(e) fo u )] rather than the original InfoMax objective since the embeddings f(z) have a lower dimension than the original samples and the choice of the transformation for the views gives more flexibility. [Wang & Isola (2020) simplifies the usual CL. loss InfoNCH et al. (2020) into an alignment (or reconstruction) and uniformity (or entropy) term. While the alignment term trains the encoder to assign similar representations to positive views, the uniformity term encourages feature distribution to preserve maximal information i.e.; maximal entropy. Recently, (Rodríguez- Gälvez et al. (2023) demonstrated that these terms could be derived from the maximization of 7 (VV(V); fo(V+) and that several clus- tering methods could be retrieved from this formulation. We build onto these works introduce the CL framework required to develop the proposed CA losses.
Contrastive Analysis. Contrastive Analysis (CA) methods are designed to separate salient latent variables (iee patterns that are specific to the target dataset) from common latent variables (i.e: patterns that are shared between background and target datasets). Recently, contrastive Variational Auto-Encoders were designed to capture higher-level semantics Abid & Zou 2019); Zou et al. 2022); Weinberger et al.] 2022); Louiset et al. 2023). These methods usually rely on a latent space split into two parts, common and salient, estimated by two different encoders. To limit information leakage between common and salient spaces, three types of regularization have been proposed. First, usual solution is to introduce an explicit regularization on the salient encoder to minimize the back- ground information expressivity Abid & Zou| (2019): Weinberger et al. (2022); Louiset et al. (2023); Zou al. (2022) 2023). This regularization forces the salient vectors of the backgrounds to close to (information-les) vector, often equal to O). A second idca, proposed in MM-cVAE Weinberger al. (2022), is match the common space distributions of the target and background samples by minimizing their Maximum Mean Discrepancy (MMD) Gretton et al.) (2012). This regularization reduces the information that would enable discriminating targets from background samples within the common space. In cVAE Abid & Zou (2019), and LopusE
Louiset et al. (2023), authors mini- mize the Mutual Information between the common and salient spaces.
Contrastive Analysis is not disentanglement nor style vs. content separation. CA is not about disentanglement, which aims to isolate independent variation factors in a single data-set Locatello
ett al. 22020) Chen al. (208)); N al. (2017). In contrast, CA seeks to separate common from
3

---

Published as a conference paper at ICLR 2024
target-specific generative factors without requiring the isolation of independent factors usually de- fined in supervised manner using external attributes. Furthermore, CA is not about separating style from content (Kazemi et al. 2019);; ; von Kügelgen et al. (202I), where content is usually defined as the invariant part of the latent space, namely the part shared across different views. In contrast, style refers to the varying part that accounts for the differences between views. Content and style depend on the chosen semantie-invariant transformations, and they are defined for a single dataset. In CA, we do not necessarily need transformations or views, and we jointly analyze two different datasets. Mutual Information Minimization. Mutual Information minimization has gained significant at- tention in diverse applications such as disentangling Kim & Mnih (2018): N et (017)), domain adaptation Gholami et al. (2018), style/content identification Kazemi et al. 2019). and Informa- tion Bottleneck compression Alemi (2020). Typically, it can serve as a regularizer to diminish the dependence between variables. However, computing the value of Mutual Information is hardly pos- sible in cases where closed forms of density functions, joint or marginal, are unknown. In most machine learning setups, access is limited to only samples drawn from the joint distribution. To accommodate, most estimation methods (lower bound, upper bound, and reliable estimators) focus on sample-based estimation. However, most of these works either require strong assumptions about one of the distributions (e.g, its form) (Alemi (2020); Poole et al. (2019)) or the introduction of an independent neural network approximate distribution in sample-based variational manner. For instance, CLUB Cheng et al. (2020) derives an upper bound of the Mutual Information / [ Y) by either assuming the closed-form of p(y|z) or, in variational form, estimating it with pa- rameterized neural network go ())z)) Another example concerns Total Correlation methods Louiset et al. (2023): Kim & Mnih| (2018) that leverage the Density Ratio trick [Sugiyama et al. (2012); Nguyen et al. (2010) to estimate the density ratio between the joint distribution and the product of the marginals. This technique demands optimizing an independent discriminator to discriminate samples drawn from the joint distribution from those drawn from the product of the marginals.
3 THE INFOMAX PRINCIPLE FOR CONTRASTIVE ANALYSIS
Let X = ( Nx and Y = {ur be the background and target data-sets of images respectively. As it is commonly done in Contrastive Analysis Abid & Zou (019); WWinberger et all (2022); Louiset et al. (2023), we suppose that both z; and Y/ are drawn i..dd from the same conditional distribution p=l-lc, s). that is parameterized by unknown parameters 0 and that depends on two latent variables: the common generative factors e E RD shared between X and Y, and the salient (or farget-specific) generative factors
RD,, which are only present in and not in X. The separation between e and s can be considered a weakly supervised learning problem since the only level of supervision is the population-basse based label X Y. The user has no knowledge about the common and salient generative factors at training (or test) time. By grounding our method on the InfoMax principle [Bell & Sejnowski (1955): [Hjelm et al. (2019). and since we want the common factors e to be representative of both datasets, we propose to maximize the mutual information I between e and both datasets X and Y. Similarly, we propose maximizing the mutual information between the salient factors and only the target samples Y. Since we want the background samples x to be fully encoded by c, we enforce the salient factors s of x to be always equal to a constant value s' (i.e., no information): x; po(ar/c,, 8; 8'). Mathematically, we do that by minimizing the Kullback-Leibler divergence D between p(s (z) and 8(x'), a Dirac Delta distribution centered at s'. Furthermore, to enforce the separation (i.e., independence) between and s, also propose to use [(,, s) = 0 as a regularization constraint.
Our objective is to separate and infer the common e and salient s factors given the input data X and Y. We use two probabilistic encoders, fa_ and fa,- parameterised by 0, and 0., to approximate the conditional distributions p(cl-) and p(sl-) respectively. The two encoders are shared between X and Y. Furthermore, as commonly done in recent representation learning papers, we assume to have multiple views v of each image x (or y) generated via stochastic augmentation function t: = t(.). By denoting c = fo, (v), = (w), 8, = fo. (t(x)), our goal becomes finding the optimal parameters 0" = {0-, )r} that maximize the following cost function:
arg max (o(((I;)) + (x::c)) + (5)
s.t. DKL(s=116(s')) = and (c,,s= =0
(I)
Common InfoMax
Salient InfoMax
Information-Iess hyp.
Independence hyp.
4

---

Published as a conference paper at ICLR 2024
In Sec. 3.1 we show how to estimate the common terms, [(x; c) and [ c), via formulation similar to the alignment and entropy terms introduced in Wang Isola (2020). In Sec. Sec.B.2, we take into account the information-less hypothesis (i.e. background embeddings should always be equal to an information-less vector the salient space) to estimate the salient term T(y: 8). Ultimately, in
Sec. 3.3 we propose strategy enforce the independence hypothesis i.e. [ 8) 0, that prevents
information leakage between the common and salient space.
3.1
RETRIEVE INFONCE FROM INFOMAX FOR COMMON SPACE
In this section, demonstrate that I (x;; c) and (y: c) can be estimated via the multi-view alignment and uniformity losses inspired by Wang & Isola (2020). Full derivation can be found in Appendix Let fee be the common encoder and e fec (8(.) be the common representations. The MI
Sec. A
[xx; c) (same reasoning is also valid for [(y; can be decomposed into:
(xx;e) --E-EEpp Empp, (c(z)) + (())
Alignment
Entropy
Entropy (Uniformity). As in Wang & Isola (2020), the entropy can be computed with non- parametric estimator described in Ahmad Lin 1976). To do so, we compute the approximate density function p(c,) with Kernel Density Estimator as in Parzen (1962); Rosenblatt (956), based on views by (random augmentation of an image with index j) uniformly sampled from both the target dataset fo, (t(y)) p(cly) and the background dataset fo, (t(x:) p(c|.)). We choose a Gaussian kernel with constant standard deviation т, which results in an L2 distance between the
views. However, in practice, we constrain the outputs fee (-) to be unit-normed, which is equivalent to directly choosing a von Mises-Fischer kernel with concentration parameter 1. As in Wang &
Isola 2020), optimize a lower bound of this estimator in practice, called -unar-
Cuiit log
Nx Ny
Σ 1=1 N
1
Nx+
exp
-11 (v.) -fe (12
log / (3)
Nx + NY
2T
Constant term
Differently from Wang & Isola (2020), we propose to estimate the conditional entropy H (c)x) with a re-substitution entropy estimator. We compute the approximate density function p(eg(ar)) (ara) with a Kernel Density Estimator based on samples uniformly drawn from the conditional distribution of ~ p(cla,). where of [o(( and of are K views obtained via the stochastic process t(.). As for the entropy term, we choose a Gaussian kernel with constant standard deviation т to derive an L2 distance between the views. Our formulation generalizes Wang & Isola (2020), directly retrieve a multi-view aligmment term between K positive views of the same image and not a single-view alignment as in Wang & Isola (2020). However, in practice, to reduce the computational burden, we also choose a single view K=1, as in Wang & Isola (2020). Combining the background alignment (clxr) and the target alignment -aaeenttH(g (cly))
Aligmment:
obtain:
NY
K - f1 (v:) fee ( (c/)
1
Latign
log
exp
log( V2TT
(4)
Nx + NY
K
Constant torm
On the relation with I(fa (v), (v'). Many recent representation learning works (Chen & He (020)); Wang & Isola (2020) maximize the MI between two views and v of 2: (ff((v) fe(e").
Inspired by the InfoMax principle, we propose instead maximizing I(fo As shown in
Tschannen al. (2019), by directly applying the data processing inequality, one demonstrate that 1 (Fe(u), fo is lower bound of (v)
3.2 DERIVE THE BACKGROUND-CONTRASS
ALIGNMENT AND UNIFORMITY TERMS
In this section, consider the maximization of the salient term 7(s; y), which is decomposed into an alignment and uniformity term as before, constrained by the information-less hypothesis:
arg max (s:
Eynp, Hslyy)
H(s)
s.t. DKL (60()))((')))>)))))))))))))))(
Target-only Alignment
of --ntrop
Information.ee less hyp.
Intuitivelyy if [oo (-) ||2 = 1. the L2 distance between two representations can be simplified into negative dot-product: Il.f (v:) foo (v;)le =2 2f (v:)² (oy). Full proof in Appendix See A.2.1
5

---

Published as a conference paper at ICLR 2024
Target-only alignment. To estimate the target samples' alignment term, the same estimation method as in 3.1 Namely, we derive an alignment term between two views 8; = t(y)) and o = H(f)) of the same target image Y/- As in Sec. 3.1 we use re-substitution estimator with a Gaussian Kernel Density Estimation with constant standard deviation and K = 1 in practice.
1
Ny
1 K
K
- flg (vs) for (c1011
Cy-aligo
log
exp
log( (VTT)
(6)
Ny
2T
Constant term
s' -Uniformity. Concerning the Entropy term, as in Sec. 3.1 we propose to develop the salient entropy with re-substitution entropy estimator. Again, we use lower bound of HS) called -oit- Then, we estimate the density p(s) with a Gaussian Kernel Density Estimator based samples uniformly drawn from the target dataset fag (f(y))
p(sly) and from the back- ground dataset (t())) ~ p(s[z)) Importantly. the information-less hypothesis constrains the salient encoder to produce background embeddings always equal to the information-less vector s': fog (+()) ~ 8(s'.. Using this hypothesis in the computations (see Sec. B.2 of the Supplementary)
and ignoring constant terms, we obtain:
Ny
fo. (t(y.)
1 Ny
llfo, (t(yi) fo. (t(y;))
Cg'-unif = log
exp
exp
Ny
2T
To respect the Information-less hypothesis, we write Eq. als Lagrangian function, with the straint expressed as a 3-weighted (3 2 0) KL regularization. Assuming that Sa follows a Gaussian distribution centered on fo, [a)(x) with standard deviation т (constant hyper-parameter), the KL divergence as an L2-distance between fo, (1x)) and s', as in He et al. (2019):
derive
115o, (1(4)) x112
F(0s,B;,,,,.3 Àg Ayraaanign AgL
Σ
(8)
Nx
2T
3.3 ON THE NULL MUTUAL INFORMATION CONSTRAINT
In Eq. [ to avoid information Icakage between common and salient space, constrain prob- lem so that the MI between è and 8 null. Another common choice would be to simply minimize
T(c, s) instead than forcing it to be equal 0. In Tab. and 2 we show that the latter choice (i.e.,
[[c, 8
0) clearly outperforms (variational) MI minimization methods, as vCLUB Cheng et al.
al. (2023), (see Sec. F.7 Minimizing H(c) H(s) is detrimental: By def., [[c: 8) -H(c, s)
2200 vL Poole et al. (2019), vUB Alemi (2020), and TC
H(c) + ((s) 2 O, which entails ((: s) S (c) + (s). Thus, a trivial way to minimize I(c; would be minimizing H (c))+H( (s). However, it reduces the quantity of information contained in either
s, which could be detrimental. Furthermore, the Common and Salient InfoMax losses of our framework seek to maximize ((c) and (s) rather than minimizing them. This is why, instead of minimizing 7 ((;; s), we propose to simultaneously maximize (c, s), (c) and (s), until H (c, = H()) + H(s), respect the constraint (cc s) of
To estimate and maximize H(c, s). we propose a new method, called kernel-based Joint En- assumptions about the form of the pdf nor neural network-based approximation (Cheng et al. (2020); Alemi (2020); Poool et al. (2019)) Inspired by Holmes al. (2007), we develop Hc, with re-substitution entropy estimator: (c, Nx+NY
tropy Maximization (k-JEM). that requires
N F.y logg(c,, 8;). We estimate the density pa(c, 8;) with a Gaussian
Kernel Density Estimation with a constant standard deviation parameter 7 with samples (c, 8) uni- formly drawn from the target dataset (t(w),, for (1(y)) p(c, sly) and from the background dataset (fes (t(z)), fee (t(x)) p(c, s(x). The indices and j refer to two different samples in the dataset. Full computations Appendix, Sec. D
Nx+Ny
N
||cc c,112
-1|s 8,
-H( Nx NY
Σ log
Σ
exp
exp
(9)
Nx + Nv
2r
2T
j=1
In this work, implicitly assume that the encoders fo, and fo, can model any distribution.

---

Published as a conference paper at ICLR 2024
4 DISENTANGLING ATTRIBUTES IN THE SALIENT SPACE
Here, we propose to explore an extension of the salient contrastive loss in the case where indepen- dent fine-grained attributes about the target dataset {a; E Y are available. We assume the existence of Ds attributes and each attribute a is generated by a single factor of generation sy of the target dataset. We also make the hypothesis that the given attributes describe the entire salient variability of the target dataset and thus construct our salient encoder to output (exactly) Ds la- tent dimensions. We aim to construct salient space where each salient latent dimension Sth. only depends on its corresponding attribute ade, By leveraging the attributes in a supervised manner, we re-write Eq. by replacing (1:; s) with the sum of all attribute Supervised InfoMax terms:
N
arg max (x;e)) [(y;
s.t. DKT. (sx(((x))(())) 0 and [(e, s) 0 (10)
D3
d-th SuptnfoMax
Taking inspiration from Dufumier et al, (2021a/b), we then decompose each d-th attribute Super- vised InfoMax term in a supervised alignment and uniformity term:
Ny wa(af 0z)
|| 89lle ff(s La-th SupInfoMax
I(a
(11)
where the indices i and j refer to two different samples in the target dataset and the scalar weight Ka(af
wa(a (aa aa)) Σj=ı K A (ag,a9)
measures the similarity between their attributes. We define KA as
Gaussian kernel and the entropy 1 (s²) is also estimated, as before, with Gaussian kernel.
5 EXPERIMENTS
Here, we measure our method's ability to separate common from target-specific variability factors. We train Logistic (or Linear) Regression on inferred factors to assess whether the information about a characteristic present in both populations, or only in the target one, is captured in the com- mon (C) in the salient (S) latent space. We compute (Balanced) Accuracy scores (=(B-)ACC), or Area-under Curve scores (=AUC) for categorical variables, Mean Average Error (=MAE) for con- tinuous variables, and the sum of the differences (frot) between the obtained results and the expected ones.
Hyper-parameters. We empirically choose T 0.5 for all experiments and losses. The other hyper-parameters are Ae and Äs. which weigh the common terms and salient terms. respectively, and X, which weighs the independence regularization. The choice of these weights depends on the ratio between common and target-salient information quantity, which might differ among datasets. Architectures and hyper-pan syper-parameters are chosen as the nop-perfomming ones for each experiment.
SOTA CA methods We have compared the performance of our method with the most recent and best-performing CA-VAE methods whose code was available: cVAE Abid Zou (2019) 5 con VAE
Aglinskas et al. 2022 6 MM-cVAE Weinberger et al. (2022) and Sep Louiset al. (2023). In each experiment, all VAE use the same encoder-decode architecture, as described in the Sup-
plementary F The architecture used for SepCI.R is also described in Sec. F
Digits superimposed on natural backgrounds. In this experiment particularly suited to CA and inspired from Zou et al. (2013), we consider CIFAR-IO images as the background dataset (y 0) and CIFAR- 10 images with an overlaid digit as the target dataset (y 1). In Tab [ our model out- performs all other methods in correctly capturing the common factors of variability (i.e: objects) in the common space and the target-specific factors of variability (i.e: digits) in the salient space.
CelebA accessories dataset. We consider subset of CelebA Liu et al.] The target class (y = 1) contains images of celebrities wearing glasses or hats. The background class (y 0) contains im- ages of celebrities without accessories. In Tab 2 SepCIR correctly captures the information that enables distinguishing glasses from hats only in the salient space, and it puts the information to distinguish men from females in the common space. Our method globally outperforms all other methods (smallest 5tae). "Best Expected" reports perfect results (100%) when the attribute should be present in that latent space and a random result (50%) when it should not.
*If it is not true, one add Salient InfoMax term (Eg I and increase the salient space dimension. 'Here, for eVAF, we use the fixed version of the TC regularization described in Louiset et al. 2023) 'Here, VAF corresponds to eVAE method without the TC regularization, as in Aglinskas al. (0022
7

---

Published as a conference paper at ICLR 2024
Table 1: Digits on CIFAR-10 (B-ACC). Details in Sec F.7
Table 2: CelebA accessories (B-ACC).
DIGITS
OBIECTS
HATS/GL
SEX
boo 4
S1
C4
4
C1
S
C4
eVAE
90.6
23.0
33.4
90.2
83.89
66.56
60.25
60.60
82.32
CONVAE
86.2
21.0
10.6
35.6
89.8
81.64
65.94
61.53
58.93
86.90
MM-CVAE
88.8
19.6
12.2
32.0
93.6
84.60
66.43
60.56
61.57
80.82
SEPVAE
90.6
17.8
10.6
36.6
81.2
84.46
65.19
60.12
59.20
81.65
SULCLR-VCLUB SYM
94.4
18.0
14.6
97.0
98.98
59.62
65.20
54,23
71.61
SEPCLR-VCLUB C S
95.2
39.4
9.2
27.2
106.2
98.81
73.71
61.77
53.72
82.95
SEPCLR SEPCLR-VCLLBB -
95.2
57.0
8.8
31.8
118.8
98.66
95.95
67.65
73.16
91.78
SE
95.0
15.4
96.4
98.83
56.94
57.97
$1.38
64.60
SEPCLR-vLIO
94.0
23.0
10.0
31.8
87.2
99.04
93.17
63.35
59.13
89.91
SEPCLR-VLIO C
95.4
41.0
9.2
28.8
106.0
98.46
94.33
65.00
71.77
89.10
SEPCLR-YUB SYM
94.6
42.0
29.0
98.68
63.59
56.09
SEPCLR-VUB C
92.8
23.4
7.8
22.6
95.8
98.73
94,40
66.58
71.37
90.88
SEPCLR-VUB
96.6
41.8
8.6
28.6
105.2
98.78
93.92
62.94
61.27
96.81
SEPCLR-TC
95.2
10.2
24.2
139.4
98.97
98.76
60.39
74.96
85.22
SEPCLR-MMD
94.6
62.2
53.4
98.95
67.50
71.83
65.47
74.91
SEPCLR-NO K-JEM
95.6
94.4
9.0
42.0
145.8
99.03
66.68
98.48
79.48
86.65
SEPCLR K-MI
96.2
19.8
8.0
65.8
45.8
98.96
77.10
63.07
71.08
70.13
SEPCLR-K-JER
96.2
73.2
32.0
98.57
55.21
62.52
78.00
41.16
BEST EXPECTED
100.0
10.0
10.0
100.0
0.0
100.0
50.0
50.0
100.0
0.0
Neuroimaging: parsing schizophrenia's heterogeneity. Separating healthy from pathological la- tent mechanisms that drive anuto-anatomical variability in schizophrenia is challenging. Yet, this ability could help understand and anticipate the development of these diseases. Given healthy MRI scans and patients with schizophrenia, we aim to capture pathological patterns only in the salient space that should correlate with clinical scales (such as positive symptoms: SAPS, and negative symptoms: SANS) while not being biased by demographic variables (age, sex or acquisition sites), which should be encoded the common space. As in Louiset et al. (2021) 1 2023), we gather Tlw VBM Ashburner (2000) warped MRIs (1283 voxels) and evaluate our method in cross-validation scheme. In Table 3, we can clearly see that our method outperforms all others.
Table 3: Separate healthy from pathological variability Schizophrenia disorder. Best in bold.
AGE MAE
SEx B- B-ACC
SITE B B-ACC
S
$
$
CVAE
6.43::0.18
7.27=0.25
75.06=3.488
74.99:2.13
65.12:4.06
59.6245.42
CONVAE
6.40++0.26
7.46+0.18
1.84
1..72+1.32
60.42+3.67
54.46++2.46
MM-CVAE
6.
7.10=0.34
72.80=3.95
12.15+2.47
63.24=1.41
56.69199.84
SEPVAE
6.40:10.13
0.28
74.1921.81
72.61
63.8912.16
44.10:15.78
SEPCLR-K K.JEM
6.64++0.1
7.72+0.45
76.5+
1.89
66.9445.06
42.40
SANS
MAE
SAPS MAE
DIAGNOSIS
C1
S
Ct
S
C
S
CVAE
5.89110.7
4.3520.26
4.65:10.34
2.98.1.18 18
60.6612.63
68. 68.24:15.42
CONVAE
6.17:10.45
3.95-0.28
4.50-0.37
2.76:10.18
61.85+2..0
58,53:14.87
MM-CVAE
6.78-+0.54
4.92+0.58
4.52+0.33
3.16++0.05
64.25+2..98
70.94++4.08
SErVAE
7.05:10.67
4.1420.39
4.79-1067
2.60=10.27
60.90:11.75
79.15:13.39
SEPCLR MEICLR-KJJER JEM
9.17:12.49
3.74=0.12
5.54:10.70
2.52:10.16
60.16::1.19
1959
Chest and eye pathologies subtyping. We propose two experiments using subsets of CheXpert Irvin (2019) and ODIR dataset (Ocular Disease Intelligent Recognition dataset) to assess the abil- ity of our method a controlled environment, About CheXpert, we have healthy X-ray scans (back- ground) and diseased scans (target) divided into 3 distinct subtypes: cardiomegaly, lung edema, and pleural effusion. In the ODIR dataset, there are healthy (background) and diseased fundus images (target) which are divided into 5 subtypes: Diabetes, Glaucoma, Cataract, Age macular degenera- tion, and pathological Myopia. Sex-related patterns should only be captured in the common encoder. Disentangling dSprites while contrasting with background. To evaluate CA method enriched with target attributes, we provide a novel toy dataset. Background dataset X consists grid. Target dataset Y consists of dSprites item added upon the grid of digits. dSprites only exhibit generation factors (shape, zoom, rotation, X position, Y posi- tion). Using Eq. [ we train our salient encoders supervised manner to capture and disentangle each attribute in a single salient space dimension (Fig. 22). The common encoder is instead trained capture the background variability (Fig. 2bb. Quantitatively, 1st salient dimension distinguishes captur
MNIST
digits (1-4) regularly placed
shapes (B-ACC= 98, 23%) while the concatenation of other salient dimensions and common di- mensions does not AB-ACC= 36.08%) 2nd predicts zoom attribute (R² 0.977) while others don
8

---

Published as a conference paper at ICLR 2024
0.002. 3rd predicts rotation (R2 = 0.947), others don't R? = 0.0. 4u predicts horizontal translation
(R² = 0.995), others don't 't R = 0.017. 5α predicts vertical translation (R² = 0.995), others don't t
R 0.009. This shows that our method correctly separate common from salient information and disentangle salient factors (in a supervised manner) at the same time.
Table 4: CheXpert X-ray scans (B-ACC).
Table 5: ODIR fundus images (B-ACC).
SUBTYPE
SEX
S
SUBTYPE
SEX
O +
87
C
8
C
St
C
S
Ct
CVAE
45.77
49.27
54.24
81.26
93.48
46.13
43.91
49.11
$1.86
120.03
CONVAE
42.31
52.53
60.88
79.30
108.8
49.80
52.01
50.82
47.01
131.86
MM-CVAE
42.50
50.89
57.04
80.19
102.24
42.79
43.66
54.91
53.76
131.02
SEPVAE
42.20
51.10
56.38
79.95
102.34
38.64
41.44
52.91
52.62
124.75
SEPCER R-K-JEM
61.30
52.85
61.57
80.25
89.87
68.54
47.71
52.48
59.62
97.03
BEST EXPECTED
100.0
33.0
50.0
100.0
0.0
100.0
25.0
30.0
100.0
0.0
3
1
3
3
1
3
3
1
3
3
4
4/
2
2 4
2 4
X
4
4
4
_
4
3
4
3
3
2. 4
2 4
2 4
2 4
! 3
1 3 4
3
4 3
$ 3
3
2 4
4
2 4
X
4
o
4
3 à 4
2
2-
4
2
3 2 4
3 2 4
/
/ 3
/ 3
/ 3
/ 2
4 2 4
3
2 1
2
2 +
2 4
2
4
X 4
/ 3
A 3
6 3
/ 3
/ 3 à 4
3 4
3
/
3
3 3 2 4
2 4
2 X
2 4
2
(a) Given the latent vector of the upper left image, (b) Given the common latent vector of image (left col- we modify only one salient dimension in cach row umn), fetch the image in the dataset whose inferred com- while freezing the others, then fetch the image in mon latent vector is the closest in terms of L2 distance.
the dataset whose latent vector is the closest.
Figure 2: Qualitative results on attribute seribute-superviseed SepCIR
6 LIMITATIONS AND PERSPECTIVES
An important question in Contrastive Analysis, is the identifiability of the models. Namely, under which conditions can the models recover the true latent factors of the underlying tata-generating process. Recent works have shown that non-linear models, VAEs included, are generally not iden- tifiable. To obtain identifiability, two different solutions have been proposed: 1) either regulariz- ing Kivva et al. (2022) the encoder or 2) introducing an auxiliary variable so that the latent factors are conditionally independent given the auxiliary variable [Hyvrinnn et al. (2019); Khemakhem eet al. (202)). In CA, neither of these solutions may be used [ Even though SepCLR effectively separates common from salient factors, does not assure that all true generative factors have been identified (like all CA methods). This is serious limitation of CA methods that we leave for future works. Intriguingly. we also noticed that adding a reconstruction loss during the training degrades performance, see Sec. G.2 in Appendix. However, adding powerful generator, in Zou et al. (2023); Carton et al. (2024), on top of the frozen encoders would allow synthesizing new images and increase interpretability.
7 CONCLUSION
In this paper, leverage the power of Contrastive Learning learn semantically relevant repre- sentations for Contrastive Analysis. We reformulate Contrastive Analysis as a constrained InfoMax paradigm. Then, we propose to estimate the Mutual Information terms via alignment and uniformity terms. Importantly, we motivate a novel independence term between common and salient spaces computed via Kernel Density Estimation (KDE). Our method outperforms related works on toy, natural, and medical datasets specifically made to evaluate the common/salient separation ability.
"hhe dataset label could be considered auxiliary variahle, but does not make and independent

---

Published as a conference paper at ICLR 2024
REFERENCES
Abubakar Abid and James Zou. Contrastive Variational Autoencoder Enhances Salient Features, February 2019.
Abubakar Abid, Martin J. Zhang, Vivek K. Bagaria, and James Zou. Exploring patterns enriched in a dataset with contrastive principal component analysis. Nature Communications, 9(1):2134, May 2018.
Aidas Aglinskas, Joshua K. Hartshorne, and Stefano Anzellotti. Contrastive machine learning re- veals the structure of neuroanatomical variation within autism. Science (New York, N.Y.), 376 (6597):1070-1074, 2022.
I. Ahmad and Pi-Erh Lin. A nonparametric estimation of the entropy for absolutely continuous distributions (Corresp.). IEEE Transactions on Information Theory, 22(3):372-375, May 1976.
Alexander A. Alemi. Variational Predictive Information Bottleneck. pp. 1-6. PMLR, February 2020.
Luigi Antelmi, Nicholas Ayache, Philippe Robert, and Marco Lorenzi. Sparse Multi-Channel Vari- ational Autoencoder for the Joint Analysis of Heterogeneous Data. pp. 302-311, PMLR, May 2019.
John Ashburner. Voxel-based morphometry 'he methods. Neurolmage, pp. 805 821, 2000.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning Representations by Maximiz- ing Mutual Information Across Views. June 2019.
Carlo Alberto Barbano, Benoit Dufumier, Edouard Duchesnay, Marco Grangetto, and Pictro Gori. Contrastive learning for regression in multi-site brain age prediction. In JEEE 20th International Symposium on Biomedical Imaging (ISBI), 2023a.
Carlo Alberto Barbano, Benoit Dufumier, Enzo Tartaglione, Marco Grangetto, and Pictro Gori. Unbiased Supervised Contrastive Learning. In International Conference on Learning Represen- tations (ICLR), 2023b.
Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 365(6356):161-66,, 1992.
A. Bell and T. Sejnowski. An information-maxxinn maximization approach to blind separation and blind November 1995.
deconvolution. Neural Computation,
Florence Carton, Robin Louiset, and Pietro Gori. Double InfoGAN for Contrastive Analysis. In International Conference Artificial Intelligence and Statisties (AISTATS), 2024.
Ricky Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating Sources of Disentanglement in Variational Autoencoders. In Advances in Neural Information Processing Systems, volume 31, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. pp. 1597-1607, November 2020.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational Lossy Autoencoder. In ICLR, 2017.
Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning, November 2020.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and L. Carin. CLUB: A Con- trastive Log-ratio Upper Bound of Mutual Information. June 2020.
Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu. Qingyi Chen, Jue Wang, and Hongkai Xiong. Motion- aware Contrastive Video Representation Learning via Foreground- background Merging, March 2022.
10

---

Published as a conference paper at ICLR 2024
Benoit Dufumier, Pietro Gori, and Edouard Duchesnay. Contrastive Learning with Continuous Proxy Meta-data for 3D MRI Classification. In Medical Image Computing and Computer Assisted Intervention MICCAI 2021, pp. 58-68, 2021a.
Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, and Edouard Duchesnay. Conditional Alignment and Uniformity for Contrastive Learning with Continuous Proxy Labels. In Med- NeurlPS, Workshop NeurIPS, 2021b.
Benoit Dufumier, Carlo Alberto Barbano, Robin Louiset, Edouard Duchesnay, and Pietro Gori. Integrating Prior Knowledge in Contrastive Learning with Kernel. In International Conference on Machine Learning (ICML), 2023.
Rong Ge and James Zou. Rich component analysis. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML 2016, pp- 1502-1510, June 2016.
Behnam Gholami, Pritish Sahu, Ognjen Rudovic, Konstantinos Bousmalis, and Vladimir Pavlovic. Unsupervised Multi- Target Domain Adaptation: An Information Theoretic Approach, October 2018.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengehao Wang. Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self-supervised Pretraining of Visual Features in the Wild, March 2021.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A keree two-sample test. The Journal of Machine Learning Research, 13:723-773, March 2012.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. pp. 9726-9735, June 2020.
Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, and Xiangyu Zhang. Bounding Box Regression With Uncertainty for Accurate Object Detection. pp. 2883-2892, June 2019.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization, 2019. arXiv:1808.066: 06670 [cs, stat].
Michael P. Holmes, Alexander G. Gray, and Charles Lee Isbell. Fast nonparametric conditional density estimation. In Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence, UAI 2007, pp. 175-182, Arlington, Virginia, USA, July 2007.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard E Turner. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning. In AISTATS, 2019.
Jeremy et Irvin. CheXpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence, pp- 590-597, Honolulu, Hawaii, USA, January 2019.
Andrew Jones, F. William Townes, Didong Li, and Barbara E. Engelhardt. Contrastive latent variable modeling with application case-control sequencing experiments, February 2021.
Hadi Kazemi, Seyed Mchdi Iranmanesh, and Nasser Nasrabadi. Style and Content Disentanglement in Generative Adversarial Networks. pp. 848 848-856, January 2019.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational Autoen- coders and Nonlinear ICA: A Unifying Framework. In Proceedings of the Twenty Third Interna- tional Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, June 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised Contrastive Learning. In Advances Neural Information Processing Systems, volume 33, pp. 18661-18673, 2020.
Hyunjik Kim and A. Moih. Disentangling by Factorising. February 2018.
11

---

Published as a conference paper at ICLR 2024
Diederik P. Kingma and Max Welling. Auto-Encodinn Variational Bayes. In International Confer- ence on Learning Representations, ICLR, 2014.
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. In NeurlPS, 2022.
Lingpeng Kong, Cyprien de Masson d' Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yo- gatama. A Mutual Information Maximization Perspective of Language Representation Learning. September 2019.
Didong Li, Andrew Jones, and Barbara Engelhardt. Probabilistic Contrastive Principal Component Analysis, April 2021.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical Contrastive Learning of Unsu- pervised Representations. October 2020.
R. Linsker. Self-organization in a perceptual network. 21(3):105-117, March 1988.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3730-3738.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A Commentary on the Unsupervised Learning of Disentangled Representations. Proceedings of the AAAI Conference on Artificial Intelligence, 34(09). 2020.
Robin Louiset, Pietro Gori, Benoit Dufumier, Josselin Houenou, Antoine Grigis, and Edouard Duch- esnay. UCSL: A Machine Learning Expectation- Maximization Framework for Unsupervised Clustering Driven by Supervised Learning. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/KKDD), Lecture Notes in Computer Science, pp. 755-771, 2021.
Robin Louiset, Edouard Duchesnay, Antoine Grigis, Benoit Dufumier, and Pietro Gori. SepVAE: contrastive VAE separate pathological patterns from healthy ones, July 2023. Workshop on Interpretable ML in Healthcare at International Conference on Machine Learning (ICML).
Siddharth N. Brooks Paige. Jan- Milleillem van de Meent, Alban Desmaison, Noah Goodman, Push- meet Kohli, Frank Wood, and Philip Torr. Learning Disentangled Representations with Semi- Supervised Deep Generative Models. In Advances in Neural Information Processing Systems, volume 30, 2017.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-588) November 2010.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Pre- dictive Coding, January 2019.
Emanuel Parzen. On Estimation of Probability Density Function and Mode. The Annals of Math- ematical Statistics, 3363):1065-1076, 1962.
Mary Phuong, Max Welling, Nate Kushman, Ryota Tomioka, and Sebastian Nowozin. The Mutual Autoencoder: Controlling Information in Latent Code Representations. In Arxiv, 2018.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On Variational Bounds of Mutual Information. Proceedings of the 36th International Conference on Machine Learning, pp. 5171-5180. PMLR, May 2019.
Borja Rodríguez-Gäávez, Arno Blaas, Pau Rodríguez, Adam Goliñski, Xavier Suau, Jason Ramapu- ram, Dan Busbridge, and Luca Zappella. The Role of Entropy and Reconstruction in Multi- View Self-Supervised Learning, July 2023.
Murray Rosenblatt. Remarks on Some Nonparametric Estimates a Density Function. The Annals of Mathematical Statisties, 27(3):832-82 837,, September 1956.
12

---

Published as a conference paper at ICLR 2024
Lukas Ruff, Robert A. Vandermeulen, Nico Gömitz, Alexander Binder, Emmanuel Müller, Klaus- Robert Müller, and Marius Kloft. Deep Semi- cemi-Supervised Anomaly Detection. September 2019.
Adria Ruiz, Oriol Martínez, Xavier Binefa, and Jakob Verbeek. Learning Disentangled Representa- tions with Reference-Bante Based Variational Autoencoders. ArXiv, January 2019.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):3003668 May 2023.
Kristen A. Severson, Soumya Ghosh, and Kenney Ng. Unsupervised Learning with Contrastive Latent Variable Models. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 4862-4869, 2018.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the Breg- man divergence: a unified framework of density-ratie estimation. Annals of the Institute of Sta- ristical Mathematics, 6405): 1044, October 2012.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning Video Representations using Contrastive Bidirectional Transformer, September 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Multiview Coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Lan-Michael Frahm (eds.), Computer Vision - ECCV 2020, pp. 776-794, 2020.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised Learning from a Multi-view Perspective. October 2020.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On Mutual Information Maximization for Representation Learning. September 2019.
Julius Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf. Michel Besserve, and Francesco Locatello. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style. In Advances in Neural Information Processing Systems, volume 34, pp. 16451-16467, 2021.
Ke Wang, Harshitha Machiraju, Oh-Hyeon Choung, Michael Herzog, and Pascal Frossard. CLAD: A Contrastive Learning based Approach for Background Debiasing. arXiv, 2022.
Tongzhou Wang and Phillip Isola. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. May 2020.
Chen Wei, Huiyu Wang, Wei Shen, and Alan Yuille. CO2: Consistent Contrast for Unsupervised
Visual Representation Learning. September 2020.
Ethan Weinberger, Nicasia Beebe-Wang, and Su-In Lee. Moment Matching Deep Contrastive Latent
Variable Models, February 2022.
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal Contrastive Training for Visual Representation Learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6991-7000, June 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. pp. 12310-12320. PMLR, July 2021.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE: Balancing Learning and Inference in Variational Autoencoders. Proceedings of the AAAI Conference Artificial Intelligence, 33(01): 5892, July 2019. ISSN 2374-3466 doi: 10. 100.1609/aaii.v333 v33i01 3330158855 330003338.. URL https:
//ojs. aaai. org/ / inde. hhtp:///AA.A...aaaccc////es/////
Number: 01.
James Y Zou, Daniel J Hsu, David C Parkes, and Ryan P Adams. Contrastive Learning Using Spectral Methods. In Advances in Newral Information Processing Systems, volume 26, 2013.
Kaifeng Zou, Sylvain Faisan, Fabrice Heitz, and Sebastien Valette. Joint Disentanglement of Labels and Their Features with VAE. pp. 1341-1345, Bordeaux, France, October 2022.
13

---

Published as a conference paper at ICLR 2024
Kaifeng Zou, Sylvain Faisan, Fabrice Heitz, and Sébastien Valette. Disentangling high-level factors and their features with conditional vector quantized VAEs. Pattern Recognition Letters, 172: 172-180, August 2023.
14

---

Published as a conference paper at ICLR 2024
A RETRIEVE THE INFONCE LOSS
Let X = {<=)]]= be the data-set of background images Iú, and Y [ be the data-set of target images y/s- Imput samples are assumed to be independently generated from latent unobserved variables C {c; R 1 Our objective is to estimate an encoder foc that infers latent factors of generation c from the imputs (and its views v).
fi-1-
To do so, entitle the latent codes produced by the common encoder foe (4(.), where ((.) v the views generated from either a or y via a stochastic augmentation function i(.). The objective is to construct an encoder fea (t()) that invariant data augmentation. From the InfoMax perspective, we seek the optimal parameters 0* that maximize the MI between and ~ fee (t(x)). Foremost, we decompose the MI [a;; c) into:
[(;;c)= = -- -Egop,
H(c + H(c)
(12)
Aligument
Entropy
but the same reasoning is valid for the target dataset: [G:y)= Eyyr, H(cly) Hc)..
A.1 DERIVE THE UNIFORMITY TERM FROM THE ENTROPY TERM
In this section, we propose to make the correspondence between the concept of Entropy, well-known in Mutual Information literature, and the concept of Uniformity introduced Wang & Isola (2020). The entropy can be derived with non-parametric estimator described in Ahmad & Lin (1976) with samples uniformly drawn from both the target dataset and the background dataset.
+x+ Ny
1
((c) Nx NY
Σ
log ((c;)
(13)
Then, we compute the approximate density function p(c,) with Kernel Density Estimator, based samples uniformly drawn from both the target dataset fo, (t(y)) p(cly) and the background dataset fo, (t(x)) p(clz):
Nx+Nv
Ny Σ J=1
Ac()) = Nx +
Σ log
Kc(c,
(14)
Nx
+ Ny
For simplicity, we choose Gaussian kernel with constant standard deviation to derive an L2 distance between the views. This enables us to obtain:
(((c) = Nx Ny
1
Nx+Ny
Σ 1
log
Σ exp 1=1
- -
log 2
(15)
Nx
NY
27
Constant
where Cj = fo(y;) and = fo().. And where v; and Uj are the views obtained by feeding the input with index (can be target or a background sample) through the stochastic data augmentation function t(.). In practice, Wang & Isola (2020) minimize the asymptotic lower bound of this term entitled Uniformity term. Using Jensen's inequality, we obtain:
1
Nv+Ny
Σ
log Nx + NY
exp
-115((r))- So(
Nx Ny
2T
-ir(e) Iog(
(16)
1
Nx+Ny
Nx+
|ee - for (v))
log
exp
Nx + NY
Nx + NY
2T
Custonn
Given a bounded support, minimizing Lumit encourages the latent vectors to match a uniform distri- bution (e.g: spherical uniform distribution unit-norm support in Wang Isola (2020)).
15

---

Published as a conference paper at ICLR 2024
A.2 DERIVE THE MULTI- VIEW ALIGNMENT TERM
Differently from Wang & Isola (020), we propose to estimate the conditional entropy on back- ground samples x with re-substitution entropy estimator.
1 Nx
H(cz))
log (og((c;(a)))
(17)
Nx
We compute the approximate density function ((c)(24)) with Kernel Density Estimator based on samples uniformly drawn from the conditional distribution of p(claz), where of = ff( and of are K views obtained via the stochastic process (())
Nx
K
-H(c)z) = Nx
log
Kolfec fec K
Σ Km
(18)
Ke is chosen choices enable us to retrieve a Multi-Vise View Alignment term with positive views rather than only in Wang & Isola (2020):
Mises-Fisher kernel with constant concentration parameter ń = These
((ca)) log(C(n)) 1
exp -Ilfee (v:) fae (
log
(19)
Nx
27
By estimating the conditional entropy target samples both, we can retrieve the Alignment term written in Eq. only one view in this paper: K
((lyy) in the same fashion and summing For computational reasons, we restrict to
A.2.1 ON THE CONNECTION BETWEEN THE GAUSSIAN KERNEL, AND THE VON MISES-FISHER KERNEL.
Let us note the kernel similarity between two representations: for (x;) and (2,) as K (Joo),))) So (az)). Assuming that we are given a Gaussian kernel with constant standard devi- ation o, this term can be estimated as:
1 exp (1J00 S (xr))
KGuussian (fee (x:), fee (2,))
(20)
V
2T
Now, we can divide the square norm into three terms:
(x:))1 250 (xi)² "oo (x1)) + I| (rr)
Kausssian (Sor (xx), foe (x²)))
/2TT exp
2r
(21)
Let assume that foo (x,) and foo (Iz) are unit-normed, then this estimation get simplified into:
1
-1 + (2:) ( (x,))
K Kaaussian (f (fee (z:), fee (Zz))
exp
(22)
V2
which can be further simplified:
fec (x:)² Soc
Kaussian (feo (1t). fee (2z)) el /
exp
(23)
Ignoring the normalization terms, we recognize the von Mises-Fisher kernel with concentration hyper-parameter 4:
K KYMF = C(k) exp feo (.o))
1
B DERIVE THE BACKGROUND-CONTRASTII INFONCE LOSS IN THE SALIENT SPACE
In this section, we propose deriving the salient [(s; v) into novel loss entitled BC-InfoNCE. Foremost, let us decompose the constrained Mutual Information maximization:
arg max-
Eyyop H (sly)
((s)
S.L. DKL (s (8(ss))
(24)
Target Aligument
Entropy
Information. less hyp.
16

---

Published as a conference paper at ICLR 2024
B.1 ALIGNMENT TARGET SAMPLES:
In order to estimate the target samples' alignment term, we use the same estimation method as in
3.1 First, we derive an alignment term between two views fog (v;) and fos (vf²) of the same target image using re-substitution estimation:
Ny Ey-p, (slyy) = Ny Σ (0s111))
(25)
Then, the density p(s;1y;) is estimated with Kernel Density Estimator based on samples uniformly drawn from Pelpu i.e: {fo(t(y:) ( Hem where Uy,)* = of are K views uniformly drawn from the stochastic trputtrranfoommation process t(y:):
1 N
K N
Eyrr, (slu)
log
(z(fo((s),
(26)
Kz is chosen as von Mises-Fisher kernel with constant concentration parameter x and only K positive view is chosen. These choices enable us derive the target alignment term:
Ny
[[ -fo(u)) )
E Ey H(syy)
(27)
NY
2T
B.2 S'-UNIRORMITY:
Now, concerning the Entropy term, we propose to develop the salient entropy with resubstitution entropy estimator from samples drawn from X U..
H(S) = (Ny + Nx)
log (v)))
(28)
vEI(XUY)
Then we estimate the density P(Jes (v)) with Gaussian Kernel Density Estimator based on latent vectors drawn from the target view f0g (I()) and from the background views fag (1xx)).
1lfe, (v) fe, (vt )
(1(s) = Ny + Nx
Σ log Ny + Nx
Σ (+E((XY))
exp
(29)
RER(XUY)
We consider the asymptotic form of the Entropy. Therefore, we pull the log out of the exterior sum. In practice, it is equivalent considering lower bound of the Entropy. Now, separating the background and the target datasets inside the log yields:
Ny
Nx
-I1fo, (w:) - So, (x;)
exp Lost Lniunofomm
Ny + Nx
Ny
Nx
Σ
exp
i=l
Ø=1
Ny
Ny
-1So, (vs) - So. (uy))1
Σ
exp
Ny + N
Ny + NX
1=1
#=1
(30)
Nx
Nx
fl fe. (r2)
Σ
Σ
exp
Ny + Nx
Ny + Nx
7
1=1
1=1
Nx
Il,e, (x:) - fe. (2)))
exp
Ny + Nx
Ny +
Nx
17

---

Published as a conference paper at ICLR 2024
Importantly, the information-les hypothesis constrains the specific encoder to produce background embeddings aligned on the information-ees vector s'. This property implies that background sam- ples should not have any variability expressed in the latent space. Assuming that the salient encoder respects this property yields fes(((x)) s', it enables express (S) as:
Nx
-looo, - -s'11
Nx
Nx
-exp L uniform
-2
exp
+
Ny x
Ny
Nx
7
Ny
Nx
Ny
Nx
Ny 1
Ny
-Ilso, (v:) fo, (u;)11
exp
Ny + Nx
Ny
+
Nx
7
j=1
(31)
Assuming that the target and background datasets are balanced: Nx Ny = N and ignoring the constant terms, we obtain:
NY
So (y:)
8112
Ny
11, (vi)
fo, (v:))1
Cs'uniform
log
exp
exp
Ny
7
2NY 5=1
7
(32)
B.3 ON THE INFORMATION-LES, HYPOTHESIS:
To respect the information-les. hypothesis, we re-write Eq. as Lagrangian function, with the con- straint expressed as a 3-weighted (3 2 0) KL regularization. Assuming that 8E follows a Gaussian distribution centered fo, (x) with constant standard deviation permits deriving the KL diver- gence into an L2-distance between fa, (x) and 8'. Let us re-write 5q.55nnde the KKT conditions:
Nx 115o, (xs)
1 Nx
F(0s,B:2,0..3 = by alignmment Ly uniformity + B
2112
(33)
1=1
C RETRIEVE THE SUPERVISED INFONCE LOSS
The Supervised counterpart of the InfoNCE loss has been introduced Khosla al. (2020). Com- pared to SimCLR, it consists of choosing positive pairs from the same class, while the negative pairs term remains unchanged. Let associated discrete continuous labels yı. and Z {z+)), the associated latent codes 2, Let us introduce the maximization of Mutual Information between the labels Y and latent vectors Z. The Mutual Information can be decomposed as follows:
{ be data-set of images 2;s { be their
I(z;
Eyro, H(lly)
H(2)
(34)
Supervised Alignment term
Uniformity term
The Supervised counterpart of the InfoNCE loss has been introduced in Khosla et al. 2020). In this section, we show that can be derived from the MI between Y and fe()( Compared to InfoNCE, it consists in aligning positive views (t(x,), t(x,)) from the same class y4 Yj via supervised alignment term - Eyy-py H=lyy), while the entropy term estimation (=z) remains the same. Using the re-substitution estimator and the KDE, we derive the supervised alignment term into the alignment term of Cia in Khosla et al. 2020
Σ exp -Ilfo(u,) jEP())
- -f(cc))
Eggo, H(2ly) log( (og((V2T))
log P(()
(35)
2T
where P(i) is the of indices of samples belonging to class y and IP()] is its cardinality. In Dufumier et al. (2021a), the authors proposed a generalized version of SupCon, which accounts for continuous label y.
18

---

Published as a conference paper at ICLR 2024
C.1 ON THE DISTINCTION BETWEEN C AND SUP
In Khosla et al. 2020), the authors show that it is preferable to optimize Capp a variant of Cup where positive samples summed outside of the logarithm. We propose derive Caup rather than C by simply computing lower bound of the Alignment term via Jensen's inequality:
[[fo(v.)* (f(o))
Ey-p, (=ly)) log( 2TT)
Σ
(36)
N
P(O)
2T
jEP(O)
C.2 QUANTIFY THE JENSEN GAP FOR SUPINFONCE:
In Eq 36 we derived lower bound of the Conditional Entropy of the Supervised InfoMax formula- tion via Jensen's 's inequality. In this paragraph, we propose to a) quantify Jensen's Gap between both formulations and b) describe under which condition these formulations are equal (tight bound). The Jensen's Gap be computed
1
N
[ISo(())* - So(u))1B
-Ilfa(us)* - -oo
JGAP =
log P(i)
exp
N
2T
P(i)
2T
jEP(()
JEP(1)
GAP
(37) where JGAP 2 0. Let us note JGAP if and only if D O. We simplified D into the difference between a LogSumExp and a SumLogExp of ((u:)T.fo(v') Using the fact that LogSumExp consists of a smooth approximation of the max function, DGAP = if and only if:
max max IIfo(u)) fo(v;)1B+ + log N
N Σ
in
(38)
3
where = Mj. ie: Us. Uy and u/ are views from images from the same class
C.3 THE CASE OF A CONTINUOUS y:
In Dutumier al. 202 la), the authors proposed a generalized version of SupCon, which accounts for continuous label It consists of adding weight wo (V(V Uj) before the similarity term. Let us explain how to retrieve this formulation. From Eq. 34 we use the resubstitution estimator:
1
Ey-p, H2lly) + log 2TT)
log p(2.l)))
(39)
From there, we can use a Kernel Density estimation for the conditional distributions in the case where we only have access to samples from the joint distribution:
N
N E Ky Us (f(a())))
EEy-p, H(lyy) + log(v2TTT) = N log
(40)
N プソ Ky m3)
By choosing Ky as gaussian kernel: (yt yU) ov2= exp
(u 2
and Kz as von Mises-
Fisher kernel as usually done in Contrastive Learning literature, we retrieve Dufumier et al, (2021a)'s in formulation.
1
N
ue (c.. exp -11so(x.) 1o(x;)1
- Eyy-,, H=2ly) log( 2TT) =
log
(41)
2r
==1
Ky (H
where wo (Ht, 43)
Σ Ky Ky (v..g))
Now, the Jensen's inequality can be utilized to retrieve Dufumier et al. (2021 's exact formulation.
19

---

Published as a conference paper at ICLR 2024
D MAXIMIZE THE JOINT ENTROPY VIA KERNEL DENSITY-BASED ESTIMATION
In Sec. 3.3 we proposed a method to estimate and minimize - (c, 8) without requiring any as- sumptions about the form of its pdf nor requiring a neural network-ber based approximation (Cheng ct al. 2020); Alemi 2220); Poole et al. (2019)). Inspired by Holmes et al. (2007), develop
(cc, 8) with a re-substitution entropy estimator:
Nx+Ny E
(c, Nx Ny
log p(c, 8,)
(42)
To do so, we estimate the density po(c, si) with a Kernel Density Estimation:
Nx+Ny
-H(css)= Nx+
1
Σ log
1
Σ
Kolc,, ()s(ss)))s()ss)))
(43)
Nx NY
k=1
where ej and 8j are drawn from the joint distribution p(c, s). In practice will draw pairs (c, from (Jaa(z), Jug (x)) and (Jeg (y). fa where x and are respectively uniformly drawn from X and Y. Importantly. as in Sec. S.c.B.2, the information-less constraint still holds: (z, (x) = Vx. For simplicity, we choose Gaussian kemnels for Ke and Ks with constant standard deviation parameter т, which simplifies the estimation of the joint entropy into:
1
Nx+Ny
Nx+
1l0 Cj
||:
s,112
(c, Nx NY
log
exp
exp
(44)
Nx + NY
2T
2T
E CAPTURING INDEPENDENT ATTRIBUTES AND DISENTANGLE WITH CONTRASTIVE LEARNING
E. SUPERVISED DISENTANGLEMENT
We can also use our framework to derive a supervised disentangling loss with known variability factors. In this section, we propose to explore an extension of BC-InfoNCE in the case where independent fine-gn mme-grained attributes about the target dataset: fa= E R 1 are available. Given this set of independent observed characteristics, we can leverage these observations in a supervised manner to identify the independent factors of generation of the target dataset.
We assume the existence Ds attributes and construct our salient encoder so that outputs Ds latent dimensions. Our objective is to construct salient space where each salient latent dimension Sd only depends on its corresponding attribute a², Let us re-write Eq. by replacing the salient InfoMax term by each d-th attribute Supervised InfoMax term:
D.
arg max I I(yrc)) Ds d=1
Σ
s.t. DKL (8(s)) 0 and [(c, (45)
d-th SupInfoMax
From there, we take inspiration from Dutumier et al. Supervised InfoMax term supervised alignment and uniformity term:
2021a) to decompose each d-th attribute
Ny
|| 81ll2 + ((())) = Ca-th SupInfoMax
1(a4; a
Wo (a(a a²)
(46)
Ny
2r
We propose to develop the Entropy term for each d-th salient dimension as Sec.
F DATASETS AND IMPLEMENTATION DETAILS
F. DSPRITES WATERMARKED ON GRID OF DIGITS EXPERIMENT
To evaluate the Contrastive Analysis method enriched with target attributes, we provide a novel toy dataset. The background dataset X consists of MNIST digits (1-4) regularly placed grid.
20

---

Published as a conference paper at ICLR 2024
The target dataset Y consists of dSprites item added upon the foreground of this grid of digits. dSprites is dataset introduced to evaluate disentanglement. Its images are of size 64x64 pixels. Its elements only exhibit 5 generation factors, see Fig. B making it easy to evaluate the disentanglement. Possible variations are 1) shape (heart, ellipse, and square), 2) size, 3) position in X, 4) position in Y, and 5) orientation (i.e. rotation). To construct the Contrastive Analysis dataset use in this paper, we randomly sample MNIST images of digits I, 2, 3, and 4 and regularly place them on grid. We create 25,000 background images with this method. Then, we superimpose random dSprite element on 25,000 distinct digit grids to create 25 000 target images. We use the same method derive 5,000 test images equally balanced between the target and background classes. Importantly, we constrain the dSprites elements to have rotation attribute between -45 and +445 degrees. Downstream task performances are computed on the projection head.
Figure 3: Illustration of the dSprites dataset and its different independent variability factors: shape, z0om, rotation, Y position, and position.
F.2 MNIST DIGIT SUPERIMPOSED ON CIFAR-10 BACKGROUND
MNIST digit superimposed on CIFAR-10 background is a simple intuitive dataset inspired from Zou et al. (2013). We consider as the background dataset (y 0) CIFAR-10 images, and as the target dataset ( = 1) CIFAR-10 10 images (background) with an overlaid digit (target pattern), see This experiment is particularly suited to CA, expect our model to successfully capture the background variability (i.e: natural objects semantic) in the common space and to capture the digits variability in the salient space. In practice,
Fig. 4
used train of 50000 images (25000 Cifar- 10 images, 25000 Cifar-10 10 images with random MNIST digits overlaid) and an independent test set of 1000 images (500, 500). Images are of size 32 32. Pixels were normalized between and 1. In terms of Data Augmentation for the stochastic transformation process t(.), we remained close to SimCLR Chen et al. (2020), as we used RandomCropp(izz=4(4 24), scale=(0.2, 1.0)) augmen- tation, then a HaaddomHorzzonallPpppp-----
augmentation, KandommColorJiteer(.. 0.4, 0.4, 0.1) applied with a probability 0.8 followed from GranddonGGaSSSul(p-0.... augmentation.
Concerning the Neural Network architecture, both common and salient encoders were chosen as ResNet-15 18 with representation linear layer as follows: linear(512, 32) and a non-linear projector layer as follows: (linear(32, 128), batch norm( 128), reluÔ. linear 32)). We used Adam opti- mizer with learning rate 5e-4, batch size of 512, and trained during 250 epochs.
As for the SepCLR's syper--arametery we chose X = 1, Xs B 1000. and X 10. As related works, downstream task performances are computed before the projection head Chen et al.) (2020).
Concerning Contrastive Analysis VAE methods, took inspiration from experimental setups in Louiset et al. (2023). Namely, we used a standard encoder architecture composed 4 convolutions (channels 3, 32, 32, 32, 256), kernel size 4, and padding (1, 1, 0). Then, for each mean and
21

---

Published as a conference paper at ICLR 2024
6
Figure 4: the Superimposed MNIST digits on CIFAR background dataset. Target images are CIFAR- 10 images overlaid with MNIST digit. Background images are CIFAR-10 images.
standard deviations predicted (common and salient), we used two linear layers going from 256 to hidden size 256 to (common and salient) latent space size 32. The decoder was set symmetrically. We used the same architecture across all the CA- CA-VAFs concurrent works we evaluated. Interestingly, we also tried with ResNet-18 18 encoders but the results actually remained similar. The learning rate was set to 0.001 with an Adam optimizer. The models were trained during 250 epochs with batch size equal to 512. We used B. 0.5 and B, 0.5,
2, y = le 9, 0.025 For cVAE, we used 3e = 0.5 and 3g = 0.5, 6 = and x = 0 for con VAE. For MM-cVAE we used the same learning rate, Se 0.5 and 8g = 0.5, the background salient regularization weight 100, common regularization weight of 1000.
Concerning Mutual Information minimization methods, used the same hyper-parameters for k-JEM, except for A. X was set to 0.1 for CLUB, as in the original paper Domain Adaptation section Cheng et al. (020). Please note that we also tried values of 1 and 10, but it did not give better results. We also chose 0.1 for vUB and voluut. For TC, we used À = 10. For MMD, we used 50; motivate this choice in the Sec. G.I
F.3 CELEBA ACCESSORIES
In CelebA with accessories Weinberger et al. (2022), we consider a subset of CelebA [iu et al.] contains sets, target and background, from subset of CelebA Liu et al.) one with images of celebrities wearing glasses or hats (target) and the other with images of celebrities not wearing any of these accessories (background). Importantly, and contrarily MM-eVAE Weinberger et al. (2022) and Sep Louiset et al. (2023). we take care to balance the distribution of males and females in the background and the target dataset avoid gender bias with respect the accesories. We used train set of 20000 images, (10000 no accessories, 5000 glasses, 5000 hats) and an independent test set 4000 images (2000 no accessories, 1000 glasses, 1000 hats). Images are of size 128 128, normalized between 0 and 1.
In terms of Data Augmentation for the stochastic transformation process I(.), we remained close to SimCLR Chen al. (2020), as used RandomCCrop(siz==1123 128), scale=(0.2, 1.0)) augmen- tation, then a Randommorrzzontlllpppp0..333333 augmentation. a Random 0.4, 0.4, 0.1) applied wit probability 0.8 followed from KandomGrayySclee(pp)) Concerning the Neural Network architecture, both common and salient encoders were chosen as ResNet-18 with representation linear layer as follows: linear(512, 16) and a non-linear projector layer as follows: (linear(16, 128), batch norm( 128), reluO, linear(128, 16)). We used an Adam opti- mizer with learning rate 5e-4, batch size of 256, and trained it during 250 epochs.
augmentation.
As for the SepCLR's s hyper- parameters, we chose, as in MNIST superimposed on CIFAR-10 exper- iment, Ag 1, Às = á 1000, and X 10. As related works, downstream task performances are computed before the projection head Chen et al. (2020).
Concerning Contrastive Analysis VAE methods, we took inspiration from experimental setups in Louiset et al. (203)). Notably, we used images of size 64.64 pixels. Namely, we use standard encoder architecture composed of 5 convolutions (channels 3. 32, 32, 64, 128, 256), kernel size 4, stride 2, and padding (1, 1, 1, I, 1). Then, concerning the mean and standard deviations predicted (common and salient), we used two linear layers going from 256 to hidden size 32 to (common and salient) latent space size 16. The decoder was set symmetrically. We used the same architecture across all the CA- VAES concurrent works we evaluated. The learning rate was set to 0.001 with an Adam optimizer. The models were trained during 250 epochs with batch size equal to 512. We used
22

---

Published as a conference paper at ICLR 2024
3. 0.5 and 3g 0.5, K 2, 10, 0g 0.025. For cVAE, used Be 0.5 and 3, = 0.5, == 2 and k = for conVAE For MM-cVAE, we used the same learning rate, 3. = 0.5 and B, 0.5, the background salient regularization weight 100, common regularization weight of 1000.
Concerning Mutual Information minimization methods, used the same hyper-parameteer k-JEM, except for A. À was set to 0.1 for CLUB, as in the original paper Domain Adaptation section Cheng et al. (220).. Please note that we also tried values of 1 and 10, but it did give better results. We also chose 0.1 for vUB and vLlout. For TC, we used À = 10. For MMD, we used
for
50; we motivate this choice in the Sec. G.I
Figure 5: Celeba accessories dataset. The upper row consists of background images. The lower row shows target images.
F.4 CHEXPERT
In the CheXpert subtyping experiment, we select a subset of CheXpert separated in the background dataset: 10,000 healthy X-rays and the target dataset: 3,000 with edema, 3,000 with pleural effu- sion, and around 2,000 images with cardiomegaly. Images are resized to 224x224 pixels. Pixels are normalized between 0 and
For SepCLR, in terms of Data Augmentation for the stochastic transformation process ((.), we mained close to SimCLR Chen et al. (2020), as we used a RundomCrop(ssz==2224 224), scale=(0.2, 1.0)) augmentation, RandomColorlitteri from a RandonnGrayScale(p=00..2 augmentation, a paanddooRoaationddggeeeessee domHooiizonal/Pippp000.. augmentation.
0.4, 0.4, 0.1) applied with probability 0.8 followed
and then Ran-
Concerning the Neural Network architecture, both common and salient sizes are 32. Both common and salient encoders were chosen as pre-trained ResNet-18 with representation linear layer as fol- lows: linear(5 (51,, 12, 32) and a non-linear projector layer as follows: (linear(32, 128), batch norm( 128), relu(,, linear( 128, 32)). We used an Adam optimizer with learning rate of 5e-4, a batch size of 256, and trained it during 200 epochs. As for the SepCLR's 's hyper-parameterd we chose Àc 1, As related works, downstream task performances are computed before the projection head Chen et al. (2020).
As = 1, = 10, and
Concerning the Contrastive VAEs, we use the same common and salient encoders. For the decoders, chose architecture composed of linear layer, taking into input the concatenation of common and salient space, mapping it to a size 256. Then deconvolution layers were used with a kernel size of 4, stride of 2, and padding of 1 with filters (256 to 512, 256, 128, 64, 32, 16, 3). Output images are of size 256 x 256 and are cropped to 224 x 224. The final activation layer is chosen as a sigmoid layer.
We used the same architecture across all the CA-VAEs concurrent works we evaluated. The learning rate
0.001 with Adam optimizer. The models were trained during 200 epochs with batch size equal to 256. We used 3. = 0.5 and B, 0.5, и. = 2, 7 1e 9, σ, = 0.05. For cVAE, 2 and x for VAE. For MM-eVAE, used the same learning rate, 8. = 0.5 and 8a = 0.5, the background salient regularization weight 100, common regularization weight of 1000.
used 3g 0.5 and 3, 0.5,
F.5 ODIR (OCULAR DISEASE IMAGE RECOGNITION)
In the ODIR subtyping experiment, we select a subset of the ODIR dataset separated into back- ground and target dataset. Train dataset contains 1890 healthy images, 363 diabetes images,
23

---

Published as a conference paper at ICLR 2024
278 glaucoma images, 281 cataract images, 242 age-related macular degeneration images, and 227 pathological myopia images. On the other hand, TEST dataset contains respectively 210 healthy, 37 diabetes, 26 glaucoma, 39 cataract, 23 macular degeneration, 30 myopia images. Pixels are normal- ized between 0 and 1.
For SepCLR, in terms of Data Augmentation for the stochastic transformation process ((.), we remained close to SimCLR Chen et al.] (2020), as we used a RandomCrop(ssze=222 scalc=(0.75,, 1.0)) augmentation, RandommColorJiter(04. 0.4, 0.4, 0.1) applied wit probabil- ity 0.8 followed from a Candom@raayScalepp000.. augmentation, a RandomRorationdeerrss--- and then a Random Verticalllip(p=0.55 augmentation.
224),
Concerning the Neural Network architecture, both common and salient encoders were chosen as pre-trained R with a representation linear layer as follows: linear(511, 32) and a non-linear projector layer as follows: (linear(32, 128), batch norm( 28). reluO, finear(128, 32)). We used an Adam optimizer with a learning rate of 5c-4, a batch size of 256, and trained it during 200 epochs. As for the SepCLR's
we chose λ 1, Äs = 1, 3 100, and A 10. As related works, downstream task performances are computed before the projection head Chen et al. 2020).
Concerning the Contrastive VAEs, we use the same common and salient encoders. For the decoders, chose architecture composed of linear layer, taking into input the concatenation of common and salient space, mapping it to a size of 256. Then 7 deconvolution layers were used with a kernel size of stride of 2, and padding of 1 with filters (256 to 512, 256, 128, 64, 32, 16, 3). Output images are of size 256 x 256 and are cropped to 224 x 224. The final activation layer is chosen as a sigmoid layer.
We used the same architecture across all the CA-VAEs concurrent works we evaluated. The learning rate was set to 0.001 with an Adam optimizer. The models were trained during 200 epochs with batch size equal to 256. We used B. 0.5 and B, = 0.5, n. = 2, 1e 9, σ = 0.05. For cVA,, we used 3, 0.5 and 3. = 0.5, x = 2 and x for con VAF, For MM-eVAF, we used the same learning rate, &e = 0.5 and 8, = 0.5, the background salient regularization weight 100, common regularization weight of 1000.
F.6 SCHIZOPHRENIA EXPERIMENT
In this study, we analyzed neuroimaging data from several sources including the SCHIZCONNECT database (which includes 368 healthy controls and 275 patients with schizophrenia) and the BSNIP database (which includes 199 healthy controls and 190 patients with schizophrenia). The data used in this study was collected from various scanners and locations and included brain scans from in- dividuals in the United States. Images are of size 128 x 128 x 128 with voxels normalized on a Gaussian distribution per image. Experiments were run 5 times with a different train/val split (re- spectively 75% and 25% of the dataset) account for initialization and data uncertainty. Inspired by Louiset et al. (002). common and salient convolutional encoders were chosen as 3D-convolutions (channels 1, 32, 64, 128, 256, 512), kernel size 4, stride 2, and padding 1 followed by batch normal- ization layers. Then, we used a linear layer from 32768 to representations (sizes 128 for common set non-linear with hidden sizes 128 for
and 32 for salient). Then, the projection heads common and 32 for salient, with batch normalizationí 1 28) and relu() activation functions.
For SepCLR, the data augmentations were inspired from Dufumier al. (2021a), that is: horizon- tal flip with probability 0.5; blur with probability 0.5, sigma=(0.1, 0.1); noise with probability 0.5, sigma=(0.1, 0.1); CutOut with probability 0.5, patch size equal to 32x32x32, RandomCrop of size (96x96x96) with probability 0.5. The models were trained during 50 epochs with batch size equal to 32 with an Adam optimizer of learning rate of 0.0005. As for the SepCLR's 's hyper- papeee-ar,eeeess, we chose Ac = 1, Às = 1, 3 = 1, and À = 5. As related works, downstream task performances are computed before the projection head Chen et al. (2020). Importantly, the classification task computed with a layers MLPs in order to be comparable with Sep Louiset et al. (2023)
Concerning the Contrastive Analysis VAES methods we compared with, we use the same experi- mental setup in terms of hyper-parameter and architecture as in Louiset et al. (2023). Concerning the architecture, in details, the common and salient convolutional encoders were chosen as 3D-convolutions (channels 1, 32, 64, 128, 256, 512). kernel size 4, stride 2, and padding followed by batch normalization layers. Then, we used a non linear layer from 32768 to directly predict mean and standard deviations (sizes 256 for common and 256 for salient) with 2048 as hidden
24

---

Published as a conference paper at ICLR 2024
size with batch normalization and relu activation functions. The decoder symmetrically, except has 6 transposed convolutions (channels 512, 256, 128, 64, 32, 16, 1), kernel size 3, stride 2, and padding 1, followed by batch normalization layers.
F.7 MUTUAL INFORMATION MINIMIZATION METHODS
To compare with k-JEM (kemel-Joint Entropy Maximization), we used the implementation of sev- eral Mutual Information variational upper bound, namely vCLUB Cheng et al. (2020), vUB Alemi
(0200 and oul Poole et al. (2019) available at at / thub. com/ L.i nea r 95/CLUB)
tree/master Interestingly, these methods can be implemented with variational approximation S from C, vice-versa (C' from S), or symmetrically (mean of both). We tried all three possibilities with different weights and chose the best results each time
in Tab [ and Tab 2
We also compared with the exact Mutual Information estimator TC of Louiset et al. (2023) and Abid
& Zou (2019) inspired by the Total Correlation introduced in Kim & Mnih (2018).
In Sec. 3.3 we motivated the idea of minimizing the negative joint entropy (-H(C, S)) rather than implemented k-MI, version of k-JEM where we also minimize the entropies H(C) + H (S). To do so, we estimate (C)) in the common entropy estimation in Eq. [5 and (S) as in the salent entropy estimation in Eq. 29 Interestingly, we can see that k-MI indeed underperforms compared to k-JEM.
the Mutual Information (H) H(S) H(C,S), To prove our point,
G
SUPPLEMENTARY RESULTS
G.1 ON MUTUAL INFORMATION MINIMIZATION VERSUS TARGET AND BACKGROUND DISTRIBUTIONS MATCHING
In Contrastive Analysis, practitioners make use of various regularizations respect properties estab- lished priori. Recent works agree that background input should be mapped to a single information- less vector in the salient space. However, regularizations have been proposed in order to reduce the information leakage between the common and the salient space: match the distributions of targets and backgrounds in the common space, 2- minimize the mutual information between the common and salient distributions. our framework, the latter was naturally derived from the Info- Max principle. Tab aadd Tab 7 we propose compare both strategies on a) CelebA accessories and b) Digits superimposed on CIFAR-10 to assess their effect on the common space. In both ex- periments, we observe that the stronger the regularization is, the less common information (objects and sex) is captured. Also, we observe that k-JEM 's ability to diminish target-specific information (digits and accessories) remains relatively consistent across the regularization strength. Concerning MMD (Maximum Mean Discrepancy), high regularization strength is needed to reduce target- specific information despite its detrimental effect on capturing common patterns. We conclude that low-strength k-JEM regularization (we choose À = 10 in practice) is the right trade-off for captur- ing common patterns while canceling salient patterns.
Table Digits watermarked on CIFAR-10 (B-ACC). Comparison of k-JEM with MMD given different strengths.
DIGITS
OBJECTS
òror +
S
C4
$
C1
SEPCLR-NO K-JEM
95.6
94.4
9.0
42.0
145.8
SEPCLR-10 MMD
95.4
86.8
10.8
48.2
56.8
SEPCLR-50 MMD
94.6
21.2
9.0
62.2
134.0
SEPCLR-100 MMD
95.2
13.8
11.0
56.4
53.2
SEPCLR-10 K-JEM
96.2
11.0
10.4
73.2
32.0
SEPCLR-50 K-JEM
95.2
13.2
8.6
59.2
47.4
SEPCLR-100 K-JEM
95.0
12.0
9.2
52.4
53.8
BEST EXPECTED
100.0
10.0
10.0
100.0
0.0
25

---

Published as a conference paper at ICLR 2024
Table 7: CelebA accessories (B-ACC). Comparison of k-JEM with MMD given dif- ferent strengths.
HATS/GLSS
SEX
Ôror +
S
C
$
C1
SEPCLR-NO K-JEM
99.03
66.68
98.48
79.48
86.65
SEPCLR-10 MMD
98.99
81.53
60.87
76.19
87.14
SEPCLR-50 MMD
98.95
67.50
65.47
71.83
62.19
SEPCLR-100 MMD
99.03
53.25
67.12
52.51
68.83
SEPCLR-10 K-JEM
98.57
55.21
62.52
78.00
41.16
SEPCLR-3 K-JEM
98.83
58.27
62.45
68.38
53.51
SEPCLR-100 K-JEM
98.73
62.00
68.92
57.10
75.09
BEST EXPECTED
100.0
50.0
50.0
100.0
0.0
G.2 ON THE
RECONSTRUCTION TERM
Contrastive Analysis, jointly performed with generative process, enables performing salient or common characteristics swapping, salient attribute generation or deletion, and novel sample genera- tion. Therefore, investigated the addition of decoder jointly trained with the encoder parameters to reconstruct the input images (with a Mean Square Error Cost Function) during the optimization process. We added reconstruction term from the concatenation of the common and salient space (as in CA-VAEs but without the need for a -a-pammtterrzaiion trick) with the same decoder as in CA-VAEs, and degrades the results. Intriguingly, Tab. and Tab, (), which could be explained by the fact that the reconstruction task tries to conserve unnecessary noisy information in the latent space. However, an interesting perspective could be to include and train generator or decoder for generation and interpretability purposes, given frozen representations learned with SepCLR.
found that it tends degrade the results (see
Table 8: Digits watermarked on CIFAR-10 (B-ACC). On the impact of a reconstruc- tion term in addition SepCLR.
DIGITS
OBJECTS
S
C4
$
C+
SEPCLR-K-JEM SEPCLR- BEPCLR-K-JEMM WITH 0.1 REC
96.2
11.0
10.4
73.2
32.0
98.8
10.8
40.6
47.2
85.4
SEPCLR BEPCLR-K-JEM WITH REC
94.4
22.2
51.8
27.4
132.2
BEST EXPECTED
100.0
10.0
10.0
100.0
0.0
Table 9: CelebA accessories (B-ACC). On the impact of a reconstruction term in addition SepCLR.
HATS/GLSS S
SEX
OTOr 1
C
$
C
SEPCLR-K-JEN SEPCLR-K-JEM WITH 0.1 RBC
98.57
55.21
62.52
78.00
41.16
97.27
67.81
67.53
62.38
75.69
SPPCLR-KJJER WITH REC
91.51
68.87
62.77
64.39
78.98
BEST EXPECTED
100.0
50.0
50.0
100.0
0.0
G.3 ON THE COMPARISON WITH CONTRASTIVE METHODS
In this section, propose to compare SepCLR with self-supervised methods that not based on the moooerr-eccoe architecture. As there are no Contrastive Learning methods tailored for Con- trastive Analysis, propose design naive and simple strategy compare with SepCLR. First,
26

---

Published as a conference paper at ICLR 2024
infer the common features with the features of SimCLR trained the background dataset only (as it should have common features only). Then, we propose to infer the salient space with SupCon method trained to discriminate the background samples from the target samples. This way, such method should capture target-specific patterns while discarding common features. Additionally, we compare with SimCLR trained both datasets get reference point (even though, in that case, the common space and the salient space are the same unique space, which cannot perform the sep- aration of common and salient patterns). See Tab. 10 Tab. 11 and Tab. 12 for the results. SepCLR always performs better in terms of ōwt.
Table Comparison of SepCLR-K-JEM with Contrastive methods on Digits watermarked CIFAR-I (B-ACC)
DIGITS
OBJECTS
S
C4
$
C+
SIMCLR ON TG AND BG
44.0
44.0
94.6
94.6
180.0
SIMCLR + SUPCON
41.4
51.4
19.0
50.0
159.0
SEPCLR-K-JEM
96.2
11.0
10.4
73.2
32.0
BEST EXPECTED
100.0
10.0
10.0
100.0
0.0
Table 11: Comparison of SepCLR-k-JEM with Contrastive methods CelebA
accessories (B-ACC).
HATS/GLSS
SEX
ôuur
St
C1
S
C
SIMCLR ON BG AND TG
98.92
98.92
84.16
84.16
100.0
SIMCLR + SUPCON
97.93
82.15
59.98
80.76
63.44
SEPCLR-K-JEN
98.57
55.21
62.52
78.00
41.16
BEST EXPECTED
100.0
50.0
50.0
100.0
0.0
Table 12: Comparison of SepCLR-k-JEM with Contrastive methods on ODIR dataset (B-ACC).
SUBTYPE
SEX
òzor +
S
C4
S $4
C
SIMCLR ON BG AND TG
66.10 10
66.10
57.20
57.20
125.0
SIMCLR + SUPCON
68.70
57.17
51.94
58.41
107.0
SEPCLR-K-JER
68.54
47.71
52.48
59.62
97.03
BEST EXPECTED
100.0
25.0
50.0
100.0
0.0
G.4 ABLATION STUDY
In the main text, we investigated the effect of a null Mutual Information constraint by removing the proposed loss (No k-JEM) by minimizing the Mutual Information estimate (k-MI) rather than the Joint Entropy estimate (see Tab. and Tab. 2}. Here, we propose further ablation study in Tab. [3 and Tab. 14 We report the results of method when removing all proposed losses by one. We can observe that each loss is important since, when removing it, we either degrade the capture of salient patterns
fail disregard the common features in the salient space.
Table 13: Ablation Study on Digits watermarked on CIFAR-I0 (B-ACC).
DIGITS
OBJECTS
8mor +
S
C
S
C1
SEPCLR-X ( (NO K-JEM)
95.6
94.4
9.0
42.0
145.8
SEPCLR-P 3 0 (NO INFOLESS REG)
96.2
11.6
10.4
71.8
34.0
SEPCLR R-Às = (NO SALIENT TERM)
93.4
42.0
18.6
40.0
90.25
SEPCLR-AC 0 (NO COMMON TERM)
94.4
10.4
18.8
20.4
94.4
SEPCLR-K-JEM
96.2
11.0
10.4
73.2
32.0
BEST EXPECTED
100.0
10.0
10.0
100.0
0.0
27

---

Published as a conference paper at ICLR 2024
Table 14: Ablation Study on CelebA accessories (B-ACC).
HATS/GLSS
SEX
dror
S
C
$ $
C
SEPCLR - = O (NO K-JEM)
99.03
66.68
98.48
79.48
86.65
SEPCLR (NO INFOLESS REG)
99.12
53.88
68.82
77.29
46.29
SEPCLR Às = (NO SALIENT TERM)
77.50
87.73
53.30
77.55
85.98
SEPCLR Àc (NO COMMON TERM)
98.38
56.32
66.44
53.09
71.29
SEPCLR K-JEM
98.57
55.21
62.52
78.00
41.16
BEST EXPECTED
100.0
50.0
50.0
100.0
0.0
G.S PERFORMANCES ON THE BACKGROUND DATASETS
In the main evaluated method the ability linearly predict common attributes in the common space only and salient attributes in the salient space only on target samples only (as they are generated from both common and target-speeific factors of variability). In this section, we evaluate the ability to linearly predict common attributes only in the common space. In Tab, [5 and Tab [6 we can see that the common performances remain good on background samples while the salient space is non-in -on-informeeive as it is supposed to be. We can also notice that, compared to concurrent CA VAE methods, our method is still the best perffrerroning one in terms of 8, as predicts common attributes way better.
Table 15: Balanced Accuracy results Digits watermarked CIFAR-10 on background samples.
DIGITS
OBJECTS
δ +
S
C 4
$
C1
MM-CVAE
X
X
18.2
32.8
75.4
SEPVAE
X
X
20.0
34.4
75.6
SEPCLR- K-JEM
X
X
28.0
74.0
44.0
BEST EXPECTED
X
X
10.0
100.0
0.0
Table 16: Balanced Accuracy results samples.
CelebA accessories background
HATS/GLSS
SEX
ô +
S St
C+
S
C
MM-CVAE
X
X
64.27
70.48
43.79
SEPVAE
X
56.42
70.19
36.23
SEPCLR K-JEM
X
X
64.10
86.63
27.47
BEST EXPECTED
X
X
50.0
100.0
0.0
G.6 ON THE IMPACT OF LUNIF OR LLOG-SUM
As shown in Sec. A we estimate the entropy using resubstitution entropy estimator. This results in one of the terms of the standard Contrastive loss (i.e., InfoNCE) that accounts for the negative samples. As shown in Wang et Isola, this term has the same minimizer as the Lunar loss when the number of negatives tends to be infinite. We decided to use the Lunif loss instead of the Contrastive loss because is computationally less expensive, and it has been shown by Wang et Isola to lead to good representations and good downstream task performance. Furthermore, we have also compared the two losses in the CIFAR SICARIO-MNEL (0-MIIST dataset the CelebA accessories (see Tab, 17 and Tab, [8] and found that the results are slightly better or similar using Lunif-
G.7 ON THE IMPACT OF THE ENCODERS
In this section, we justify our choices terms of architecture. In Tab. 19 and Tab. 20 we show the performance of SepVAE and SepCLR Digits watermarked on CIFAR-10 and CelebA with
28

---

Published as a conference paper at ICLR 2024
Table 17: Balanced Accuracy results on Digits watermarked on CIFAR-10.
Comparison between Luiif and Llog-sum-exp estimate and minimize (C)) and (S).
DIGITS
OBJECTS
0
S
C4
S
C1
SEPCLR-K-JEN (LUnIF)
96.2
11.0
10.4
73.2
32.0
SEPCLR-K- K-JEM (LOG-SUM-EXP)
96.6
11.6
11.0
71.6
34.4
BEST EXPECTED
100.0
10.0
10.0
100.0
0.0
Table 18: Balanced Accuracy results on CelebA accessories.
Comparison between Luiit and Llog-sun-exp to estimate and minimize H(C) and (S).
DIGITS
OBJECTS
0
S
C
$4
C1
SEPCLR-K K-JEM (Cosn) SEPCLR-K. ----R--JJEE K-M (LOG- EUM-EP)
98.57
55.21
62.52
78.00
41.16
98.73
55.06
61.36
76.94
40.75
BEST EXPECTED
100.0
10.0
10.0
100.0
0.0
accessories with different architectures. SepVAE with ResNet-18 performs less better similarly than the one described our original paper. Conversely, SepCLR with ResNet-18 performs bet- ter. Overall, SepCLR remains largely better than SepVAE, consistent method across Contrastive Analysis VAEs.
Table 19: Results of several different encoder architectures Digits water- marked on CIFAR (B-ACC).
DIGITS
OBJECTS
dror +
S
C4
$4
Ct
SEPVAE
90.6
17.8
10.6
36.6
81,2
SEPVAE RESNET 18 ENCODER
90.8
23.2
10.2
34.0
88.24 24
SEPCLR WITH SEPVAE'S ENCODER
75.6
28.8
16.2
52.6
96.8
SEPCLR-K-JEE K-MMM
96.2
10.4
73.2
32.0
BEST EXPECTED
100.0
10.0
10.0
100.0
0.0
Table 20: Results of several different encoder architectures on CelebA acces- sories (B-ACC).
HATS/GLSS
ÔTOT
C
S
C1
SEPVAE SEPVAE WITH RESNET-18 18 ENCODER
84.46
65.19
60.12
59.20
81.65
86.13
67.47
60.04
61.93
81.45
SEPCLR K-JEM WITH SEPVAE'S ENCODER SEPCLR K-JEM
97.89
60.01
51.07
70.51
42.68
98.57
55.21
62.52
78.00
41.16
BEST EXPECTED
100.0
50.0
50.0
100.0
0.0
G.8 DSPRITES ELEMENT SUPERIMPOSED ON A DIGIT GRID
We show supplementary qualitative results on the salient space disentanglement in Fig. 6 We qual- itatively show that the common space captures background variability rather than foreground vari- ability in Fig. 7 We qualitatively show that the salient space captures foreground variability rather than background variability in Fig.
We also show quantitative results in Tab. 21 computing the Mutual Information Gap (MIG) score measure the disentanglement in the "Sprite-MNiST experiment. Results are reported below, and it can be noticed that the proposed method obtains good results (MIG is bounded by 0 and 1, where indicates a perfect result).
29

---

Published as a conference paper at ICLR 2024
shape
{
3
3
3
1 3 4
/
1
3
3
3
5
/ 3
3
3
2
2
4/
4
30
4
3
4
4
Jun8
2 4
3 2 4
3
3
1
4
3
3
3
I 3
3
2 4
2 7
2 4
2 4
4
2 4
2 7
3 4
2
4
3
3
3 2 4
/ 3 2
/ 3
3
3
5
3
3 4
2 4
>0 4/
2 4
+
3
f 3
6
3
3 2 W//
3
3
3 4
2
2
X
2 //
30 4/
vurical
3
3
/ 3 2
3
3 4/
3 4
3
2
2
2V 4
4
>
Figure 6: Attribute Supervised SepCLR on dSprites superimposed on a digits grid. Given an image, sampling of the nearest neighbor images in the latent space given small displacement given axis of the salient space. Each row represents the variation of only one element of the salient factor s while keeping fixed. We can see certain disentanglement: shape (line 1), zoom (line 2), orientation (line 3), X and Y position (lines 4 and 5).
1
3
3 x 4
1
1
3
3
3
\ 3 Y
1 3 4
/ 3
/
3
/ 3 2 4
( 2
3
2 4
4
4
_ 4
2 4
2 4
= 4
3 2 4
!
1 3
3
3
$ 3
3 _ 4
h 3 2 4
A 2 4
3
/ 3 2 4
/ 3 2 4
/ 2
3
4
3 4
4 4
2. 4
2 4
X 4
2
3 2 4
3 2 4
/
/ 3
/ 3
/ 3
/ +
/ 3
/ 3 e 4
/ 3
/ + 2 4
/ 3 2
/ 3 + 4
2 1
2
2 4
2
4
2
/ 3 2 #
A 3 4
6 3
/ 3
3 4
3 2
/ 3 2
3 2 4
/ 7 4
/ x
/ 3 7
/ 3 2
2
2 4
3
4
3
2 4
2
digits grid. Given random target images on the left, we sample the nearest neighbors in the dataset with respect to their L2 distance in the common space only. We can that the dSprite object remains the same while the MNIST digit grid in the background changes across the neighbors.
Figure 7: Attribute Supervised SepCLR on dSprites superimposed
/3
3
3
3
1 3 Z 4
3
3
3
3
f 3 2 %
/ 3
I 3 2 %
2 4
2 4
2 4
2 4
2 4
4
.2 4
2 4
2 4
/ 5
1 3
8 2 4
/ 48 2 4
3
3
3
3
/ 3 4
2 4
2 4
2 4
2 4
4
S 4
2
2
2
1 3 2
3
1 3 2 4
/ 3 4
/ 3
3
3
3
3
MB3
2 4
2
2 4
2 4
2 4
5 4
2 4
2. 4
4 2 4
4 3
I 2 4
4 4
/44 3 H
1 3 2 / 2 4
4 3
3
1 d
/
4 H
4 2 4
2 4
7
4
2
3
4
Figure 8: Attribute Supervised SepCLR on dSprites superimposed on a digits grid. Given random target images the left, sample the nearest neighbors in the dataset with respect their L2 distance in the salient space only, we can see that the dSprite object remains the same while the MNIST digits in the background change across the neighbors.
30

---

Published as a conference paper at ICLR 2024
Table 21: Computation of MIG on the salient space dSprites on MNIST digit grid experiment.
ZI (SHApE)
Z2 (ZooM)
Z3 (ROTATION)
Z4 (TRANS X)
Z5 (TRANS Y)
AVG
BEST EXPECTED
ATTR SUP SEPCLR K-JEM
0.915
0.909 0.003
0.674 0.007
0.823 0.007
0.835
0.831
RANDOM VECTOR
0.002
0.0008
0.003
G.9 QUALITATIVE RESULTS ON CELEBA WITH ACCESSORIES
In this section, propose display qualitative results the CelebA accessories dataset. In Fig. and Fig. T0 we propose to respectively display a 2D 1-SNE plot for the salient and common latent space of ScpCLR-k-JEM on the larget dataset (portraits of celebrities with accessories). Yellow points represent people with hats, Purple points represent people with glasses. We can clearly see that our method has correctly encoded the patterns related the accessories in the salient space and not in the common space.
Figure 9: 2D t-SNE plot of the salient space of SepCLR-k-JEM on CelebA with accessories. We highlight in yellow and purple the actual labeled subgroups (people with hats or with glasses), re- spectively. We can see that the two subgroups are clearly separated in the salient space. Furthermore, we train K-Means (K=2), which successfully identifies the two subgroups, and we propose to dis- play the 6 nearest images from both centroids. Interestingly, and people different genders but with the same accessories (hats cluster O and glasses in cluster 1). This clearly shows that our method has correctly encoded the patterns related to the accessories in the salient space and not the general ones (e.g., background, pose, gender, etc.).
observe various backgrounds, poses,
Cluster 0: Hats
Cluster 1: Glasses
31

---

Published as a conference paper at ICLR 2024
Figure 10: 2D t-SNE plot of the common space of SepCLR-k-JEM CelebA with accessories (target dataset). We highlight in yellow and purple the actual labeled subgroups (people with hats or with glasses), respectively. We can see that the two subgroups overlap in the common space. This clearly confirms that our method does not encode the patterns related to the accessories in the common space.
32


Reviews:
[{"review_id": "90CMGlIehS", "reviewer": "Reviewer_Btev", "summary": "The paper provides a novel concept of contrastive learning for separating common and silent patterns. The approach utilizes two encoders, one responsible for learning silent representation and one for common. The authors propose the criterion to train the model that is based on the InfoMax principle. Due to the fact the direct optimization of the criterion for the problem is difficult and general for the problem, they propose a set of assumptions that allow to training of the model directly using a gradient-based approach. The approach is evaluated using big number of use cases.", "questions": "I would like to ask the authors to respond to the weaknesses section. I will also would like to ask about selecting KDE as a model for this case. Can KDE be replaced with the normalizing flow instead?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation of the paper is good, and the work is clear and well-written. \n- The proposed method is very interesting and sounds good technically. The flow of the proposed solution seems to be accurate. The authors clearly formulate the problem and general criterion given by eq. 1. Further, they decompose each component and propose a well-justified form of the component for given encoders. \n- The experiments are well-motivated, and the results seem to confirm the hypothesis stated in this work. It is very beneficial that datasets are from a variety of domains, going beyond standard benchmarks.", "weaknesses": "- The model is designed only to model $p(c|\\cdot)$ and $p(s|\\cdot )$. It would be nice to see some approximation of the distribution over data $p(\\cdot|s,c)$. I think that the proposed architecture can be enriched with the decoder that models this probability. \n- I am not quite sure if setting the same architecture for the proposed and reference methods is a good approach. In my opinion, the best architecture for each individual method should be used in experiments. \n- Only VAE-based methods are used as reference approaches. VAEs are doing the additional jobs, they have decoders and serve as generative models, while SEPCLR is only learning the common and silent representation. It would be nice to see the comparison with the models from similar groups, either SEPCLR as VAE, or reference methods that do not preserve autoencoding properties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eFnE8afrSc", "reviewer": "Reviewer_fALd", "summary": "This paper presents a theoretically grounded approach based on contrastive learning, SepCLR, to separate salient features from common features given a weak supervision in the form of a target dataset that contains both salient and common features and a background dataset that contains only common features. A series of mutual information based objective functions and regularization terms are presented along with clear motivation and background behind each of the terms, and the approximate loss functions are derived to achieve these objectives. Experimental results on multiple vision and medical benchmarks demonstrate that a good separation of latents is achieved and the method out-performs prior work. Visualizations of retrievals support their experiments.", "questions": "1. Could you provide an ablation study on the various objectives and their impact?\n2. In Table 1, 2, what is the performance on the background dataset? Is the salient latent non-informative as it is supposed to be? Is the common term informative?\n3. Is the task switch from classification to regression affecting the proposed approach's effectiveness in Table 4?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper is well-organized and well-written. It covers the necessary background on contrastive analysis, provides the theoretical and intuitive motivations on their various objective functions and puts them in context with prior literature, provide the derivations, and also discuss their limitations well.\n\nThe main novelty lies in their information-theoritic formulation of the various objectives and more importantly, exploiting contrastive learning and other prior literature to make the various objectives tractable. Another novelty is that they proposed joint entropy maximization to prevent information leakage between the common and salient latents as opposed to mutual information minimization, as the latter strategy can lose information.\n\nResults in Table 1 and 2 show good latent separation on synthetic vision tasks, and that the proposed approach outperforms prior approaches. In addition, they also reveal that the proposed joint entropy maximization is better than other strategies to prevent information leakage. Table 3, 4, and 5 show similar results on the medical domain and the retrieval visualizations support their findings.", "weaknesses": "This paper presents several objectives, but it's unclear which objectives are important and how they work together. This makes it unclear for a practitioner to transfer the results to a different problem. So, I encourage the authors to ablate the different objectives. \n\nAnother weakness is that in Table .4, which is perhaps an important real-world application of the proposed approach, the improvements over prior work is not much. This is in contrast with other experiments i.e. Table 1, 2, 3, 5. The reasoning for this is unclear and also not provided.  \n\nNit: MMD abbreviates to maximum mean discrepancy and is referred in the paper as moment matching distance [1]. \n\n[1]: A Kernel Two-Sample Test https://jmlr.csail.mit.edu/papers/v13/gretton12a.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GEMagnvX1j", "reviewer": "Reviewer_etAa", "summary": "This paper proposes a technique called SepCLR that uses contrastive learning in order to learn representations that separate common elements from salient ones for the downstream task at hand. The authors examine the performance of their method in creating separate representations for these two elements, and demonstrate improvements over previous work on the same subject.", "questions": "I would be grateful if the authors could clarify the points I made above regarding the design decisions made for the method and the details of its formulation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method proposed by the authors is novel, as far as I am aware. Limiting the shared information between the common and the salient space explicitly is an interesting way to ensure that the two encoders model different aspects of the data, leading to less overlap in the information between the two encoders.\n\n- The authors have performed extensive experiments on a variety of datasets, and have also examined several different variations of their proposed method, as can be seen in Table 1. I also appreciate the fact that the datasets used are not only standard ones like MNIST or CIFAR-10, but also come from the medical domain (although I would also appreciate results involving more complicated datasets like CIFAR-100 or ImageNet, which have more object classes available).\n\n- I also appreciate the detailed analysis that the authors provide for the datasets used in the appendix.", "weaknesses": "- As far as I understand, there is an inherent limitation for the method in that knowing the labels for the target dataset is required during training. This limits the applicability of SepCLR in the unsupervised setting, which is also the one most commonly examined by contrastive learning works.\n\n- I believe that there are some issues with the proposed method, that I would be grateful if the authors could elaborate on:\n\n  - The authors make some decisions when designing the loss that go against what is commonly done in related contrastive learning papers. In particular, the loss they propose has the formulation of $L_{unif}$ as found in Wang & Isola [A], but the most commonly used formulation is that of InfoNCE, which differs in that the resulting loss is a sum of Log-Sum-Exp functions, instead of a single Log-Sum-Exp. Similarly, in the alignment term they use a formulation closer to $L_{out}$ from Supervised Contrastive Learning [B], but the same paper notes that another formulation that simply sums the inner products, named $L_{in}$, is better experimentally (the authors examine this in the appendix, but do not explain why they chose $L_{out}$). I would be grateful if the authors could elaborate on these design decisions.\n\n  - Related to the above, it seems that the alignment terms in the common space and in the salient space are different (and similar to $L_{out}$ and $L_{in}$ respectively). I would be glad if the authors could explain why this is the case.\n\n  - In Equation (7), the first term in the sums essentially forces the representations of the salient encoder to be far from the constant vector $s’$. It’s not immediately clear to me why this term is there - it doesn’t seem to arise from optimizing $\\hat{H}(S)$, and the informationless hypothesis only comes into play in Equation (8). I think the authors need to explain this part a bit more.\n\n  - Finally, the zero mutual information constraint is somewhat misleading - I understand the point the authors make that minimizing $I(c;s)$ is not the best thing to do, but at the same time, the proposed method does not directly force $I(c; s) = 0$. There is no guarantee that maximizing $H(c,s)$ does not affect the maximization of $H(c) + H(s)$, nor that the final solution will have $H(c,s) = H(c) + H(s)$. I believe that the authors should be clearer about this point.\n\n- I also believe that some points regarding the presentation of the paper can be improved:\n\n  - Tables 1 and 2 contain several variants of SepCLR, but it is not clear what each of them signify. The authors should better explain the variants of SepCLR in this table.\n\n  - Section 4 seems out of place, as it does not come up later in the main paper, and is in fact extremely similar to Section E in the Appendix. I believe that this part should be moved away from the main paper, as currently it throws the reader off (despite the paper having good structure overall).\n\n- Finally, I believe that it would be good to include the baseline of simply training the model using the entirety of the dataset via e.g. SimCLR. While I’m fairly sure that this will not perform as well, it’s still something good to include to get a sense of why the two different encoders are necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vSktyz8D92", "reviewer": "Reviewer_F95G", "summary": "The paper proposes a novel theoretical framework for Contrastive Analysis based on the InfoMax principle, leveraging Contrastive Learning to estimate the common and salient terms, and suggests a strategy to reduce the information leakage between the common and salient spaces. Specifically, the framework consists of two InfoMax terms for the common space and the salient space, and k-JEM for preventing information leakage. In addition, the authors propose a Supervised InfoMax term to disentangle the salient factors. \n\nThe key contributions are:\n\n1) Reformulating Contrastive Analysis under the InfoMax principle with two Mutual Information terms to maximize - one for common factors and one for salient factors unique to the target dataset. \n\n2) Leveraging Contrastive Learning losses to estimate these Mutual Information terms - retrieving InfoNCE for the common factors and proposing a new background-contrasting loss for the salient factors.\n\n3) Introducing a new strategy called k-JEM to maximize joint entropy for reducing information leakage between common and salient spaces.\n\n4) Extending the framework with a Supervised InfoMax term to disentangle salient factors when attributes are available. \n\nThe experimental results on 5 datasets show k-JEM outperforms other mutual information minimization techniques and significantly promotes separating salient factors and common factors.\n\nOverall, the proposed SepCLR framework and k-JEM regularization demonstrate strong empirical results for contrastive analysis on both visual and medical datasets.", "questions": "Here are some potential questions about the paper:\n\n1. How does Equation 7 constrain `s'` to be information-less? This equation seems to promote the embeddings to be uniformly distributed rather than constraining s' specifically. Some clarification on how the information-less hypothesis is enforced would be helpful. \n\n2. The alignment terms in Equations 4 and 6 look different - one uses a log summation inside the log, while the other does not. What is the reason for this difference in formulations between the common space alignment (Eq 4) and salient space alignment (Eq 6)? Some explanation or intuition here could help the reader understand.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Here are some strengths of the paper:\n\n1. The proposed theoretical framework provides new insights into Contrastive Analysis by formulating it under the InfoMax principle and identifying key mutual information terms to estimate. This enlightens future work on estimating these terms for contrastive analysis. \n\n2. The paper proposes a strategy to disentangle target-specific attributes within the salient space in a supervised manner when attributes are available. This extends the framework's capabilities.\n\n3. The paper provides an extensive discussion and comparison of several mutual information variational upper bound methods (vCLUB, vUB, vL1out, TC) as well as the strategies of mutual information minimization and distribution matching for reducing information leakage.\n\n4. The derivation of the InfoNCE loss and its alignment and uniformity terms from the InfoMax principle is clearly explained, connecting contrastive learning and information theory foundations.", "weaknesses": "1.  The choice of metrics could also be expanded and analyzed in more detail. For example, the reasoning behind expecting certain accuracy scores is not fully clear. Furthermore, it is not comparable between accuracy of 0% and 20% for (digits,C) on CIFAR-10. \n\n2. The evaluation is limited to a small set of datasets and tasks. A more comprehensive evaluation on a wider variety of datasets and downstream tasks could strengthen the results. There is limited discussion of hyperparameter sensitivity and scalability to larger datasets. Analyzing the impact of key hyperparameters and demonstrating scalability would be useful.\n\n3. In the disentanglement experiment (Figure 2), some entanglement seems to remain between factors. Using quantitative metrics like MIG and DCI could help analyze this. The sprite changes in Figure 2(b) could also be explained. \n\n4. Some architectural and mathematical details are unclear:\n- The encoder architectures and whether they are independent could be specified. \n- The notation for views v and number of samples Nx, Ny could be clarified.\n- Formulas and descriptions could be expanded for readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y8otIgKw3w", "reviewer": "Reviewer_4LjD", "summary": "Ths paper discusses Contrastive Analysis, a sub-field of Representation Learning that aims to distinguish common and salient factors of variation between healthy and diseased datasets. \nCurrent models based on Variational Auto-Encoders have shown poor performance in learning semantically expressive representations. \nIn contrast, Contrastive Representation Learning has shown significant advancements in various applications. \nThe proposed method, called Sep-CLR, leverages Contrastive Learning to acquire semantically expressive representations suitable for Contrastive Analysis by utilizing the InfoMax Principle and optimizing Mutual Information terms.\nThe paper provides both theoretical and experimental analysis.", "questions": "1. It would be better if higher-resolution images could be provided in Figure 1.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The theoretical analysis is reasonable and easy to follow.\n2. The proposed method outperforms baselines by a significant margin on several datasets.", "weaknesses": "1. The submission format of the paper should change to ICLR 2024. It is ICLR 2023 now.\n2. It will be better to give some qualitative results on not only the mnist dataset but also X-ray or other real-application data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)