Towards Principled Objectives for Contrastive Disentanglement
Authors: Anwesa Choudhuri
Ashok Vardhan Makkuva
Ranvir Rana
Sewoong Oh
Girish Chowdhary
Alexander Schwing
Venue:  (2020)
Keywords: Disentanglement
Contrastive

Abstract: Unsupervised learning is an important tool that has received a significant amount of attention for decades. Its goal is `unsupervised recovery,' i.e., extracting salient factors/properties  from unlabeled data. Because of the challenges in defining salient properties, recently, `contrastive disentanglement' has gained popularity to discover the additional variations that are enhanced in one dataset relative to another. %In fact, contrastive disentanglement and unsupervised recovery are often combined in that we seek additional variations that exhibit salient factors/properties. 
Existing formulations have devised a variety of losses for this task. However, all present day methods exhibit two major shortcomings: (1) encodings for data that does not exhibit salient factors is not pushed to carry no signal; and (2) introduced losses are often hard to estimate and require additional trainable parameters. We present a new formulation for contrastive disentanglement which avoids both shortcomings by carefully formulating a probabilistic model and by using non-parametric yet easily computable metrics. We show on four challenging datasets that the proposed approach is able to better disentangle salient factors.


Under review as a conference paper at ICLR 2020
TOWARDS PRINCIPLED OBJECTIVES TRASTIVE DISENTANGLEMENT
FOR CON-
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised learning is important tool that has received significant amount of attention for decades. Its goal is 'unsupervised recovery,' i.e., extracting salient factors/properties from unlabeled data. Because of the challenges defining salient properties, recently, 'contrastive disentanglement" has gained popularity discover the additional variations that are enhanced in one dataset relative to another. Existing formulations have devised variety of losses for this task. However, all present day methods exhibit two major shortcomings: (1) encodings for data that does not exhibit salient factors are not pushed to carry no signal; and (2) introduced losses are often hard to estimate and require additional trainable parameters. We present new formulation for contrastive disentanglement which avoids both shortcomings by carefully formulating a probabilistic model and by using non- non-nonnnnnnn non-aarmmttric yet easily computable metrics. We show four challenging datasets that the proposed approach is able to better disentangle salient factors.
INTRODUCTION
Unsupervised machine learning requires to extract latent information from unlabeled data points. For instance, given a dataset of digits we want data points to be grouped according to the depicted number, e.g., by clustering into a set of groups. Many algorithms have been developed and commonly their goal is 'unsupervised recovery,' i.e., to extract salient factors/properties from unlabeled data.
However, unsupervised recovery is inherently challenging because saliency of fac- hor/property/feature can be hard to define and in case of visual data, can depend on the viewpoint. Recent work (Abid et al., 2017; Severson et al., 2018: 8;; Ruiz al., 2019) has therefore advocated for 'contrastive disentanglement'. Given two sets of data without obvious correspondence between their members, discover the additional variations that are enhanced in one dataset relative to another. For instance, one dataset may depict grass while the other illustrates grass with digits overlaid. In this case the digits are the additional variations/eehaanemeenn contrastive setting is useful across many important applications in unsupervised learning, e.g., when comparing to control groups or when disentangling variations from data.
that need to be disentangled. This
For this contrastive setting a set of techniques have been proposed very recently to disentangle variations (Abid et al., 2017; Severson et al., 2018; Ruiz et al., 2019). Common to all those techniques the idea that observed data modeled the linear or non-linear combination of two transformed latent signals. For example, one latent signal enables reconstruction of one entangled dataset (e.g., grass dataset), while second latent signal models the 'additional variations' of the second dataset when compared to the first (e.g.. the digits). Furthermore, these methods introduce additional losses in order to better disentangle background factors from salient features. While those losses were shown to improve results, they exhibit two key shortcomings: (1) the formulation ignores information about the prior, i.e., the salient features of the background data should be zero; and (2) additional KL-divergence-basse losses which are hard to estimate in practice are introduced to improve disentanglement. However, reasonably accurate estimation of those losses requires additional trainable parameters, which increases the model size.
To fix those two shortcomings we instead propose two new losses for disentanglement. The first actively encourages the salient factors be zero if additional variations not available. We show that this loss originates from a careful probabilistic derivation of the disentanglement setting. The second par-mmratt loss encourages the background distributions to be identical. Different from

---

Under review as a conference paper at ICLR 2020
5
Lws
5
x
X
Enc
Dec
LVAE
y
Zy
Vice
Le
Our VAE framework to achieve contrastive disentanglement. We propose the two new losses Lw₃ and Le to improve disentanglement. Clustering in z space has overlapping clusters because the common factors are encouraged be distributionally similar. Clustering in the space has clearly distinet clusters representing target, background, and intra-targes classes.
Figure 1:
existing methods we propose to use a non-parametric loss which easy to compute and does not require additional trainable parameters.
We demonstrate our contributed loss functions on a variety of datasets as highlighted in Table 1. We validate through a set of challenging qualitative and quantitative benchmarks that our first new loss gives rise to a pair of disentangled latent features. Further, we empirically establish that our second new loss encourages the common factors of the target and background data to have identical distributions, while baseline losses do not exhibit a similar behavior.
BACKGROUND
In this section, we formally state the contrastive disentanglement objective and recall existing approaches address as well as their shortcomings. Let )= and p be two unlabelled datasets with each (1) () Rd. We refer to (ax(t) } as the target dataset, whereas ()0 is the background dataset. We assume that the samples (x)() are drawn i.i.d. from target distribution (z)),, whereas { are drawn i..d. from a background distribution (3)). For example, in the Grassy- -NISS dataset (detailed in Section 4), the target dataset (0) is the set of images with MNIST digits superimposing image patches showing grass, and the background set ( is comprised of images showing only grass. The target dataset exhibits features that have commonality with the features of the background data, e.g., grass, in addition to some unique interesting features that are specific only to the target data, e.g., digits. Thus the primary objective in the contrastive disentanglement setting is the following:
Goal: Contrastive disentanglement aims to extract a pair of disentangled latent features (s, 2) such that the salient feature s encodes the generative factors of interest, e.g., digits, that are enhanced in (x(*) } relative to ( }, whereas the background feature explains the common sources of variation these two datasets, e.g., grass background.
A key aspect of this setting is that the salient feature s is constant across the background dataset { and is enhanced only in the target set (x(1)). However, apart from the unlabelled datasets of images, no additional information about these generative factors of interest, such as the background- target pairs nor target images with a specific salient feature are provided. Thus the key challenge is to exploit this implicit information inherent to the structure of the two datasets to learn disentangled features. We now review some relevant approaches to learn these disentangled factors.
VAF. Given training samples generated from a single unknown distribution p(xr), variational auto- encoder (VAE) models the joint distribution of the observation and the latent space through parametric deep nets (æ, 2). Hereby z is the lower dimensional latent vector and æ is the high dimensional observation, such as image. The joint distribution factorize into pa (x. z) p( p(z)pe (2))(x)2)) z)) where the prior p(z) is assumed to be a standard Gaussian and the conditional pe [x) 2) is assumed to be parameterized by deep net referred to the 'decoder.' Since the posterior pe x) is hard to compute, VAES approximate this posterior with a parameterized deep net qo (z (z), called the "encoder." In particular, (=(2(z)) (z) too is assumed to be factored Gaussian with mean and diagonal
2

---

Under review as a conference paper at ICLR 2020
p0(222,8)
Po |/
(a) Target latent model
(b) Background latent model
Figure 2: Latent variable model for target and background. Shaded circles represent observed variables.
covariance determined by the encoder deep net. For training, the encoder and decoder parameters Φ and 0 are learnt by minimizing the objective function L(O, Φ), given by
T, Φ) = p())) [KI.. (go (=| || p(=)) - T (x|a²) log po (a||=)]
(1)
where L(O, Φ) serves as an upper bound the negative log-likelihood of the data.
This classical VAE framework for a single dataset can be extended to the contrastive setting which involves multiple datasets (Ruiz et al., 2019 Abid & Zou, 2019). The main idea is that the samples generated from the two unknown distributions, i.c., the target distribution (z()) and the background distribution ps (2()) ). can be captured by a pair of independent generative latent factors (z, s). Here is thought of as a 'common' factor that captures variations in samples from both the distributions, whereas s is the 'salient' factor that encodes generative factors of interest that are unique only to the target samples. Thus the generative model for the target dataset is modeled by a conditional Gaussian distribution, i.e., D; ~ pe [x) Zi, 8.), where denotes the set of decoder parameters. Since the common factor alone encodes the background dataset, model it as Y Po z 8j = =0), where we set the constant salient factor to zero, without loss of generality. Further, similar to the VAE setup above, approximate the posteriors over the latent features by factored conditional Gaussian distribution, i.e., (zi, 8:) Jat; do 1=022/2 az; (s(c)) and (zj, 82) [y ~ qos (z W )o (* (8(y)). where & denotes the set of encoder parameters which capture and the mean and diagonal variances of these approximate posterior distributions. Note that both datasets share the same encoder and decoder.
Another key component of the generative model is the latent prior. Since both of the indepen- dent latent factors z and s encode the target samples, we define the target latent prior pr(z, 8) as (z(z.s) s) _ p. (z)p; (s) (z 0, La.) (s; 0, Ia.), where (zz 0. Ia.) denotes standard Gaussian distribution dz-dimensional Fuclidean space, and (s; 0, Id, ) defined similarly. Fur- thermore, to accommodate the special structure of the contrastive disentanglement setting, i.e., the background { is captured by z alone with s being a constant, we define the background latent prior Po (z(z, (2,5) 8) as
po (z s) A p)(z)p((s) (z; 81s 0),
(2)
where 0{s = 0} denotes the Dirac distribution centred zero. Figure illustrates this latent variable model. Using this latent variable model setup, Ruiz et al. (2019) and Abid & Zou (2019) derive the following loss LVAE as an upper bound for the sum of negative log-likeline likeelioloood of both the datasets:
LVAE = (p,(,) [KLL (2(()))) (x) (s)) Eg ((le))) log pa (x (r|z, s]]+
(3)
(p.(w) [Dtt (go II (T))) Ega(= log z, 8 = 0)].
This classical VAE loss LVAE defined above is the basis for eVAE and sRbVAE which we discuss below. However, we already want to highlight an important observation regarding the loss in Eq. (3): by using the above latent variable model for the contrastive setup and plugging them in Eq. (1), the actual combined negative log likelihodo is bounded by a loss that contains a crucial KL divergence term between the posterior 4 (sly) and the Dirac latent prior pb (s) ò{s 0} in addition to the LVAE defined above. We elaborate on this further in Section 3.
eVAE. In a recent work, Abid & Zou (2019) introduced the Contrastive Variational Auto Encoder (cVAE) for learning disentangled latent features. In particular, they consider the same latent variable model setup above. In addition, to further encourage the disentanglement between the latent factors z and 8, they add the following total correlation (TC) term to above objective in Eq. (3):
LTC = Ep(x) [KL. (ğ || g (z(xx)) (z)=))g((())) (s(x)))
where q A (2, s(x)) denotes the joint conditional probability of the latent features. The overall objective for the e eVAE framework thus given by:
min LOVAE A LVAE A LTC. (0.0)
(4)
3

---

Under review as a conference paper at ICLR 2020
where À > is hyperparameter.
BRb-VAE. Recently, Ruiz et al. (2019) introduced the Symmetric Reference- based Variational Auto Encoder (sRb- for learning the disentangled features (s, z). Utilizing the fact that minimizing the loss LVAE in Eq. (3) is equivalent to minimizing the KL divergence over the joint distributions (g (z, 2, 8, c) || (ar, 2, 8, c)), where E {0, }} indicates if æ is either from the background or target dataset, the authors add an additional KL divergence term to make the overall objective symmetric, i.e.,
min LARD-VAE & LVAE + DKL 2, 8, || (0.0)
2, 8,,
(5)
Note that the distribution go 2, 8, is defined as go 8, - p(c)p(zlc)uo (z)
where p(c = 0) - p(c = 1) = 1/2 and p(øle= = (x)). and p( 0) - (x) denotes the larget and background distributions respectively. Similarly, po(x, s, e) _ p(c)p(z, s slc) z, s). where p(z, s/c = 1) = (z 8) and p(z, s|c- = 0) = pb(2, 8) denote the target and background latent priors.
Since both vAE and ERb-VAE build upon the VAE framework, their losses involve the same LVAE defined in Eq. (3). However, this VAE loss is not completely reflective of the assumptions about the background latent prior ph (z, 8) defined in Eq. (2). In particular, due to the absence of the KL divergence term DKL (9)(l)) || 8(s
Shortcomings:
0}) in Eq. (3), the existing objective LVAE does not enforce the salient latent feature s to be zero for the background dataset, which is contrary to our modeling assumptions. Intuitively, by more strictly encouraging the salient latent feature to be zero, we will prevent information from being encoded in this vector. In addition, to encouraging disentanglement, both aforementioned approaches add new KL divergence based losses to the standard objective. However these new divergence terms involve estimation of ratios of densities which are quite hard to approximate. Hence they utilize the classical trick of discriminator neural nets (Kim & Mnih, 2018). This makes the overall training procedure more complex since this discriminator should upproximate this ratio at each training step of the encoder parameters (0, Φ). This also increases the number of trainable parameters together with an added difficulty of finding the right discriminator architecture to estimate this ratio. In contrast, in this paper, we provide a simple and principled approach to promote disentanglement that performs at least as good as both of these methods on a variety of datasets (see Section 4). In addition, our architectural complexity is identical to classical VAE as opposed to the complex architectures of existing approaches.
APPROACH
In this section, we address the aforementioned shortcomings and present our novel approach towards the desired goal of contrastive disentanglement. In particular, we introduce two novel loss terms that are reflective of the structure inherent to the contrastive setting and that help attain disentangled representations. Whereas the first loss term enforces the information about the common background features to be encoded in z alone, the second loss term encourages the distribution of the common factors be the same across both the datasets. Figure 1 illustrates approach. We now diseuss details about each of these loss terms below.
3.1 CORRECT VAE LOSS
To gain intuition for our approach, recall the central aim of contrastive disentanglement: (1) encode information about the common features within both datasets in the common factor alone; and (2) encode salient features of interest, that are unique to the target samples, in just s. While this is incorporated in the background and target latent priors, the classical VAE loss LVAe given in Eq, (3), and used by Abid & Zou (2019); Ruiz et al. (2019), does not capitalize on this. To see this, note that while the reconstruction error term (xlg) log Po Po 2, $ = 0) enforces the decoder to reconstruct the background image y from just z with s being zero, the KL divergence loss (x by) (z))) does not enforce the information about the background to be encoded in only z since it does not penalize any leakage of the background features into the salient feature s. To address this issue in a principled manner, we propose to add
loss term that exploits the background latent prior from Eq. (2) and that achieves our desired goal of disentangled representations.
We now derive our new loss mathematically. Denoting the respective negative log-likelihoods of the background and target data L(po(y)) A Epa(w)l- (o()) and L(po(=)) e
4

---

Under review as a conference paper at ICLR 2020
Bm((a) log po(pe)))] (on [(po(x)) = E (() (m) (s)))))) p (x())):(s))
obtain from the latent variable model setup in Section 2 that
(ala)) Po s)],
and
((po(y)) - Ep(c) [ (s))) p=(=))6(s)) - (x w)as(ale)) Po 0)]
Eptwi (2 (s 0(s = 0})]
+ B ( (=1y) pb(2)) - (= Pog
=Eprty) (1,,)) (1a. (3) o. )
+ Bpv)) [DKL(g po())) g. log (3) (z) 0)]
where the last equality follows from Lemma 1, and lc(x) 0 if x = and oo otherwise, for some fixed constant c, and (H))a(y) (y)) denotes the onean-variang output pair of the encoder for salient feature 8. Thus adding the bounds for both the target and background distributions, we obtain
(x)) + (p)) & ( ((H Mo We now define the upper bound in Eq. (6) as our new VAE loss, i.e.,
(o (y))) + LVAE - Lo + LVAE.
(6)
TA
Eps(s)) [1 ((o.0); ((Ho.s (1). ( (v))) + TYAE-
(7)
Note that our indicator term above strictly enforces the salient feature to be zero for the background dataset. This directly reflects the problem structure of contrastive disentanglement. In particular, this constrains the encoder to encode the background information in the common factor z alone, encouraging contrastive disentanglement. We empirically validate this claim in Section 4.1 and
show that our new loss LYAE outperforms the existing approaches on variety of datasets and benchmarks. Since the indicator loss is non-differentiable, for the sake of implementational ease, we approximate it by quadratic loss B E||l,, (1))l² 20 + (l. ( In fact, this loss term is motivated by the following fact (which follows from Lemma 2) that
W2 (sly), ofs 0}) = (1))l + (()),=
where (., denotes the second-order Wasserstein squared distance. Hence in view of the above equation, our Wasserstein distance based quadratic regularizer acts differentiable surrogate to the non-differentiable KL divergence term (which is exactly the indicator term) Eq. (7) above.
3.2 WASSERSTEIN LOSS FOR DISTRIBUTIONAL SIMILARITY
In the above section, discussed why the new loss L LVAE is able achieve better disentangled representations than current state-of-the-art approaches. In addition to obtaining a disentangled set of latent features (z, s), an equally important and additional core goal of contrastive disentanglement is that the distribution of the common factor z remain identical across both datasets. However, experiments in Section 4.2 indicate that our proposed (ours) loss as well losses of existing frameworks do not meet this criterion. While the assumption of the same latent prior for the common factor across both datasets, i.e., (2(2) (r()), and sharing of the encoder architecture for both datasets implicitly enforce this desired requirement, empirical evidence suggests that current losses do (ours) not strictly enforce it. This motivates to define a new loss LA,E VAF that addresses this distributional mismatch via
(ours) L LW VAB L (ours) LVA
Xx ((2)).)())
X 300.
where (=) (1/B) EI 0( 0(== 2 denotes the empirical marginal distribution of the com- mon factors 2(1) for the target data, (z) is defined similarly for the background data, B is the batch size, and >х= >> is hyperparameter. Here W2(,,) denotes the second-order squared Wasserstein distance, defined as
inf (Ø) A W3 (z), (o.(()) Ten(B]IBB
ΣT. 1j
C2 -1= - 2
where [B] (1...., B}. B). [B] denotes all joint probability distributions on [B] x [B] whose first and second marginals are both uniform distributions on [B], i.e., I[[B), [B]) {T RBxB
BxB
T1 2 = B))) T 1B (1/B) 1 b), and 1 a is the vector of all ones with size B.
5

---

Under review as a conference paper at ICLR 2020
Target Clustering
Lw2
Dataset
Method
2r
Sx
(Lower
(Lower CA(%)
better)
(Higher is better) CA(%)
is
SS
SS
better)
eVAE
12.59
0.18
89.02
0.46
8.91
sRb- VAE
64.94
0.40
65.14
0.43
12.34
MNIST
Ours (Eq. (7))
12.43
0.19
91.23
0.52
8.88
Ours (Eq. (8))
11.91
0.04
92.17
0.55
4.60
cAE
26.61
0.27
26.89
0.28
9.04
ARb-VAE
22.60
0.16
23.42
0.16
38.12
NL MNIST
Ours (Eq. (7))
16.36
0.13
29.44
0.12
9.02
Ours (Eq. (8))
17.56
0.19
32.37
0.16
4.69
cVAE
52.06
0.35
60.01
0.39
38.26
sRb- VAE
55.12
0.34
56.17
0.36
43.87
CelebA
Ours (Eq. (7))
51.53
0.31
59.97
0.38
35.51
Ours (Eq. (8))
55.05
0.34
64.55
0.42
11.07
cVA
19.02
0.17
17.75
0.15
35.76
sRb- VAE
18.31
0.15
19.85
0.15
41.34
Affectnet
Ours (Eq. (7))
18.16
0.14
19.06
0.16
30.04
Ours (Eq. (8))
16.06
0.14
22.66
0.18
10.11
Table 1: Comparison of our methods baselines cVAE (Eq. (4)) and sRb-VAB (Eq. (5)) evaluated on metrics CA (clustering accuracy (%)) and SS (silhouette score). Lower CA and SS on z are better, meaning that the common latent vectors are indistinguishable. Ideal CA using x are 10% on L MNIST and NL MNIST, 50% CelebA, and 11.11% on Affectnet. Lower Lw, is better, which means that 2, and žy are distributionally similar.
Intuitively, this Wasserstein loss term Lw, gauges similarity of the common factors of both datasets and penalizes any distributional mismatch. This loss also ensures that the relevant target signal is encoded solely in the salient latent feature Indeed, in Section 4.2 we empirically verify this claim (ours) and establish that across a variety of datasets our new loss I VAE. outpeforms existing approaches.
This Wasserstein loss has two major advantages over the KL divergence loss terms used by previous works: (1) both the loss well as its gradients be efficiently computed; and (2) this new loss does not require any additional trainable parameters whereas estimation of the KL divergence often requires discriminator net, since its direct estimation from samples is hard to compute (Kim & Mnih, 2018). To compute the gradients for Lw,, since the objective linear in Eq. (9), it follows from Danskin's theorem (Danskin, 2012) that Volw, Ei, T7, V VaCij where 1" is the optimal probability matrix in Eq. (9). We compute the optimal matrix T by using the Python Optimal Transport library (Flamary & Courty, 2017) which uses an efficient linear programming solver from Bonneel et al. (2011).
EXPERIMENTS
In this section, we empirically show that our novel losses T.(ors)) and T.(our)) T.,,VAE- Eq. (8) respectively, perform better than the current state-of-the-arr methods as highlighted in Table 1. In particular, in Section 4.1, we validate through set of qualitative and quantitative benchmarks that the correct VAE loss L gives rise a pair of disentangled latent features as opposed to the existing approaches (also highlighted in Figure 5). Further, in Section 4.2, empirically establish that L loss constrains the target and background common factors to have identical distributions, while the previous losses do not exhibit similar behavior. For the sake of brevity, this section, we refer (ours) as 'Ours (Eq. (7))' and [ours) VAE as 'Ours (Eq. (8))'.
VAE: defined Eq. (7) and variety of datasets
Datasets
Linear and NonLinear Grassy MNIST. We construct synthetic datasets with signal and back- ground images linearly and non-linearly combined respectively. The Linear Grassy MNIST dataset (L-MNIST) constructed following Abid & Zou (2019). The background category ()) consists

---

control the non-linearity of this dataset by changing the threshold value. Even Under review as a conference paper at ICLR 2020
L MNIST
NL MNIST
CelebA
Affectnet
Ours (Eq. (8)) Figure 4: Unsupervised recovery of salient features on the Linear Grassy MNIST dataset. Teal circles
sRbVAE
cVAE
Ours (Eq. (7))
represent s component of grass. Other smaller clusters with multiple colors represent digits 9.
of randomly cropped 28 x 28 sized image patches from the grass category of ImageNet (Deng et al., 2009). To construct the target set {xx0}} we choose images of digits from 0 to 9 from the MNIST dataset (LeCun al., 1998) and linearly combine them with random grass images from the background. To construct the non-linear Grassy MNIST dataset (NL, MNIST), we use a thresholding operation before superimposing the MNIST digits with background patches y i.c., (:()) [u, v, c] = I[y [u, v,c] c] > 0.3) (u, v, c] + (1 p) [u, ul). where p = 0.5, (6) denotes the ,th pixel of the MNIST digit image t, and
if y (+) [u, v, > 0.3 and 0 otherwise. Here u and denote the respective row and column pixel indices of the image and e is the channel index. We observe NL-MNIST to be a very challenging dataset on which the state of the art methods perform poorly. Clean class labels are available for evaluation, unlike other non-linear and challenging datasets like the CelebA (Liu et al., 2018) () and Affectnet (Ruiz et al., 2019) data, which can often have noisy labels. Besides,
though the target samples (x) are generated using samples from ())), we do not have access to these pairs (x))), ()) during training. In Figure we show samples from both datasets.
CelebA. We create the target and the background sets using the CelebA dataset (Liu et al., 2018) as described by Abid & Zou (2019). To form the target dataset, we use subset of the CelebA (Liu et al., 2018), namely, faces with caps and eyeglasses. The remaining categories are shuffled and samples randomly picked as background images.
AffectNet. We use this challenging dataset for disentanglement of facial expressions. To form the faces with neutral expressions described by Ruiz et al. (2019). The target set is constructed with faces expressing some emotions, namely, happiness, sadness, surprise, fear, disgust, anger and contempt. These expressions become the salient features.
background,
Evaluation Metrics
Clustering Accuracy and Silhouette Score. To empirically evaluate that the learned salient features 8x of the target dataset correspond to the ground truth class labels, we use the silhouette score (SS) and clustering accuracy as our metrics. For both the metrics, first we perform t-SNE (Maaten & Hinton, 2008) on the target salient features to embed them in a two dimensional latent space and cluster them using the standard k-means. The clustering accuracy computes the proportion of the samples whose class labels align with the ground-tre -trut labels. The SS metric gauges the similarity between the clustering labels and the ground-trul truth class labels by comparing the intra-cluster distances to inter-cluster distances. The SS score takes values between - to 1. The higher the score, the stronger the correspondence between the cluster labels and the class labels. Furthermore, to verify that the common factors 2₂ of the target dataset don't 't encode salient features, we cluster the common factors using the same procedure as above and empirically validate whether the clustering accuracy and the silhouette score are close to that of random label assignment. For example, for Grassy MNIST, allotment of random labels (MNIST digit values) would achieve 1 10% accuracy.
Wasserstein loss between common factors. We use the Wasserstein loss Lw, defined in Eq. (9) to measure the similarity between the empirical distributions of the target common factors and the background common factors respectively. The closer it is to zero, the larger the similarity.
7

---

Under review as a conference paper at ICLR 2020
9
8
2
9
8
2
9
8
2
9
8
2 2
9
8
2
9
8
2
9
8
2
9
8
0
0
9
4
9
8
2
7
Z
9
8 2
9 8
2
cVAE
sRbVAE
Ours (Eq. (7))
Ours (Eq. (8))
Figure 5: Disentanglement of different methods. Rows from top to bottom represent target images
from the L-MNIST dataset, decoded images from passing (2, (z, 0), and (0, 8) through the decoder respectively. Note that the latent vector (z, 8) is obtained after the forward pass of the target image through the encoder.
ARb--AA
92.25
40
(So.
91.75
25
30
Accuracy
91.50
Waszerstein
25
91.00
15-
90.50
10
90.25
60
50
100
120
140
160
180
200
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Training Epochs
(a)
(b)
Figure 6: (a) Lwa loss across training. Ours (Eq. (8)) optimizes over Lw; (red plot) and hence has a very low Lw, value after 200 epochs. (b) Clustering accuracies over X influencing Lw, for Linear MNIST. X₂ achieves the best value across multiple trials.
Affectnet metric: Classification. In addition, we also use classification accuracy to gauge the correspondence between the salient latent feature and the ground-truth class label. particular, train a shallow classifier with the latent feature and the respective class label as the input-output training samples and report the classification accuracy of this trained model.
4.1 CORRECT VAE LOSS
This section empirically demonstrates that our correct VAE loss (ours) significantly improves disentanglement over that of cVAE and VAE. Quantitatively, in Table 1, we conduct thorough comparison of our approach to prior work across all the aforementioned datasets. We observe that the correct VAE loss is able consistently obtain better disentangled features as reflected in the clustering accuracies of both the salient features and the common features Zz. Qualitatively, in Figure 4, for the L-MNIST dataset, the salient features of the target and background samples, i.e., 8 and 8y. are embedded in two dimensional latent space using tSNE and then clustered using k-means. Note that s, is expected to be close to zero, whereas s is supposed to encode the information about the MNIST digits. As shown in Figure 4, clusters obtained through our loss well separated compared to that both e eVAE and VAE. Indeed, we achieve clustering accuracy of 91% on the L--NIST dataset over all the 10 digit classes, while cVAE and sRb-VAL attain accuracies of 89% and 65% respectively. Figure further illustrates that the target common factors zn and the salient factors 8z. obtained with eVAE and sRb-VAL VAE, contain information about the salient features and the background features respectively. This is contrary to the goal of contrastive disentanglement. Instead, our reconstructed samples show no leakage of information.
8

---

Under review as a conference paper at ICLR 2020
4
7
C
4
7
3
L MNIST
NL MNIST
CelebA
Affectnet
Figure 7: Reconstruction results with Ours (Eq. (8)). The rows from top represent the background images from the dataset, reconstructed background, target images from the dataset, and reconstructed target images respectively.
5 S
6
5
6
6
6
S
S
S S
S
S
S
S
5
5
5
5
5
5
5
5
5
5
3
3
5
5
3
3
3
3
3
3
3
3
8
3
3
3
<
CelebA Figure 8: Interpolation results. Interpolating from 81 82 along the diagonal, keeping ø1 fixed. Top
L MNIST
left: [2, $1] decoded. Bottom Right [21, 82] decoded.
4.2 Tw, Loss
In the earlier section, we empirically demonstrated that the correct VAE loss is able to obtain better disentangled representations than the baselines. However, as highlighted in Figure 6, all these losses do not enforce the distributions of the common factors across the target and background datasets to be identical. This illustrated by a high Wasserstein loss (also shown in Table 1). Instead, in the
presence of our new loss I. VAZ.VAE> we observe that the corresponding loss values reduce considerably. This is highlighted in Figure 6 for the L-MNIST dataset and in Table I across all the datasets. In addition, we also observe in Table that encouraging the distribution of the common factors to (ours) be identical across both datasets via T. /W,, VAE further enhances the disentanglement between the common and the salient features. This is shown via the higher clustering accuracies compared to baselines. Intuitively, enforcing the common factors have the same distribution implicitly constrains the unique features the target dataset to be encoded the salient features alone. This phenomenon also illustrated in Figure 5. We found the hyperparameter choice A= datasets. Figure (right) shows the clustering accuracies of digits for L-MNIST as we vary X
[ours)
perform best across all
reconstruction results of the background and target images different datasets, obtained from the approach described in this section. Figure 8 shows the results obtained by interpolating 2 classes of the target salient factors in 2 dimensions. This is performed the L MNIST and CelebA datasets.
Figure 7 shows
RELATED WORK
Unsupervised contrastive analysis has been proposed very recently in work by Abid et al. (2017) and Severson et al. (2018). Inspired by the principal component analysis (PCA), Abid et al. (2017) describe "contrastive PCA' (cPCA), a model which discovers dimensionnnl structure that uniquely characterizes dataset compared another. The approach linearly transforms latent representations

---

Under review as a conference paper at ICLR 2020
Happiness
Sadness
Surprise
Fear
Disgust
Anger
Contempt
DIP-VAE-II
0.548
0.245
0.401
0.389
0.268
0.391
0.463
SVAE
0,583
0.251
0.389
0.349
0.260
0.391
0.469
B-TCVAE
0,563
0.277
0.393
0.349
0.256
0.427
0.467
RbVAE
0.536
0.393
0.379
0.311
0.320
0.383
0.421
VRR
0.587
0.405
0.387
0.327
0.344
0.425
01.883
Ours(Eq. (7))
0.575
0.488
0.389
0.365
0.356
0.395
0.501
Ours(Eq. (8))
0.579
0.491
0.401
0.374
0.358
0.398
0.506
Table 2: Affectnet (Ruiz et al., 2019). Per class classification accuracy on each class of emotions.
which also linearly combined. To obtain more expressive transformations kernels have also been investigated (Abid et al., 2017). Since linearity restricts expressiveness of the model, in more recent work, Severson al. (2018) introduce "contrastive latent variable models' which permit to non-linearly transform the latent representation. Importantly contrastive latent variable models still combine transformations linearly.
Disentangling of representations (Bengio et al., 2013) has also been investigated in the computer vision community. Recently, variety of approaches like 3- 8-VAE (Higgins et al., 2016), DIP-VAS (Kumar et al., 2018), FactorVAE (Kim & Mnih, 2018) 3-TCVAE (Chen et al., 2018) have been discussed. Generally those techniques operate on a single dataset with the goal to extract latent factors.
Also related is work on reference- baased variational auto-encoder (RE-VAEE) (Ruiz et al., 2018), where reference-based disentangling introduced. A model carefully analyzed and the resulting cost function is, augmented via reverse KL-distribution to disentangle factors between two datasets.
In contrast, to the aforementioned methods we develop a model that disentangles factors without the need for any heuristics. To this end we introduce two losses. The first loss encourages the encoded background data representation to avoid any signal in the salient feature. The second loss encourages the non-salient probability distributions to be identical for both the background and the target set. Both losses combined help us accurately disentangle salient representations.
Canonical correlation analysis (CCA) (Hotelling, 1936) and its probabilistic counterpart (PCCA) (Bach & Jordan, 2005) also operate on at least two sets of data. However, both require the datasets to be paired. This is also true for the non-linear extensions, e.g., the based Gaussian processes (Damianou et al., 2016).
Dimensionality reductions techniques like t-SNE (Maaten & Hinton, 2008) and multi-dimensiona scaling (MDS) (Cox & Cox, 2000) are related in that they recover non-linear data projections. Yet, they are designed explore one dataset at a time. Consequently, for contrastive analysis, those approaches are applied on each dataset separately and a manual comparison is subsequently required uncover similarities and differences.
Statistical test, e.g., two-sample t-test, Fisher's s discriminant analysis, Wilcoxon signed-rand test, WainnWWhtneyy U-test, identify differences between two datasets given features. Albeit uncovering feature differences, none of those approaches are developed to find differentiating features.
CONCLUSION
In this paper we present two principled losses for variational autoencoder based models which address contrastive disentanglement, i.e.. extracting the salient features that enhance one dataset (target) compared to another (background). Our first loss explicitly discourages expression of salient features for the background data and is derived from classical variational principles. The second loss encourages background features to be identical for both target and background data. In extensive experiments on a variety of datasets we showed the benefits of the two introduced losses.
REFERENCES
Abubakar Abid and James Zou. Contrastive variational autoencoder enhances salient features. arXiv preprint arXiv: 2019.
Abubakar Abid, Martin J. Zhang, Vivek K. Bagaria, and James Zou. Contrastive principal component analysis. arXiv preprint (009v/709.067110 16, 2017.
10

---

Under review as a conference paper at ICLR 2020
F. R. Bach and M. I. Jordan. A probabilistic interpretation of canonical correlation analysis. Technical report, University of California, Berkeley, 2005.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. PAMI, 2013.
Nicolas Bonneel, Michiel Van De Panne, Sylvain Paris, and Wolfgang Heidrich. Displacement interpolation using lagrangian mass transport. In ACM Transactions on Graphics (TOG), 2011.
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv: 2018.
Trevor F. Cox and Michael A. A. Cox. Muliidimensional scaling. Chapman and Hall/CRC, 2000.
A. Damianou, N. D. Lawrence, and C. H. Ek. Multi-view learning as nonparametric nonlinear inter-battery factor analysis. In arXiv preprint 1eXiv:660..44933 2016.
John M. Danskin. The theory of max-min and application weapons allocation problems. Springer Science & Business Media, 2012.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Proc. CVPR, 2009.
Rémi Flamary and Nicolas Courty. POT: Python Optimal Transport library, 2017. URL https:
//github.....///// wtt.:coooolaa.ea wwwom/fflaaay//eeee/
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. ICLR, 2(5)66, 2017.
Harold Hotelling. Relations between two sets of variates. Biometrika, 1936.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proc. ICMI,,
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11X1::1412.6980, 2014.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational Inference of Disentan- gled Latent Concepts from Unlabeled Observations. In Proc. ICLR, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
Adrià Ruiz, Oriol Martinez, Xavier Binefa, and Jakob Verbeek. Learning Disentangled Represen- tations with Reference- Variational Autoencoders. working paper or preprint, 2018. URL
https: //ah//haa..... iariia.... Er hhtphhall00 haahaaa91100....1111111111111111
Adrià Ruiz, Oriol Martinez, Xavier Binefa, and Jakob Verbeek. Learning disentangled representations with reference-based variational autoencoders. arXiv preprint urXiv:1907.08555 2019.
Kristen Severson, Soumya Ghosh, and Kenney Ng. Unsupervised learning with contrastive latent variable models. arXiv preprint
2018.
Cédric Villani. Topies in optimal transportation. Number 58, American Mathematical Soc., 2003.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
Lemma 1, Let P = N (s: pr, o²) be a Gaussian distribution with mean and diagonal covariance
o², and Q 0{8 } be Dirac distribution centred zero. Then
(K Q) L((o.)) ((H, (),,o))
where 1 ((. = 0. (m, σ (0, 0), and otherwise.
Proof. By definition, we know that (P Q) = Ep|log
P(a) Q if < Q and co otherwise. Since
P is a Gaussian and Q is a Dirac, we have that < Q and only (μ, o) = (), Hence the claim follows.
Lemma 2. Let P = Ns; o²) be Gaussian distribution with mean and diagonal covariance
a². and Q (s }} be Dirac distribution centred zero. Then
(P, ||a112 + 1|
Proof. Note that (P, Q) is defined as
(P. Px
inf
E||X - -1
where the infimum is over all joint probability distributions Px.y such that X has the marginal P. whereas Y follows Q. Since P is a Gaussian distribution, which is absolutely continuous with respect to Lebesgue measure in Rd, it follows from Theorem 2.12 of Villani (2003) that
W T:##-=Q
inf
Ex Ex~P T(X)
where T#P P Q denotes that the pushforward of probability measure P under the map T Rd _> Rd Q. Or equivalently, T(X) Q whenever that the only such feasible transport map is given by T(x) 0, Vr e Rdd Hence, (P Q) Ex~Pl| = || + |0
P. Since Q is a Dirac mass at zero, it follows
A.1 DATASET DESCRIPTION
Linear and NonLinear Grassy MNIST
For the background dataset {y0),, 3. we randomly crop 28 x 28 sized image patches from the grass category of the ImageNet dataset (Deng al., 2009). We total of 4986 grass patches to construct this dataset. We construct the target dataset (zx()) in the following way: We choose 4986 samples for each digit from 0 to 9 using the MNIST dataset. We use 4986 samples because it corresponds to the lowest number of occurrences for digit (digit 4). Each of these digits are then superimposed on random grass images from the background dataset. The superimposition is a linear operation described by
( = py + (1 ())
(10)
where {()) is random MNIST image and م 0.5. Even though the samples {z((0) } are generated from (0) according to Eq. (10), we do not have access to these pairs (a((), (() during training.
To construct the non-linear dataset, we use a thresholding operation before superimposing the MNIST digits on (9) i.e.
[u, U, cl = Ily [u, u, c > 0.3) [u, c] + (1 ())
[u, v))
(11)
where { is random MNIST image, p 0.5, I denotes the indicator function, u, v denote the row and column pixel indices of the image and e is the channel number.
CelebA
12

---

Under review as a conference paper at ICLR 2020
We form the target and the background sets using the CelebA dataset (Liu al., 2018) as described by Abid & Zou (2019). To form the target dataset, we use a subset of the CelebA (Liu et al., 2018), namely, faces with caps and eyeglasses. For each class, categories are shuffled and 10, 000 samples are randomly picked as background images. Each image from the target and background is cropped into 64 x 64 patches.
take 5000 samples. The remaining
AffectNet
Use this challenging dataset for disentanglement of facial expressions. Background: faces with neutral expressions. Target: Faces with other expressions, namely happy, sad, surprised, fearful, disgust, angry and contemptuous. Labels only for qualtitative evaluation. (collected from affectnet cite, adria cite) Aligned and cropped (cite adria). 96 x 96
A.2 SOME RECONSTRUCTION AND INTERPOLATION RESULTS
Figure shows the reconstructed results across the different datasets. From top, the rows represent background images, reconstructed background, target images and reconstructed target respectively. Figure 10 shows results by interpolating salient features of target across dimensions. The common factors are kept constant
A.3 ARCHITECTURAL CHOICES
We use a 6 layer architecture for our model to test it on the Linear and Non-Lincar Grassy MNIST dataset. The encoder consists of 5 convolutional layers and 1 fully connected layer. LeakyReLU acti- vation function is used as the non-linearity and channel normalization is used after each convolutional layers. The decoder consists of fully connected and transposed convolutional layers. We use the LeakyReLU activation for the initial layers and the sigmoid function for the last layer. We use batch size of 128, learning rate of 10-3 and Adam optimizer (Kingma & Ba, 2014). We performed all the experiments using an 16 dimensional latent space to capture the grass and 8 dimensional space to capture digits, i.c., 26 E R 24 E R The weight on the KLD term the VAE loss is = (Higgins al., 2017).
We adopt a 8 layer architecture for our model to be consistent with (Abid & Zou, 2019) for evaluation on CelebA dataset. We performed all the experiments using 16 dimensional latent space to capture the background faces and a 6 dimensional space to capture target variations, namely eyeglasses or caps.
For Affectnet, we use a 8 layer architecture for our model to be consistent with (Ruiz et al., 2019). We tale equal number of target images (5000) per class performed all the experiments using an 32 dimensional latent space to capture the background faces and a 32 dimensional space to capture target variations, namely eyeglasses caps.
A.4 ADDITIONAL. EXPERIMENTS
We have performed additional experiments with varied sample sizes and found that our methods still consistently outperform the existing approach from Ruiz et al. (2019) across a variety of sample sizes. Figure 11 shows the results of this experiment. The experimental details our experiments, we found that Ruiz et al. (2019) performs the best among all the other baselines as seen from Table of our paper, hence we consider Ruiz al. (2019) be our baseline. With regards to varied sample sizes, for the target dataset containing MNIST linearly superimposed on grass (L-MNIST), Similarly, for the background dataset of just grass images, the total number of samples are 4986. 2500 and 1000 respectively.
follows: In
chose the number of samples from each digit class be 1000, 2500 and 4986.
13

---

Under review as a conference paper at ICLR 2020
4
7
3
6
6
3
5
7
9
4
6
4
5
4
7
7
3
6
6
3
5
7
9
4
6
4
5
(a)
0
9
7
(b)
(d)
Figure 9: Reconstruction results across different datasets. (a) L MNIST, (b) NL MNIST, (c) CelebA,
(d) Affectnet
14

---

Under review as a conference paper at ICLR 2020
5
5
5
5
6
6
5
5
5
5
6
6
5
5
5
5
6
9
5 5
5
5
5
6
9
S 5
5
5
5
6
9
S 5
5
5
5
6
9
5 5
5
5
5
9
5 5
5
5
5
5
9
9
5
5
5
5
9
9
5 S
5
5
9
9
S
5
S
5
9
9
9
S 5
5
5
5
5
9
9
9
8
8
5
5
5
5
5
9
9
9
9
8
8 8
8
8
8
8
7
8
8
8
8
8
9
9
9
8
8
8
8
7
8
8
8
8
9
9
9
8
8
8
8
8
8
7
8
8
8
8
8
9
9
9
8
8
8
8
8
8
8
9
9
9
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
8
8
8
8
8
8
8
8
8
8
9
9
9
8
8
8
8
8
8
8
8
8
8
8
9
9
9
Figure 10: 2D Interpolation on L MNIST
Clustering Accuracy vs Varied Sample Sizes
Clustering Accuracy on Target Features(%)
cVAE[]]
90
Ours (Eqn 8)
80
75
65
1000
1500
2000
2500
3000
3500
4000
4500
5000
Sample Size
Figure 11: Clustering Accuracy(%) versus sample sizes for cVE (Ruiz et al., 2019) and for our approach (Eqn 8)
15


Reviews:
[{"review_id": "rkgcvpn-cH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is concerned with contrastive disentanglement. The considered problem is interesting and important in the community, and the proposed method seems to be of practical use, according to the empirical results. My concern is that the contribution of the paper on the theoretical or methodological side seems a bit weak.\n\nThe proposed method relies on the VAE framework for contrastive disentanglement. The authors proposed two modifications: one is to explicit enforce the posterior of the content representation to be a Delta distribution, and the other is to make sure that the background representation has the same distribution across the background and the target. In the original cVAE framework for contrastive disentanglement, the two constraints were considered in a slightly more implicit way. It is not surprising to see that the proposed method seems to outperform cVAE only slightly. To see how much we can really gain by further explicitly enforcing the two constraints, the authors may do some further studies with varied sample sizes. BTW, in the result for CelebA in Table 1, cVAE seems to outperform the proposed method, denoted by 'Ours', according to Acc of z_x, but it was indicated in the text (as well as in by the bold font) that 'Ours' performed the best--is there a typo?\n\nI acknowledge I read the authors' response and other reviews and would like to keep my original rating.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gmvxTkqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\n1. This paper fixes two problems that appeared in common contrastive disentanglement methods.\n2. The paper evaluates its method in multiple experiments.\n\nOverall, this paper does a nice contribution to improving existing methods of contrastive disentanglement. My major concern is the novelty in this paper since all the contributions can be summarized into adding two additional loss terms, which I would like to discuss below.\n\n1. [The missing term KL[q(s|y)||p_y(s)]]. One major claim of this paper is to add back the additional term KL[q(s|y)||p_y(s)] in order to push the encoder q(s|y) to converge to a point mass \\delta\\{s=0\\} for background images y. According to the mathematical formula, this term is natural, but the major concern is the non-overlapping support between q(s|y) and p_y(s), since in the paper they assume p_y(s)=\\delta\\{s=0\\}. This paper claims they fix this problem. But I do not think the solution is satisfactory, because they use another term E_q[\\mu_{\\phi,s}(y)^2+\\sigma2_{\\phi,s}(y)] to replace KL[q(s|y)||\\delta\\{s=0\\}]. If I get it correct, the replacing term E_q[\\mu(y)^2+\\sigma2(y)]=H(q(s|y)) is the entropy of q(s|y). But if we check the true divergence term, KL[q(s|y)||\\delta\\{s=0\\}]=-E_q[\\log\\delta\\{s=0\\}]-H(q(s|y)) that consists of a negative entropy term. This means the behavior of the proposed loss tries to minimize the entropy while the original term tries to maximize the entropy, regardless of it is ill-defined. Thus, the replacement of the loss term seems reasonable to me at first glance but does not quite directly fix the problem of an ill-defined KL-divergence. A more reasonable solution is to replace the KL divergence with the Wasserstein distance. Is there any difficulty if we do that?\n\n2. [The additional distributional matching term W_2^2(q_t(z),q_b(z))]. This term makes perfect sense to me, even though it is just added artificially. I'm more curious about the way to evaluate the gradient. Normally, computing Wasserstein distance is hard due to a hard linear programming matching algorithm. In this paper, the authors seem to use an existing library to compute the optimal transportation matrix. Is that a heavy computational burden? Or one could resort to some entropy regularization methods with sinkhorn iterative solvers. I'm curious about how these two methods compare with each other.\n\n3. Another concern is the numerical result seems not as good as the qualitative results.\n\nOverall, I think this paper does make some changes regarding previous works on contrastive disentanglement. But their main contributions need some more explanation and justification.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gXN-wZ5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Existing formulations for contrastive disentanglement ignore the information about the prior, i.e., the salient features of the background data should be zero, and moreover, additional KL-divergence-based losses which are hard to estimate in practice are introduced to improve disentanglement.\n\nTo resolve this issue, the authors propose new regularizations still based on VAE. Specifically, they propose to\n\nRegularization 1. penalize the generated mean and standard deviation of background latent variable towards (0,0) which means Dirac-delta distribution on 0.\n\nRegularization 2. match common latent variable distributions for background dataset and target dataset using Wasserstein distance.\n\nWhile authors point out that existing formulation in Abid & Zou (2019); Ruiz et al. (2019) ignores about the prior, the effect of regularizing salient feature seems marginal. This can be found in experimental results in Table, proposed objective (7) gives similar or worse result on L MNIST(similar), CelebA(worse), and Affectnet(worse) dataset. Also, visualization of salient features in Figure 4 show that representation balancing effects of proposed regularization terms are marginal compared to cVAE.\n\nAnother cocern is proposed regularization 1,2 might hurt the original ELBO objective. This concern is also related to beta-VAE, the paper showed that large coefficient for prior regularization simply makes the disentanglement effect for VAE. On the other hand, proposed objective (7) requires 'large constant' for approximate quadratic loss. This might induce unintentional negative effects for the objective (7) and objective (8). It would be great if authors can show the proposed regularization does not induce such unintended numerical problem. A possible solution might be showing asymptotic theoretical guarantees as in semi-implicit variational inference.\n\nThe objective (8) seems to require additional computational cost to solve linear programming problem. It is not clear if it is effective enough to bear that extra cost; it is hard to conclude that proposed regularization performs better than previous works without confidence interval.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes new regularizations on contrastive disentanglement. After reading the author's response,  all the reviewers still think that the contribution is too limited and all agree to reject.
Decision: Reject