Unsupervised Learning of Global Factors in Deep Generative Models
Authors: Ignacio Peis
Pablo M. Olmos
Antonio Artés
Venue:  (2021)
Keywords: unsupervised
autoencoders
disentanglement
generative models
representation learning

Abstract: We present a novel deep generative model based on non i.i.d. variational autoencoders that captures global dependencies among observations in a fully unsupervised fashion. In contrast to the recent semi-supervised alternatives for global modeling in deep generative models, our approach combines a mixture model in the local or data-dependent space and a global Gaussian latent variable, which lead us to obtain three particular insights. First, the induced latent global space captures interpretable disentangled representations with no user-defined regularization in the evidence lower bound (as in beta-VAE and its generalizations). Second, we show that the model performs domain alignment to find correlations and interpolate between different databases. Finally, we study the ability of the global space to discriminate between groups of observations with non-trivial underlying structures, such as face images with shared attributes or defined sequences of digits images.

Under review as a conference paper at ICLR 2021
UNSUPERVISED LEARNING OF GLOBAL FACTORS IN DEEP GENERATIVE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We present novel deep generative model based non i.i.d. variational autoen- coders that captures global dependencies among observations in a fully unsuper- vised fashion. In contrast to the recent semi-supervised alternatives for global modeling in deep generative models, our approach combines a mixture model in the local or data-dependent space and a global Gaussian latent variable, which lead us to obtain three particular insights. First, the induced latent global space captures interpretable disentangled representations with no user-defined regular- ization in the evidence lower bound (as in beta-VAr VAE and its generalizations) Sec- ond, we show that the model performs domain alignment to find correlations and interpolate between different databases. Finally, study the ability of the global space to discriminate between groups of observations with non-trivial underly- ing structures, such face images with shared attributes defined sequences of digits images.
INTRODUCTION
Since its first proposal by Kingma Welling (2013). Variational Autoencoders (VAEs) have evolved into a vast amount of variants. To name some representative examples, we can include VAEs with latent mixture models priors (Dilokthanakul et al. (2016)), adapted to model time-series (Chung et al. (2015)). trained via deep hierarchical variational families (Ranganath et al. (2016), Tomezak & Welling (2018)), or that naturally handle heterogeneous data types and missing data (Nazabal et al. (2200).
The large majority of VAE-like models are designed over the assumption that data is i.i.d., which remains valid strategy for simplifying the learning and inference processes in generative models with latent variables. A different modelling approach may drop the i.i.d. assumption with the goal of capturing higher level of dependence between samples. Inferring such kind of higher level dependencies can directly improve current approaches to find interpretable disentangled generative models (Bouchacourt al. (2018)), to perform domain alignment Sreinze-Demi & Meinshausen 2017 or to ensure fairness and unbiased data [Barocas et al. (2017)).
The main contribution of this paper is to show that deep probabilistic VAE non i.i.d. model with both local and global latent variable can capture meaningful and interpretable correlation among data points completely unsupervised fashion. Namely, weak supervision group the data samples is not required. In the following we refer to our model as Unsupervised Global VAE (UG-VAE). We combine clustering inducing mixture model prior in the local space, that helps separate the fundamental data features that an i.i.d. VAE would separate, with global latent variable that modulates the properties of such latent clusters depending on the observed samples, capturing funda- mental and interpretable data features. We demonstrate such result using both CelebA, MNIST and the 3D FACES dataset in Paysan et al. (2009). Furthermore, we show that the global latent space can explain common features in samples coming from two different databases without requiring any domain label for each sample, establishing a probabilistic unsupervised framework for domain alignment. Up to knowledge, UG-VAE is the first VAE model in the literature that performs unsupervised domain alignment using global latent variables.
Finally, demonstrate that, even when the model parameters have been trained using unsu- pervised approach, the global latent space in UG-VAE can discriminate groups of samples with non-trivial structures, separating groups of people with black and blond hair in CelebA series of

---

Under review as a conference paper at ICLR 2021
numbers in MNIST. In other words, if weak supervision is applied at test time, the posterior distri- bution of the global latent variable provides with an informative representation the user defined groups of correlated data.
2 RELATED WORK
Non i..d. deep generative models are getting recent attention but the literature is still scarse. First we find VAE models that implement non-parametric priors: Gyawali al. (2019) the authors make use of a global latent variable that induces
te--parametric Beta process prior, and more efficient variational mechanism for this kind of IBP prior are introduced Xu et al. (20T9). Second, both Tang et al. (211)) and Korshunova et al. (2018) proposed non i..d. exchangable models by including correlation information between datapoints via an undirected graph. Finally, some other works rely on simpler generative models (compared to these previous approaches), including global variables with exed-complexity priors, typically a multi vartiiv Gaussian distribution, that aim at modelling the correlation between user-specified groups of correlated samples (e.g. images of the same class in MNIST, or faces of the same person). In [Bouchacourt et al. (2018) or Hosoya (2019), authors apply weak supervision by grouping image samples by identity, and include in the probabilistic model global latent variable for each of these groups, along with a local latent variable that models the distribution for each individual sample. Below we specify the most relevant lines of research, in relation to our work.
VAEs with mixture priors. Several previous works have demonstrated that incorporating a mix- ture in the latent space leads to learn significantly better models. In Johnson et al. (2016) authors introduce a latent GMM prior with nonlinear observations, where the means are learned and remain invariant with the data. The GMVAE proposal by Dilokthanakul et al. (2016) aims at incorporat- ing unsupervised clustering in deep generative models for increasing interpretability. VAE model Tomezak & Welling (2018), the authors define the prior as mixture with components given by approximated variational posteriors, that are conditioned on learnable pseudo-inpun This approach leads to improved performance, avoiding typical local optima difficulties that might be related to irrelevant latent dimensions.
the VAMP
Z
S.
Cc
X;
2,
((a)
4, E [1.M]
Zx
x
0x
N
xx
X
0x
g
X,
(a) VAE
(b) GMVAE
(c) ML-VAE
(d) NestedVAE
Figure 1: Comparison of four deep generative models. Dashed lines represent the graphical model of the associated variational family. The Vanilla VAE (a), the GMVAE (b), and semi-supervised variants for grouped data; MI.VVA (c) and NestedVAE (d).
Semi-supervised deep models for grouped data. In contrast to the i.i.d. vanilla VAE model in Figure (a), and its augmented version for unsupervised clustering, GMVAE, Figure T (b), the graphical model of the Multi-Level Variational Autoencoder (ML-VAE) in Bouchacourt et al. (2018) shown Figure (c). where G denotes the number of groups. MI-VALL includes local Gaussian variable S; that encodes style-related information for each sample, and global Gaussian variable Cc to model shared in a group of samples. For instance, they feed their algorithm with batches of face images from the same person, modeling content shared within the group that characterize person. This approach leads to learning a disentangled representations at the group and observations level, in content-style fashion. Nevertheless, the groups user-specified, hence resulting in semi- supervised modelling approach. In Vowels et al. (2020) authors use weak supervision for pairing samples. They implement two outer VAES with shared weights for the reconstruction, and Nested
2

---

Under review as a conference paper at ICLR 2021
VAE that reconstructs latent representation off one to another, modelling correlations across pairs of samples. The graphical model for Nested VAE depicted in Figure (d).
UNSUPERVISED GLOBAL VAE
We present UG-VAFF deep generative VAE framework for modeling non-i.i.d data with global dependencies. generalizes the ML-VAE graphical model in Tigure[[ce to remove the group pervision, ùi include a clustrring-inducnng prior the local space, and tii) propose a more structured variational family.
z
4
B
(a) Generative model
(b) Inference model
Figure 2: Generative (left) and inference (right) of UG-VAE.
3.1 GENERATIVE MODEL.
Figure 2 represents the generative graphical model of UG- VAE. A global variable 3 H9 induces shared features to generate a group of B samples X (x1, x C RDD. and G is the number of amortize the learning of the model parameters. During amortized variational training, groups are simply random data mini-batches from the training dataset, being G the number of data mini-batches. We could certainly take B N (the training size) and hence g 1, but this leads to less interpretable global latent space (too much data to correlate with a single global random variable), and slow training process.
groups we jointly
Conditioned to B, data samples are independent and distributed according to Gaussian mixture
local (one per data) latent variable Z (zi,..., ZB) Rdd and [di..., dg) S {I, K} are independent discrete categorical variables with uniform prior distributions. This prior, along with the conditional distribution p(z, (d, B), defines a Gaussian mixture latent space, which helps to infer similarities between samples from different batches (by assigning them to the same cluster), and thus, de plays a similar role than the semi-supervision included in [Bouchacourt et al. (2018) by grouping. Our experimental results demonstrate that this level of structure in the local space crucial to acquire interpretable information at the global space, and specially, if we fix d, for all the samples within batch, that the global variable is able tune different generative factors for each cluster.
The joint distribution for single group is therefore defined by: Pe X Z, d, 3) =
(1)
where the likelihood term of each sample a Gaussian distribution, whose parameters are obtained from a concatenation of z, and B as input of decoder network:
p X Z, 3)
I ((x z. B)
II N (mo (izi, Bl), Ea, (Z; A=l
In contrast with Johnson et al. (0166). where the parameters of the clusters are learned but shared by all the observations, in UG- VAE, the parameters of each component obtained with networks fed
3

---

Under review as a conference paper at ICLR 2021
with B. Thus, the prior of each local latent continuous variable is defined by mixture of Gaussians, where d, defines the component and |3 is the input of a NN that outputs its parameters:
B
B
pZ]dd () = II
p 4, 3) = II
N
(1)) (4.
(3). 2g (1t)) (3)).
(3)
1=1
2=1
hence we trained as many NNs as discrete categories. This local space encodes samples in represen- tative clusters to model local factors of variation. The prior of the discrete latent variable is defined as uniform:
B
p(d) = II Cat(TT)
Th = 1/K
(4)
i=1
and the prior over the continuous latent variable & follows an isotropic Gaussian, p(3) = (000
3.2 INFERENCE MODEL
The graphical model of the proposed variational family is shown in Figure 2(b):
(Z. d, X (Z| Zd (
(5)
where we employ encoder network that maps the input data into the local latent posterior distri- bution, which is defined as Gaussian:
(Z II [Z.
II N (us, (x:), Es. (x:))
Given the posterior distribution of z, the categorical posterior distribution of d; is parametrized by NN that takes z, as input
B (d Z) II g(d (.) = II Cat(TTp.
B
1=1
The approximate posterior distribution of the global variable /3r computed as a product of local contributions per datapoint. This strategy, as demonstrated by Bouchacourt et al. (2018), outper- forms other approaches like, for example, a mixture of local contributions, as it allows to accumulate group evidence. For each sample, NN encodes x, and the Categorical parameters Te. (z:) in local Gaussian:
B II N [Mox((x; Toe (2:))) Dos ([xs, Tou (zz))))
( Z) N (pg. E3)
If we denote by br and Σ, the parameters obtained by networks fho and Σ Ed,, respectively, the parameters of the global Gaussian distribution are given, following Broomley (2003). by:
B
A = ΣΦ
A
i=1
(9)
B
p3 = (Ap)
Aip
1=1
where Ag Ez is defined the precision matrix, which model diagonal matrix.
3.3 EVIDENCE LOWER BOUND
Overall, the evidence lower bound reads as follows:
[(0, Φ; X, Z, d, 3) = Ey(() [C,(0, o; x,, z., d, B)] - By(a) [Del (g(3) 2)1x()))p(()))
(10)
The resulting ELBO is an expansion of the ELBO for a standard GMVAE with new regularizer for the global variable. As the reader may appreciate, the ELBO for UG-VAE does not include extra

---

Under review as a conference paper at ICLR 2021
hyperparameters to enforce disentanglement, like other previous works as 3-VAE, and thus, extra validation is needed apart from the parameters of the networks architecture, the number of clusters and the latent dimensions. We denote by L, each local contribution to the ELBO:
33] B((,)) [DKL K (g(z, xx) plz dd 3)) DKL (g(d//z))lp(d)))
L, db 2,, d. 22
(II)
The first part of equation 10 is expectation
the global approximate posterior of the so-called local ELBO. This local El BO differs from the vanilla ELBO proposed by by/Kingma& & Welling (2013) in the regularizer for the discrete variable d,, which is composed by the typical reconstruction term of each sample and two KL regularizers: one for Z, expected over d,, and the other over d4. The second part in equation[Oii
regularizer on the global posterior. The expectations over the discrete variable d; are tractable and thus, analytically marginalized.
In contrast with GMVAE (Figure in UG- VAE, [ is shared by group of observations, there- fore the parameters of the mixture are the same for all the samples in a batch. In this manner, within each optimization step, the encoder ( Z.) only learns from the global information obtained from the product of Gaussian contributions of every observation, with the aim at configuring the mixture to improve the representation of each datapoint in the batch, by means of 33 X) and p( pXX IZ, (). Hence, the control of the mixture is performed by using global information. In contrast with ML-VAE (whose encoder ((CalX) X) is also global, but the model does not include a mixture), in the 33 encoder incorporates information about which component each observation belongs to, as the weights of the mixture inferred by (d]Z) are used to obtain ((3 X, Z). Thus, while each cluster will represent different local features, moving will affect all the clusters. In other words, modifying 3 will have some effect in each local cluster. As the training progresses, the encoder ( X, Z) learns which information emerging from each batch of data allows to move the cluster in a way that the ELBO increases.
4 EXPERIMENTS
In this section we demonstrate the ability of the UG-VAE model to infer global factors of variation that are common among samples, even when coming from different datasets. In all cases, we have not validated in depth all the networks used, we have merely rely on encoder//ecoder networks pro- posed in state-of-te of-t--of-the-rr art VAE papers such as Kingma & Welling 20133) Bouchacourt et al.) (2018) or B
Higgins et al. (2016). Our results must be hence regarded as proof of concept about the flexibility and representation power of UG-VAE, rather than fine-tuned results for each case. Hence there room for improvement in all cases. Details about network architecture and training parameters provided in the supplementary material.
4.1 UNSUPERVISED LEARNING OF GLOBAL FACTORS
In this section we first asses the interpretability of the global disentanglement features inferred by UG-VAE over both CelebA and MNIST. In Figure 3 we show samples of the generative model as we explore both the global and local latent spaces. We perform linear interpolation with the aim at exploring the hypersphere centered at the mean of the distribution and with radius d; for each dimension To maximize the variation range across every dimension, we move diagonally through the latent space. Rows correspond to an interpolation dimension (p(3) follows a standard Gaussian). As the local p(z) d, 33) (equation 3 depends on
the global B3 between |-,, 1] every
and [,, if denote μ4 uu= ((3,, the local interpolation goes from [u2 3, Mat 3. Medd - 3] to [M20 + 3, M21 + 3, 3..., Med + 3). The range of #3 for the local interpolation is determined to cover the variances _ (B) that observe upon training the model for MNIST and CelebA. The every image in Figure correspond to samples from a different cluster (fixed values of d), in order to facilitate the interpretability of the information captured both local and global levels. By using this set up, we demonstrate that the global information tuned by ,3 is different and clearly interpretable inside each cluster.
The total number of clusters is set to 20 for CelebA and K = 10 for MNIST. Three of these components are presented in Figure 3 We observe that each row (each value of ()) induces
5

---

Under review as a conference paper at ICLR 2021
shared generative factor, while is in charge of variations inside this common feature. For instance, in CelebA (top), features like skin color, presence of beard or face contrast are encoded by the global variable, while local variations like hair style or light direction In a simple dataset like MNIST (bottom), results show that handwriting global features as cursive style, contrast thickness characterization of whether these generative factors are local/global is based on an interpretation of the effect that varying z and provokes in each image within batch, and in the whole batch of images, respectively. In the supplementary material, we reproduce the same figures for the all the clusters, in which we can appreciate that there significant fraction of clusters with visually interpretable global/local features.
controlled by the local variable.
encoded by B, while the local defines the shape of the digit. The
We stress here again the fact that the UG-VAE training fully unsupervised: data batches during training are completely randomly chosen from the training dataset, with no structured correlation whatsoever. Unlike other approaches for disentanglement, see Higgins et al. (20T6) or Mathieu ct al. (2019), variational training in UG-VAE does not come with additional ELBO hyperparameters that need to be tuned to find a proper balance among terms in the ELBO.
hair
hair
light
Z
Z
Z
skin color
beard
contrast
e
d
d =
d
shape
shape
shape
cursive
3
3
thickness
0 2
2 2 2
2
2
2
contraas
6
6
6
6
2
2
0
3
2
0
2
2
2
2
6 6
6
6
6
2 2 &
2 Z Z & <
0
2
2
0
Q
2
7
6
6 6
6
2
3 0 6 6 d 4
2
4
22222
6
3
3
3
8
6
A 0
2
22
6
6 6 &
6
8 6 6
8
4
0
2 3
2
2 22
6
0
a
d
d=7
= 9
Figure 3: Sampling from UG-VAE for CelebA (top) and MNIST (bottom). We include samples from local clusters from total of K 20 for CelebA and K 10 for MNIST. CelebA (top), the global latent variable disentangles in skin color, beard and face contrast, while the local latent variable controls hair and light orientation. MNIST (bottom), ,3 controls cursive grade, contrast and thickness of handwriting, while z varies digit shape.
One of the main contributions in the design of UG- VAE the fact that, unless we include clustering mixture prior the local space controlled by the global variable [3, unsupervised learning of global factors is non-informative. To illustrate such result, in Figure we reproduce the results in Figure 3 but for a probabilistic model in which the discrete local variable d not included. Namely, we use the ML-VAE in Figure[[cc but trained it with random data batches. In this case, the local space is uni uni-modal given / and we show interpolated values between -1 to 1. Note that the disentanglement effect of variations in both 33 and mild and hard interpret.

---

Under review as a conference paper at ICLR 2021
Z
9
9
B
B
5
3
9
5
9
8
4
5
8
8
0
2
(a) CelchA
(b) MNIST
Figure Sampling from ML- VAE, trained over unsupervised data.
4.2 DOMAIN ALIGNMENT
In this section, we evaluate the UG- performance in an unsupervised domain alignment setup. During training, the model is fed with data batches that include random samples coming from two different datasets. In particular, FACES Paysan et al. (2009), a dataset of 3D scanned faces, with a proportion of 50% samples from each dataset inside each batch.
train model with mixed dataset between CclebA and 3D
Upon training with random batches, in Figure we perform the following experiment using domain supervision create test data batches. We create two batches containing only images from CelebA and 3D FACES. Let 31 and B2 be the mean global posterior computed using (8) associated for each batch. For two particular images in these two batches, let = and 2σ be the mean local posterior of these two images, computed using (3). 5 shows samples of the UG-VAE model when we linearly interpolate between B1 and B3 (rows) and between ø1 and 22 (columns Certainly ,3 is capturing the domain knowledge. For fixed z, e.g. ø1 in the first column, the interpolation between 31 and B2 transferring the CelebA image into the 3D FACES domain (note that background is turning white, and the image is rotated to get a 3D effect). Alternatively, for fixed 3, e.g. B1 in the first row, interpolating between 2 and 22 modifies the first image into one that keeps the domain but resembles features of the image in the second domain, as face rotation.
In Figure ()) we show the 2D 1-SNE plot of the posterior distribution of ,3 for batches that are ran- dom mixtures between datasets (grey points), batches that contain only CelebA faces (blue squares), and batches that contain only 3D faces (green triangles). We also add the corresponding points of the B1 and (3 interpolation in Figure (a). In Figure c)., reproduce the experiment in (a) but interpolating between two images and values of ,3 that correspond to the same domain (brown in- terpolation line in Figure ())).. As expected, the interpolation of / in this case does not change the domain, which suggests that the domain structure in the global space is smooth, and that the interpolation along the local space z modifies image features to translate one image into the other. In Figure 6 experiments with more datasets are included. When mixing the 3DCars dataset (Fidler al. ) with the 3D Chairs dataset (Aubry et al. (2014), we find that certain correlations be- tween cars and chairs are captured. In Figure (a), interpolating between racing car and an office desk chair leads to a white car in the first domain (top right) and in a couch (bottom left). In Figure
6 (b), when using the 3D Cars along with the Cars Dataset (Krause et al. (2013),, rotations in the cars are induced.
Finally, in the supplementary material we show that, as expected, the rich structured captured by UG-VAE illustrated in Figure 5 is lost when we do not include the clustering effect the local space, i.e. if we use MLEVAE with unsupervised random data batches, and all the transition between domains is performed within the local space.
Note that since both , and z are deterministically interpolated, the discrete variable d plays no role to sample from the model.
7

---

Under review as a conference paper at ICLR 2021
21 22
clabA FACES
3
9,
23
-20
(a) CelebA-FACES
(b) ,3 TSNE 21D space.
(c) FACES-FACES
Figure 5: UG-VAE interpolation in local (columns) and global (rows) posterior spaces, fusing celebA and FACES datasets. In (a) the interpolation goes between the posteriors of sample from CelebA dataset and sample from FACES dataset. In (c) the interpolation goes between the poste- riors of sample from FACES dataset and another sample from the same dataset.
21 + Z2
Z2
3
3
32
3D Cars-3D Chairs
(b) 3D Cars-Cars
Figure 6: Extended experiment: UG-VAE interpolation in local (columns) and global (rows) poste- rior spaces, fusing 3D Cars with 3D Chairs (d) and 3D Cars to Cars Datasct (c).
4.3 UG-VAE REPRESENTATION OF STRUCTURED NON-TRIVIAL DATA BATCHES
In the previous subsection, we showed that the VAE global space able to separate certain structure in the data batches (e.g. data domain) even though during training batches did not present such an explicit correlation. Using UG-VAE trained over CelebA with unsupervised random batches of 128 images as running example, in this section we want to further demonstrate this result.
In Figure 7 we show the i-SNE 2D projection of structured batches using the posterior 3 distribution in (8) over CelebA test images. In Figure 7aa, we display the distribution of batches containing only men and women, while in Figure (bb) the distribution of batches containing people with black or blond hair. In both cases we show the distribution of randomly constructed batches as the ones in the training set. To some extend, in both cases we obtain separable distributions among the different kinds of batches. A quantitive evaluation can be found in Table | which we have used samples from the & distribution batches. When random batches are not taken as a class, the separability is evident. When random batches are included, it is expected that the classifier struggles to differentiate between batch that contains 90% of male images and a batch that only contain male images, hence the drop in accuracy for the multi-case problem.
train supervised classifier to differentiate between different types of
8

---

Under review as a conference paper at ICLR 2021
An extension with similar results and figures for another interpretation of global information captur- ing are exposed in the supplementary material, using structured grouped batches in MNIST dataset. In this experiment, the groups digits that belong to certain mathematical series, including numbers, odd numbers, Fibonacci series and prime numbers, and we prove that the model is able to discriminate among their global posterior representations.
random
random
Male
Black Hair
Farale
20
Blond
20
-10
-20
-20
-10
10
20
-20 -10
10
20
Figure 7: 2D t-SNE projection of the UG-VAE 9 posterior distribution of structured batches of 128 CelebA images. UG-VAE is trained with completely random batches 128 train images.
Table 1: Batch classification accuracy using samples the posterior B distribution.
Batch categories
Classifier
Train accuracy
Test accuracy
Linear SVM
1.0
0.95
Black (0) vs blond (1)
RBF SVM
1.0
0.98
Linear SVM
0.91
0.54
Black (0) vs blond (I) vs random (2)
RBF SVM
0.85
0.56
Linear SVM
1.0
0.85
Male (0) vs female (1)
RBF SVM
1.0
0.85
Linear SVM
0.84
0.66
Male (0) vs female (1) vs random (2)
RBF SVM
0.89
0.63
5 CONCLUSION
In this paper we have presented UG-VAE, unsupervised generative probabilistic model able capture both local data features and global features among batches of data samples. Unlike similar approaches in the literature, by combining a structured clustering prior in the local latent space with a global latent space with Gaussian prior and a more structured variational family, we have demonstrated that interpretable group features can be inferred from the global latent space completely unsupervised fashion. Model training does not require artificial manipulation of the ELBO term to force latent interpretability, which makes UG- VAE stand out w.r.t. most the current disentangiement approaches using VAEs. The ability of UG-VAE to infer diverse features from the training set is further demonstrated in a domain alignment setup, where we show that the global space allows interpolation between domains, and also by showing that images in correlated batches of data, related by non-trivial features such as hair color or gender in CelebA, define identifiable structures in the posterior global latent space distribution.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mathieu Aubry, Daniel Maturana, Alexei A Efros, Bryan C Russell, and Josef Sivic. Seeing 3d chairs: exemplar part-based 2d-3d alignment using large dataset of cad models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3762-3769, 2014.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness in machine learning. NIPS Tutorial, 1, 2017.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Thirty-Second AAAI Con- ference Artificial Intelligence, 2018.
Paul Bromiley. Products and convolutions of gaussian probability density functions. Tina- Vision Memo, 3(4):1, 2003.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988, 2015.
Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint arXiv: 1.02648, 2016.
Sanja Fidler, Sven Dickinson, and Raquel Urtasun. 3d object detection and viewpoint estimation with a deformable 3d cuboid model. In Advances in neural information processing systems, pp- 611-619, 2012.
Prashnna Gyawali, Zhiyuan Li, Cameron Knight, Sandesh Ghimire, B Milan Horacek, John Sapp, and Linwei Wang. Improving disentangled representation learning with the beta bernoulli process. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 1078-1083. IEEE, 2019.
Christina Heinze- Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. arXiv preprint arXiv:1710.11466; 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAE: Learning basic visual concepts with a con- strained variational framework. 2016.
Haruo Hosoya. Group-based learning of disentangled representations with generalizability for novel contents. In IJCAI, pp. 2506-2513, 2019.
Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan Adams, and Sandeep Datta. Composing graphical models with neural networks for structured representations and fast infer- ence. Advances in neural information processing systems,
2016.
Diederik Kingma and Max Welling. 4rXiv.1312.6114,200 2013.
Auto-encoding variational bayes.
arXiv preprint
Iryna Korshunova, Jonas Degrave, Ferene Huszár, Yarin Gal, Arthur Gretton, and Joni Dambre. Bruno: A deep recurrent model for exchangeable data. In Advances in Neural Information Pro- cessing Systems, pp. 7190-7198, 2018
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei 3d object representations for fine- tne-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) Sydney, Australia, 2013.
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders, In International Conference on Machine Learning, pp. 4402-4412, 2019.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. Pattern Recognition, pp. 107501, 2020.
10

---

Under review as a conference paper at ICLR 2021
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296-3091 Ieee, 2009.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International Conference on Machine Learning, pp. 324-333, 2016.
Da Tang, Dawen Liang, Tony Jebara, and Nicholas Ruozzi. Correlated variational auto-encoders. arXiv preprint arXiv:/955.053.0 2019.
Jakub Tomezak and Max Welling. Vac with a vampprior. In International Conference on Artificial Intelligence and Statisfics, pp. 1214 2018.
Matthew Vowels, Necati Cihan Camgoz, and Richard Bowden. Nestedvae: Isolating common factors via weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9202-9212, 2020.
Kai Xu, Akash Srivastava, and Charles Sutton. Variational russian roulette for deep bayesian non- parametrics. In International Conference on Machine Learning, pp. 6963-6972, 2019.
Aron Yu and Kristen Grauman. Fine- aeaengr visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 192-199, 2014.


Reviews:
[{"review_id": "c3vuQ1R7lWh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a deep generative model based on the non i.i.d. VAE framework in an unsupervised version. The model which combines a mixture prior in the local latent space with global latent space has three advantages: First, the latent space can capture interpretable features. Second, the model performs domain alignment. Third, the model can discriminate among their global posterior representations. Although this paper has mild improvement on the basic VAE structure, the model displays a good interpretability power, and the setup of the latent variables are illustrated reasonably in the paper.\n\nStrength:\n1. This paper models on non-i.i.d data in an unsupervised version, which provides a flexible model.\n2. This model provides a good interpretability power. This paper demonstrates how the features are controlled by the global and local variables, and also it shows the necessary of including the mixture prior in the local space to acquire interpretable information.\n3. This model performs domain alignment, the global variable β can capture domain knowledge. \n\nCritiques:\n1. The paper only discussed one prior distribution for the latent variable d and β. The power of choosing other types of prior distribution is unknown.\n2. For the domain alignment experiment, only two datasets are used, it will be more convincing to include more datasets.\n\nI have read authors' feedback and will keep my original score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "aiPvQDL0uQy", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article introduces a VAE-based method for separating local variation factors from global variation factors in the data in an unsupervised manner. It achieves so by designing a graphical model with a mix of example-local and batch-shared variables, and training it using the ELBO. The article provide an detailed experimental analysis on MNIST and CelebA, and experimental evidence that all parts of the model (notably the discrete d variable) are relevant.\n\nThe article provides a well detailed description of the proposed UG-VAE, and how it compare to similar models from the literature (notably ML-VAE, from which it is inspired). The experimental analysis is reasonably convincing, though the interpretation of the provided figures in the text is a little more optimistic than I would agree.\n\nThere is however one point in particular I would like to see clarified for this paper to be accepted: the justification of the structural design. I have several questions/remarks regarding it:\n\n1. Equation 8 shows that q(β | X, d) actually depends on the categorical parameters of q(d|Z), rather than the value of d itself. As such, from a correctness perspective, the distribution is q(β | X, Z) (with weight sharing with q(d | Z)): it does not depend on the value of d. I suspect this choice was made to escape the computational cost of marginalizing the whole batch of variables d in the joint distribution for β, but this changes the meaning of the model.\n\n2. The analysis of what features get stored in β versus Z is not very insightful. The text of the paper present this as if it was obvious, but does not seem that obvious to me why \"beard\" is a global feature but \"hair\" is a local one. The same non-obviousness applies to all examples.\n\n3. I don't get why the training procedure is supposed to work. It empirically does to some extent, but there is no clear theoretical justification. If the same β is used for the whole batch, given the batch is randomly sampled, what part of the dynamic would indeed drive the model toward extracting some \"common\" features into this variable?\n\nFor this last point in particular, the proposed structure seems pretty close to an other one that would be I believe much more natural: have the global parameter β not shared across the entire batch, but depending on the class d the encoder assigned to the datapoint. Is that something that has been considered, and if yes why was it not satisfactory compared to the proposed model?\n\nI believe it is necessary to clearly address these points for the article to be accepted.\n\n--------\n\nSmall remarks & typos:\n\n- Just after equation 9, there is a ^-1 missing on the sigma in the text\n- the notation for the KL-divergence is inconsistent between equations 10 and 11 ( D_KL(...) vs KL(...) )\n- the text in 4.A about how the z feature is explored in figure 3 is not completely clear. Is the interpolation done only along the diagonal, moving from (μ1 - 3, μ2 - 3, ...) to (μ1 + 3, μ2 + 3, ...) ? If so, why ? If it is rather only moved along one particular dimension, then I suggest rewording this sentence to make it clear.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WtsEdbF_YS8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Response to rebuttal: the authors have drastically improved the quality of the submission with the new experiments and clarifications, I have therefore increased the score to a weak accept.\n\n-------------------------------------------\n\nThis paper introduces a non-iid VAE architecture that uses a mixture of gaussian latent space and a global latent variables shared among all the elements of a mini batch to capture global information in correlated datapoints in an unsupervised way.\n\nOverall the paper in well written, and I believe in focuses on two important research directions, namely unsupervised learning of disentangled representations and domain alignment. The model itself is novel and well explained, but I feel the technical explanation is missing intuition on how the model can learn disentanglement in beta from purely random batches, which is not obvious to me. \n\nMy biggest concern is in the experimental section, that I did not find convincing enough for a number of reasons:\n1. I find it hard to understand if the improvements come from the introduction of the d or the beta latent variables, or a combination of both. How does the model perform in ablation studies in which you remove just one of this components while leaving the others unchanged? \n2. In the single-datasets experiment in section 4.1 how do you define what constitutes a local vs global factors?  Currently some of the chosen factors in Figure 3 seem quite arbitrary. Why is light a local factor but contrast a global one? Why is hair local but beard global? \n3. The quality of the images is not great to be honest (beta-VAE paper has more convincing ones, just to name a single work), and it is not easy to understand whether the low quality results are due to the fact that as you say you have not validated in depth the networks used or because of flaws in the methodology\n4. How would a beta-VAE perform with the same setup of the experiment in 4.1? I would not be surprised if it could capture the same features as your model. It is true as you claim that your method does not require the tuning of the beta hyperparameter in the ELBO, but the UG-VAE needs tuning of the dimensionality of d and beta, and is a more complex architecture than a beta-VAE so it is harder to implement and will take longer to train.\n5. It is not clear to me in Figure 4.1. why you are traversing z space in this way, but perhaps I misunderstood what you are doing. How are you guaranteed that you will follow the data manifold? The ML-VAE results might be off just because of this. \n6. I believe the more exciting application of this model would be for domain alignment. Why haven't you focused on more multi-datasets experiments?\n7. How would a gm-vae baseline with 2 clusters perform with the same setup of the experiment in section 4.2? \n\nIn its current state I believe this paper is not ready for acceptance, but I hope the authors will be able to clarify some of my concerns in which case I will increase the score.\n\nMinor comment:\n* The second paragraph of the introduction is giving a lot of details on related work. I would recommend to move this discussion to the related work section, and leave the introduction for higher level discussions that only aim at giving intuition to the reader.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SL8-bibhplU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel deep generative model based on noni.i.d. variational autoencoders that captures global dependencies among observations in a fully unsupervised fashion. The proposed model combines a mixture model in the local or data-dependent space and a global Gaussian latent variable, which captures interpretable disentangled representations with no user-defined regularization in the evidence lower bound. The proposed model is being evaluated in two tasks: (1) disentanglement, and (2) domain alignment.\n\nPros:\n(1) The paper is very well-written and easy to understand. Especially the model figures (figure 1 and 2) are very intuitive and makes it much easier to understand the intuition and difference between the proposed model and previous related works.\n(2) The paper is trying to handle a very interesting task, which is to relax the assumption in a lot of previous works in VAE field. The i.i.d data assumption destroys the correlation between data points in the same dataset. Relaxing this constraint will enable wider adoption of this line of models, which can be very useful.\n\n\nCons:\nMy major concern on this paper is the quality of the evaluation section.\n(1) first of all, there is no quantitative or qualitative comparison between the proposed method and previous works. Although disentanglement is a task that is hard to quantify, it is hard to show that UG-VAE outperforms other methods.\n(2) One of the contribution mentioned is that UG-VAE does not need to tune a bete parameter as in beta-VAEs. But tuning a hyper-parameter for the disentanglement task is not necessarily a negative thing, as UG-VAE also needs to set the total number of lusters. When setting K= 10, it is inducing prior knowledge, and not completely unsupervised anymore.\n(3) It is not convincing that the global disentanglement features are learned in the global latent variables. From Figure 3, it just shows some dimensions control certain attributes. It is also not clearly discussed what is defined as \"global attributes\" and waht is \"local attributes\".\n(4) \"Composing graphical models with neural networks for structured representations and fast inference\" from NeurIPS 2016 also has a mixture model as latent space. It will provide readers more insight if the authors could discuss the similarities and differences between these two papers.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims at learning disentangled representation at different level without the supervision signal of group information. To achieve this, the proposed UG-VAE model uses both global variable $\beta$ to represent common information shared across all data, as well as a mixture of Gaussian prior for the local latent variable $p(z) = \int p(z|d)p(d)d$ where $d$ represents the assignment of the group for a particular datapoint. Experiments considered evaluation on unsupervised global factor learning, domain alignment and a downstream application task on batch classification.

Reviewers agreed that the proposed model seems interesting and novel, however some reviewers raised clarity concerns on how to interpret the learned representation by UG-VAE. Revision has addressed this clarity issue to some extent, although some doubts from some reviewers still exists. Also reviewers raised concerns on less competitive experimental results, and the authors have updated the manuscript with improved results. 

To me the main issues of the experimental section are (1) no quantitative result is provided regarding global factor learning and domain alignment, and (2) there is no other benchmark being studied in the experimental section. In my view, at least some other VAE representation learning baselines can be included in the batch classification section in order to demonstrate the real benefit of learning global factor based representations in downstream tasks. 
Decision: Reject