Isolating Latent Structure with Cross-population Variational Autoencoders
Authors: Joe Davison
Kristen A. Severson
Soumya Ghosh
Venue:  (2020)
Keywords: variational autoencoder
latent variable model
probabilistic graphical model
machine learning
deep learning
continual learning

Abstract: A significant body of recent work has examined variational autoencoders as a powerful approach for tasks which involve modeling the distribution of complex data such as images and text. In this work, we present a framework for modeling multiple data sets which come from differing distributions but which share some common latent structure. By incorporating architectural constraints and using a mutual information regularized form of the variational objective, our method successfully models differing data populations while explicitly encouraging the isolation of the shared and private latent factors. This enables our model to learn useful shared structure across similar tasks and to disentangle cross-population representations in a weakly supervised way. We demonstrate the utility of our method on several applications including image denoising, sub-group discovery, and continual learning.

Under review as a conference paper at ICLR 2020
ISOLATING LATENT STRUCTURE WITH CROSS-POPULATION VARIATIONAL AUTOENCODERS
Anonymous authors Paper under double-hlind review
ABSTRACT
A significant body of recent work has examined variational autoencoders as a powerful approach for tasks which involve modeling the distribution of complex data such as images and text. In this work, we present framework for modeling multiple data sets which come from differing distributions but which share some common latent structure. By incorporating architectural constraints and using a mutual information regularized form of the variational objective, our method suc- cessfully models differing data populations while explicitly encouraging the iso- lation of the shared and private latent factors. This enables model to learn useful shared structure across similar tasks and to disentangle pross-population representations in a weakly supervised way. We demonstrate the utility of our method on several applications including image denoising, sub-group discovery. and continual learning.
INTRODUCTION
Unsupervised learning of latent representations is widely used for dimensionality reduction, density estimation, and structure or sub-group discovery among other applications. Methods for recovering such representations typically rely on the assumption that the observed data a manifestation of only a limited number of factors of variation (Locatello et al. 2019 Bengio et al. (013). The variational
Welling 2013), a combination of a non-linear latent variable model and an amortized inference scheme (Dayan et al. 1995). is a popular method for recovering such latent structure. VAEs and their extensions have received considerable attention in recent years and have been shown useful for modeling text (Miao et al. 2016), images (Gulrajani al. 2016), and other data exhibiting complex correlations (Gómez- (Gómez-Bombarc Bombarelli et al. (0188.. However, barring a few exceptions (Bouchacourt al. 2018 Severson et al. 2019). this line of work has assumed the data to be independent and identically ly distributed.
autoencoder (VAF) (Kingma
In this work, we consider the task of modeling independent but not identically distributed data. We are particularly interested in analyzing data comprising two or more distinct but related sub- populations. Such data arise frequently in practice. For instance, patients suffering from an ailment may exhibit symptomatic heterogeneity based on gender or environmental factors, documents in corpus may exhibit semantic syntactic similarities based on genre or authorship, images may cluster depending on the image subject. Our goal is to provide rich descriptions of such data by re- covering latent representations that disentangle factors of variation common all populations from those that are unique to a particular population. Motivated by this challenge, we propose non-linear latent variable models that explicitly account for the heterogeneity in the data. Coupled with an inference procedure that encourages high (low) mutual information between the latent representa- tion and the data within (across) a population, we show that our models are indeed able to recover population-specinf representations that are salient and disentangled across differing populations as well as shared representations that isolate commonalities between populations.
Through careful experiments, vet the effectiveness of approach and demonstrate that the learned representations are useful for a diverse set of applications including image denoising, unsu- pervised sub-group discovery, and continual learning.

---

Under review as a conference paper at ICLR 2020
Φ
Rsi
4r
Z
0,
0
0
X,
Ii... m
((...)
Figure 1: Graphical model of CPVAE (left) vs. standard VAE (right)
2 METHOD
2.1
CROSS-POPULATION VARIATIONAL AUTOENCODER
We propose generative model of data instance X belonging population using two latent variables Zys and tkg. The latent variables are mapped to the observed space using non non-linear map- pings, So, (Z&z) and fo, (txz), each parameterized by a neural network. While we share the param- eters 0, among all populations, 0% are only shared among instances belonging to the population k. This construction encourages the model to capture common latent structure in 0,, while allowing 0x to focus on factors of variation unique to the particular population k. We combine the contributions from the two mappings using a function g. The generative procedure can summarized as,
Zis (O,
Ek NO,, Zkis thi ~ pla(So (z:), fer (tw))),, Vie (1.......4 m... vK E ,L....,K,
(1)
Xic
where p is an appropriately chosen distribution for modeling the observed data. In our experiments, additive function g(a, b) 6, and select p to be the Gaussian distribution with mean parameterized by g and an isotropic diagonal covariance matrix, V. We emphasize that other choices of function easily be incorporated into the model. Exploring the space of aggregation functions is planned future work.
2.2 AMORTIZED VARIATIONAL INFERENCE
Amoritized variational inference (Dayan et al. 1995 Gershman & Goodman 2014) with the aid
of reparameterized gradients (Rezende et al.) 2014 Kingma & Welling (013) is straightforward to implement for the model described in equation We assume that the variational approximation factorizes conditioned on the observation,
(Z t&s xki) = 9a. (Zss xxx)go
(twy Xi:).
(2)
We parameterize variational distribution with a single, shared inference network with two distinct outputs, one for each latent variable. Finally, the model and variational parameters jointly learned by optimizing the evidence lower bound (ELBO),
K
((0,0) =
Bys tu (i.)) [log p(xgi Zisi, tas; Os, Oz)
(3)
DxL. (go tae (pss))((Zsss tx:))
Importantly, in the above setup, the per-population model and variational parameters Og and du are learned only from the data in that population, allowing them to learn population specific representa- tions. We refer to this combination of the model described in equation I and the inference network defined above the cross-population variational nuto-encoder (CPVAE), owing to its similarity with the VAE (Kingma & Welling 2013). See Figure for a graphical representation of the CPVAE and comparison the VAE.
2

---

Under review as a conference paper at ICLR 2020
2.3 MUTUAL INFORMATION REGULARIZED INFERENCE
By employing population-specific mappings, CPVAE prevents latent factors of variation unique to one population from 'leaking' into the private representation of another population. However, the model structure does not prevent leakage between the shared and population specific representa- tions, since every data instance generated by combination of the shared and population specific representation. In fact, when CPVAE is trained by maximizing the ELBO in equation we find that the private representations of a population often exhibit latent features from the shared space and vice versa.
Inspired in part by the VAE (Zhao et al., 2017). , we discourage such inferences by minimizing the mutual information between population-specifie latent variables and data from non-corresponding populations while maximizing the mutual information within population.
Let X {xx1,.... XinyJ denote the set of all data instances belonging to population k. let ty. Zk be the set corresponding latent variables, and X- = {xy; & k K} denote the set of all non-corresponding populations. We learn the CPVAE by maximizing,
J(0, L) + Σ nh (xx(x - Ig Ix(x=xitx))
where ig ((tx) and Eje Yoh (tij Xjs) where Xji E x- i.e., the result of encoding members of the non non-corresponding populations, x- with the parameters on of the current popula- tion's 's inference network. This function can be simplified a tractable objective by re-writing the mutual information term:
Eyg,, (x) log 9on (6x(tx) (tx)
(x15 tx)
Toh (tx|/xx)
-- (x) log Jor (tx) (((x))
Tou (txlxx))p(t)
(5)
p(tx)
Eg (eo) log Joh (t&) p(ta))
-Epo(xa) Beo, (talxx)) log
Udi (tx(
DKL (gor (te. (p())x)))((((t))]]] DKL (goa (tx)llp(tt)).
11k:
where we assume 4o (Xk, tx) (xx (tx xx) and (xx) is the empirical distribution. Note that when we take Ig (x*; t²) ( th) in our objective, the intractable marginal KL term conveniently cancels out. The complete objective function is
J(0,6)=
Σ
Bg (2.) (log p(xes
Zkis Luv; 0s. 0k)]
DxL. (1o, (aes (Rss\xx:)llp(Zaa
K
n2
T
DKL (gou (Ezz1x3e) (le(ts;))
K
jyk
(6)
This objective differs from the standard variational objective only in that the divergence of the private latent prior for any population t is minimized for its contrasting populations x- and not its corresponding population XĄ. That is, we minimize DKL (9oz (fx|x=))] rather than DKL. (go, (t& xx (tt()))) As we will show, this modification reduces entanglement between pri- vate latent vectors and enables the model to learn more salient representations of each population. Moreover, since Kullback Leibler divergence is always non-negative, (0, σ) in equation remains lower bound to the marginal likelihood.
In some scenarios, it may desirable to increase the importance the cross mutual information term (xxx; tx) to our objective, further discouraging shared features from leaking into population- specific representations. To this end, we can simply add a scaling constant a 2 to the final term in equation 6 In experiments, we often find substantially improved results by beginning
3

---

Under review as a conference paper at ICLR 2020
with 1 and gradually annealing its value over training. A summary of the complete training procedure can be found in Algorithm
Algorithm 1 Training procedure of CPVAE
Initialize conditional parameters {0.] (B): Vk E K]:
Initialize variational parameters & = {$.) U {Øx; vk K};
repeat
Sample mini-batch from each population {{xM)) iiiik= for k 1.... K do
Sample shared codes Zς qo. (zy xxs); Sample private codes t; ~ 96n (tx (x)); for j k E 1.... K do
Sample fictitious codes ij- qon (t3 xx);;
end end
Calculate ((
()) as in (6)
Update gi+1, , <- 0", according to ascending gradient estimate of J(0, 6):
until convergence;
RELATED WORK
Substantial recent work has explored methods for the unsupervised learning of disentangled repre- sentations. Though lacking a formal definition, the key idea behind disentangled representation is that it should separate distinct informative factors of variation of the data (Bengio et al., 2013 Lo- catello et al. 2019)) Several techniques have been proposed to encourage VAES to learn di disentangles latent representations. Some examples are 3 3-VAE (Higgins et af.. 2017), Annealed (Burgess et al. 2018), FactorVAE (Kim & Mnih 2018). 3-TCVAE (Chen al.) 2018), and DIP-VAE (Kumar et al. 2017). each of which proposes some variant of the VAE objective to encourage the variational distribution be factorizble. Recently, it has been proposed that is not possible recover dis- entangled features without inductive bias or supervision (Locatello et al.) (201)). CPVAE makes no assumptions about the composition of variational factors but imposes a form of weak supervision where population assignment is known and uses that information in choosing the model architecture and learning algorithm.
A few other techniques have been proposed to use weak supervision for learning improved represen- tations. Multi-study factor analysis (De Vito et al. 201 b) has similar aims and structure as com- pared to CPVAE but uses linear model and focuses on applications related to high-throughput bio- logical assay data. Contrastive latent variable models (Severson et al. 2019 Abid & Zou 2019) have non-linear variants but focus the case where dataset is the target to be compared/contract to another dataset. Multi-level variational autoencoders (AEs) have been proposed as a way incorporate group-level data in unsupervised learning (Bouchacourt al. 2018). After dividing data into disjoint groups according to some factor of interest, the MI-VAE framework models latent structure both the level of individual observations and of entire groups. This method effectively separates latent representations into semantically relevant parts, but differs from the CPVAE frame- work which models both private and shared structure at the observation level. nutput-interpretabble VAES (oi-VA VAEs) have been proposed to leverage data that can be partitioned into within sample groups in an interpretable model (Ainsworth et al.) (2188). The model structure is such that the components within each group are modeled with separate generative networks. Mappings from the latent representations to each of the groups are encouraged to be sparse via hierarchical Bayesian priors to improve interpretability. The primary difference between oi- VAE and CPVAE is that oi- VAE uses groupings over components, X; E _ X; = [x,...... X,K] and D = {x, ) and CPVAE uses groupings over instances, X5; Rdd and D = {xx,) 1K There has also been recent work in learning disentangled representations from sequential data (Hsu et al. 2017 Li & Mandi 2018). These models share a representation across all elements of the sequence to learn global sequence dynamics while local aspects modeled via time step specific representations. This problem is somewhere between the standard disentangled representation learning, where the goal is to learn disentangled features with prior knowledge, and weak supervision the sequential nature of the
4

---

Under review as a conference paper at ICLR 2020
9
8
8
9
7
8
8
8
Figure 2: Grassy-MNIST reconstructions with two populations: digits 0-4 and digits 5-9. First row: original images. Second row: full reconstructions. Third row: private space (digit-only) reconstructions. Fourth row: shared space (background only) reconstructions. Note that this model has never seen the original digits or the original backgrounds.
Method
SHARED ARI PRIVATE ARI
VAE
.1881
CPVAE, NO MI CPVAE, a
.1926 .0006
.0031
.2281
Adjusted rand index (ARI) for discovered clusters in the shared vs. private latent spaces.
Table 1:
data inherently provides some structure. Our work presented here focuses on non-sequential data. Moreover, unlike us these works do not employ mutual information regularized inference, which we find be crucial for recovering disentangled shared and private representations.
EXPERIMENTS
In order to determine the effectiveness of our approach as well as demonstrate several possible applications, we evaluate it with number of experiments on tasks including image denoising, sub- group discovery, classification, and continual learning.
Each experiment employs the following setup: the encoder and decoder are strided convolutional neural networks with ReLU activations and batch normalization after each layer. The decoder em- ploys residual skip connections which have been shown to help prevent posterior collapse in VAEs (Dieng et al. 2018). We optimize using Adam (Kingma & Ba 2014) with a base learning rate of 0.001.
4.1 DENOISED GENERATIVE MODELING APPLIED TO GRASSY-MNIST
We evaluate our model on a synthetic dataset of handwritten digits from the MNIST (LeCun et al.
1998) superimposed on grassy backgrounds from ImageNet (Russakovsky et al. 2015) (see Fi Figure 2 top row for example images). In Abid et al.] (2018); (everssn et al. (2019); Abidd & Zou (011), the authors train contrastive models on this dataset along with the original grass images in order to learn more salient latent representations of the digits.
In our experiment, we split this synthetic data into two populations consisting of digits 0-4 and digits 5-9, We use shared latent dimension of 100 and population-speciss latent size of 25. We show that our model is able to effectively separate the complex background representations in the shared latent space from that of the digits in each private space. Crucially, the model is never shown either the original background grass images nor the original non-noisy digits.
A sample of reconstructed images can be seen in Figure The first and second rows show the original digits with noise and the full CPVAE reconstructions, respectively. The third and fourth show the reconstructions when the shared or the private latent spaces ignored, respectively.
5

---

Under review as a conference paper at ICLR 2020
S-NE of GpvAE digits O-4
L-SNE of VAE digit
40
-00
-40
-20
20
40
Figure 3: Visualization of subgroups within MNIST digit population after t-SNE projection of latent space for our method (left) vs standard VAE (right). Each color represents one of the five digits within the population.
TRAPE
Figure 4: CPVAE reconstructions of CelebA dataset images. Top row: original images, Second row: reconstructions with male and shared space decoders. Third row: reconstructions with non mane and shared space decoders. Bottom row: shared decoder only.
These results qualitatively demonstrate CPVAE's ability to separate the shared and private feature representations.
As evaluation of the salience of the private space representations, also measure the ability of our model to do sub-group discovery within each population. K-means clustering is applied to the private space latent representations and compared to the true class labels using the adjusted rand index (ARD, which measures the correspondence of the cluster assignments with the true labels (Menezes & Roth 2017). The results are Table [ and TSNE visualization of one of the private latent spaces for this task compared with that of standard VAE is in Figure 3 Our model outper- forms the standard VAF, demonstrating the ability our model to learn improved representations by incorporating proslaponn structure. Incorporating the MI objective (a 1) further improves ART scores in the private space and worsens scores in the shared space, suggesting that the MI term effectively mitigates the leakage of population-specific features in the shared space.
4.2 DISENTANGLING LABELED ATTRIBUTES IN CELEBRITY IMAGES
CPVAE allows us to model data such that latent structure which corresponds to some labeled at- tribute of interest can be isolated. To demonstrate this, perform experiment the Large-scale CelebFaces Attributes (CelebA) dataset (Liu et al. 2015). This data consists of 202,599 aligned and cropped pictures of celebrities with 40 binary attributes labeled for each image. See Figure

---

Under review as a conference paper at ICLR 2020
DATASET
ANNEAL a
a
No MI
RESNET50
MNIST
99.5
98.2
97.3
99.6
MNIST-GRASS
77.5
75.1
70.5
85.7
CIFAR-10
76.2
47.9
42.2
84.1
Table 2: Maximum likelihood classification test accuracies (%) for our model with and without mutual information terms evaluated on different datasets. For reference, we also include accuracies from ResNet50 classifier.
We train CPVAE model on this dataset with two populations determined by the male attribute label. Under this setup, the model is incentivized to learn representations of gender-specific features the private latent spaces while the shared space infers the remaining factors of variation. As a qualitative evaluation of this model, we autoencode sample of images with each private space and examine the resulting reconstructions. By reconstructing an image with the non-corresponding pop- ulation's 's decoder, we get reconstructions that closely resemble the original image but with features that appear traditionally male when constructed from the male space and female when constructed from the non-male space. See Figure 4 for sample of these results. This serves as additional evidence of our model's ability to separate population-specific features from shared latent structure.
4.3 MAXIMUM MARGINAL LIKELIHOOD CLASSIFICATION
As an additional evaluation of our model's ability to learn disentangled population-specte repre- sentations, we test our model's ability to classify unseen data points into their corresponding popu- lations. We assign an instance x. to the population that maximizes its marginal likelihood,
k = arg max maxp xx 02 0.) = arg max E (pizentte)) [p(x,, Zgs, tki 0x. 0z)]
KEK
KEK
We use importance sampling compute the intractable expectation,
p(zke this
E Ep(xanttee [p(x,
Zikis thii Ok, 0z)] tulle
p(x. Zkii tui Bz 0,)
Tob (Zics tui x.)
(8)
In order achieve a high accuracy, the model must learn features in each population-specific latent space which are unique to its corresponding set. We therefore evaluate our model's classification per- formance on several labeled image datasets of varying difficulty MNIST, CIFAR-10 (Krizhevsky ett al.) 2209)) and the Grassy MNIST described in experiments above. In each case, we define distinct population for each class and evaluate its performance on a held-out that our goal with this experiment is not to demonstrate tfite-of-hhe-eee art classification performance, but provide convenient, quantitative benchmark for evaluating the quality of the model's 's learned representations. For reference, also provide classification accuracies from a ResNet-50 convolu- tional model (He et al. 2015 LeCun et al.) 2015) trained by maximizing pkk: x.) for five hundred contextual epochs. Note that these CNN scores are not the state of the art for each task, but serve reference point for understanding our model's performance. We compare this against our CPVAE models both with and without the mutual information regularization as well as variant where the weight on the mutual information term, a, is gradually annealed over training. Additional details about model architectures and the annealing scheme can be found in the supplement.
set. We emphasize
The results can be seen in Table 2 Our model approaches the performance the convolutional neural network despite not being trained directly maximize classification performance. We find that performance improves by utilizing our mutual information regularized objective, particularly when a is annealed over training. This result provides compelling evidence that our model is able to effectively learn private space representations which are unique to each corresponding population.
4.4 CONTINUAL LEARNING
Neural networks are widely known to be susceptible to a phenomenon known as catastrophic for- getting (McCloskey & Cohen 1989), wherein model's performance previously learned task
7

---

Under review as a conference paper at ICLR 2020
Sunflowerr
Daisy
Tasks
Tasks
Iris
Task
Task
Daffodil
Task
Task
Pansy
Images
CPVAE Reconstructions
Task 5
VAE Reconstructions
Task 5
Figure 5: Image reconstructions for CPVAE vs VAE in continual learning setting. Each column shows the reconstructions for the labeled task as well as previous tasks, demonstrating CPVAE's ability to retain previously learned information by utilizing both population-spedte and shared la- tent vectors.
degrades rapidly as tasks encountered. We show the merit of our method in mitigating this phenomenon in the context variational autotencoders. By modeling sequentially arriving tasks as distinct populations,
method models the structure unique each task in the population-specific latent spaces and learns general information useful to all tasks in the shared space.
We perform experiment the 17 Category Flower dataset (Nilsback & Zisserman 2006). We evaluate our model's ability to reconstruct images from five sequentially-arriv sunflower, daisy, iris, daffodil, and pansy. Each category is assigned its own population and train- ing proceeds in two phases: first, tasks and are learned together using the mutual information regularized objective. This step allows the model to infer structure that shared between the two tasks that can then be utilized for learning later tasks. For example, the shared space may learn to represent backgrounds while the private spaces learn to represent the corresponding flowers. Next, each remaining task is learned in sequence with the evidence lower bound.
image categories:
The results can be seen Figure 5 where we show the reconstructions for an image from each task throughout the training procedure for CPVAE in comparison VAE. In contrast to the VAE whose reconstructions are profoundly impacted by introduction new tasks, the reconstructions from our model are relatively unaffected.
CONCLUSION
In this work, presented framework for using a VAE-like architecture model multiple sets of data which are independent but come from differing distributions. We developed an architec- ture which encourages the isolation of shared and private latent factors, and presented mutual information regularized version of the evidence lower bound which discourages entanglement of the shared and oppulaiin-sppecifi latent vectors. Our experiments on the Grassy MNIST dataset demonstrated our model's ability to learn more salient representations and to effectively separate the shared and private latent factors on the task of image denoising. We also showed the effectiveness of our regularized objective in learning population-specific representations on several image classi- fication tasks. Lastly, we demonstrated the value of learning population-specific representations in the context of continual learning where method retains performance previously encountered tasks in comparison to a VAE.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
A. Abid, M. J. Zhang, V. K. Bagaria, and J. Zou. Exploring patterns enriched in a dataset with contrastive principal component analysis. Nature Communications, 9:2134, 2018.
Abubakar Abid and James Zou. Contrastive variational autoencoder enhances salient features.
CoRR, 100/1992.04601, 2019.
04601
K. Ainsworth, Foti, A. K. Lee, and E. B. Fox. oi-vae: Output interpretable vaes for nonlinear group factor analysis. In ICML,, 2018.
Y. Bengio, A. Courvile, and P. Vincent. Representation learning: A review and new perspective. IEEE Transactions on Puttern Analysis and Machine Intelligence, 35:1798-1828, 2013.
D. Bouchacourt, R. Tomioka, and S. Nowozin. Multi-level variational autoencoder: Learning dis- entangled representations from grouped observations. In AAAI, 2018.
C. P. Burgess, I. Higgins, A. Pal, L. Matthey, N. Watters, G. Desjardins, and A. Lerchner. Under- standing disentangling in 8-vae. arXiv preprint arXiv: 00099034000999 2018.
T. Q. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. In NeurlPS, 2018.
Dayan, G. Hinton, M. Neal, and S. Zemel. The helmholtz machine. Neural Computation, 5:889-904, 1995.
R. De Vito, R. Bellio, L. Trippa, and G. Parmigiani. arXiv:6611.0635013 2018a.
Multi-study factor analysis.
R. De Vito, R. Bellio, L. Trippa, and G. Parmigiani. Bayesian multi-study factor analysis for high- throughput biological data.
2018b..
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. arXiv preprint arXiv: 04863,, 2018.
S. Gershman and N. Goodman. Amoritized inference in probabilistic reasoning. In Annual Meeting of the Cognitive Science Sociery, 2014.
Rafael
Jennifer Wei, David Duvenaud, José Miguel Hemnáneez-Lobtoo Benjamín Sánchez-Lengeling Dennis Sheberla, Jorge Ryan Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven contin- representation of molecules. ACS central science, 4(2):268 276, 2018.
Timothy D Hirzel,
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. arXiv preprint arXiv:1611.05013. 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR, abss151120035 2015. URL URt.http://axxxv.oor http://aaxxiv..o/bbbbbbbse www. httpa/ab///1111.333 03385
I Higgins, I. Matthey, A. Pal, Burgess, Glorot, M. Botvinick, Mohamed, and Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
W.-N. Hsu, Y. Zhang, and J. Glass. Unsupervised learning of disentangled and interpretable repre- sentations from sequential data. In NIPS, 2017.
H. Kim and A. Mnih. Disentangling by factorising. In ICML, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /tee:///2..90 2,800, 2014.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.61 2013.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
A. Kumar, P. Sattigeri, and A. Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In ICLR, 2017.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. In IEEE, pp. 2278-23244 2324, 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((553):436, 2015.
Y. Li and S. Mandt. Disentangled sequential autoencoder. In ICML, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
F. Locatello, S. Bauer, M. Lucie, G. Rätsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions the unsupervised learning of disentangled representations. In ICMI., 2019.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Telmo Menezes and Camille Roth. Natural scales in geographical patterns. CoRR, abs/1704.010336, 2017. TRLbttp:: //apiv////xvvv..//a arxiv. org/ abs 1704. /tt.a//1..1..11..... 01036
Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In Interna- fional conference on machine learning, pp. 1727-1736, 2016.
M-E. Nilsback and A. Zisserman. A visual vocabulary for flower classification. In Proceedings of the IEEE Conference Computer Vision and Patter Recognition, volume 2, pp. 1447-1454, 2006.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint a 2014.
O. Russakovsky, Deng, Su, J. Krause, S. Satheesh, Ma, Z. Huang, A. Karpathy, Ag Khosla, M. Bernstein, A. C. Berg, and F.-F. Li. Imagenet large scale visual recognition challenge. Inter- national Journal of Computer Vision, 115:211-252, 2015.
Kristen A Severson, Soumya Ghosh, and Kenney Ng. Unsupervised learning with contrastive latent variable models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 4862-4869, 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational au- toencoders. CoRR,
2017. URL
02262
10

---

Under review as a conference paper at ICLR 2020
Appendices
A
GRASSY MNIST GENERATIVE SAMPLES
3
3
5
3
3
5
6
6
6
7
8
8
8
Figure 6: Generative samples from CPVAE trained on Grassy MNIST. Each row is from a population corresponding to different digit.
0
0
0
O
/
/
/
/
6
3
1
6
6
2
2
2
2
2
2
7
5
3
2
2
3
3
3
3
2
7
3
3
3
9
3
3
3
4
7
4
4
4
4
7
4
4
4
4
4
4
4
9
5
5
5
S
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
3
6
5
C
6
6
7
7
7
7
2
7
7
7
7
7
7
7
7
7
7
8
8 9
8 8
6
8
6
8
8
8
8
8
8
8
9
9
9
9
9
7
9
9
9
7
9
9
9
Figure 7: Generative samples from the private space of CPVAE trained on Grassy MNIST. Each row from population corresponding to different digit.
11

---

Under review as a conference paper at ICLR 2020
B EXPERIMENT DETAILS
B.1 MAXIMUM MARGINAL LIKLIHOOD CLASSIFICATION
The weight on the mutual information term annealed according to a(e) min(1.1 1000) where is the training epoch number. For MNIST, the shared and private latent dimensions are and 5, respectively. For CIFAR-10, the latent dimensions are 10 and 20. For Grassy MNIST, they are 30 and 5. We augment each set by randomly translating the images by up to 10% in each direction during training which we find to help reduce overfitting and increase test performance.
B.2 CONTINUAL LEARNING
The continual learning experiment uses a total of K populations, one for each task. The dimensions of the latent vectors are 10 for the shared space and 10 for each private space. The VAE which we use for comparison has a latent size of 60 that the total latent size of each model the same. During the first training phase we use base learning rate of 0.001, and lower it to 0.0001 during tasks 3-5.
12


Reviews:
[{"review_id": "r1lMN7j9FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studied the problem of learning the latent representation from a complex data set which followed the independent but not identically distributions. The main contributions of this paper are to explicitly learn the commonly shared and private latent factors for different data populations in a unified VAE framework, and propose a mutual information regularized inference in order to avoid the “leaking” induced by the shared representations across different populations. The isolation of the commonly shared and population specific latent representations learned by the proposed are empirically demonstrated on several applications. However, I have some concerns regarding this paper as follows.\n(1) It is not clear why the private representation exhibits latent features from the shared space when using equation 3 and how this phenomenon hurts this CPVAE model.\n(2) In equation 1, how to define the isotropic diagonal covariance matrix in the Gaussian distribution p? Is it parameterized by g?\n(3) In equation 3, what is the prior distribution of p(z_ki, t_ki)?\n(4) In equation (4)(5), why could the marginal KL term be canceled out when using I_q(x_k; t_k) - I_q(x_-k; \\tilde{t}_k)?\n(5) The mutual information regularized inference involved the KL term between any two private factors from different populations. It might be not efficient for optimization. Thus, it will be helpful if the authors provide the model efficiency analysis compared with other baseline methods.\n\nMinor comments:\n(1) what is the symbol “n_k”? Did it denote the number of examples for the k-th population?\n(2) For mutual information regularized inference, it used two different notations: “I_q(x_k; t_k) - I_q(x_-k; t_k)” and “I_q(x^k; t^k) - I_q(x^-k; t^k)”.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJxnlUtN9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to model multiple datasets from differing distributions with shared latent structure and private latent factors. The main techniques include architecture design which encourages the isolation of shared and private latent factors and a mutual information-based regularizer. The paper is clearly written and easy to follow. I enjoyed reading it. The experiments support the claim of learned population-specific representations.\n\nHowever, I found that the paper has some weaknesses:\n1. The novelty is not enough. All the techniques involved in the paper are not new but from existing literature. The idea is not new. The authors also mentioned several previous works in Section 3, e.g. Multi-level  VAEs, oi-VAEs.\n\n2. More importantly, I did not see any baselines in the experiments except vanilla VAE. As far as I understand, previous methods can be easily adapted to these tasks. For example, [1] tried continual generative modeling for a sequence of distinct distributions. Many important baselines are missing in the experiments, which makes it hard for me to evaluate how significant the work is.\n\n3. What if the populations are not exclusive? The regularizer enforces them to be isolated but they are not in fact.\n\n4. How did you choose the annealing schedule of $\\alpha$ in Section B.1?\n\nMinor:\n\npage 2  eq (1) z_i -> z_{ki}\n\npage 3 last paragraph “it may desirable”\n\nReferences:\n\n[1] https://arxiv.org/pdf/1705.08395.pdf\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg6uOEatH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a novel model called \"Cross-Population Variational Autoencoder (CPVAE), which is designed to model data from different distributions sharing some common structure. The proposed generative model utilizes both shared and private per-population latent variables. In order to restrict shared latent variables from \"leaking\" into private representations, the authors introduce an information-theoretic regularizer. This regularizer forces private population representations to: (a) maximize mutual information with input samples from their population, and (b) minimize mutual information with input samples from other populations. In other words, private representations are forced to be \"meaningful\" on the corresponding population alone.\n\nQuality:\nThe paper is well-written. I find the proposed method to be quite interesting. The derivations appear to be correct except for possibly the cancellation in Eq. 5, which I originally missed.\n\nSignificance:\nIn my opinion, if sound, the approach discussed in this paper may lead to interesting practical applications and may inspire other methods based on similar ideas.\n\nOriginality:\nEven though, as authors point out, there is a substantial amount of work in this field, I believe that their approach is novel and has its own merits.\n\nClarity:\nThe paper is well written and the material is presented with clarity. In my opinion, the only exception is Section 4.4, which could definitely benefit from a few additional sentences describing the training procedure in more detail. Right now I find it a bit confusing. It would appear that the shared encoder / decoder continue to be trained as new populations arrive. Would this mean that catastrophic forgetting can actually impact this shared representation? And if it changes by a sufficient degree, can it reduce the quality of the generated samples for older populations? If so, I think these points should be mentioned in the text.\n\nQuestions and suggestions:\nExperiments described in the paper are sufficiently convincing, but there are a few questions that could potentially be better clarified in the paper.\n\n1. After reviewing the text again and seeing the comment of Reviewer #1, I am also confused about the cancellation in  I_q(x_k; t_k) - I_q(x_{-k}; \\tilde{t}_k). Is it not true that marginal distributions q_\\phi(t) and q_\\phi(\\tilde{t}) in the KL divergence term in Eq. 5 are different? Unfortunately, the final optimization objective relies on the cancellation of these terms and if they do not cancel, the approach may not be theoretically justified despite producing interesting and compelling results. (This affected the final rating. I will be able to change the rating once this point is clarified.)\n\n2. Another issue is related to the special case when there are several very similar populations. Consider, for example, a case when there are two nearly-identical populations out of many. Using very similar latent variables for two similar populations would be penalized by the regularizer (not too significantly though). I assume that depending on the embedding sizes and the value of alpha (which authors introduce in Section 2.3), the model would either choose to use shared latent variables to encode these populations, or would allow for two nearly-identical private latent representations to exist. I think this is a conceptually important special case that could be mentioned and possibly explained in the paper.\n\n3. I think the paper would also benefit from a clarification regarding the \"mixing\" function g. Choosing this function to be a simple sum of arguments seems restricting and may be insufficient for some datasets. It does not appear to be the case, but are there any restrictions on g? Can it come from a parametrized function family with parameters being optimized during training?\n\n4. I think the paper would benefit from a more detailed discussion in Section 4.4 (see above).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a hierarchical Bayesian model over multiple data sets that                                                      
has both data set specific as well as shared parameters.                                                                           
The data set specific parameters are further encouraged to only capture aspects                                                    
that vary across data sets by an addition mutual information contribution to the                                                   
training loss.                                                                                                                     
The proposed method is compared to standard VAEs on multiple data sets.                                                            
                                                                                                                                   
The reviewers agree that the main approach of the paper is sensible. However,                                                      
concerns were raised about general novelty, about the theoretical justification                                                    
for the proposed loss function and about the lack of non-trivial baselines.                                                         
The authors' rebuttal did not manage to full address these points.                                                                 
                                                                                                                                   
Based on the reviews and my own reading, I think this paper is slightly                                                            
below acceptance threshold.
Decision: Reject