Wyner VAE: A Variational Autoencoder with Succinct Common Representation Learning
Authors: J. Jon Ryu
Yoojin Choi
Young-Han Kim
Mostafa El-Khamy
Jungwon Lee
Venue:  (2020)
Keywords: Wyner's common information
information theoretic regularization
information bottleneck
representation learning
generative models
conditional generation
joint generation
style transfer
variational autoencoders

Abstract: A new variational autoencoder (VAE) model is proposed that learns a succinct common representation of two correlated data variables for conditional and joint generation tasks. The proposed Wyner VAE model is based on two information theoretic problems---distributed simulation and channel synthesis---in which Wyner's common information arises as the fundamental limit of the succinctness of the common representation. The Wyner VAE decomposes a pair of correlated data variables into their common representation (e.g., a shared concept) and  local representations that capture the remaining randomness (e.g., texture and style) in respective data variables by imposing the mutual information between the data variables and the common representation as a regularization term. The utility of the proposed approach is demonstrated through experiments for joint and conditional generation with and without style control using synthetic data and real images. Experimental results show that learning a succinct common representation achieves better generative performance and that the proposed model outperforms existing VAE variants and the variational information bottleneck method.

Under review as a conference paper at ICLR 2020
WYNER VAE: A VARIATIONAL AUTOENCODER WITH SUCCINCT COMMON REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A new variational autoencoder (VAE) model is proposed that learns a succinet common representation of two correlated data variables for conditional and joint generation tasks. The proposed Wyner VAE model is based on two information the- oretic coblems--dsstiburee simulation and channel synthesis-in which Wyner's common information arises as the fundamental limit of the succinctness of the common representation. The Wyner VAE decomposes a pair of correlated data variables into their common representation (e.g., shared concept) and local rep- resentations that capture the remaining randomness (e.g., texture and style) in respective data variables by imposing the mutual information between the data variables and the common representation as regularization term. The utility of the proposed approach is demonstrated through experiments for joint and conditional generation with and without style control using synthetic data and real images. Experimental results show that learning succinet common representation achieves better generative performance and that the proposed model outperforms existing VAE variants and the variational information bottleneck method.
INTRODUCTION
This paper aims to develop a new probabilistic framework for generation tasks (i.e., learning the distribution of given data and sampling from the learned distributions) for two high-dimensional random vectors. To motivate the main idea, consider the following cooperative game between Alice and Bob. Suppose that given an image of child's photo, Alice sends its description Z to Bob who draws a portrait of how the child will grow up based on it. The objective of this game is to draw a nice portrait, and thus Alice needs to help Bob in the process by providing a good description of the child's s photo photo-aa any redundant information the description may confuse Bob in his guessing process. What description does Alice need to generate and send from the child's photo?
Cuff (013) formulated this game of conditional generation as the channel synthesis problem in network information theory depicted in Fig. [ Given joint distribution q(x, q( ((x)((y (x), Alice and Bob want to generate Y according to g xx) based on a sample from xx)). In this problem, Alice wishes find the most succinct description of X child's photo) such that Y (her adulthood portrait) can be simulated by Bob according to the desired distribution using this description and local randomness V (new features to draw portrait of adults that are not contained in photos of children). The minimum description rate for such conditional generation is characterized by Wyner's common information (Wyner 1975 El Gamal and Kim 2011) denoted by (X; Y) and defined as the optimal value of the optimization problem
minimize subject to variables
1x(X, Y; Z) X Z Y q(z|x,y),
where X + Z Y denotes Markov chain from X to Z to Y and I,(X, Y;Z) denotes the mutual information between (X, Y) and Z, where (X, Y, Z) ~ q(x, (z(x),
The same quantity J(X; Y) arises as the fundamental limit of the distributed simulation of correlated sources studied originally by A. Wyner (1975) in which two distributed agents wish to simulate a target distribution q(x, y) (i.e., joint generation of (X, Y)) based on the least possible amount of

---

Under review as a conference paper at ICLR 2020
Encoder M(*))
Message
Decoder y"(m.
Eocoder
Decoder V(a,
M (l 2n1)
Z
(a) Channel synthesis
(b) Single-letter characterization (a)
U Decoder u²))
U
Decinder x(R, w)
X
Message
M E (1.... 2
Decoder (mm
Decoder y(z. v)
Y
V
(c) Distributed simulation
(d) Single-letter characterization of (c)
Figure 1:
Schematics for channel synthesis from X to Y (a,b), and distributed simulation of (X, Y) (c,d). (a,c) and (b,d) correspond to the operational definition and the single-letter characterization of each problem, respectively. The local randomness U and V make the decoders stochastic.
shared common randomness. (See Fig. I (c.d).) In this sense, the joint distribution q(x, y) and the conditional distributions q(y[x), (xxly) have the same common information structure characterized by the optimization problem (.) which involves learning the joint distribution in its nature. We call the joint encoder g(z|x. y) (or equivalently, the corresponding random variable Z) as the common representation of (X, Y), and the mutual information I (X, Y; Z) then can be viewed measure of the complexity of Z.
The goal of this paper is to propose new probabilistic model based on these information theoretic observations, to achieve a good performance in joint and conditional generation tasks. We apply the idea of learning succinct common representation design new generative model for pair of correlated variables: seeking a succinct representation Z in learning the underlying distribution based on its sample may also help reduce the burden on the decoder's 's side and thereby achieve better generative performance.
The rest of the paper gradually develops our framework as follows. We first define probabilistic model based on the motivating problems, which we aim to train and use for generation tasks (Section 2.T). and then establish general principle for learning the model based on the optimization problem (Section 2.2). We propose one instantiation of the principle with standard variational technique by introducing additional encoder distributions (Section 2.3). The proposed model with its training method can be viewed
variant of variational autoencoders (VAEs) (Kingma and Welling 2014 Rezende et al.) 2014), and is thus called Wyner VAE. (See Appendix A for brief introduction on VAEs.) The encoder components introduced in Wyner VAE allow to decompose data vector into the common representation and the local representation, which can be used for sampling with style manipulation (Section 2.4). We carefully compare our model and show its advantages over the existing VAF variants /Vedantam et al. 2018 Suzuki et al. [016]] Sohn et al.) 2013 Wang al. 2016) and the information bottleneck (IB) principle (Tishby et al. 1999) (Section 3 which is a well-known information theoretic principle in representation learning. In the experiments, we
empirically show the utility of our model various sampling tasks and its superiority over existing models and that learning a succinct common representation achieves better generative performance in generation tasks (Section 4).
WYNER VARIATIONAL AUTOENCODER
2.1 PROBABILISTIC MODEL
We first define a probabilistic model for joint and conditional sampling tasks based on the single-letter characterizations of the motivating problems (Fig. (b.d). We assume that all distributions to be
2

---

Under review as a conference paper at ICLR 2020
introduced below belong some standard parametric families such Gaussians, and use (q, (or (p, 8)) to denote the parameters and the distribution of encoders (or decoders and priors).
In both channel synthesis and distributed simulation, Z ~ po(z) signifies the common randomness fed into the deterministic decoders Xy (z, u) and Yo(z, v), while U Po (u) and V ~ po(v) signify the local randomnesses for each decoder. We take the sources of randomness (or priors) pa(z), Po (u), Pv(v) as standard parametric distributions such as Gaussian from which sample can be easily drawn. U (or V) is interpreted as a local description of X (or Y) given the common description Z. We can also view U (or V) as an intrinsic randomness of the stochastic decoder Po(x|z) (or po ( (z).. which is the induced distribution by the decoder x (z. u) (or yo v)) and the prior (u(u) (or pa (v)). To perform joint sampling, we need the priors pa (z(.), Pa (u), (v(v) and the decoders (z, u) and Yo (z, v) (Fig. (d). For conditional sampling of Y given X x, we need the marginal encoder q (2 22xx) to be defined in addition to the prior (v and the decoder (z, v) (Fig. (b)). We focus on how to learn 9s (2)xx) in what follows, since 96 (zly) can be dealt with the same way by the symmetry of the Markov chain X + Z -..
After Wyner, we name the entirety of the components (the marginal encoders qo (z]x), qo ()) the priors pe (z), po()), po (v), and the decoders pe (x|2,, u), conditional sampling tasks as the Wyner common representation model or the Wyner model in short.
Pe (y' 2. v)) that is essential for joint and
2.2 OBJECTIVE FUNCTION
We now propose objective function for learning each component in the Wyner model based on the optimization problem (I. In practice, the data distribution q(x, y) ) is replaced by the empirical distribution gemp (x. defined by the given samples. To train the components for joint sampling in the Wyner model, we solve the optimization problem by incorporating a new consistency constraint
g(x, (gsz2x)/()x
|z):
with the priors po(z)) (z), (u)),, po(v) and the decoders (z, u), ye(z, v) new optimization variables.
The Markov condition X + - Y can be removed under the new constraint (2) We then relax the constraint ( with inequality constraint
Lxy-xy
(z(poz))pll (z)pe(x) (xp(xxz))p(()))) (y) (z),
e,
for some > 0, and convert the problem (1) into an unconstrained form as in Zhao et al. (018)):
minimize
Lxy-xx + Alg
variables
2 (o x, y), po Po (u). po (v), xo(z, u), yo(z.)
Here we use a shorthand notation I, := IX,,, ; Z), and use À > 0 to denote the reciprocal of the Lagrange multiplier. We can choose D(p, g) as any proper distance or divergence measure between distributions such as f-divergencess
divergence, Wasserstein distance, or maximum
mean discrepancy (Zhao et al.[ (2018).
To find the marginal encoder gz(z)x) that is consistent to q(y ((y/x)]a(z]x; y) for conditional gen- eration, aim to match a joint distribution induced by the Markov chain X Z - Y. i.e., q(x)qs pe (y (y),, with the decoder distribution po (z) (x (x (ylz) and the encoder distribu- tions (x, y 19 (2|xx)))) y) of the joint model. That is, wish to find ga (z(z(x) that minimizes
Lx (x)
(4)
and
Cxy-ty := ((g(()gg
q(), (2)
(5)
The objective functions Ly-y and -xy x for q0 (zly) can be defined in a symmetric manner. The final objective function for training the Wyner model with succinet common representation learning then becomes
L : Lxy-xx + Al, + Xx-xxxx-x + -y + ay Ly-ry + Oxy -xyy
(6)
where the weights a' 's are nonnegative hyperparameters.
Yet, to minimize the objective function in practice, need choose which divergence/distance metric ((p, 4) to use, and also need to address computationally intractable terms the induced
distributions Pol (x 12).pr Po (z)z) and the mutual information term In-in 3 and (5)
3

---

Under review as a conference paper at ICLR 2020
2.3 VARIATIONAL RELAXATION
We propose an instantiation of the objective function with a specific choice of divergence function with standard variational technique. We choose the metric D(p, q) the reverse KL divergence DKL (gll)), which is common choice in the variational inference literature (see, e.g.. Blei et al.
(2017)). ) To remove the intractable induced distributions, we relax the objective function 3 by introducing variational encoders do (u(u,, x) and 4s yv)
Lxy-ixy = DKL (x y)
(2) (ypo)y|z))
(7)
(KL x, y) x 2.
y,,,,
(8)
= -DKL(gxx, 1ol ( x y 2o g) (v ( (u) u)po v))
(9)
=: Lxy
(10)
where (8) follows from the chain rule and nonnegativity of KL divergence (see, e.g..
Cover and Thomas (2006). po(u, yx, denotes the induced conditional distribution by Pe(u), (v)), Pa (x u), Pe (y z. v), and px(2/z, u) (or po (y/z,, v)) denotes the distribution induced by the decoder Nz (z, u) (or yo v)). Note that the intractable distributions po(x Po (z) no
longer appear in the upper bound (9).
For the mutual information term I [X Z with (X, Y, Y) ~ g(xx,)) y), we use the following standard upper bound (see, e.g., Zhao et al. (0188)
1 = -x DKL (go (z) X, Y) (2))) & By((x.) [PLL (1o(2|
N,X,,,)IN Y))lpo(z)) = T. (11)
where (z) denotes the induced marginal distribution by (x. and to (z) Note here that the relaxation gap is DRL (īo (z) (o))))) which again upper bounded by the joint KL divergence (7). Therefore, all relaxation steps in (8) and (IT) become tight when the joint distributions induced by data distribution, encoders, priors, and decoders are perfectly consistent with each other, i.c..
qxx y yo (z|x go (u z, x) (v/z. y) po(z) po (u) (v) (x z, u)po (y /2 v).
We note that these relaxation techniques are standard in the literature, the tightness of which deserves a separate future study; for recent related work, refer to Cremer et al. (2018) Poole et al. (019).
To sum up, the objective function for the joint model (3) is relaxed
Exy + A7g-
(12)
Note that the additional information regularization with À > 0 is only on the common representation Z, but not on U, V. which we call local representation. By increasing À > 0 to proper degree that does not impedes fitting, we can "reroute" the information flow from (X, Y) through the common representation 2 ( x, y) to the local representations (u/u,, x), go y)
Following similar steps in 7.. 9 the objective function in can also be upper bounded as
Lx-xx SLx- == DKL (1( x)
po (u)ps(x|2, u)).
(13)
For (5). we choose Lxy-y ==
DKL (g(x, y y) (z) x. y (x)))x x) (y/z)), which can be viewed the expected conditional ELBO (Sohn et al. 2015). It can be also subsequently relaxed as
Lxy- S Liyy-v DKL (g(x, y 2s x, )as (v)
(2)x))
v)). (14)
After all, the final relaxed objective function is given as follows:
Z = -xy-xy + AI, + Xx-xxXx-x + Iy + axy -y -x + axy-xI (15)
See Figure 2 for an overview of each term in the objective function.
We call the overall framework which consists of all the components in the Wyner model and the additional encoders (z)x), y). go (u z, x). go (v /z, y) together with its training objective (15) as Wyner common representation VAE or Wyner VAE in short. After parameterizing each distribution component in Wyner VAE as standard parametric distributions such as Gaussians, whose parameters again parameterized deep neural networks, Wyner VAE can be trained efficiently by the stan- dard reparameterization trick (Kingma and Welling 2014) as in the standard VAE. (See Appendix B for the Gaussian parameterization and the corresponding objective functions.)
4

---

Under review as a conference paper at ICLR 2020
you yquolli (vj,)
velelee/l/joovviievlae ulps (yla
Lxy-xx + XXg(X, Y:Z) (eq. (3))
pelv!
Tay aay + AT AT Y: (eq. (12))
Xolk Kalx.ni
YaUR-
qpíuils xl
qa(aix) poli)
by (eq. (5))
Lx xx (eq. (4))
I Exy-ty (eq. (14))
S Lx-xx (eq. (13))
(10) 8.X) pu(vipuly a.
Figure 2: A summary of the training objective for the Wyner VAE 15
In practice, we can either train the objective function 15 training), or train the joint model first letting a's be and then train the marginal encoder a retrospective manner by freezing the joint model (two-stage training) as in Vedantam al. (2018). The hyperparameter a's may be chosen with cross-validation with validation sets. We elaborate the training schemes used for each experiment Appendix
jointly with some positive a's (joint
We remark the role of the encoders 4s (z) x, y), ).)0)) qo (ulz,x), (o(v).. The joint encoder qa (z|x, y) appears in the optimization problem 1T and plays reference distribution in learning the com- ponents of the Wyner model. The variational encoders qo (u/u,, x) and 4s (v|z, y) are introduced to remove the intractable induced distributions pe (z(x)) and pa (z y), satisfying the correct condi- tional independence structure implied by the decoder model po (z) (z) (u)po (v) (x)zz,, u)po v). that is. 9 zz u, v/x, y) (z)xx (). (ulz,, (v y,Y), or equivalently UL. Y) and V VUL(UU, | IZ. If we learn succinet common representation go yz) (e.g.. shared concept) from (X, Y), then ( x) would capture the remaining randomness U of X (e.g., texture and style). We call this decomposition of the pair (X, Y) into the common representation Z and the local representation U.V as common-loce local information decomposition of (X, Y). We refer to (Z, U, V) as the joint representation of (X, Y), distinguish it from the common representation Z. Provided that Wyner VAE achieves a good information decomposition, the variational encoders then can be used to explicitly in finding the local representations U and V from the data variables X and Y.
2.4 SAMPLING WITH STYLE CONTROL
As alluded above, the variational encoders go (u|z, go can be used in sampling tasks with style control as a local representation (i.e.. style) extractor. We illustrate how to perform conditional sampling with style control (Fig. 3 (e)). Suppose that (X, Y) is pair of correlated images generated from the common concept but from different domains. Given an image
we can f (v/ z, y) (Fig 3 (d)). We then generate Y, from an image X, similar to conditional sampling (Fig (c)), while replacing the randomly drawn local representation V ~ po(v) with the previously extracted style Vo. thereby the generated images Yo,y's are of the same style as the reference image Yo- In similar manner, we can also perform joint sampling with a fixed style given a style reference data pair (Xo, yo), by mixing randomly drawn common representation Z from the prior (z( with the extracted style variables (u, Vo).
extract the style information Vo from yo by sampling (Zo, Vo) from go (zly) g
2.5 DEGENERATE CASES IN WYNER VAE
Two degenerate cases may arise in Wyner VAE. The first case is where the common variable Z captures all information of (X, Y), while U and V capture none. Joint VAE (JVAE) (Vedantam et al.) 2018) and joint multimodal VAE (JMVAE) (Suzuki et al. 2016) inherently assume this degenerate case, as it is discssued in the next section. Wyner VAE is able to avoid such degeneracy by explicitly
5

---

Under review as a conference paper at ICLR 2020
SARIR)
pula
Z
Z
Z
sampligg
stochastic rncoustration
(c) Conditional sumpling
Coaditionll sampling style
Figure 3: Schematics for selected sampling tasks. Double arrows denote deterministic mapping.
Table 1: Summary of related work. (J: joint generation, C: conditional generation, St style control.)
C
J
S
JVAE Vedantam et 2018),
JMVAE Suzuki
2016)
O
O
X
CVAE (Sonn et al. (2015
0
X
O
VCCA private (Wang et al. 2016
0
O
O
VIB (Alemi al. 2017)
0
X
X
Wyner VAE
O
O
having the local variables and controlling the common regularization parameter À 0. On the other extreme, Z may capture no information, while U and V capture all the information of X and Y, respectively. It may happen in Wyner VAE if the regularization parameter is too large. To avoid the degeneracy, we need to choose proper À by cross-validatie
RELATED WORK
In this section, we compare the proposed Wyner VAE to the existing models, deferring detailed description of the encoder, decoder, prior components and objective functions of each model to Appendix C
We provide a summary for capabiliitees of each model in Table [
VAE models. Wyner VAE can be viewed as a generalization of the probabilistic model (i.c., encoder and decoder) assumed in two existing joint VAFs- JVAE (Vedantam et al. 2018) and VM (Suzuki al. 2016) alluded in the previous paragraph. These models implement a similar idea of performing joint and conditional generation tasks via symmetric Markov chain X - W Y, where W is the joint representation of (X, Y). other words, these models can be derived by removing the local variables U and V in Wyner VAE. In Section 4 we demonstrated that the local variables in Wyner VAE help generate a variety of samples compared to JVAE.
joint
The same decoder structure of Wyner VAE with the "shared" (Z) and the "private" (U, V) latent variables has been also studied in the context multi-view learning (Shon et al., 2006 Ek et al. 2008
Salzmann al. 2010, Damiiaouu et al.) 2012) mostly based on linear analysis such canonical analysis
correlation anal ysis (CCA). More recently, variational CCA-private (VCCA-private) (Wang et al. 2016) was proposed learn the decoder model with variational encoders qo (z)x), 4s (u(x), and yv), with the encoder model (z, v/x, = go (o(2)))4p (x (u]x)g&(vly) to directly capture the conditional model from X Z to Y. On the other hand, Wyner VAE relies on the conditional inde- pendence structure (z u, v)) = 4 (z x, y) (u x)4 (v y), which is naturally induced by the decoder model. We argue that this choice of encoder model Wyner VAE may capture better semantic meaning of the local (private) random variables U and V, thereby leading better generative performance; see, e.g., Fig. 6
Conditional VAE (CVAE) (Sohn et al. 2015) directly models the conditional distribution g(y x)) obtained by simply conditioning every component the vanilla VAE for q(y) with the conditioning variable X. If Y is an image and X is an attribute of the image, latent representation V in CVAE needs to capture the redundant information of Y. which is not contained in X, i.c., style information of Y given X. Wyner VAE can be viewed a common conditioning variable, being capable of bidirectional sampling in its nature. Yet. if X high-dimensional, the conditional models like CVAE in general tend overfit the input data of
combination of two CVAEs with Z as

---

Under review as a conference paper at ICLR 2020
Table 2: Wyner model vs, the IB principle Tishby al. 1999).
Wyner model
IB principle
Motivating problem Probabilistic model
Channel synthesis, distributed simulation
Lossy compression Z X Y
X Z Y
Direction of inference
Bidirectional
Unidirectional
Measure of succinetness
T(X, Z
I(X: Z)
Measure of fit/relevance Optimal quantity
D(p, q) J(X; Y)
I(Y; Z)
N/A
X (Dutordoir et al. 2018). To address this problem, a subsequent related work, bottleneck conditional density estimation (BCDE) (Shu et al. (0117). proposed to learn joint and conditional VAE models simultaneously by softly tying the parameters of the two models for regularization. We note that Wyner VAE naturally addresses such problem by using a unified single probabilistic model for both joint and conditional distribution learning, finding succinct common representation Z for regularization.
Information bottleneck principle. The information bottleneck (IB) principle (or method) (Tishyy) iet al. 1999) is widely known information theoretic approach in representation learning especially for discriminative tasks, i.e., when the target variable Y is function of X and/or even discrete. Motivated by lossy compression, the IB principle proposes to find a compressed representation Z from the input variable X (i.e., (z(x))) while maximizing the relevance of Z in predicting the target variable Y as the minimizer of the optimization problem minimize (a(2)x) BI, (X; (X:Z) YY; Z),
where (X, Y, Z) ~ y(x,
The foremost difference between the IB principle and our approach is in the underlying Markov chains: our symmetric Markov assumption X + Z + Y is more natural than Z - X - Y of IB, when guessing Y based Z representation of X. Further, our framework aims find certain common information structure characterized by Wyner's common information with proper analogies to generation tasks of our interest, whereas the compressed-froot -foood--f-o------ yet relevant-to- representation Z in the IB principle lacks its operational meaning, relying on a rather weak analogy to lossy compression. We summarize other differences various aspects Table 2 In particular, we compare Wyner VAE with variational IB (VIB) (Alemi et al. (2017) in the experiments, which is an instantiation of the principle based on variational technique that can implemented with neural networks. As empirically shown below, VIB is not suitable for conditional generative tasks if the target variable Y high-dimensional.
4 EXPERIMENTS
We empirically demonstrate that Wyner VAE outperforms JVAE, CVAE, VCCA-private, and VIB, for join//conditional generation tasks and style manipulation on various datasets. We defer the implementation details and training schemes used for each experiment to Appendix F
Synthetic data. We first performed an experiment with a mixture Gaussians (MoG) dataset as a toy example. We considered a pair of 10-dim. MoG random vectors (X, Y) only correlated through label Z Unif([1, 2, 3, 4, 5]) (common information) and 5-dim. Gaussian random vectors U. V ~ (0, Is) (local randomness in each variable). We used the Gaussian latent variables (Z, U. V) of dimensions (10, 10, 10). trained each model for 500 epochs (separate 50 epochs for each marginal encoder for JVAE and Wyner VAE) with a training data of size 50k, and summarized the numerical results in Fig. 4 and Table which were evaluated with a test data of size 10k. See Fig. Pin Appendix E. for some visualizations.
In Fig. (a.b), Wyner VAE with À = 0 performed best for fitting joint distributions, but did not excel in conditional log-likelihoods. We observe that the performance of Wyner VAE on the test data gets improved throughout training by increasing A: À = 0.05 achieved good conditional performance without too much sacrifice in the joint performance, while larger value of X(= 0.1) interfered fitting to the distribution, failing to capture the essential common information structure. Overall, Wyner VAE with À control outperformed the other models, CVAE tends to overfit quickly noted earlier.
7

---

Under review as a conference paper at ICLR 2020
negative logiikelinoed
repaativ -kallhodd
Mutual information
34.0
Ang
And
33.-
And 225
14.2
2-0-223
And
33.6
J-d
VOCAgr 1WA2
VCD4-ph
CYAE
25.4
16.0
150 200
150
30C oppcrs
352
400
450
300
300
232
250
300
402
450
100 228
250 300
250
420
453
Figure 4: Numerical evaluations for the MoG experiment. For each point of the plots, we trained 10 different models and plotted average values with the shaded region that shows the standard deviation. (Two largest and smallest outliers were dropped for each point.)
Table 3: Best negative log- oog-likelihood (nll) values during 500 epochs of training for MoG dataset.
Joint all
Conditional nll
JVAE Vedantam al. 2018
32.82 + 0.14
32.80 = 0.76
Wyner VAB (A
32.54 + 0.11
16.16 = 0.14
Wyner VAE (A
0.05)
32.64 1 0.18
16.03 1 0.07
VCCA-private WWang et al
2016
32.77 1 0.04
15.96 1 0.04
CVAE (Sohn et al. 2015)
16.11 + 0.06
VIB Alemi 2017 (3 0.1)
503.91 = 2.79
JVAE and VIB performed extremely worse test conditional log-likelihoods compared to others as in Table 3 JVAE failed to capture the common information structure the training epochs increased, while VIB was only able to capture the average behaviors - we demonstrate how these models failed in this toy dataset in Appendix E.T Although VCCA-private achieved the best conditional likelihood with a comparable joint likelihood performance, this model fails to learn more complex distributions as illustrated in Fig. 6 and Appendix E.2
E.3
Fig. (c) shows that À can control the common mutual information Ix(X, Y; Z) in Wyner VAE - in particular, À E (0.05, 0.1) kept the (estimated) mutual information at constant level. We remark, however, that the MoG dataset has J(X; Y) = log 5, while the mutual information estimates are not around the true value (Fig. 4). We attribute this mismatch to modeling the underlying discrete latent variable with a continuous, Gaussian vector. Note that we deliberately used the standard Gaussian model assuming that we do not have a prior knowledge on the dataset generating process.
MNIST and SVHN datasets. We performed experiments with image datasets MNIST Le
Cun 1998) and SVHN (Netzer et al.] 2011), by randomly pairing digit images only through their labels. In particular, we constructed two dataset, MNIST-MNIST add-1 dataset, where
label(Y Y,) = label(X,) + +1 (mod 10), and MNIST-SVHH dataset, where label(Y,) = IXbel(X.).
For MNIST-MNIST add-1 dataset, we trained Wyner VAE with different choices of A {0.0, 0.l,0..,.... 0.5) to show the effect of X on the generative performance of Wyner VAE. Fig. 5 (a) corroborates our main claim that À > 0 helps learning succinet representation in terms of small XX, Y: Z) and there exists a sweet spot (A = 0.1) that strikes the balance between the fitting and the succinctness.
We also evaluated the label accuracy using the pre-trained Le-Net5 (LeCun et al. 1998) classification network of accuracy 99.1% %, and the per-pixel variance of samples from conditional generation. Note that both high accuracy and high variance are desired for good generative models. Fig. (b) shows that by rerouting the information flow through Z to U, V, À > helps U, V capture the style of images with small sacrifice in label accuracy.
We present image samples to visualize the effect of in Wyner VAE and the superiority of Wyner VAE over the existing models. We performed four sampling looks-conditional generation, conditional generation with style control, joint stochastic generation, and joint sampling with style control; see Fig. 3 for both MNIST-MNIST add-1 and MNIST-SVHI datasets, but here present only few
8

---

Under review as a conference paper at ICLR 2020
negattie log 1X,, Y:2)
Accurscy (96)
Variance
200
100
0.06
250
200
Accumacy
40
0.08
500
Conchiinal
0
0.01
Figure 5: Numerical evaluations of Wyner VAE for conditional generation of MNIST-MNIST add-1
dataset. The plots were generated similarly as Fig.
See also Table i the Appendix.
D
/
/
7
9
/
1
/
1
0 2
9
3
1
7
2
7
2
7
2
4
2
2
1
2
2
2
2
2
9
9
9
7
2 3 7
3
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4 4
4
4
4
4
4
4
4
4
4
4
4
4
4
#
4 S
4
5
5 5
5
5
5 5
5
5
5
5
5
5
5
5
5
5
5
5
S
5
5
Insut
(x1) Wyner VAE
[b1] Wyner VAF (A=
(c1) Wyner VAF
(61) IAKE
(VA)
[[]] VGCAgrrvate
(A
( 0.2)
Styleref
4 4
4
*
4
4
4
*
4 4 4
4 4
4 *
4
4
4
4 G
#
S
6
6
6
6
6
6
0
6
6
5
4
G
6
4
9
5
6
7
7
7
7
7
7
3
7
7
7
7
7
8
8
8
8
8
8
&
8
8
N/A
8
E
8
2 Q Q
8
9
9 9
9
9 9
9
9
9
9
9
9
O
0
0
0
0
0000
0
0
8
0
Q
0
8
Input dabel rel)
(42) Wyner VAE (A = 0)
(02] Wyner VAE
(62] Wyerr VAE
(AD)
[e2]
(12 VOCA- privte
(2 = 0.1)
(A (A=0..)
Figure 6: Samples from Wyner VAE and the other models for MNIST-MNIS" add-1 dataset. (al-fl)
Conditional sampling. (a2-f2) Conditional sampling with style control. both tasks, the leftmost column denotes the conditioning input to the models.
samples for illustration. We refer the interested reader Appendix E.2 E.3 for full comparison. We defer all the samples from VIB to the appendix, as it only generated same "average" images.
Fig. o presents samples of MNIST-MNIS pairs by conditional generation with and without style control. Fig. o (al-cl) and Fig. 6 (a2-c2) demonstrated how the variations in the generated images and the style information captured V are affected by varying X, respectively. (We illustrate how we performed conditional generation with style control for CVAE and VCCA prrivate in Appendix ( = 0.1 showed the best conditional generation results, consistent with Fig. 5 (a). JVAE generated images without much variation. VCCA-private erred frequently in guessing the labels, which implies that the shared representation Z VCCA-pre -privatevvateeee does not capture the "common information".
VCCA-private
Fig. 7 presents samples of MNIST-SVHN pair from conditional sampling with style control (Fig. 7 (a.b)) and two variations of joint sampling tasks (Fig. 7 (c-f)). Fig. 7 (c.d) show joint stochastic reconstruction: Z is inferred from the label reference data, and samples are generated jointly by drawing local randomness (U, V) ~ po (u)po (v): see Fig. 3 (b). Fig. (e.f) show joint sampling with style control: similarly to conditional sampling with style control, generated joint samples by drawing common randomness Z Po (z) with a specified local information from the style reference. In all cases, we observe that À 0.1 achieves better style manipulation over À indicating that X > 0 helps separating style information from common information.
MNIST quadrant prediction dataset. We performed quadrant prediction task (Sohn et al. 2015) with a static, binary MNIST dataset (Larochelle and Murray (01T), using the Bernoulli observation model for decoders. Specifically, split each digit image into parts into left (X; conditioning)

---

Under review as a conference paper at ICLR 2020
Style
3
3
3
3
3 3 3 3
3 3 3 3 3
3
S
5 6
5
1 5 6
5 6
5 6
J
15
3
3
6
6
6
label ref.
(c) Wyner VAE (A
0)
(d) Wyner VAE
0.1)
7
Z
8
9 7 9 8
0 8
7 2 6
8
8
2+ style
8
918 Wyner VAE (A 0.1)
2
6
3
8
Ipptt (lbbl me))
Wyner (a = 0)
Wyner
(2 = 0.13
(e) Wyner VAE (2 0)
Figure 7: Samples from Wyner VAE for MNIST-SVHN dataset. (a.b) Conditional generation with style control. (c,d) Joint stochastic reconstruction. (e,f) Joint generation with style control. 0.1 helps local latent variables capture style information, and the generated samples exhibit the effect compared to À
Table 4: Best nll values during 1000 epochs of training for MNIST left-right prediction task. The conditional nll value for CVAE is taken from Sohn al. (2055).
Joint all
Conditional nll
JVAE /edantam et al.
2018
91.05 1 0.03
45.22 1 0.03
Wyner VAE 0)
87.74 1 0.05
43.87 + 0,03
Wyner VAE (A 0.15)
89.66 0.03
43.85 + 0.03
CVAE (ohn et al. 2015
44.73
and right (Y; target). In this case, the most succinct common representation Z is nontrivial object in contrast label (or its bijection) in the previous experiments. Table 4 summarizes negative log-likelihoods for Wyner VAE, JVAE, and CVAE on the test dataset. Note that Wyner VAE with with performs best, while an additional regularization parameter 0.15 only gives a marginal improvement in conditional performance. For this specific case, having the local variables U and V is enough to capture the local information, as the underlying data seems to have much complex common information structure compared to a discrete label in the previous cases.
CelebA dataset. We performed an experiment with CelebA dataset (Liu et al. 2015). ), which is degenerate case in the sense that X is function of Y. While any bijection of X is common representation Z attaining J(;; Y) in this case, we demonstrated some merits of using Wyner VAE over CVAE in terms of better common representation learning in Appendix E.4
CONCLUDING REMARKS
Cuff's channel synthesis and Wyner's distributed simulation another manifestation of Occam's razor by finding the simplest probabilistic structure that connects one random object to another. The proposed Wyner VAE finds this succinct structure in a disciplined yet efficient manner, and provides theoretically sound alternative to the information bottleneck principle. The experimental results demonstrated the potential of our approach as a new way of learning joint and conditional generation tasks with optimal representation learning that can be further developed and refined for more complex dataset such as auditory, text, or a pair of those.
A few remaining questions from experiments are in order. First, this paper does not address how close the estimated common information from a learned Wyner VAE model to the true Wyner's s common information (X; Y). and it may be interesting devisc better estimation technique of Wyner's is common information from data. We emphasize that, however, the main goal of this paper is to demonstrate the advantages of the probabilistic structure of Wyner VAE and the regularization parameter À in various joint and conditional generation tasks. Second, it would be interesting investigate the relation between the variational approximation gaps and the quality of learned representations in Wyner VAE (Cremer et al. 2018 Poole et al.) 2019).
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexander A.
Alemi, lan Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In Int. Conf. Learn. Repr. 2017. URL
Id=HyxOzBceq.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. J. Am. Stat. Assoc., 1 112(518)8359-87) 2017.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley, New York, second edition, 2006.
Chris Cremer, Xuechen Li, and David Duvenaud. Inference suboptimality in variational autoencoders. In Proc. Int. Conf. Mach. Learn., pages 1078-1086, 2018.
Paul Cuff. Distributed channel synthesis. IEEE Trans. Inf. Theory, 59(11):7071-7098 November 2013.
Andreas Damianou, Carl Ek, Michalis Titsias, and Neil Lawrence. Manifold relevance determination. In Proc. Int. Conf. Mach. Learn., 2012.
Vincent Dutordoir, Hugh Salimbeni, James Hensman, and Marc Deisenroth. Gaussian process conditional density estimation. In Advances Neural Information Processing Systems, pages 2385-2395, 2018.
Carl Henrik Ek, Jon Rihan, Philip HS Torr, Grégory Rogez, and Neil D Lawrence. Ambiguity modeling in latent spaces. In Int. Workshop Mach. Learn. Multimodal Interaction, pages 62-73. Springer, 2008.
Abbas El Gamal and Young-Han Kim. Network Information Theory. Cambridge University Press, Cambridge, 2011.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning basic visual concepts with a con- strained variational framework. Int. Conf. Learn. Repr. 2017. URL /attps://openreeieew
hett/forum? id=
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes In Int. Conf. Learn. Repr., 2014. URL
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Adv. Neural Info. Proc. Syst., pages 3581-3588 2014.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. Int. Conf. Artif. Int. Stat., pages 29-37, 2011.
Yann LeCun. The MNIST database of handwritten digits. http://yaan..ecun.cootu mnist// 1998.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-2) 2324,, 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit, pages 3730-3738, 2015.
Lars Maalóe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. In Proc. Int. Conf. Mach. Learn., pages 1445-1453, 2016.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.

---

Under review as a conference paper at ICLR 2020
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. Proc. Int. Conf. Mach. Leam., pages 5171-5180, 2019.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proc. Int. Conf. Mach. Learn., pages 1530-1538, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proc. Int. Conf. Mach. Learn., pages 1278- 1286, 2014.
Reuven Y Rubinstein and Dirk Kroese. Simulation and the Monte Carlo method, volume 10. John Wiley & Sons, 2016.
Mathieu Salzmann, Carl Henrik Ek, Raquel Urtasun, and Trevor Darrell. Factorized orthogonal latent spaces. In Int. Conf. Artif. Int. Stat., pages 701-708, 2010.
Aaron Shon, Keith Grochow, Aaron Hertzmann, and Rajesh P Rao. Learning shared latent structure image synthesis and robotic imitation. Adv. Neural Info. Proc. Syst., pages 1233-1240, 2006.
Rui Shu, Hung H. Bui, and Mohammad Ghavamzadeh. Bottleneck conditional density estimation. In Proc. Int. Conf. Mach. Learn., pages 3164-3172, 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Adv. Neural Info. Proc. Syst., pages 3483-3491, 2015.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv: 1,01991, 2016.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. In Proc. 37th Ann. Allerton Conf. Comm. Control Comput.. pages 368 03--377,-15 1999.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. In Int. Conf. Learn. Repr., 2018. URL htttps://openreeiier
net/forum HkckC Rb
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correlation analysis. arXiv preprint 16Xiv:1610.03454 2016.
Aaron Wyner. The common information of two dependent random variables. IEEE Trans. Inf. Theory, 21(2):163-179, 1975.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. The information autoencoding family: A lagrangian perspective latent variable generative models. In Proc. Uncertain. Artif. Intell., pages 1031-1041, 2018.
12

---

Under review as a conference paper at ICLR 2020
A A QUICK OVERVIEW ON VARIATIONAL AUTOENCODERS
2014 is a class of Variational autoencoder (VAE) (Kingma and Welling 2014 Rezende et al.
deep generative models that aim to simulate the unknown distribution [(x] underlying the data
Xi...., N to generate new samples from this distribution efficiently. Let qemp (x) be the empirical distribution defined by the sample. Assume generative latent variable model pe pe (2))p(((())
model the underlying distribution (x).. One of the standard approach to learn each component in the model, the prior Po and the decoder pe (x/z). 22. is the maximum likelihood approach that aims to solve
maximize
log (xx),
(16)
or equivalently,
mininize DKL (xemp(x)) (x)))oo(x)),
(17)
where (x) is the induced distribution characterized by px(z)po(x(z), i.e., px(x) po (z)pp (x (2) dz. However, is often computationally hard to solve the optimization problem directly due to the induced distribution po that involves integration over high-dimensional space.
In variational Bayesian learning approach (see, c.g., BBee et al. 2017 an approximate posterior qs (z|x) (also called as an encoder) is introduced to relax the objective (16). Here we present a short derivation of the well-known VAE objective function. Note that the objective in (I6) can be upper bounded as
DKL(gemp (x) (x())) DKL (9omp (x) (15 + (x (xx) DKL (
(18)
DKL (x)xx(z)xx) po x))
(19)
DKL (1)llgmmm)mgllllllllg (x)x((2/xx po (z)po z)),
(20)
where 5o (z(xx) is the induced posterior characterized by pa (x)z). (8) follows from the non-
negativity of the KL divergence, and (19) follows from the chain rule of the KL divergence (see, e.g,
Cover and Thomas (2006)). Note that the final relaxed form (20) does not contain the intractable term (x). The variational relaxation tight if and only if DKL (46 (z|x) (zpx() = 0 for all x.
The upper bound (20) is the objective function for the standard VAE model (Kingma and Welling 2014. Rezende et al. 2014). That is, the standard VAE model aims to solve the following optimization problem:
minimize DKL (Jemp (x (. x)lppo (z)ps (x|z)). 0,0
(21)
To express the objective function in a more standard form in the literature, we add a constant h (gemmp (x)), the differential entropy of (x)p and then derive
DKL (x)gx (x) (z)po (x 22) + h(gemp(x))
(22)
-B (xc DKL
+
9 X) log
dz =: Ex [Co.s [X)
(23)
po
(X(z)
The loss function L (x) in 23 with negation is called the evidence lower bound (ELBO) in the literature, since - -L & (x) lower bounds the evidence log po(x) for each x. The KL divergence term
and the expected log loss term are called as the regularization term and the reconstruction term, respectively.
Assume that the observed variable X continuous for simplicity. The most standard parameterization of the components in the VAE model is the diagonal Gaussian parameterization
po (z) (z 00 diag(or.e))
(24)
Pe (x Z) = (x] (z())) diag (z)))
(25)
4 (2|x)) = [z 2 (x), diag (x))).
(26)
where each function may be parameterized by neural network. Here, o² denotes a vector of a proper dimension and diag(a²) denotes diagonal matrix with diagonal entries 2 Often, the covariance
13

---

Under review as a conference paper at ICLR 2020
of the prior is taken to be isotropic and constant such as oā.0 1, but o².0 also be set as independent trainable parameter as in this work. Note that the diagonal Gaussian parameterization (x(x))) is a formal modeling assumption to have tractable density, which is required to evaluate the log loss log (/px(x/z) in the reconstruction loss. This formal Gaussian noise also plays a role in estimating likelihoods. However, after training, the decoder variance (z(z) is dropped and the resulting decoder is used as deterministic: z + (z).).
With this parameterization, the loss function (x,o(x) in (23) can be estimate efficiently for each via a Monte Carlo approximation by sampling z ~ qo (z x). The overall objective function then can be minimized with gradient based optimization algorithm like Adam (Kingma and Ba 2014) based on the reparameterization trick (Kingma and Welling 2014 Rezende and Mohamed 2015
and backpropagation.
B STANDARD IMPLEMENTATION OF WYNER VAE
B. GAUSSIAN PARAMETERIZATION
As elaborated in Appendix A on VAFs, we use standard Gaussian parameterization for Wyner VAE our experiments. Concretely, we let
Po ((z)0, diag(or
(27)
Po po(u) /V diag (f),))
(28)
po po(v) / (V 0. diag(o
(29)
pe [x z, u} = .V (x u), (i
(30)
Po (y) 2, v) N yo(z, v), diag(o v))),
(31)
To (z(xx. y) .V (Vz Z012, (x. y), diag(o y),)
(32)
9e (u|z, x) .V un z x) diag (2,x))
(33)
Lo (v/22, = // ((v (z. y) diag (z,y)),
(34)
96 (z(x)) .V (z x), diag (x)),
(35)
(z yy) (z 202 (y) diag(ori.. (y))
(36)
where each function may be parameterized by neural network.
B.2 OBJECTIVE FUNCTIONS
We can rewrite the objective function Lxy in 12 terms of the expectation of ELBO as 23
DKL z, u, v)llpo v)) (xg(x.))) Eg(x,)) Co. ()+ C (X.Y)
where
Cree (x, y) = Boo(a)x))) //
1 po(y Z, v) dv
Co (x,
(x) log (x du + golv Z.y)
C C (x,y) = DKL
y) (u
DKL(q6 x, x)
+ (ygstaxxy)
DKL (ys (u Z, (po(()) + (1s(v Z. x))))ooo()))]
We remark that a 3- VAE (Higinn et al. 2017 type regularization in our joint model corresponds to imposing additional weight 3 >>
Lig (x, (x,y) y) such that the objective function becomes
Eg(x,)) C Y) + 3C X, Y)J. We note that
1 X Y: Z,
U. v) (X, Y: Z, U, V) + DKL u, Eg(x,y) DeL (go (z X, Y)g, ()lz, X)gs(v) (v/z, X (1l)po(z)po(u)po(v))
(37)
(38)
R ( (X. Y)).
(39)
14

---

Under review as a conference paper at ICLR 2020
Therefore, applying 3-VAE type regularization imposes additional regularization (X, Y; Z, U. V), which corresponds to the entire latent bottleneck (U, V, Z) for the information flow from (X, Y).
B.3 ESTIMATION OF MUTUAL INFORMATION
With the typical Gaussian parameterization as presented above, we have an easy estimate for (X, Y; Z). After training, given test dataset (x,, the mutual information I,(X, Y; can be estimated as
(x Y: = (Z) h (Z
= h(pp(z)) ((x, ((
= h(po(z))
h(go (2)x,
2
N
2
log 0,0,j
log 00 (x,,
(40)
2
2.V
1=
j=1
We also estimate the variational upper bound of [X( Y; Z) in follows:
(X Y; < T Y: Z) (x(x))) [DL (g X, Y (pp))))
DKL (2 (2|x;, ll)o
(41)
In the last expression, each KL divergence can explicitly obtained from the Gaussian parameters of qo (z)x,, y.) and (z).
C A DEEPER LOOK ON RELATED WORK
For the standard VAE Kingma and Welling 2014 Rezende et al.] 2014), we refer the interested reader to Appendix A In what follows, revisit and decompose each VAE-type model into its encoder/prior/deccoerr components, and express the objective function in the form of the reverse KL divergence DKL (pe)). where qs is the joint distribution over the data variables and the latent variables defined by the data distribution and the encoders, and pe is defined by the priors and the decoders.
JVAE/JMVAE
JVAE VVddnntam et al. 2018) and JMVAE (Suzuki et al. 2016) consist of
the joint encoder 4s (w xx, y). the marginal encoders go (w/x), (w(y))
the prior pe(w),
the decoders pe (x w). Pe ww)
They share the same joint model objective
DkL (xL (w x (x) [))]
(42)
but differ in training the marginal encoders. JMVAE trains the marginal encoder (z|x) via minimizing
Eg(x,>> ( [KL (go llloo
(43)
together with the joint model objective (42) (by adding two objective functions with an additional weight as a hyperparameter), while JVAE trains the marginal encoders separately from the joint model by training the marginal VAE as proposed, i.c..
minimize DKL (x) (w w|x)]lpe (w) wo)po( w)). (w(x)x)
(44)
We remark that pa (x w) and po w) are fully characterized by the deterministic decoders x so(w) and (w().
15

---

Under review as a conference paper at ICLR 2020
It worthwhile to compare the regularization term [X( Y; Z) in Wyner VAE with the idea of 3- VAE (Higgins et al. 2017). which empirically showed that an additional weight on the regularization term finds disentangled representation in VAE model. If apply 3-VAE type regularization in our joint model, then it corresponds to an additional weight 3 > on the mutual information on the joint representation (X,(X, Y: Z, U, V)), not only on the common representation (X, Y: Y; as in Wyner VAE. (See Appendix (.2)) In words, Wyner VAE provides finer control on the information flow from (x, Y). by manipulating the common path qo (z(xx,))) while 3-VAE blocks the entire latent bottleneck qø (z|x, y) (u/z, (v/z.,),
CVAE CVAE Sohn et 2015) for modeling 7(y|x) consists of
the encoder ds (vly,)),
the prior pv(v)x),
the decoder po (y [V x).
The objective function is then given as
xx(x) [KL (gly
( V, v,)]]
(45)
Note that without the conditioning variable x, this model boils down to the vanilla VAE for modeling ((y). If we assume a Markov chain X (yyl))).
Y, the decoder Po(y|v, x) can be replaced with
We performed conditional generation with style control with CVAE as follows: Given a style reference Yo and its corresponding Xg, we sample and keep Vo via the encoder 4e (vly, x). Given new X1 we take Yo,1 yo (Vo. X1) as a new sample. Note that this scheme assumes that Vo < 2o (v/yoo Xn) captures only the remaining information yo excluding the information on X
SSVAE/ADGM Semi-supervised VAE (SSVAE) (Kingma et al., 2014) proposed similar con- ditional VAE model for modeling ((y|x) especially when the conditioning variable x is discrete. SSVAE consists of
the encoders 4s (vllyy,)), (x(y)) (classifier).
the priors pe (v). Po (x).
the decoder po (ylvv),... v,)
Note that it has additional encoder is (xly) (classifier) top of qo (vly,, x) from CVAE, and as- sume the priors pv(v) and (xx).. which replace the conditional prior (v x)) and the data distribution ((x) in CVAE, respectively. With a paired (i.e., labelled) data, SSVAE minimizes
DKL (g(x, y y)4 (v (vvyy.s (xx
(46)
With an unlabeled data, SSVAE minimizes
DKL (x(y))g(x))) (y) (x)
(47)
Note that the label information gxx) (46) is replaced with go (xly) in (7 as the label information missing in the unlabeled data.
Formally, in this degenerate case, identifying the conditioning variable X as the common latent variable Z in Wyner VAE recovers SSVAE. Yet, SSVAE proposed in the context of (semi- supervised) classification problem which aims to learn a classifier 4s (x y), and for a high-dimensional X, it is not feasible to directly model the conditional distribution go (xly).
Auxiliary deep generative model (ADGM) (Maaløe et al. 2016) adds an auxiliary latent variable Z to SSVAE improve the performance. ADGM consists of
the encoders qe (z)))), qo (v|z, y,
y),,, x), 4o (x/z, y) (classifier),
the priors po(v) (v), Po (x),
the decoders po (y /v, x), (2
Note the new components go (zly) and (z(z)v, y) on top of the SSVAE, and the original en- coder/decoder components have additional condition on the auxiliary variable Z. However, ADGM does not impose any additional conditional independence with Z, it not directly comparable to Wyner VAE.
16

---

Under review as a conference paper at ICLR 2020
VCCA-private
As noted earlier, VCCA-private (Wang et al. 2016) consists of the encoders do (z) x), (and/or go (zly)), (o (o(v y).
the priors po (z), (u()), po(v).
the decoders (x u), (o(yiz,
Note that the prior and the decoder components are same with Wyner VAE. Hence, the objective for VCCA-private for the marginal encoder qo (z|x)) can be expressed as
DKL (g g, (u/ x) (vlx Po u (v) (o)) (x|z, u)po v)).
(48)
To model the other direction of the marginal encoder go (y 2), they minimize
DKL (as y (u/x)) (vly) (po
(x u)po |2, v)..
(49)
To learn as (2|x), 2 (zly) simultaneously, BiVCCA-private minimizes convex combination of the two KL. divergence terms.
We performed conditional generation with style control with VCCA-prival as follows: Given a style reference yo. we sample and keep Vo via the encoder Qo (vly). Given new attribute X we take Y0,1 Ve(Vo. X1) as new sample.
VIB VIB (Alemi et al. 2017 proposed a variational relaxation of the following minimization problem posed by the information bottleneck principle (Tishby et al. 1999):
minimize (II((::)) (Y;: Yo go
(50)
where (X, Y, Z) ~ q(x, g) (2)x.. x). VIB introduces two variational distributions pz(2) and (y(y/z)
that approximate (z) and de (y lz). Then, based standard variational bounds the mutual
information terms (see, e.g., Zhao et al. (0188), we obtain a variational upper bound on B1, (X; Z) TY(Y: Z)
(2)x X)]lo())) -B (x)x)))))))
(D(x)
log Po
+ l((())))
(51)
(2)
which is the objective function for VIB. Note that the relaxation gap is given by (D(ūs(z)llpozz)) + (y Z))].
(52)
D LIKELIHOOD ESTIMATION
For each likelihood of our interest, we derive naive Monte Carlo (MC) estimator and an MC estimator with importance sampling (see, c.g.. Rubinstein and Kroese (2016)). Here we only present the estimators for Wyner VAE, but the estimators for other models can derived in the same manner.
D.1 JOINT LIKELIHOOD
We wish to estimate the joint log-likelihood of the model with respect to given test data ((x(0) ( i.e. EN log (x((() y()), where
(x (z(z) (V) u)po v) dudv..
(53)
(1) Monte Carlo estimator: Let (z((), ue()) v(o) po for so l1....
1 Σ (0) (3))))()) v())). (x. y)
(54)
(2) Importance sampling: For each (x,y), let (z(*), u((), v(o)) for S....,S.
S
pe pa )oo ( )oo (+)) (pp v()))
yx = 5 mm1
Σ
(55)
2o (2( x, )o (z(0) x) (v()) (2)))
17

---

Under review as a conference paper at ICLR 2020
D.2 CONDITIONAL LIKELIHOOD
We wish to estimate the conditional log-likelihood of the conditional path of Wyner VAE from x to y, i... qo xepo Pe (y z, v), with respect to given test data {(x(*) i...
N
log To.o (y (1(1))))
(56)
where
(v)x( go(z v)
(57)
(1) Monte Carlo estimation: Given let (z((), v(())) -z0))~/(( fors
for ==........ I....,S.
1
(y po (o(y1z(0), v(()).
(58)
(2) Importance sampling: For each (x, let (z(+), v(+) ~
for 1....
S.
1 (z())) (y 5 4o (z()))/ y (v(o)/2()) v(())
(59)
E ADDITIONAL EXPERIMENTAL RESULTS
E. MIXTURE OF GAUSSIANS
Here we present the test log- likelihood evaluation of JVAE and VIB in conditional generation tasks of the mixture of Gaussians (MoG) dataset; recall Fig. Both JVAE and VIB performed worse as the training epochs increased, that is, they overfit to training data (Fig. g). We visually illustrate how they failed in Fig. Fig. a shows the outlook of the MoG dataset in our experiment. Fig. 9 (el,e2)
show that JVAE captured all the components at the beginning, but then collapsed to few components afterwards. On the other hand, Fig. 9 (f1, show that VIB only captured the average behaviors, although gradually adapting to the underlying data.
Test conditional negative log hgglliellhoodd
IVAE
[3=0.001]
(8=0.01)
Vis (fww0.1
200
50
100
150
200
250
300
350
400
450
500
Figure 8: Conditional nll values of JVAE and VIB for MoG dataset. For each point of the plots, we trained 10 different models and plotted average values with the shaded region that shows the standard deviation. (Two largest and smallest outliers were dropped for each point.)
Alll the scatter plots were generated based the Gaussian kernel density estimation.
18

---

Under review as a conference paper at ICLR 2020
(eltuennn Dala
(il VuE (.6))
Jyz VRZ 0.05) ulter 300upad
DXR2 alt 500 sposht
ME0VEERRgpall ufla s00eacdl
IAZIIVAL afte 500
OF 01) afeer 900 rpomy
Figure 9: Visualization of conditionally generated samples for MoG dataset. Each axis of the scatter plots corresponds to the first coordinate of X; and Y, respectivelyf The X data points were from the test data, and the Y data points were generated from the conditional models based the test data. One sample was generated for each data point.
E.2 MNIST-MNIST ADD-1
Table 5: Accompanying table for Fig, Summary of numerical evaluations of MNIST-MNIST
add-1 1 experiments. For each row, we trained 10 different models and dropped two outliers for each average and standard deviation.
Joint nll
Conditional nll
I,(X, Y:
Accuracy (%)
Variance
IVAE /Vedannam al.
2018
1173.40
13.98
514.26
12.03
65.89 0.51
98.84
0.11
0.0032 0.0005
Wyner VAE(=-U)
1198.95 5.20
172.90
45.18
41.35 + 0.78
98.01 + 0.15
0.0254 = 0.0036
Wyner VAE 0.05)
1195.09 10.35
58.86
21.78
33.57 + 0.84
97.46 =
0.0353 0.0027
Wyner VAE 0.10)
1212.66 10.61
-0,51 6.94
24.52 :: 0.89
91.33 + 1.31
0.0454 0.0009
Wyner VAB (A 0.15)
1220.64 10.71
3,35 1 6.04
19.73 - 0.78
84,90 + 2.68
0.0480 = 0.0006
Wyner VAB
1230.32 9.30
16.37
79.94 2.83
VCCA-private [Wang et al. 2016
1254.89 7.17
5..64
59.63 + 1.25
0.0548
0.0003
CVAE (Sohn nl. 2015
15.39 6.41
97.69 0.28
0.0404
0.0004
VIB Alemi
15 0.001)
733.86 13.88
96.54 - 0.24
0,0000
0.0000
19

---

Under review as a conference paper at ICLR 2020
D
7
9
3
9
9
1
2
2
2
a
7
2
2
2
2
2
9
9
2
1
3
2
2
2
2
2
2
2
3
3
3
3
3
3 3 4 5
3
3
3 4 5
3
3
3
3
3
9
3 4 3 f 7
9 7
3
3
3
3
3
3
4
4
4
4
4
4 5 6
4
4
4 5
4
4
4
4 5
4 5
4
4 4 5 5 6
4
4 5 6
4 5 6
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
1
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
3
0
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
8
8
8
8
0
8
8
8
8
8
8
8
8
9
9
9
9
8
9
9
9 O
9
9
9
9
9
3
0
9
9
9
9
9
9
9
0
0
O
O
0
O
8
O
O
0
O
5
9
O
0
O
O
0
0
Input
Wyner VAE
Wyner VAE 0.1)
Wyner VAE 0.2)
CVAE
VCCA-prrivtee
IVAE
VIB
Figure 10: Conditional generation.
Style
4
4
4 *
4
4
*
4
4 4
*
4
4 4
*
4
4
4 3
H
D
1
1 /
9
/
1
1
1
2
2
a
a
2
2 0 3 4 5
2
2
2 / 3
9
7
7
Q
2
q 3 4 5
2 5 7
2
3
3
3
3
3
3 5 4 S
3
3
3
3
7
2 4
4
4
4
4
4
7 4
4 4
4
4
4
4
4
5
5
5
5
5
5
5
S
$
5
5
6
6
6
6
6
6
0
6
4
4
4
6
4
6
4
9
6 Q 6 8
N/A
N/A
7
7
7
7
7
7
7
7
7
7
3
7
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
f
8
8
6
8
8
8
8
9
9
9
9
9
9
9
9
9
9
0
0
0
0
0
0
O
8
Q
Q
0
0
Input
Wyner VAE 0.
Wyner VAE (2
Wyner VAE
CVAE
VCCA-private
IVAE
VIB
ref.)
(2 0.2)
Figure 11: Conditional generation with style control.
Label ref.
Joint stochastic reconstruction
Style ref.
Joint generation with style control
Wyner VAE (A 0.0)
4 4 4 4
4
5
9
9
L
8
O
9
5
S
5
5
5
0
O
3 9
6
Wyner VAE (A 0.
4
T
'( S 4 4 4
{
4 S 4 S S 4 4 4 4 4
7 S 9 S
7
4
0
3
4
7
9
9
L
9
8 1
9 6
5
5
5
S
5
1
4
5
2
/
0
3
4 9
4
4
4
V84
3
S
R
6
8 9 0
9 0 & 2
(2 0.2)
5
5
5 5
5
1
S
2
3
&
4
4
4
0
S
4
3
3
VCCA-private
5
5 5 4 4 4 4 4
5
5
5
2
3
4
/
4
4
IVAE
N/A
5
S
5
5 5 S S S S S
Figure 12: Joint generation.
20

---

Under review as a conference paper at ICLR 2020
E.3
MNIST-SVHN
0
0
90
10
38
81
98
98
88
1
16
5
9
Z
2
25
2
3
8
5
3
3
3
31 31
4
111
9
CI
5 6
15
6
5
5
5
6
5715
15
95
95
6
3
6
5
6016
16
268
16
5
6
26
7 & 7
717
7
1/
7
2 81 8
707 6
7
2
g
16
9
8
99
9
Input
Wyner VAE 0.
Wyner VAE (2=
Wyner VAE =2= 0.1]
CVAE
VCCA-private
IVAE
VIB
Figure 13: Conditional generation.
Style ref.
4
4
0
0
0
10
0
0
0
0
0
0
0 1
0 I
0
1
1
1
7
1
v
2
2
2
2
2
2
2
2
2
2
2 3 4
3
3
3
335
3
3
3
3
3
3 4
T
3
T
4
4
4
4
4
4
4
4
4
4
4
N/A
N/A
5
5
5
15
5
5
5
5
5
5
5
5 6 7 8 9
6
6
6
6
6
6
6
6
6
6
6
7 8
7 8
7
7
7 8
7
7
7
7
7
7 8 9
7 9
8
8
8
8 9
8
6 9
4
Input (label ref.)
Wyner VAE 0.
Wyner VAE 0.05)
Wyner VAE
CVAE
VCCA-private
JVAE
Figure 14: Conditional generation with style control.
Label ref.
Joint stochastic reconstruction
Style ref.
Joint generation with style control
Wyner VAE (A 0.0)
Z
2
2 2
Z
2 2
2 Z
2
Z
0
3
0
9 8
0
9
8
8
5
24
2
0
3
00
Wyner VAE 0.05)
2
2
2
2
2
2
2
2
3
2 2
Z
5
8
6
5
0
2
<
& 3
24
1
2.
2
02.
2.2
322
2
5
20
9
61
9
2
Wyner VAE (A 0.1)
2
2
2
2
1
2
2
5 7
5
J
0
0
J
6
6 B
2
2
2
2
5
9
3
0
0
8
8
Z
7
4
2
2
2
2
Z
Z
S
J
[
2
Z
Z
2
8
VCCA-private
24
L
2
2
2
2454
2
2.
2-
84
24
S4
Z
Z
2
Z
Z
Z
Z
Z Z
Z Z
JVAE
N/A
24
2.2
2.
Figure 15: Joint generation.
21

---

Under review as a conference paper at ICLR 2020
E.4 (FACE, ATTRIBUTE) PAIRS FROM CELEBA DATASET
CelebA dataset (Liu et al. 2015) consists of pairs of a face image and a 40-dim. binary vector that contains attributes information of the face. We performed conditional generation of face images (Y) given an attribute vector (X). Since an attribute X is a function of a given face image Y. we let the dimension of the local variable U be 0 in this case. Fig. presentss samples of CelebA faces from Wyner VAE, JVAE, and CVAE Wyner VAE with À = 0.1 generated a variety of faces with the correct attributes, while JVAE generated images with little variations as previously observed and CVAE generated diverse images but often with wrong attributes. See also Appendix [E.4.2 for the results with style control and numerical evaluation the performance of Wyner VAE on CelebA dataset.
Input attributer Arthedepstiooss
[5] Wyner VAE (A 0)
Antrccike
Bigllp
Busley spchrows
Heaey makcup
Wyner
beard
(A = 0.1)
mar
dearing rigg Epstict
Young
(cl IVAE
Sample image dataset
(d) CVAE
Samples from Wyner VAE, JVAE, and CVAE for CelebA dataset. Multiple face image samples were conditionally sampled given an attribute vector listed at the leftmost column.
Figure 16:
Note that this is a special case where the target variable Y is a function of a conditioning variable X, which is degenerate case in the sense that any random variable Z that is bijective with X can serve as an optimal common representation that achieves J (X; Y). and Wyner VAE and CVAE may have comparable conditional generation performance. In practice, however, Wyner VAE outperforms CVAE since Wyner VAE learns a good representation Z of X that is helpful in generating Y conditionally, while CVAE directly uses the raw X for conditioning.
For a qualitative evidence, we present a few Atribute->Face samples in Fig. 17 from models trained with more data in the paper, from truly unseen attribute likely-female-feaure VAE can produce plausible images as it finds good representation Z of an unseen attribute, while CVAE fails.
bald)
Input attributes
Wyner VAE (2 = 0.15)
CVAE
Attractive
Bald Big_Upp
Heavy Makeup
No, Beard
Wearing, Lipstick
Figure 17: Attribute Face (CelebA) samples from Wyner VAE and CVAE.
E.4.1 NUMERICAL EVALUATION
We present an additional numerical evaluation of Wyner VAE to corroborate the effect of À > 0. With CelebA models, we evaluated both conditional paths, i.e., face to attribute classification (face2attribute) and attribute to face generation (attribute2face). For the attribute classification, we counted the number of corrected classified binary attributes out of 40. The image variance is evaluated with per pixel, while all pixel values were normalized between 0 and 1. Table 6 summarizes the results. We can observe that both the accuracy and the variance for conditional generation from a given attribute maximized the around X 0.2. Note that JVAE performs worse in classification
22

---

Under review as a conference paper at ICLR 2020
Table 6: Numerical evaluation Wyner VAE and JVAE for CelebA dataset.
Face2attribute accuracy (%)
Attribute2face variance
Wyner VAE (A 0)
89.26
0,0482
Wyner VAE (A 0.05)
89.21
0.0493
Wyner VAE (A 0.1)
89.29
0,0516
Wyner VAE (A 0.15))
89.32
0,0527
Wyner VAE (A = 0.2)
89.28
0.0544
JVAE Vedantam al.
2018
88.11
0.0073
accuracy with a comparably very small per-pixel variance implying much less variations in the generated samples.
E.4.2 ADDITIONAL ATTRIBUTE2FACE GENERATION RESULTS
Here we present additional conditional generated samples (with and without style control) from CelebA models. We used the sample images and attributes shown in Fig. [8 for these experiments.
For conditional generation in Fig. 19 samples were generated only based on the sample attributes.
For conditional generation with style control in Fig. 20 Wyner VAE and CVAE first extracted style information from sample images in the leftmost column, and then generated new samples from the original attribute added with new binary attribute specified in the topmost row for each column. Hence, the second column corresponds to the reconstruction of the style reference images from the faces and the corresponding attributes. We remark that JVAE is not capable style manipulation, and the results from conditional generation with JVAE given reference.
Sample image
from dataset
Arched_ Arched_EEycrree Eycbrows, Attractive, Lips, Big Nose, High Cheekhones, No Beard, Pale_Slap Smiling, Wavy_Hair, Wearing
Lipstick, _eering_Lipstick,
Wearing, Necklace
Big Lips, Blond_Hair, Blurry, Barrow_Eyes, No_ Beard, Smiling, Wearing_Lipstics
Bags_Under__ree Eyes, Eyeglasses, Male, Receding_ Hairline, Sideburns, Weerinng_Necktlee Young
Arched_ Fyybrows,, Attractive, Big_Lips, Brown_Hair, Heavy Makku_Ma MoutthSiightly...mmm Beard, Smiling, Wavy_Hair,
Wearing_ Wearing_Lipstick, Wearing, Necklace, Young
o. Clock __ocklock Shadow, Attractive, Begs Uyesr_Eyes, Big, Nose, Male, Pointy_Noos Wearing_ Necktie
Figure 18: Sample images and their attribute vectors from CelebA dataset.
23

---

Under review as a conference paper at ICLR 2020
Sample image from dataset
Conditional generation of CelebA faces from the attribute vector of the sample image
Wyner VAE (== 0.0)
Wyner [A= 0.
Wyner VAE [A= 0.2)
JVAE
Figure 19: Conditional generation (attributefface).
24

---

Under review as a conference paper at ICLR 2020
Conditional generation of CelebA faces with style tranfer and attribute addition
Added attribute
Style ref.
Wyner VAE 0.0)
Wyner VAE (2 1)
Wyner VAE (2 0.2)
CVAE
IVAE
Figure 20: Conditional generation with style control (attributefface). Note that JVAE is not capable of style manipulation, and the results were simply generated from attribute2face generation and are given as a reference. Hence, the leftmost column is the sample image as in the conditional generation experiment for JVAE.
25

---

Under review as a conference paper at ICLR 2020
F EXPERIMENT DETAILS
We used the same parameterization, same latent dimensions, and the same network architecture across the different models to be fair comparison. For simplicity, we used the standard Gaussian parameterization of each component for all the implemented models as in the standrad VAE. (See Appendix A and Appendx B For the prior distributions po(z), Po(),, po(v), we let the isotropic variances o 0.a. o 1.0 2.8 be trainable. With this degree of freedom, the neural networks select necessary dimensions in the latent spaces over Z, U, V by assigning small variances to unused dimensions. For the MoG experiment, we used constant 1/2 for the decoder variance, so that the log-loss corresponds to the /2-squared loss. For the rest of the experiments, we allowed the diagonal variances be trainable: we allocated one trainable decoder variance per channel, independent from the latent inputs. We found that this trick results in sharper images across the models.
We set the dimension of the latent variable W in JVAE as the sum of the dimensions of Z, U, as W corresponds the joint representation. Similarly, since the latent variable of CVAE corresponds to the local randomness Wyner VAF, we let the dimension of V in CVAE be equal to the dimension of V in Wyner VAE. We used (10,10,10) (32,32,32), (128, 128, 128), and (128,0) 2, 0,128) as the latent dimensions of (Z, U, V) for MoG, MNIST-MNIST add- I, MNIST-SVHN, and CelebA, respectively.
All log-likelihood values in the experiments were estimated by importance sampling: see Appendix D We used 5 100 importance samples for each data point. The mutual information (X, Y; Z) was estimated in straightforward manner with the test dataset under the Gaussian parameterization; see Appendix B
Computing infrastructure
We used NVIDIA TITAN X (Pascal) for our experiments.
Implementation
We implemented all models using Keras with tensorflow backend.
Datasets For the mixture of Gaussians dataset, we generate the paired dataset (X,, Y.) E x as follows. Let j:: 21, 5, + R x R be function defined as
((1) =
= (4, (). = (- 4), ((4) = (-4, -4) (4, -4).
Then, we let
X; = 2 [1 U,
Y. = [u2 11) +V [22 12. V
(60)
[5 U.}
where Z; ~ ( 2, 4, 5)), U. V, ~ ~N(0,, Is) are drawn independently. Here, 11, E R denotes the all-1 vector.
For MNIST-MNIS add-1, we constructed 50k add-1 pairs from the MNIST training dataset. For MNIST-SVHN domain adapation, we constructed 50k MNIST-SVHN pairs from MNIST and SVHN training datasets. For testing, similarly constructed 1k paired images from MNIST and SVHN test datasets each case. For CelebA experiments, we set aside 5k samples for test dataset, and used the rest in training.
denote a 5 x Convolution- u3s2-k- k- (acti i denote 3 x 3 Convolution / Saannolttioo-aachooo----ooo k filters and stride 2 x 2, respectively. Let res-k k be a residual block that contains two 3 convolutional layers with le filters in each (i.e., c3sl-k-LRellle c3s1-k) and a skip connection from the input the output. Let fc-k- (activation be fully-connected layer with is units and non-linear activation.
Network architectures Let
MitchNorm-activatii with k filters and stride x 1. Let d3s2 (activation) with
For MNIST-MNIST add-1, MNIST-SVHN, and CelebA, each Wyner VAE consisted of an outer encoder/decoder pair and core joint Wyner model. The outer encoder/decoder pair was introduced to pre-process raw input data. We summarized the network architectures for the Wyner models in Table 7 Table 8 and Table 10 Note that we padded zeros around the 28 28 MNIST images to make them of size 32 x 32.
hhtps://kers..o
26

---

Under review as a conference paper at ICLR 2020
Table Network architecture for MoG experiments.
Core Wyser VAB
9 8
Pe (x/ja, u), Po (y) v)
Kc-25
Eo-255-Ro50
Fo: 258 Betdd
fo Beed
Ke-256
fo-256-RelJ
(So-10.2o-10)
Ic-10
Table 8: Network architecture for MNIST-MNIST and MNIST-SVHN experiments.
Core Wyner VAR
Outer encoder
Ouser decoder
Po (xl2, u)
ob 91--2211Re22
U.30 2-1 21-22-22--RRR LR
20-512-1Rela (fe-32, (F- fe-128)
1c-512-1Re10
d8 64-1.Be1
nea- 12.
Fo-4095
63w2 128 LRRLD
128
res-128
re5-120
res- 128
rea- 12.
r25-128
1332-66-11ReG1
res-128
0362-32--RRe1
63x2-256
65 Sigroid cfst 53 gignid
Table 9: Network architecture for MNIST quadrant prediction experiments.
Wyzer
(a (x)
9o (M/2, ga v.
Eo-500-22 S00-101
Eo-
10-500-BC0 20-500-820
X0-500-310
(#0-20.Ec-20)
(#c-13.fe-15)
Ic-392
Table 10: Network architecture for CelebA experiments.
(ottribute)
y (inage)
Core Wyner
envoder
encuder
decoder
0,
Pa u), Po
fo-512-1RR11J Fo-512
112-1R11
c5e1-32 -32-1/
3332-1 28-1Ret0
To-1024-1Re10
to-1024-1Rell 1224-111111
Fo 40--1 Siggiid
d3w2 64 -iteld
rex-128 128
fo-1024
4352-120-1Rell
res-120
to-1024-1Be10
t8-1028-LRetU
rea-1 28
res-120
(fo-256, Fo-256)
fo-512/fo-1638 Fo-16384
28
128
2e5-128
13322-6411Ro14
rea-: 2
d3x2: 256
e5x1l 3 Signoid
Training We used the Adam optimizer Kingma and Ba 2014 with learning rate 10-4 4 in training MoG, MNIST-MNIST, MNIST-SVHN and CelebaA models. We used learning rate 5 10-4 in training MNIST quadrant prediction experiment.
For MoG dataset, we trained each model for 500 epochs with batch size 100 and trained each marginal encoder for separate 50 epochs for JVAE and Wyner VAE. For MNIST-MNIST, MNIST-SVHN, and CelebA experiment, each model was trained for 100 epochs with batch size 128, For MNIST quadrant prediction experiment, each model was trained for 1000 epochs with batch size 20.
For the marginal encoder (z) in JVAE or Wyner VAE, we trained the joint models by setting ax-xx Qxy-y Ox ay-ty = 0 and trained only the marginal encoder for every 50 epochs by minimizing Lx-xx with freezing all the components in the joint model for 20 epochs. For
MNIST-MNIST, MNIST-SVHN, and CelebA experiments, we trained ge (2) x) for 1 epoch after every 1 epoch of the joint model training as suggested in Vedantam al. 2018)) Note that the outer encoder/decoder pairs in Tables g and TO were trained with the core joint Wyner model, and they were fixed in the marginal encoder training. For MNIST quadrant prediction experiment, we trained the joint model and the marginal encoders ( and go (zly) by minimizing the final objective (15) by setting all a' o's to be for simplicity without fine-tuning.
We empirically observed that for some experiments the two-stage training with Oxy- = axy- still effective. Note that we applied the joint training scheme and used Oxy- Oxy+ > 0 only for MNIST quadrant prediction experiment to achieve better conditional log-likelihood performance.
27


Reviews:
[{"review_id": "rJxG8duDqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for learning latent-variable models of paired data that decomposes the latent representation into common and local representations. The approach is motivated by Wyner’s common information, and is made tractable through a variational formulation. On experiments with MoG, MNIST, SVHN, and CelebA, the paper shows that penalizing the complexity of the common representation can improve style-content disentanglement and conditional sampling.\n\nWhile I found the formulation and motivation from Wyner’s common information and Cuff’s channel synthesis interesting, the resulting model and experiments were unconvincing. There are many terms in the Wyner model proposed, and there are no ablations demonstrating which terms are important and which are not (e.g. do you need both L_xx and L_xy?). On the toy MoG experiment, the common information is known but trained models do not recover the right amount of information. For representation learning, accuracy decreases as the penalty on the common information increases, and for NLL, the joint and conditional NLL is often similar to existing work (CVAE and JVAE). The main win appears to be for style-content disentanglement, but the results there are qualitative and often change the content when only style is changed. It’s also puzzling as to why CVAE overfits so severely in a subset of the experiments. Without a more thorough evaluation of what terms in the loss matter, and showing that the technique recovers something like Wyner’s common information (by extracting the right information on a toy model), I cannot recommend this paper for acceptance.\n\nMinor comments:\n* Eqn 1: the constraint “Subject to X-Z-Y” is confusing. Do you mean X-Z-Y is a Markov network (undirected) and not a Markov chain (directed), in which case X -> Z <- Y does not correspond to X-Z-Y? (This is addressed in Eqn 3-4, but should be fixed here)\n* How is Wyner’s common information related to the multivaraite mutual information I(X; Z; Y)?\n* When describing distributed simulation, please include the variables and objective like you do for common information\n* s/Markovity condition/independence assumption?\n* Why are both losses in Eqn 4 and Eqn 5 needed? Couldn’t you use either to train q(z|x)?\n* Eqn 10 (and most of your objectives) are still intractable due to the q(x)/q(x,y) terms, you should note that it is constant and dropped from the objective\n* “Style control”: could you define what you mean by style in this context? Prior work could likely also do “style control” by e.g. interpolating subsets of dimensions.\n* Related work: https://openreview.net/forum?id=rkVOXhAqY7 (CEB) that may result in a similar objective as Wyner’s common info\n* When comparing to JVAE/JMVAE, it seems like the main difference is suing a latent-variable in the decoder, but the framework is still the same. It’d be useful to spend more time comparing/contrasting with this prior work.\n* Fig 4: would be useful to have a picture of the samples (right now they’re just in appendix)\n* Fig 4: in the data generating process, I believe I((X, Y); Z) = ln(5) = 1.6 nats, but none of your models converge to rates around there. Why not?\n* Unlike other approaches (JVAE, CVAE) you have additional terms in your loss for conditional prediction at training time. It seems like these may be giving you gains, and it’d be useful to perform ablations over the terms in Eqn 15 to figure out what is impacting performance in Fig 4. E.g. if you set \\alpha_{x -> y} and \\alpha{y -> x} to 0\n* Why do the CVAE models overfit? Have you tried optimizing the variational parameters on the test set if it’s due to amortization gap?\n* Fig 5b: what’s the variance you’re plotting? In representation learning, we often just care about downstream accuracy, and it looks like \\lam = 0  performs best there.\n* Table 3: isn’t this showing that \\lam = 0 works the best for joint NLL and close to best for conditional NLL on the MoG task? What’s the discrepancy with Fig 5 where conditional NLL is highly dependent on \\lambda for MNIST?\n* Fig 6 (a1-f1): for this MNIST add 1 dataset, the only common info should be the label. But it looks like the labels do change for non-zero values of lambda (b1 there’s a 2 -> 7), c1 1 -> 9)\n* Table 4: would be useful to train CVAE yourself, as the small differences in numbers could just be due to tuning / experimental setup. You also should bold everything that’s within stderr (i.e. \\lambda = 0 and \\lambda = 0.15 are equally good)\n* Fig 9: would be useful to include CVAE, and Wyner VAE in this plot, i.e. is it only using the mode information in the latent variable when doing conditional sampling?\n* CelebA results look interesting, but there’s been other work on generation from attributes that presents more visually compelling results, e.g. https://arxiv.org/abs/1706.00409", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skllo4L6tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a variational auto-encoder approach for paired data variables, with a separate notion of common and individual randomness. On top of data reconstruction, Wyner's common information is added as a regularization term to promote the succinctness of the latent common representation in particular. Besides, KL divergence based constraints are proposed to encourage consistency of different modeling components. Although the consistency imposed by KL can be weak, this helps avoid making unrealistic assumptions.\n \nTo make the proposed objective function tractable, the authors make use of several standard techniques such as variational bounds and sampling-based approximations. The proposed approach can be tailored for multiple purposes, including joint/conditional generative modeling, two variable auto-encoding, style extraction and control, as illustrated in Figure 2 and Figure 3. It also supports the multi-stage training scheme, adding more flexibility. The authors did a great job in the literature review and experimental comparison. Table 1 and Table 2 demonstrate the position and distinctions of the proposed Wyner VAE approach. The comparison to the information bottleneck is also interesting.\n \nThe numerical studies are extensive. The authors demonstrate performance improvement quantitatively over several baseline competitors, although the margin is not big. With appropriate regularization, the qualitative results indeed look better subjectively.  The proposed approach stands out as a \"swiss army knife\" approach for two-variable VAE, which could support versatile needs. The shared and individual randomness are proved to be useful in style extraction and style control in the experiments. Overall, I feel the experiment evaluations and explanations are convincing and informative.\n \n\t• The Alice Bob example is helpful to understand the main purpose of this paper \n\t• The authors introduced the theoretical background of Wyner's common information --- the two contexts it arises. These texts are interesting to read, but I feel the connection to the following learning problem is weak. It would be nice to have more justifications on the choice in the context of VAE learning (optimality?), as opposed to other seemingly more natural choices. For example, the sum of mutual information I(X; Z)+I(Y; Z), or other forms of common information\n\t• Although the proposed approach involves many terms, in general, the presentation is good with clear notations. I am just confused about p_tilde (u,v|x,y,z) in (8). Is it simply p(u)p(v)? \n\t• There is a typo in (19) and (20)", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxfbuwRYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a variational autoencoder for a pair of correlated observation variables, motivated by Wyner's common information. An instantiation of the model using reverse KL divergence metric is also provided for model inference. Experimental results on simulated dataset and real image datasets show the effectiveness of the proposed generative model. The paper also provides a comprehensive appendix with technical and experimental details. Overall, the paper is technically sound and well supported by theory and experiments.\n\nHere are a few specific comments and questions about the technical content:\n1) For most of the data modeling tasks in real applications, we will most likely have one observational variable (speech segment, document or image). How would the proposed VAE model be applied when the two correlated variables are not explicitly observed or defined? It would be nice if the authors can provide some discussion on the general applicability of the  proposed model.\n\n2) A common challenge of information theoretic VAE is the analytical intractability of divergence measure. For example in Zhao et al. 2018, the maximum mean discrepancy measure is used to approximate the divergence. It is unclear to me whether the objective function in Section 2.3 (using the reverse KL divergence measure) is analytically tractable or not. If not, what kind of approximations (e.g., sampling) did the author used?\n\n3) In the experimental section, the authors showed that for joint and conditional distribution modeling tasks, the optimal regularization parameter might be different. This implies that for a specific task, the user is responsible to choose a proper  \\lambda value. I wonder whether the authors can provide some general guidelines on how to choose this important parameter in practice, as the results seem to be highly dependent on the choice.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlVHKMaqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes Wyner VAE, a variational autoencoder for a five-variable graphical model. The five variables consist of two observable views, X and Y, with a shared latent variable Z, while the two other latent variables U and V control the randomness of X and Y, respectively, independent of others. The overall objective is motivated by Wyner's mutual information minimization. Using a few common techniques to bound KL divergence, the paper arrives at a few terms that are reminiscent to the variational lower bounds, including terms that constraints the hidden variable to be similar to the prior, and two cross-modal reconstruction terms. The paper includes a comparison between similar models, both in terms of formulation and experiments. The results show that Wyner VAE has certain nice properties that is lacking in other models.\n\nI am giving a score of 6. The formulation and the lower bound for learning are the strengths of the paper. The lack of discussion on some of the common problems for VAEs and the experiments are the weakness of the paper.\n\nThe paper deserves credit to motivate the problem from Wyner's viewpoint. However, after the derivation, the framework falls back to common VAEs, and is not very different from VCCA-private. Comparing Wyner VAE and VCCA-private, I can see why it is hard to justify the difference of the two based on the formulations, but I also find it unconvincing from the experiments to conclude that one is better than the other, especially when there are many hyperparameters to tune. The paper should state this clearly when comparing the two.\n\nIn terms of the common problems for VAEs, the paper does not address a few degenerate cases. The first case is where all the information gets pushed to the shared latent variable Z, leaving little private information for U and V. The second case is that little information about both views gets pushed to the shared latent variable Z, while the decoders are doing all the heavy lifting to reconstruct the observables. To avoid these degenerate cases, more constraints might be needed for the models, such as the ones proposed in (Hsu et al., 2017). Finally, as with most VAE models, the paper does not discuss how well lower bounds approximates the likelihood and the resulting learned representation when the gap between the likelihood and the objective is large. \n\nThe experiments in the paper are somewhat lacking. Figure 7 is where the capability of the model is fully demonstrated: a clear distinction between the shared and the private views are shown. It would be great if the paper can include experiments on a real-world multi-view data set, such as images paired with texts, speech paired with images, speech paired with video, etc.\n\nUnsupervised learning of disentangled and interpretable representations from sequential data\nWei-Ning Hsu, Yu Zhang, James Glass\nNeurips, 2017\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper adds a new model to the literature on representation learning from correlated variables with some common and some "private" dimensions, and takes a variational approach based on Wyner's common information.  The literature in this area includes models where both of the correlated variables are assumed to be available as input at all times, as well as models where only one of the two may be available; the proposed approach falls into the first category.  Pros:  The reviewers generally agree, as do I, that the motivation is very interesting and the resulting model is reasonable and produces solid results.  Cons:  The model is somewhat complex and the paper is lacking a careful ablation study on the components.  In addition, the results are not a clear "win" for the proposed model.  The authors have started to do an ablation study, and I think eventually an interesting story is likely to come out of that.  But at the moment the paper feels a bit too preliminary/inconclusive for publication.
Decision: Reject