Information Theoretic Regularization for Learning Global Features by Sequential VAE
Authors: Kei Akuzawa
Yusuke Iwasawa
Yutaka Matsuo
Venue:  (2021)
Keywords: variational autoencoder
VAE
disentanglement
global features
sequential models
representation learning
mutual information

Abstract: Sequential variational autoencoders (VAEs) with global latent variable $z$ have been studied for the purpose of disentangling the global features of data, which is useful in many downstream tasks. To assist the sequential VAEs further in obtaining meaningful $z$, an auxiliary loss that maximizes the mutual information (MI) between the observation and $z$ is often employed. However, by analyzing the sequential VAEs from the information theoretic perspective, we can claim that simply maximizing the MI encourages the latent variables to have redundant information and prevents the disentanglement of global and local features. Based on this analysis, we derive a novel regularization method that makes $z$ informative while encouraging the disentanglement. Specifically, the proposed method removes redundant information by minimizing the MI between $z$ and the local features by using adversarial training. In the experiments, we trained state-space and autoregressive model variants using speech and image datasets. The results indicate that the proposed method improves the performance of the downstream classification and data generation tasks, thereby supporting our information theoretic perspective in the learning of global representations.

Under review as a conference paper at ICLR 2021
INFORMATION THEORETIC REGULARIZATION FOR LEARNING GLOBAL FEATURES BY SEQUENTIAL VAE
Anonymous authors Paper under double-blind review
ABSTRACT
Sequential variational autoencoders (VAEs) with global latent variable have been studied for the purpose of disentangling the global features of data, which is useful in many downstream tasks. To assist the sequential VAEs further in ob- taining meaningful 2, an auxiliary loss that maximizes the mutual information (MI) between the observation and is often employed. However, by analyzing the sequential VAEs from the information theoretic perspective, we can claim that simply maximizing the MI encourages the latent variables to have redundant in- formation and prevents the disentanglement of global and local features. Based on this analysis, we derive a novel regularization method that makes & informa- tive while encouraging the disentanglement. Specifically, the proposed method removes redundant information by minimizing the MI between & and the local features by using adversarial training. In the experiments, we trained state-space and autoregressive model variants using speech and image datasets. The results indicate that the proposed method improves the performance of the downstream classification and data generation tasks, thereby supporting our information theo- retic perspective in the learning of global representations.
INTRODUCTION
Uncovering the global factors of variation from high- nigh-dimensional data is a significant and relevant problem representation learning (Bengro et all 2013). For example, global representation of images that presents only the identity of the objects and is invariant to the detailed texture would assist in downstream semi-supervised classification etll, 2019). In addition, the representation is known to be useful in the controlled generation of data. Obtaining the representation allows us to manipulate the voice of the speaker in speeches (Yingzhen Mandì, 201)) or generate images that share similar global structures (e.g. the structure of objects) but varying details (Razavi all 2019).
Sequential variational autoencoders (VAEs) with global latent variable have played an important role in the unsupervised learning of the global features. Specifically, we consider the sequential VAEs with a structured data generating process in which an observation x at time t (denoted as x;) is generated from a global feature and local feature 8t- Then, the of such sequential VAEs can acquire only global information invariant to t. For that a disentangled sequential autoencoder (DSAE), which combines state-space models (SSMs) with a global latent variable 2, can uncover the speaker information from speeches. Furthermore, Chen et (201Z); Gulrajani et all (20IZ) proposed a VAE with a PixeICNN decoder (denoted which combines autoregressive models (ARMs) and 2. In both methods, the
(Yigzzhen & Mandi (2018) demonstrated
as PixeICNN-VALE hidden state of the sequential model (either SSMs or ARMs) designed capture local information, while an additional latent variable captures global information.
Unfortunately, the design of the aforementioned structured data generating process alone is insuf- ficient to uncover the global features in practice. A typical issue is that latent variable is ignored by a decoder (SSMs or ARMs) and becomes uninformative. This phenomenon occurs as follows: with expressive decoders, such as SSMs or ARMs, the additional latent variable cannot assist in improving the evidence lower bound (ELBO), which the objective function of VAEs; therefore, the decoders will not use (Chen et all 2n Alemi et all 2018). The phenomenon in which the latent variables are ignored is referred to as posterior collapse (PC). To alleviate this issue, several studies have proposed regularizing the mutual information (MT) between x and be large, e.g.,

---

Under review as a conference paper at ICLR 2021
1. Maximize To:
Maximize
20
K
MRxx ()
+
Hix (2)
x s)
fots [2)
Minimize Nead
1xx; 21))
Increase (
1x Kes
[or; x)
Increase
K
Mic 1ap
Na (a)
disentangled
Htzl
entangled and
and
(a) Ml-maximizing regularization
(b) Proposed method
Figure 1: Comparison of (a) MI-maximizing regularization and (b) the proposed method, using a Venn diagram of information theoretic measures of æ, 2, and 8.
using 3- VAE (Alemi et all 2018). A higher MI [(;; 2) indicates that x has significant information regarding æ; this regularization prevents from becoming uninformative.
In this paper, we further analyze the MI-maximization and claim that merely maximizing [(x; insufficient to uncover the global factors of variation. Figure (a) summarizes the issue of the Ml-maximization. As illustrated in the Venn diagram, the MI can be decomposed into [(x; z) =
=[s) + (x;; 8). Maximizing the first term (x; (s) beneficial, as it measures the infor- mativeness of about x given a local feature s. However, maximizing the second term [x;; 2; might cause a negative effect, because would also increase [(2;)), In other words, maximizing [ 2) would encourage latent variables have redundant information. For example, when / [(x; 2) becomes so large that z retains all (local and global) information of x. the downstream classification performance would be degrated. Also, when local variables still contain global information due to large (22; s). it becomes difficult to control speaker information in speeches using a DSAF, See Appendix A for empirical evidence that the Ml-maximization increases (2; s), as discussed above.
Based on the analysis, we propose new information theoretic regularization method for disen- tangling the global factors. Specifically, our method minimizes (22; s), in addition to maximizing [(x;2] 2) similar to prior work (Figure (-)).. As [(z;s) s) measures the dependence between 2 and s, our method encourages 2 and have different information, i.e., the disentanglement of global and local factors. We call our method CMI-maximizing regularization, as it is the lower bound of the conditional mutual information (CMI) [a;; s). Furthermore, we introduce an adversarial training technique for estimating the CMI. A simple way to estimate it would be considering [(x; 2) and (z;;) independently, but might result compounding approximation errors. Instead, we use the formularization of 8-VAE and adversarial training (Ganin et all 2016), which reduce the number of terms to be approximated. Specifically, we approximate the upper bound of [(2; s) using density ratio trick (DRT) (guuye et al, (208)), where adversarial classifier models the density ratio. Once we estimate the bound, [[2; 8) can be minimized via backpropagation through the classifier.
In our experiments, we used DSAE and PixeICNN-VAE illustrative examples of the SSM and ARM variants. In addition to evaluate the quality of global latent variable as in previous studies, we also evaluated the ability of controlled generation using novel evaluation method inspired by Ravun & Vinyals (2011).. In the experiments, the CMI-maximizing regularization consistently out- performed the Ml-maximizing one on image and speech datasets. These results support (i) our in- formation theoretic view of learning global features: the sequential VAEs can suffer from obtaining redundant features when merely maximizing the MI. Also, the results support that (ii) regularizing [a:; z) and [(z; 8) is complementary: learning global features can be facilitated by not only making informative, but also the control for which aspect of information (global or local) goes into
Our contribution can be summarized as follows: (i) through our analysis, we reveal the potential negative side-effect of MI-maximizing regularization, which has been standard learning global representation with sequential VAEs. Then, the analysis encourages the sequential VAE commu- nity to seek for new regularization approach. (ii) In order to learn good global representation, we proposed regularizing [(x; 2) and I(2; s) at the same time. [[2; 2) and [[x;2) are robustly shown to work complementary by our experiments using two models and two domains (speech and image datasets). This finding would help improve various sequential VAES proposed before.
2

---

Under review as a conference paper at ICLR 2021
M
$
%
A
(a) DSAE (SSM with global z)
(b) PixeiCNN-VAE (ARM with global :)
Figure 2: Graphical models for (a) DSAE and (b) PixelCNN-VAE
PRELIMINARY
2. SEQUENTIAL VAES FOR LEARNING GLOBAL REPRESENTATION
Here we first explain the standard VAE; then, we give overviews of the DSAE and PixcICNN-VA VAE. Both models are shown be interpreted as having two types of the latent variables, global z and local 8;; although it is not explicitly stated for PixeICNN-VAL VAF. Here, is designed to influence particular timesteps or dimensions of x (c.g., single-frame in a speech or a small area of pixels in image). On the other hand, influences all the timesteps of æ, although of DSAE and PixeICNN VAE are imposed on different architectural constraints (discussed in Appendix ()).
Variational autoencoder (VAE) Let p(z) f p(=)p(=|z)dds be latent variable model, whose decoder p(æ|2) is parameterized by a deep neural network (DNN). Using an encoder distribution (<211)) which is also parameterized by DNN, the VAEs maximize ELBÓ:
LELBO == Epa(=) [Bg(=(e) [lo
[[ogp([/>)) p(+|=)] - DRL (=<)\=))l<<=))]
(1)
Here, Pa(x)) denotes the data distribution. ELBO contains terms: the reconstruction error and the Kullback-Leibh (KL) divergence between encoder (2(22) and the prior p(2).
Disentangled sequential autoencoder DSAE (Yingzhen Mandi, 2018) an extension of the SSMs for modeling the global and local features of sequential data separate latent variables. Using disentangled variables, DSAE can control the outputs (e.g., perform voice conversion). DSAE
has global latent variable and local latent variable and generates an observation at time / from and 8,. The ELBO can be expressed as follows:
LeSM == Recon - KT((2) - KT.(s),
where Recon = -B (x,,,
log (og((1)))) KT((=) = Pate
KL(s) ee
DEa DKI(g(se/ 2, (10))))
Here, p())((1-1) 1)) is prior, ((2(()r)) and (( (TrT, 8t-1) are encoders, p(xr/88, 2) is a decoder, and ( 2,8) 8) ((()))(((()))((((=)))))) 2). Furthermore, dhet denotes all the elements of the sequences up to t, and r denote I = ØgT. Figure (+(a) illustrates the data generating process.
VAE with PixeICNN decoder PiXelCNN- VAE designed to take advantage of both PixeICNN and VAEs. VAEs are known to fail in terms of capturing the local features of images, such as textures and sharp edges. Conversely, PixeICNN good at capturing the local features, but often fails to generate globally coherent images and has no latent variables. Then, successfully trained PixeICNN- VAFS would generate high-fidelity data and induce latent variable 2, which maintains only the global information by discarding the local information (Gulaajan et al. 2017))
PixelCNN- VAE can be interpreted as structured VAE which x; is generated from the global latent variable and local variable as follows. First, the autoregressive decoder is expressed as (2))r(2) = I7 x2(). This means that for every timestep t, x, is sampled from p(z,12, using previous observations Z<t and the latent variable 2. Secondly, the decoder can be decomposed as p(zz22, xxr) = p(z,|z, fZ using a neural network f (Pix- eICNN). Finally, we introduce random variable and its distribution q(sx(anee) = p(se(xee)) =
assume that
3

---

Under review as a conference paper at ICLR 2021
6(st - f(x=e), where 8 denotes the Dirac's delta, and ((5)(==11) p(s{(z<<) is employed to simplity the notation. With this notation, the decoder can be decomposed as 2(z))2,2<1) = p(zz,2, (()()) (<x)) (details the practical decomposition using PixelCNN have been provided Appendix [D. Thus, It can be regarded to be generated from z and 8t, which sampled from p( (2nee) (see, Figure (-()).. Furthermore, the ELBO is given follows:
ARRM = Recon - KL(z).
2.2 MUTUAL INFORMATION- ANPORMTTON-MAXIMIZNNG MAXIMIZING REGULARIZATION FOR SEQUENTIAL VAES
Despite the intentional data generating process of the sequential VAEs, the global latent variable z often becomes uninformative. To alleviate this issue, MI-maximizing regularization methods are often employed to encourage z to have æ information. Note that, here we consider the MI defined by the encoder (which corresponds to the representational MI in Alemi el all (2018)):
[x;; 2) (pp(())())))) [log pa(z)g(=(1))
(4)
pa(z))q(2)
A representative example of the MI-maximizing regularization B-VAE, which was shown to work well in Alemi aD (2018) and used baseline in ell (2019) (other methods are presented in Section E). Because the ELBO (Eq. M contains a positive lower bound and a negative upper bound of I [2; 2), the MI can be controlled by balancing the two terms using a weighting parameter B. The concrete 3- B-E objectives for DS and PixeICNN-VAE are:
VSSM Recon (*L((2) KL(s),
(5)
VARM == Recon - 3KK(2))
(6)
Alemi et all (2018) use B < 1 to regularize [(2; 2) to be large, although 8- was originally invented to encourage the independence of each dimension of with 3 by Higgins at (2017).
PROPOSED METHOD
3.1 DECOMPOSITION OF MUTUAL INFORMATION
Sequential VAES with a global latent variable can in principle uncover global representation of data by exploiting its structured data generating process. Previous studies for the sequential VAEs have further regularized mutual information [2x; z) to be large in order to alleviate posterior collapse (PC) (further discussed in Section H). Unfortunately, the MI maximization is insufficient to uncover the global factor of variations, because it cannot control the type of information going into z. More specifically, as indicated in Section the MI is decomposed as follows:
[xx;)) = [x;;;;) + I (z; s|s) [2;; - [(2; + [(r;
Simply maximizing [(x;; =) can increase (2;;)) in the right- side, which is observed in our pre- liminary experiment in Appendix A. When 1 (2;;) becomes large, z is likely to have redundant local information, conversely s becomes to have global information. Which phenomenon occurs could depend on the network architecture, as discussed in Appendix Q In both cases, the performance of downstream tasks, e.g., classification from 2 to the labels, or controlling the global characteristics of the decoder output using 2, is likely to be degraded. Furthermore, as well as the MI I [2; 2). this MI T(z; 8) is defined by the encoder distribution ((2, 8) (Appendix E)) Then, although the graphical model of DSAE is designed such that and are independent, T(z; s) is not necessarily zero, i.e.,
p(z, 8) = p(z)p(s) does not necessarily mean q(z, s) = g())g(s).
3.2 CONDITIONAL MUTUAL INFORMATION MAXIMIZING REGULARIZATION
Considering the limitations of MI regularization, need method that can encourage both (i) the increasing of [(x; 2) to prevent = from becoming uninformative, and (ii) the decreasing of [(2;s) s) to prevent and from having information that is irrelevant the global and local structure, respec- tively. Therefore, we propose maximizing the following objective as a regularization approach:
T(x; (2)
(8)
4

---

Under review as a conference paper at ICLR 2021
As (2; measures the mutual dependence between and minimizing (2; encourages and s not to have redundant information. Then, the induced global variable would have more information, while and maintains only the global and local information, respectively. The o a weighting parameter for balancing the two terms. In practice, we found that a = works reasonably; therefore, we used for the reminder of the study.
Furthermore, noteworthy that our method closely related to CMI estimation. Specifically, when assuming a 2 1,
[x;; =|s) = =) - (2; - 8(1)) 2 [x;; - (2; =: ICMP-
It indicates that TCMP ===; (2; is equal to the lower bound of CMI (further discussed in Appendix DD. CMI is known to be useful in selecting the features that are both individually informative and two-by-two weakly dependant (Eleurei, M003). Therefore, maximizing [(x;; z|s) as regularization would make the features 2 and s informative but disentangled.
Then, we present one of the tractable instances estimate IcMr' A simple way to estimate ICMI' may be to consider I [x;; 2) and s2);)) independently; however, it must approximate both. [(x; 2) and [2;; s), which may complicate optimization. Fortunately, when we assume a = 1, we can reduce the number of terms to be approximated to only one, utilizing the 3-VAR formularization. First, we express /CMP as follows (the derivation is given in Appendix Bi:
IcMr' [x; z) [(;;a) s) Epa(z) [DRL. (a(=|=)
(xI(((z, ()))(())()))
(10)
The first term the same as KL(z) Eqs. [S] and E, and is used for the 3-VAE formulariza- tion. The second term is the upper bound of (2;;)) because (rz,(g(z, s) () [(2;)) +
(xL(g())lp(z))
Because the second term is difficult to calculate analytically, we estimate it using the DRT Nguyen
atall 2008: Sugryama et all 2012)), as performed in standard generative adversarial networks (Mo hamed & Lakshminarayanan 201Z). By introducing the labels
1 for samples from q(2, s) and y = 0 for those from p()) we re-express these distributions in conditional form, i.e.,
p(z, sly = 0). The density ratio betwwen g(2, 8) and ((z)((8) can be computed using these conditional distributions as follows:
q(2 ((z := sly, = 1) and p(=)q(s)
((2,8) p(2, sly 1) p(v 1/2,8)
(11)
p(z)g(s) p(z, sly = 0) pVy s)
where we used Bayes' rule and assumed that the marginal class probabilities equal, i.c. p(y 0) = p(y = 1). Here, p(yl2,s) s) can be approximated with a discriminator D(2, 8). which outputs D- = when 2,8 wi.id. q(2, 8), and D O when 2, approximated as follows:
wiid. (())p(=). Then, Eq. I can be
D(2,s)
TCMP | (PkL((((()))(((]))) Bycs.s) s) [log 1- D(2,)2
ICMI-DRT DRT-
(12)
We parameterize s2,,s) with DNN and train it alternately with the VAE objectives. Specifically, we train D to maximize the following objective with Monte Carlo estimates:
[g( (oog (22, s)] + Bp(c)ots
D(2, s)]]
Now, we can introduce the concrete objectives of the DSAEs and PixelCNN- VAES with CMI regularization term. Adding ICMI-DRT regularization term to Eqs. & and E with a weighting parameter 7, we obtain the objective functions of our proposed method that need to be maximized:
JSSM == LssM + 7 - Recon KT((s) (1 - (T) (2; JARM = LARM + I - RReoon (1 - (2;
(13)
(14)
D(2,8)
where (2(2;)) = Eg(2,,) log 1 D(2,s)
Considering (1- (1-))K((3 (2) is equivalent to the weighting technique in 8- VAE (note that 1 1- 3 and see Eqs. 5 and R) the proposed method consists of the 3-VAE objective and (1' s). As noted in Section 2,2 3- VAE is effective for alleviating PC. However, because 3 3-VAE alone is insufficient for decreasing the redundancy of and s, minimizing 7'(2, 8) is employed.
5

---

Under review as a conference paper at ICLR 2021
Finally, we discuss alternative choices to estimate the second term of Eq. III While we chose to approximate the term with a discriminator, it can also be approximated with other distance such as maximum mean diseripancy (MMD), or it can be minimized via Stein variational gradient (sce, Zhaa at (2019). However, a weakness of these methods is that they are difficult to apply efficiently in high dimensions. Unfortunately, because the second term treats the random variable [z, 12,81:--9 sT), the dimension size becomes high when I is large. On the other hand, adversarial training requires only one assumption, i.c., D perfectly approximates the true density ratio, In practice, while this assumption does not always hold true (Moyer et all 2018: wassawa et all 2020), it is also empirically known that original objectives (in our case, minimizing DKL (g(2, 8) Ip (g(()) }) can be reasonably achieved (Ganin et all 20TG). In addition, various studies (e.g., |Myyto et all (2018); Iwasawa et all (2020) have proposed techniques to improve the robustness of adversarial training, and has been shown to scale to high dimensions when carefully designed (Brock etal, 20T9).
4 RELATED WORKS
This study is closely related to the literature on disentangled representation. Locatello el al (2019)
claimed that pure unsupervised disentangling (Cheneeaal 2016 Higgms et al, 2017: Kim Mnth 2018) is fundamentally impossible, whereas using rich supervision (Kulkarn et all. 2015) can be costly. Thus, the use of inductive bias or weak supervision (Shu etl 2020) has been encouraged. The assumption that data are generated from global and local factors is representative example of the inductive bias. Such data generating process can be probabilistically expressed by the sequential VAEs with a global latent variable. Then, the sequential VAEs have been studied for disentangling styles and topics of texts (Bowman et all 2016), object identities from the detailed textures of images eLll 2017, content and motion of movies (Hsieh all 20T8), and the speaker and linguistic information of speeches (Hss et all 2017: Yingzhen & Mandi, 2018). Although this paper focused on DSAE and PixeICNN-VA VAE as examples, our method could be also combined with them.
Unfortunately, the design of the structured data generating processes alone is often insufficient to learn the global features. To address this issue, Boowman et (2016); Chen et all (2017) initially proposed to weaken the decoder because PC often occurs when using highly expressive decoders. Subsequently, various methods have been proposed control the MI (xx; with a regularization term, which does not require problem-spee specific architectural constraints of [Bowman et all (201));
Chen all 7) Concrete examples of MI-maximizing regularization methods are as follows: VAFE (Zhao et all 2019) estimates T(x;; 2) using the MMD or adversarial training. --VAE: Alemi all (201)) proposed targeting a specific rate (the KI. term value) via 33-AEE, , and observed that the objective with 3 < 1 produces solutions to alleviate PC. 3-VAE is simpler than Info
since it does not require an approximation of zx) Auxiliary loss: (Lucas & Verbeek, 2UEE
uses the auxiliary tasks of predicting x from z, which approximates the minimization of conditional entropy (x)2). The minimization of H (x]2)) is equivalent to maximizing (ar; 2) because the data
entropy (xx) is constant. Discriminative objective: (Hsu et all, 2017) predicts a sequence index from z, which also approximates H (z]2z) minimization in the finite sample case.
Various studies have attemted to separate relevant from irrelevant information via information- theoretic regularization. Namely, the studies in the literature regarding domain-invariant representa- tion proposed to learn the invariant representation using adversarial training (Gann 2016; Xid
ctall, 2017; all 2018), variational information bottleneck frameworks (Moyer et alj, 2018), or Hilbert-Schmidt independence criterion (Jarswal et all 20TT). Our regularization term of minimiz- ing [(2; s) is inspired and similar to these studies; however, it differs in considering PC at the same time (i.e.. maximizing [(2; 2)). Also, the separation could be achieved by the design of network ar-
chitectures, as was performed in VQ-VAF2 (Razavi et all 2019). Our proposal is the regularization term and orthogonal to such architecture choices.
Also, our analysis is similar to that of Moyer et all (2018), but differs in two ways. Firstly, while the phenomenon that "large 1 (x; 2) results in large / (z;; 8)" was discussed by them, the whole mech- anism that "MI-maximizing regularization for alleviating PC has negative side-effect to increase [(2; 8)" has been overlooked in the sequential VAE community. Secondly, by explicitly considering the relationship between the two latent variables and s, our analysis is able to highlight new problem. For example, Moyer et al. (2018) consider the relationship between the latent variable z and the observed nuisance factor s. Then, their focus is only removing the redundant information

---

Under review as a conference paper at ICLR 2021
from 2. On the other hand, our analysis highlights the need to consider removing the redundant information from s at the same time as removing the redundant information from 2. Although the former has been overlooked, it is an important issue in applications such voice conversion.
Some studies have also proposed methods for alleviating PC, which are complementary to MI max- imization. Lucas et all (2019) argued that the variance of the decoder influences the stability of local stationary points corresponding to PC. He al (2019) proposed a method that remedies ill-training- dynamics. Our study differs in aiming at obtaining informative and disentangled representation with sequential VAEs, although they could, in principle, be combined with our method.
From technical perspective, our work is also related to feature selection technique based on CMT (Eluuret, 2002)). CMI is known to be useful in selecting the features that are both individually in- formative and two-by-two weakly dependant. Then, the CMI-based technique different from the MI-based one in considering the independence of the features. Moreover, it is different from previ- ous studies for disentangled representation learning, e.g.. Higgins et all (201 iiim & Mnili (2018); all (2II)) control only the independence of latent factors. Also, Mukherjee et all (2(1)) first proposed the estimation of CMI using DNNs; however, our method is different in utilizing the en- coder distribution of VAEs to improve the estimation (Zhao et all, 2019; Poole et all, 2019).
5
EXPERIMENTS
5.1 SETTINGS
We performed experiments to confirm the effect of regularizing both [(x; 2) and I(2; s) for learning good representation, using DSAE and PixeICNN VAE as representative examples of the sequential VAFs. We used the speech corpus TIMIT (Garotolo elll (992)) for the DSAF, and evaluated repre- sentation quality using speaker verification task, as was performed in previous studies (Hsu et all, 2017: Yingzhen & Mandi, MITS). For PixeICNN- VAE, we trained the VAE with a 13-layer Pixel- CNN decoder on the statically binarized MNIST and Fashion-MNIST (X1ao et al, (0177) datasets. Using the trained models, we performed linear classification from 2 to class labels to evaluated representation quality, was performed in (Razavi et all 2019), and then evaluated the ability of controlled generation. z, which has a dimensional size of 32, was concatenated with the feature map outputted from the fifth layer of the PixeICNN (which corresponds to s, see Appendix E), and was passed to the sixth layer. Further details are given in Appendix G..
As the proposed method, we employed the objective functions JsSM and JARM in Eqs. I and [ (denoted as CMI-VAE). We implemented a discriminator D as a CNN that receives and : as inputs (Appendix M. and trained it alternately with the VAEs. As baseline methods, we employed 3 3-VAE (see, Section 2.2). The objectives of 3-VAE are given in Eqs. R and E and are equal to EMI-VAE, except for not having the I' (2; s) term. Moreover, we employed the regularization method proposed in Makhzani & Frey ()017); Zhao et all (2019), which directly estimates and maximizes I [(x; =) with adversarial training (denoted as VI.VAE). In the method, l(x; 2) is added to ELBO (Eqs. and R) as regularization term, along with weighting term 7 (details can be found in Appendix H).. The performances of the models were verified by changing the value of 7.
5.2 SPEAKER VERIFICATION WITH DISENTANGLED SEQUENTIAL AUTOENCODERS
For quantitative assessment of the global representation of DSAE, we evaluate whether z can un- cover the speaker individualities, which are the global features of speech. Specifically, we extract and s<7 from the test utterances using the mean of the encoders of the learned DSAE. Subsequently, we performed speaker verification by measuring the cosine similarity of the variables and evaluated the equal error rate (EER). Here, EER is measured for both and s (denoted as EER(2) and EER(S), respectively). and s<r is averaged over each utterance prior to its measurement. A lower EER(2) is preferable because indicates that the model has an improved global representation, containing sufficient information of the speakers in linear-separable form. Furthermore, a higher EER(s) is preferable because it indicates that s does not have the redundant speaker information. In addition, report KL(²) (see, Eq. D) which approximates the amount of information in 2.
Table M1 presents the values of KL(z) and EER for the vanilla DSAE, 3- VAE, and CMI-VAE, Note that our results for vanilla DSAE differ from those reported in Yingzhen & Mandi (MS) (DSAF*
7

---

Under review as a conference paper at ICLR 2021
Table 1: KL term and EER values of DSAE trained using TIMIT. Each model was trained with
weight y. The 1 and & indicate that the purpose was to obtain a high and low score, respectively.
Model
KL(2)
EER(2) 4
EER(s) 1
DSAE"
0.00
4.82
18.89
DSAE(our implementation)
18.00
11.01
18.64
+ B-VAE
0.40 (3 0.6)
53.28
3.88
29.45
CAI-VAE
54.13
3.43
30.96
3-VAE
0.80 (3 0.2)
145.88
4.33
38.84
CMI-VAE
145.09
3,99
41.30
B-V
0.90 (3 1e
202.52
4.55
39.42
CMI-VAE
199.89
4.39
41.25
3-VAE
0.99 = 2)
364.71
6,33
38.63
CMI-VAE
361.03
5.06
40.08
in the table), which may be due to differences in the unreported training settings. The table presents that (1) a lower y (such as 0) provides lower EER(s), which indicates that have global information instead of owing to PC, without regularizing (a; 2). Furthermore, (2) given fixed % CMI-VAN MAI-VAE consistently achieved a lower EER(2) and a higher EER(s) while having the same level of KL(2) compared to 3-VAB. This indicates that regularizing (=;; is complementary Ml-maximization (3-VAE), yielding better 2 and that have sufficient global or local information but are well compressed. Note that 7 0.8 yields a higher EER(z) than ? 0.4. which may be due the fact that the independence of each dimension of 2 is worsened by increasing 7, as indicated in Higgins etall (20T7), and the induced non-linear relation cannot be measured by the cosine similarity. In fact, 2 0.8 presented a better performance in the voice conversion experiment in Appendix D. indicating that z with y 2 0.8 has more global information, although the EER(2) lower. Also, note that the EER(Z) reported in Hsu et (2I) is lower than the results for CMI-CAE here. However, believe that our claim, "regularizing / (a;; z) and (2; is complementary". defended even if we could not achieve state-of-the tite-of-the-rrr art results.
5.3 VAES WITH PIXELCNN DECODER
Unsupervised learning for image classification For a quantitative assessment of the representa- tion z of PixeiCNN- VAES,, we performed logistic regression from the class labels y on MNIST and Mushion-MNISTT Specifically, first, we extracted from 1000 training samples using the mean of q(=lar), where each of the 10 classes had 100 samples, and trained the classifier with total of 1000 samples. Then, we evaluated the acccuracy of the logistic regression (AoLR) on the test data. high AoLR indicates that 2 succeeds in capturing the label information in a sinear-separable form.
Figures Mta and 3(b) present AoLR for 3-VAE, MI-VAE, and CMI-VAE, along with the ELBO and KL(z). the figures, an upper left curve indicates that the method balance better compression (low KL(=)) and high downstream task performance. As shown in the figures, given fixed the AoLRs for CMI-VAL are consistently better than those for 8- VAE and MI-VAE, although all the methods have the same level of KL(z). This indicates that CMI-VAE
extract more global information when compressing data to the same size as 3-VAE does. Note that small y (such as 7 = 0) and very large 7 degrade the AoLRs, which may be attributed to the same reason explained in Section 5.2 Furthermore, the AoLRs of MI-VAE are lower than those of 3-VAE, which may be due to the adversarial training in MI-VAE causing optimization difficulities, stated in Alemi all (201S).
Controlled generation Most previous works (Yinzhhen & Mandi, 2018:
2010) have
primarily focused on evaluating the quality of global representation. However, a better representa- tion does not necessarily improve the performance of the controlled generation, as Nie et all (2020) claimed. Then, to evaluate the ability of the controlled generation, we propose a modified version of the classification accuracy score (CAS) (Ravuri & Vinyals, 20T9), called mCAS. CAS trains a classifier which predicts class labels only from the samples generated from conditional generative models, and then evaluates the classification accuracy on real images, thus measuring the sample quality and diversity of the model. CAS is not directly applicable to con-conditional models such as PixeICNN- VAEs. Instead, mCAS measures the ability of the model to produce high quality, diverse, but globally coherent (i.e., belonging to the same class) images for given
8

---

Under review as a conference paper at ICLR 2021
0.73
0.78
0.65
0.85
0.5 6.6
0.7
0.72
0.76
0.7
0.00
0.9
0.71
0.71
0.3
0.4
0.4 5
01.00
Aal
0.70
0.3
0.8
mCA5
0.72
mCAS
E244
0.7
0.70
0.75
CMI-VAE
3-VAE
0.60
0.0
0.68
0.2
0.46
0.88
MI-VAE
0.00
0.70
0.67 0
501
0.60
50
ELRO
KL(z)
KL(a)
KL(z]
(a) AoLR (MNIST)
(b) AoLR (Fashion)
(c) mCAS (MNIST)
(d) mCAS (Fashion)
Figure 3: Comparison of CMI-VVE with 3-VAE and MI-VAE. Each maker for 3 V-V is annotated with the value of In the figures, upper left curve is desirable because it shows the method balance better compression (low KL(z)) and high downstream task performance (AoLR and mCAS, explanations in Section 5.3) Also, detailed results can be found in Appendix K
In mCAS, we first prepared 100 real images { along with their class labels {yi J where each of the 10 classes had 10 samples. Then, using the trained VAEs, we encoded each x; into 24, and decoded 2, to obtain 10 images { 1= for every 24, thereby resulting in 1000 generated images (sample images can be found in Appendix K. Finally, we trained the logistic classifier with the pairs {(is.) Vi)li E (1,..., 100). { 10}} and evaluated the performance on real test images. Intuitively, when the decoder ignores 2, the generated samples might belong to a class different from the original ones, which produces label errors. Moreover, when has excessive information regarding and the VAE resembles an identity mapping, the diversity of the generated samples decreases (recall that 10 samples are generated for every zj), which induces overfitting of the classifier. Therefore, to achieve high mCAS, should capture only the global (label) information.
Figure 3 and s(d] compares the mCAS along with KL(2) on MNIST and MNSIS..-MNIIT. In addition, the black horizontal line indicates the classification accuracy when the classifier is trained on 100 real samples [(x;; )=) and evaluated on real test images, which will be referred to as the baseline score. The following can be observed from the figures: (1) The mCAS of the three methods outperformed the baseline score, despite using only 100 labeled samples, as well as in the baseline score, indicating that properly regularized PixeICNN-VAEs could be used for data augmentation. (2) As expected, a significantly low KL(2) gives a low mCAS because the decoder of the VAE does not utilize Moreover, significantly high KL(2) also tends degrade mCAS, because the decoder might resemble a one-to-one mapping from to x and therefore, degrade the diversity. This phoenomenon can also be observed in the sample images in appendix IK. there seems be little diversity in samples drawn from B- B-V and CMI-VAE with y = 0.6. (3) The curves for CMI-VAR are consistently left to those for 8-VAE, indicating that regularizing [(2 s) is also complementary to regularizing 1 (x;; 2) at the controlled generation.
6 DISCUSSIONS AND FUTURE WORKS
Based on the experimental results, it was confirmed that regularizing [(2;s) is complementary to regularizing (2;; =), and leads to improvement in the learning of global latent variables. Here, we chose to extend 8- VAE to construct the proposed objective function because we believe 3-VAE is the simplest MI- maximiiiinttion method that requires fewer hyperparameters. widely used in (sequen- tial) VAE community (c.g., He et all (2019); Alemi et all (2018)). However, other MI estimation methods, such as discriminative objective and MMD-based InfOVAE, can be extended to CMI reg- ularization by the addition of the T(2:8) s) minimization term (see, Section B.2).. Incorporating such MI maximization methods into the estimation of CMI, or stabilizing adversarial training with some technique (Myato et all (OSS might improve the performance, and this remains issue to be ad- dressed in a future work. Also, it would be interesting to approximate I(x; 2) and I [(2; s) separately, and tune the strength of them independently. Future studies may also apply the proposed method to encourage the learning of the representation that captures the global factors of the environment such as maps, support reinforcement learning, suggested in Gregor all (2019).

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alexander Alemi, Ben Poole, lan Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. In International Conference Machine Learning, pp- 159-168, 2018.
Y Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new per- spectives. IEEE transactions on paltern unalysis and machine intelligence, 35:1798-1828, 08 2013. doi: 10.1109/PPAM..201355500
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference Computational Natural Language Learning, pp. 10-21, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2172-2180. Curran Associates, Inc., 2016.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational Lossy Autoencoder. In Proc. 5th International Confer- ence on Learning Representations, 2017.
François Fleuret. Fast binary feature selection with conditional mutual information. J. Mach. Learn. Res., 5:153 1 1555, December 2004. ISSN 1532-4435.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversane training of neural net- works. J. Mach. Learn. Res., 2016.
J. Garofolo, Lori Lamel, W. Fisher, Jonathan Fiscus, D. Pallett, N. Dahlgren, and V. Zue. Timit acoustic-phonetic continuous speech corpus. Linguistic Data Consortium, 11 1992.
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aäron van den Oord. Shaping belief states with generative environment models for RL. In Advances in Neural Information Processing Systems 32, pp. 13475-13487, 2019.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez. and Aaron Courville. Pixelvae: A latent variable model for natural images. In 5th International Conference on Learning Representations, 2017.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Borg-Kirkaatrick. Lagging inference net- works and posterior collapse in variational autoencoders. In International Conference on Learning Representations, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Jun-Ting Hsieh, Binghin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation P Processing Systems 31, pp. 517-526. Curran Associates, Inc., 2018
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances in Neural Information Processing Systems 30, pp. 1878-1889, 2017.
Yusuke Iwasawa, Kei Akuzawa, and Yutaka Matsuo. Stabilizing adversarial invariance induction from divergence minimization perspective. In Christian Bessiere (ed.). Proceedings the Twenty- Ninth International Joins Conference on Artificial Intelligence. IJCAI-20, pp. 1955-1962. Inter- national Joint Conferences Artificial Intelligence Organization, 2020. Main track.
10

---

Under review as a conference paper at ICLR 2021
Ayush Jaiswal, Rob Brekelmans, Daniel Moyer, Greg Ver Steeg, Wael AbdAlmageed, and Premku- mar Natarajan. Discovery and separation of features for invariant representation learning. CoRR, abs/ fess/1912.nn 1e1.1191..0.6... 2019. URL
http://a..///1.1..10 00646,
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Proceedings of the 35th Interna- tional Conference on Machine Learning (ICML), 2018.
Tejas Kulkarni, William F. Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphies network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gamett (eds.), Advances in Neural Information Processing Systems 28, pp. 2539-2547, Curran Asso- ciates, Inc., 2015.
Alexander H. Liu, Yen-Cheng Liu, Yu- Ying Ych, and Yu-Chiang Frank Wang. A unified feature disentangler for multi-domain image translation and manipulation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Dasa-Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems 31, pp. 2590-2599, Curran Associates, Inc., 2018.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 41 4114-4124, 2019.
James Lucas, George Tucker, Roger B Grosse, and Mohammad Norouzi. Don't blame the elbo! linear vae perspective on posterior collapse. In Advances in Neural Information Processing Systems 32, pp. 9403-9413, 2019.
Thomas Lucas and Jakob Verbeek. Auxiliary guided autoregressive variational autoencoders. In Ma- chine Learning and Knowledge Discovery in Databases European Conference, PKDD, pp. 443-458, 2018.
Xuezhe Ma, Chunting Zhou, and Eduard Hovy. MAE: Mutual posterior-divergenn regularization for variational autoencoders. In International Conference on Learning Representations, 2019.
Alireza Makhzani and Brendan Frey. Pixelgan autoencoders. In Advances in Neural Information Processing Systems 30, pp. 1975-1985. 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018. URL https: /ttt:::://oeeervveeww http://e/ooeeerreeeeeeeeeerrroon
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. CoRR, abs/1610.03483, 2017. URL https: wwt.or///bbbaa/eeeoeoee abs/ wbwp/a/1.///1////////.. 03483
Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. Invariant rep- resentations without adversarial training. In Advances in Neural Information Processing Systems, volume 31, pp. 9084-9093 2018
Sudipto Mukherjee, Himanshu Asnani, and S. Kanman. Cemi : Classifier based conditional mutual information estimation. In International Conference on Uncertainity in Artificial Intelligence, 2019.
XuanLong Nguyen, Martin J Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances in Neural Informa- fion Processing Systems 20, pp. 1089-1096. 2008
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debhath, Anjul Patney, Ankit B. Patel, and Anima Anandkumar. Semi-supervised stylegan for disentanglement learning. CoRR, abs/2003.03461 2020. htt.//ttss../aii.oorrr//////////
03461.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Proceedings of the 36th International Conference on Machine Learning, pp. 5171-5180, 2019.
11

---

Under review as a conference paper at ICLR 2021
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In Advances in Neural Information Processing Systems 32, pp. 12268 12279. 2019.
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta- VAEs. In International Conference on Learning Representations, 2019.
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disen- tanglement with guarantees. In International Conference on Learning Representations, 2020.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the breg- man divergence: a unified framework of density-rand estimation. Annals of the Institute of Sta- fistical Mathematics, 14(5):109--1044 2012.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/17008.0777, 2017. URL attp://arxiv.
BBs/
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and Garnett (eds.), Advances Neural Information Processing Systems 30, pp. 585-596. Curran Associates, Inc., 2017.
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In Proceedings of the 35th International Conference on Machine Learning, pp. 5670-5679, 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Balancing learning and inference in variational autoencoders. In The Thirdy-Third AAAI Conference Artificial Intelligence, pp- 5885-5892, 2019.
12

---

Under review as a conference paper at ICLR 2021
4.0
33
8=0.10
3.0
8=0.20
2.5
B=0.60
B=1.00
2.0
1.5
2000
4000
6000
Epochs
Figure 4: T(2;8) values of DSAE trained with 3-VAE objective.
APPENDIX
A EMPIRICAL EVIDENCE OF LIMITATION OF MI-MAXIMIZING REGULARIZATION
In this section, we evaluate the 8(2;8) values of DSAE, and the MI-maximizing regularization is shown to result in increasing it, contrary to the intention of learning the disentangled global features. Note that the [2;; s) DSAE ideally equals zero because its graphical model is designed such that z and s are independent. However, [(; s) is not necessarily zero because the representational MI is considered (see, also, Section [.]).. Because / [2; 8) is intractable, we used the value of i(z; s) estimated with DRT was used in a similar manner to Section 6.2. Namely, we used the following equation:
(22 p(z,8/y=10 =) plu = 122,8) ((=)((8) p(z, s 0) plu 0) where p(u = 0) - plu = 1) 1
(15)
p(y|z,8) s) can be approximated with discriminator D(2, s) that outputs D = when 2, mi.dd.
(2, 8), and D = 0 when 2, miad (()) Then, / (2;;) 8) can be approximated as follows:
D s)
s); R Epp(s)g(=.ala)) [log 1 D(2,s)
((235).
(16)
D2,,s) is parameterized with DNN, and trained alternately with the VAEs' objectives. The other training settings are the same as those in Section 5.2, and can found Appendix G
Figure presents f(2; 8) values of DSAE trained with the 3-VAE objective (Eqs. and B) where KL(2) is reweighted with parameter 3 = 1 y. The figure indicates that if 8 2 0.2, smaller B results in a larger (=;,s) This indicates that when we simply regularize [(x; z) to be large using 3-VAE objective, and s become to have redundant information. In contrast, when we do not regularize Txx;2) (i.e., 3 = 1), {22;)) becomes small; however, also becomes uninformative regarding
One may notice that even if / is smaller than 0.2, (2; will not increase. This is probably due the following reason. Firstly, DSAE, it is difficult for the z to have redundant local information due to the architectural constraint (Appendix D). Then, the [(2; s) might become larger when s, not 2, has redundant information. That is. regardless of the value of [3, s may retain certain degree of global information due to PC (which is supported by the experiments in Section 5.2)). Therefore, when z global information as decreases, ((2; s) increases. However, once has enough global information, there is no room for [2z; 8) increase beyond certain point.
B PRACTICAL DECOMPOSITION OF AUTOREGRESSIVE DATA GENERATING PROCESS
Here we show the practical decomposition the autoregressive data generating process IT [f 17/1p(*, (t
sx)(($(( for PixeICNN. We consider the 13-layer PixelCNN
13

---

Under review as a conference paper at ICLR 2021
used in et (2019), which has five (7 x 7)-kemell-size layers, followed by, four (5 5) layers, and then four (3 x 3) layers. Each layer has 64 feature maps with dimensions 28 x 28 dimensions. The latent variable is extracted by an encoder, linearly transformed into (28, 28, 4) feature maps, and then concatenated to the each layer of the PixelCNN feature maps after the sixth layer. We denote the output of the i-th (i E (1...., 13)) layer as ha,t, where i denotes the timestep (x and y coordinates, and E {1, ..... 28 x 28 = 784)). Then, we can put St == ho,<t and the decomposi- tion [/=1p(<=|2 22, xx))) = ih (it=)((r,]= 8x (x)g(88/(rec) holds because only ba,<t (not ha.21) are used to generate T, with causal convolution.
One might wonder whether the activations of the PixeICNN, which is the deterministic function of æ, can be treated as random variables. However, because we regularize s only via minimizing [2;; s) (Section [32). can be determined to be treated as random variables. specifically, [ s) is defined by the joint distribution g(2; s) I (=)))) 2)dr (see definition in Appendix E), where (s) 2) ((s|z) = II_"_ ((s - ((=<)) would be integrated over a random variable x. Therefore, and s have no deterministic relation and s can be meaningfully referred to as local latent variables. Furthermore, it is common to treat the activations of hidden layers as random variables and to consider their MI (or conditional entropy) in the literature on nomain-invariant representation learning ctal 7I
Note that, the definition of 8 is important factor for the "control" of what will be learned in z; however, anything is acceptable as long as 8t has sufficiently large receptive fields. For example, Chen et all (I proposed improving global representation by using smaller receptive fields for q(s, (ec)) and constraining st to more local information. Although this architectural constraint can make z informative, it requires weakening the expressiveness of PixeICNN and can degrade ELBBO (Chen et all 2017). By contrast, our method can be applied regardless of the size of the receptive fields because it prevents s from having global information with an information theoretic regularization term. Therefore, the architectural change of Chen et all (2017) was not employed and large receptive fields were used to balance sufficient ELBO and representation quality.
C DSAE AND PIXELCNN-VAL VAE HAVE DIFFERENT ARCHITECTURAL CONSTRAINTS ON GLOBAL LATENT VARIABLES
z of DSAE and PixeICNN-VAE are imposed on different architectural constraints. In DSAE, the is constrained to have no local information. On the other hand, the z of PixcICNN-VAEs has no such architectural constraints, although it is designed to capture global features via the structured data generating process. Here, we distinguish the structured data generating process from the archi- rectural constraints: the former is the constraint based on the probabilistic graphical model, while the latter is the constraint based on the neural network structures. Specifically, the z of DSAE is concatenated with and feeded into the fully connected neural network decoder for all timestep I E [1, ..... T.. so the z may have the same effects on each timestep l. On the other hand, the of PixelCNN-VAL is linearly transformed into (28, 28, 4) feature maps, and then concatenated to the each layer of PixelCNN feature maps (see, Appendix (B). Since the linear transformation creates the feature maps that depend timesteps (x and y coordinates), it becomes easy for the to have different effects on each timestep f. Note that, such linear transformation is commonly employed in previous studies of ExclCNN-VAES (e.g., He et all (2019) in order to improve expressiveness of the decoder.
Due to these architectural differences, different phenomena can be observed in DSAE and PixeICNN-VAE when using MI-maximizing regularization. First, both the models have in common that I [ s) would become larger when using the MI regularization (see, Section D.) However, in DSAE, it is likely that s has redundant global features, not that has redundant local features, be- cause it is difficult for the 2 to have local information due to the architectural constraint. On the other hand, in PixeICNN-VAL would cause different problems in the controlled generation using DSAEs and PixeICNN- DSAE can hopefully change speaker individualities while preserving the linguistic contents (i.e.. perform voice conversion), by swapping the 2 of two utterances and reconstructing them. However, if still contains speaker information due to the redundancy, the decoder can extract speaker in- formation from either s or a and there is no guarantee that 2 will be used (see, also, Appendix I) For PixelCNN-VAL previous studies (Alemi all, 2018, Razavi et all 2019) have shown that by
2 can have redundant local features. Then, these architectural differences
14

---

Under review as a conference paper at ICLR 2021
stochastically sampling z from FixiCCN-VAAE with given 2, one obtain images with different local patterns but similar global characteristics (c.g. color background, scale, and structure of ob- jects). However, when has all (local and global) information, the diversity of the generated images would decrease, because the decoder resembles one-to-one mapping from to x (see, also, Section
D THE LOWER BOUND OF (x;; =|s)
Here derive Eq. g and discuss the approximation error between [xx; and LeMr. Firstly, can take the lower bound:
[x; = 2( [(2; + [<; [xx; I(2;s),
(17)
since the MI (22 is positive. Then, the lower bound [x; 2) - [2; s) has approximation error [2; s|z). Note that the error can be small under particular condition. Namely, the error can be decomposed as:
T(2:8/z) H((=z) H= s).
Here, both (2)x)) and H (2|xx, s) thought to small when x is nigh-dimensional data such as images, movies, and audios, because observing such would enable predict accurately. Also, empirically, has been shown that the performance of inference model did not drop much even the encoders of DSAE decomposed into q(z, s/a) ((=)=))(({())) (Yingzhen & Mandf, 2018), which indicates the error is small.
In addition, long Z 1, the following condition holds:
1(x; - 12;;8) 2 1(x;)) - a2;8))
(18)
because the MI (22;8) is positive. This approximation error becomes the smallest when 1.
E DEFINITION OF [(2; s)
This paper considers the MI s2 defined by the encoder (which corresponds to representational MI Alem ct (2018)) Namely,
(2;;)) E [log {<2)g(1)2)
(19)
((2)g(s)
where the joint distribution is ((2;8) = / [ 2)dr.
F
DERIVING EQ.
I0
Here we present the deriviation of Eq. I0:
[xx; 2) 122;)) Ex(,,)) [log [(2|2)] Be( [log g(21s)
(20)
(())
((2)
B [log g(2|2) (2)2))(((((()
(21)
{(>())g(=)p(3))
By(n,a) [log (<=[2)
p(2) Bx(,,s) Dlog ((2,8)
(22)
p(2)g(s)
Eps(z) [2kt] D
(23)
G DETAILS OF EXPERIMENTAL SETTINGS
G.1 DISENTANGLED SEQUENTIAL AUTOENCODER
Data preprocessing We use the TIMIT data (Garotolo et all 1992), which contains broadband
16kHz recordings of pwonetically-ballneeeee read speech. A total of 6300 utterances (5.4 hours)
15

---

Under review as a conference paper at ICLR 2021
presented with 10 sentences from each of the 630 speakers (70% male and 30% female). Garotola at (11922) have originally split the data into train/test subset, and we further split the train subset into 90% of train and 10% of validation subset. We followed Hsu et all (2017)): Ymgzhen & Mandi (2018) for data preprocessing: the raw speech waveforms are first split into sub-sequences of 200ms, and then preprocessed with sparse fast Fourier transform obtain 201 dimensional log-magnitude spectrum, with the window size 25ms and shift size 10 ms. This results in ' 20 for the observation T1:T.
Optimization we follow Yingzhen & Mandt (2018) for model architecture, data preprocessing, and evaluation procedures. The dimensionality of 8; and were fixed at 64; we set T 20 for the observation Zgg.. We used Adam optimizer with learning rate 2e-4 for the VAE and 2e-3 for the discriminator, and trained the models for 6000 epochs to get good convergence on the training set. The VAE architecture followed fill model in Yingzhen & Mandi (2018), and the discriminator architecture is described in Appendix The discriminator is updated twice while the VAE updated once. The results averaged over three random seed trials.
G.2 PIXELCNN-VAL
Data preprocessing We use the statically binarized version of MNIST and Fashion-MNIST datasets: each pixel value [0, i] binarized with the threshold 0.5. The datasets are originally split into train/test subsets, and we further split the train subsets into 80% of train and 20% of validation subsets.
Optimization Regarding the optimization of VAEs, we used the Adam optimizer with learning rate of 0.0001, trained for 300 epochs. We reported the values for the test data when the objective function for the validation data was maximized. Regarding the discriminators, we used the Adam optimizer with learning rate 0.001. The discriminator architecture described in Appendix II and updated twice while the VAE is updated once. As for the PixeICNN architecture, Appendix E.
H DETAILS OF MI- VAE IN OUR EXPERIMENTS
We employ [x:; 2) maximization method proposed by Makhzanı & Frey (2011)) Zhao et (20119
as a baseline method in our experiment. Briefly, we add (x;; 2) to the standard VAE objectives as regularization term with weighting term 7
To estimate [x:; 2). Makhzami Frey (2017) utilize the follwing relation based on the density ratio trick:
<(2) p(=ly= = plv=
(24)
p(2) p(zly = ply 0|2)
where p④y = 0) = 1/2 and p(y = 1) = $1 Then, although conditional probability p(y|z) cannot be obtained, it can be approximated with discriminator D(=), which outputs D = 1 when Middd q(2) and D = 0 when mi.i.d. p(2). Then, [(2; 2) can be approximated as follows:
I [2;; z) = Epa(a) Dx (g(= p[)) (xz(((=))lv)
D(z)
~ E (Dxrlu((<=)) - log 1 D(z)
IMT-DRT-
(25)
D(z) is parameterized with some DNN, and trained alternately with VAEs' objectives. Namely, D is trained to maximize the following objective with Monte Carlo sampling:
Eg(a) [lg (==)] + Ex(a) [og - D(=))]
Finally, we introduce the concrete objective of PixeICNN-VAR with the regularization term IMI- Adding MMI to the objective of PixeICNN-VAE, we obtain the objective functions of MI-VAE:
D(2)
WARM -AMM + TMu DIT
Recon (1 KT.() >EElog i- D(2)
(26)
16

---

Under review as a conference paper at ICLR 2021
In short, the objective only differs from CMI maximization method in that the discrimi- nator added on the purpose of minimizing ( while our method minimizes DEL DkIlg(2, ()(2)s)) and encourages disentanglement of and
I DISCRIMINATOR SETTINGS
We have used discriminators for CMI- -AE and MI- VAE (whose details can be found in Appendix HD. For the discriminator of CMI-VAE, we first applied convolutional encoder and took mean pool- ing, obtaining the embedding of 81:7- Then, in PixelCNN-VAE, embedding and & and treated it as logit of the discriminator. On the other hand, in DSAE, we took cosine similarity of the embedding and multiplied the similarity by a learnable scale parameter, and treated it as logit of the discriminator: The encoder architectures for PixeICNN-VAL VAE and DSAE are summarized as follows, with the format Conv (depth, kernel size, stride, padding):
took innner product of the
PixeICNN-VAE
Input (28, 28, 1)
Conv2D (256, 4, 1)
BatchNorm
ReLU
Conv2D (256, 4, 2, 1)
BatchNorm
ReLU
Conv2D (z-dim, 4, 1)
DSAE
Input (20, 201)
Conv ID (256, 4, 1)
BatchNorm
ReLU
Conv ID (256, 4, 2, 1)
BatchNorm
ReLU
ConviD (=-dim, 4, 2, 1)
The discriminator architecture for MI-VAE is summarized as follows, with the format Linear (input size, output size):
Input (z-dim)
Linear (z-dim, 400)
ReLU
Linear (400, 1)
Softmax
J VOICE CONVERSION EXPERIMENTS USING DISENTANGLED SEQUENTIAL AUTOENCODER
For quantitative assessment of controlled generation by DSAE, we performed voice conversion and evaluated the models with score similar to mCAS (see, Section 5.3), which we call VC-mCAS.
First, we prepared 500 real speeches (spectrograms with T 20) {x; { along with their gender
500
17

---

Under review as a conference paper at ICLR 2021
Table 2: VC-mCAS for 3-VAE and CMI-VAE.
Model
7
VC-mCAS(mean)
VC-mCAS(max)
DSAE + VAE
0,4000
83.73
85.8
DSAE + CMI-VAE
0,4000
84.33
86.6
DSAE + B-VAE
0.8000
87.27
87.6
DSAE + CMI-VAE
0.8000
87.47
88.0
DSAE + 3-VAE
0,9000
87.20
87.6
DSAE + CMI-VAE
0.9000
87.00
87.6
DSAE + 3-VAE
0.9900
87.33
87.6
DSAE + CMI-VAE
0.9900
87.60
88.6
DSAE + 3-VAE
0.9990
87.27
87.4
DSAE + CMI-
0.9990
87.60
88.0
DSAE + 3-VAE
0.9999
86.87
87.2
DSAE + CMI VAE
0.9999
87.73
88.0
labels (male female) {u: 1,50 where each of the 2 classes had 250 samples. Then, randomly created 250 pairs { (x, Zyy (r)l): 19: 7 Vj i.e., each pair consists of one male and one female speech. Using the trained DSAE, we encoded each into and created the pairs {(z,, 84, žj 8;)ly: b A Wj and decoded 2 and 8, (zj and 8;) to obtain üig (fz,s), which ideally has the speaker characteristics of x; and the linguistic contents of zy. Thus, we obtain 500 generated samples, where each &..5 was labeled with assuming that the characteristics that tend to depend on gender (such as pitch) were successfully converted. Finally, we trained logistic classifier with the 500 pairs {( and evaluated the performance on real test speeches. Note that, because the raw äi.j has an excessively high dimension (20(T)) x 201 (features) for the logistic classifier, was averaged over the time- axis prior to its measurement. Intuitively, when the decoder ignores the generated samples might belong to a class different from the original ones, which produces label errors. Therefore, to achieve high VC-mCAS, should capture global information but should not. Aos, the generated samples should be realistic to reduce the domain gap between train (generated) and test (real) data.
Table D presents the values of VC-mCAS for the objectives of 3-VAE and CMI-VAE, Note that report the mean and best scores within three random seed trials for each 7y The table illustrates that given fixed 7, CMI-VAE nearly consistently achieved higher VC-mCAS compared to 3-VAE, indicating that regularizing I' I'(z;)) 8) is complementary to 3-VAE. Furthermore, although 7 0.8 yields higher EER(:) than those with y 0.4 Table yields higher VC-mCAS. Therefore, in addition to measuring EER, as was done in previous studies et 2017: Yingzhen & Mandi, 2018), we claim that it is necessary to consider the performance of the controlled generation for evaluating the usefulness of the global representation.
K DETAILED EXPERIMENTAL RESULTS
K.1 DETAILED EXPERIMENTAL RESULTS FOR DSAE
Table & presents the ELBBO KI, Recon, EER, and f(2;8) values of DSAE on TIMIT corpus. Re- garding the estimation of f(2: s), please refer to Appendix A Also, note that KL(2) approximates [xx; 2) because it upper bounds (ax; 2). and has been used for the metric assess whether a decoder ignores not (Bowman et atl 2016 Alemi et all 2018: all 2019). Here, the ELBO, Recon, and KL(S) values are not divided by = 20. Also, the Recon values can be nevative because the variance of our decoder learnable parameters.
As shown in the table, (i) given a fixed 7, the two methods (8-VAE and CMI-VAE) have the same level of KL(2); therefore, both the methods can be used to alleviate PC. (ii On the other hand, given fixed y, MAE achieved the lower f(2; s) values in most cases, suggesting that it facilitates the learning of good global representation. (iii) Finally, we have confirmed that even for large 7 there still reasonable reconstruction performance for the both methods.
18

---

Under review as a conference paper at ICLR 2021
Table 3: The ELBO, KL, Recon, EER, and (2) values of DSAE on TIMIT corpus. Each model was trained with a weighting parameter 7, "se" denotes standard error.
Model
ELBO
KT
Kl(+)
Rocon
BERG)
FER(S)
mean
52
mean
50
mean
SC
mean
sc
meam
mecn
DSAR
0.00
6299.00
14.06
18.00
0.09
495 02
4.45
-6813.00
13.37
11.01
0.52
18.64
1:04
1.61
0.04
DSAEE JAVE
(1400
6280
11.72
33.28
0:37
453
407
3.75
20026.16
15.45
3.38
0.115
29:45
0120
2.00
0101
DSAB CMI-AE
0.40
6388.64
4.82
54.13
2.51
468.00 06
14.03
68.00.83
16.04
3.43
0.26
30.96
0.81
2.44
0.03
DSAE
4505.20
10.76
6797.90
41.30
DSAL 2-VAE
0.90
6172.96
333
202.52
1.34
430.20
3.88
6301.87
2.13
4.35
0.15
39.42
1.06
0.01
DSAE -CMI-VEE
0.50
6192.05
6.58
199.89
1.10
429.53
7.84
-521 146
14.59
4.39
0.25
41.25
2.10
2.70
0.03
DSAE BAVAE
0.99
0019.10
11.52
364.71
2,08
433.39
9.40
6817.40
19.39
0.33
0.34
38.03
1.18
3.27
0.15
DSAE CMI-VAE
6031.71
5.78
361.03
1,85
431.5
8.67
6821.29
12.56
5.06
0.23
40.08
1.09
285
0.07
K.2 DETAILED EXPERIMENTAL RESULTS FOR PIXELCNN-VAE
Tables E and B present the ELBO. KL(2), Recon, [(2; s), mCAS, and AoLR values of PixeICNN- VAES on MNIST and Fashion-MNIST. Moreover, these tables present MCAS(SVM) and AoSVM, which are the same with mCAS and AoLR except for using support vector machine (SVM) with RBF kernel, i... more powerful non-linear classifier, instead of the logistic classifier. Regarding the estimation of [22;s), please refer to Appendix Al Also, note that KL(() approximates [(r; 2) because it upper bounds (x:; 2). and has been used for the metric to assess whether decoder ignores or not (Bowman et all 2016 Alemi et all 2018; He et all 2019). Here, the ELBO and Recon values are not divided by 28 28.
As shown in the tables, (i) given fixed y, the three methods (3-VAR, MI-VAE, and CMI-VAE) have the same level of KL(=); therefore, all the methods can be used to alleviate PC. (ii) On the other hand, given fixed % CMI-VAE achieved the lower f(2; s) values in most cases, suggesting that it facilitates the learning of good global representation. (iii) Finally, even if we used non-linear classifier SVM to calculate mCAS(SVM) and AOSVM, CMI-VAE achieved competitive or higher performance than the baselines in most cases. Note that, the exception is that given 7 0.4, there were not much differences in MCAS(SVM) for Mishion-MNISST within the three methods. One possible reason is that using the non-linear elassifier increases the number of factors to be considered, such as overfitting, and makes fair comparisons difficult. Also, we note that using a very large y for PixelCNN-VAEs might not be good idea. is because when becomes too large, the decoder of PixeICNN tends to resemble an identity mapping from Z to its output, regardless of the regularization method (e.g., see, generated samples for 7 0.6 in Appendix (K.3). To improve performance while avoiding this phenomenon, it could be useful to using a weighting parameter in Eq. & (c.g., using y 0.3 and work as noted in Section i
1), and this remains an issue to be addressed in future
K.3 SAMPLE IMAGES FOR PIXELCNN-VAF
Figures B present the generated images with PixelCNN- VAEs.
19

---

Under review as a conference paper at ICLR 2021
4: The ELBO, KI()), Recon, mCAS, AoLR, MCAS(SVM), and AoSVM values of PixeICNN- VAES on MNIST. Each model was trained with a weighting parameter
ELBO
KL(=)
Recon
mCAS
AoLR
mCAS(SVM)
AoSVM
f(::))
Model
0.0
B-VAE
56.21
3.60
52.61
0.3966
0.6094
0.5722
0.6447
1.81
0.1
D-VAE
56.28
5.33
50.95
0.5917
0.8161
0.7172
0.3458
2.11
CMI-VAE
56.24
4.80
51.43
0.5421
0.7245
0.6499
0.7809
1.97
MI-VAE
56.23
5.16
51.07
0.5325
0.7602
0.6651
0.8080
2.05
47.47
0.6924
0.8479
0.7750
0.8980
2.45
CMI-VAE
56.68
9.32
47.36
0.7229
0.8692
0.7934
0.9026
2.29
MI-VAR
56.50
8.90
47.60
0.6734
0.8208
0.7606
0.8762
2.38
0.3
3-VAE
58.33
18.04
40,29
0.7448
0.8354
0.8109
0,9027
2.85
CMI-VAE
58.03
17.52
40.51
0.7716
0.8630
0.8136
0.9182
1.80
MI-VAE
57.81
16.37
41.44
0.7399
0.8238
0.8014
0.8938
2.85
0.4
B-VAR
61.22
29.00
32.21
0.7476
0.8204
0.8063
0.8962
3.28
CMI-VAE
29.56
31.71
0.7664
0.8401
0.8122
2.21
MI-VAE
60.52
27.10
33.43
0.7461
0.7903
0.7985
0.8742
3.11
0.5
B-VAE
64.55
37.79
26.76
0.7555
0.8279
0.8060
0,9010
3.16
CMI-VAE
64.69
38.40
26,28
0.7575
0.8405
0.8075
0,9088
1.79
MI-VAE
63.57
35.62
27.95
0.7454
0.7719
0.7963
0.8582
3.18
0.6
3-VAE
68.74
46.29
22.45
0.7642
0.8306
0,8120
0.9056
3.27
CMI-VAE
69.00
47.11
21.90
0.7618
0.8486
0.8079
0.9190
2.57
MI-VAE
67.72
44.16
0.7484
0.7722
0.7949
0.8572
3.19
5-VAD
56.04
18.43
0.7531
0.8454
0.7984
3,27
CMI-VAE
74.71
56.28
18.43
0.7553
0.8486
0.8019
0.9179
1881
MI-VAE
72.92
52.88
20.05
0.7496
0.7753
0.7912
0,8601
3.19
Table The ELBO, KL(z). Recon, mCAS, AoLR, MCAS(SVM), and AoSVM values of PixcICNN- VAES on Fashion-MNIST Each model was trained with weighting parameter
ELBO
KL(z)
Recon
mCAS
AoLR
mCAS(SVM)
AoSVM
[ s)
Model
0.0
8-VAE
88.60
4.02
84.58
0.4820
0.6906
0.6342
0.7552
2.00
0.1
88,86
6.71
82.16
0.5716
0.7172
0.6556
0.7790
2.40
CMI-VAE
88,90
6.47
82.43
0.5837
0.7201
0.6618
0.7757
2.53
MI-VAE
88.84
6.72
82.12
0.5645
0.7166
0.6592
0.7742
2.37
0.2
3-VAE
89.94
10.95
78.99
0.6174
0.7092
0.6792
0.7708
2.78
CMI-VAE VAE
90.25
11.49
78.76
0.6463
0.7258
0.6885
0.7779
2.46
MI-VAE
89.78
10.55
79.23
0.6132
0.6760
0.7655
2.69
0.3
VAVAF
91.45
16.11
75.34
0.6427
0.7030
0.681T
0.7665
2.94
CMI-VAE
91.55
16.39
16.16
0.6578
0.7238
0.6873
0.7713
2.03
MI-VAE
91.14
15.09
76.06
0.6241
0.6817
0.6897
0.7507
2.92
0.4
3-VAE
93.98
24.33
69.65
0.6536
0.7062
0.6858
0.7636
3.24
CMI-VAE
93.81
23.35
70.46
0.6662
0.7145
0.6868
0.7701
2.11
MI-VAE
93.36
22.38
70.98
0.6366
0.6787
0.6886
0.7278
3.21
0.5 B-VAE
97.36
33.02
64.34
0.6547
0.6919
0.6845
0.7521
3.36
CMI-VAE
97.14
32.14
65.00
0.6651
0.7193
0.6867
0.7688
2.29
MI-VAE
96.47
31.19
65.27
0.6343
0.6506
0.6823
0.7056
3.31
0.6
3-VAE
101.76
42.40
59.36
0.6633
0.6899
0.6853
0.7524
3.39
CMI-VAE
101.83
41.97
59.86
0.6685
0.7112
0.6904
0.7659
2.43
MI-VAE
100.66
39.96
60.70
0.6535
0.6420
0.6854
0.6961
3.33
0.7
8-VAE
107.77
52.57
55,20
0.6680
0.6983
0.6900
0.7545
3.41
CMI-VAE
107.49
51.97
55.52
0.6729
0.7103
0.6910
0.7681
2.00
MI-VEE
106.05
49.14
56.91
0.6578
0.6838
3.34
0.8
V-VA
117.00
65.76
51.25
0.6649
0.7044
0.6855
0.7665
3.37
CMI-VAE
116.21
64.68
51.53
0.6634
0.7098
0.6851
0.7705
2.35
MI-VAE
113.98
60.19
53.79
0.6599
0.6108
0,6890
0.6709
3.31
0.9
A-VAE
132.62
85.13
47.49
0.6695
0.7097
0.6844
0.7787
3,47
CMI-VAE
131.01
82,97
48.04
0.6703
0.7169
0.6860
0.7780
2.74
MI-VAE
128.34
77.72
50.61
0.6565
0.6209
0.6880
0.6882
3.31
20

---

Under review as a conference paper at ICLR 2021
7
> 2 C
3
4
b
&
4 4 6
4
4
5
5
S
5
5
5
5
5 6 5
5
5
5
5
C 6
L
L
C
U
C
L
4
7
9
7 a
+
4 1
8
8
8
8
8
8
a
8
8
8
8
8
9
3
3
3
a
9
8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
(a) CMI-VAE with 0.3
(b) Ø-VAE with = 0.3
8
0
0
0
0
4
1
7
>
>
>
>
>
7
>
a
>
>
3
3
3
3
:
3
;
7
3
3
3
3
4
4
4
4
4
4
4
L
4
4
G
5
5
5
5
5
5
5
5
5
C
L
C
U
U
C
U
U
C
L
C
U
L
U
C
7
?
7
1
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
3
8
9
8
8
9
9
9
9
9
9
9
9
9
9
9
9 9
4
9
9
9
9
9
9
(c) CMI-VAE with 7 0.6
(d) 8-VAE with 7 = 0.6
Figure 5: Real images (the first column) and generated images by PixelCNN- VAES (the other 10 columns). The images each row are stochastically sampled from the decoder p(z[2) using the same z, which is extracted from x in the first column. The figures present that the diversity of the images in 7 = 0.3 is better than that in = 0.6, which may be because PixeICNN-VAE would resemble identity mapping with large y. contrast, 7 0.3 apparently produces more label eπтотs than
0.6 because the decoder ignores with a small 7 (see, e.g., the rows for 3 and 4). Furthermore, when comparing (a) (CMI-VAE with 7 0.3) and (b) (5-VAE with 7 0.3), apparently, (a) produces less label errors (see, e.g., the rows for 2 and 3). This result is consistent with the mCAS scores in Figure (Section (5.3), which indicates that CMI- VAE achieved better diversity and less label errors than 3-VAE.
21


Reviews:
[{"review_id": "p9ye8IZyHfo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nObserving the deficiency of existing MI-based sequential VAEs, where through the learning objective design, the local and global features may become disentangled, the authors propose adding a regularizer to explicitly disentangle local and global features. To address the computation tractability issue of adding the new regularizer, a density-ratio based approximation is adopted and a classifier is trained to approximate the density-ratio term, which during training is learned alternatively with the VAE objective.\nRelation between the proposed method and Conditional MI, beta-VAE, and domain adversarial training is discussed. Then the proposed method is empirically evaluated on two sequential VAE based tasks: speech voice manipulation (DSAE) and image generation (PixelCNN-VAE). The authors designed experiments that show that the proposed approach can improve global representation learning.\n\nQuality\n\n*Pros: \nThe paper is overall of good quality: the context of the problem is well explained with adequate diagrams/plots to aide understanding. The approach is well motivated and the derivation of the approximation method is mostly easy to follow. Experiments are well designed and details are provided.\n\n*Cons:\nThe density-ratio technique to approximate D(q(z, s) || p(z)q(s)) should be better explained in the paper: I had to look into the reference (Sugiyama et al 2012) to understand the derivation in Equation (11).\n\nClarity\nThe paper is clearly written.\n\nOriginality\nThe paper is based on existing work of MI-VAE families of neural networks. Instead of proposing a completely new architecture/method, the authors spot a gap in the current literature, i.e., that local and global feature representations can be disentangled using the current learning objective. The paper addresses exactly this gap.\n\nSignificance\nThe paper and the proposed method should be of some significance to the VAE and domain-adaptation community and inspire future works.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WIoGXqSMrOU", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update after discussion with authors**\nI want to thank the authors for their incredibly detailed responses and engaging so actively in the discussion. Some of my criticism could be addressed, while other issues are still somewhat open. If the main merit of the paper is to make the \"sequential VAE community\" aware of issues that have been discussed and addressed before, then I think the paper does an OK job at that (though I'm not entirely sure what community that is, the issues have been discussed before in the fields of vision and VAEs with autoregressive decoders). \n\nI want to strongly encourage the authors to be as precise as possible when describing the novelty - maximizing the mutual information that a representation carries w.r.t. some relevance variable while simultaneously minimizing information that it carries w.r.t. to another variable is NOT novel. What is novel is the application of that principle to separating \"global\" from \"local\" information in sequential data (and how to actually perform this originally intractable optimization in practice). I also want to encourage the authors to state what's known and what's new as clearly as possible and improve the quality and clarity of the \"educational\" review of why maximizing mutual information is not enough as much as possible.\n\nViewing the paper as \"showing how a known problem also appears when separating global from local information, and how to apply known solution-approaches to the problem in this specific context\", shifts the relative importance of the issues raised by me. Essentially, that view emphasizes the paper as mostly an application paper (rather than a novel theoretical contribution). Accordingly (but please make sure that that shift in view is also clear in the final paper), I am weakly in favor of accepting the paper and have updated my score accordingly.\n\n---\n\n**Summary**\nThe paper tackles the problem of separating ‘global’ from ‘local’ features in unsupervised representation learning. In particular, the paper tackles a common problem in autoencoders where the decoder (generative model) is autoregressive and conditioned on a variable. Ideally, the latter variable captures all global information (such as e.g. speaker identity) whereas the autoregressive model deals with generating local structure (such as e.g. phonemes). As the paper points out, capturing only global information (and all of it) in the conditioning variable alone is notoriously difficult with standard variational autoencoder objectives, and several solutions have been proposed in the past. In this paper the idea is to add an explicit penalty for statistical dependence (mutual information) between the global and the local random variable. This intractable objective is simplified with a series of approximations, leading to a novel training objective. Results are shown for speech data, and MNIST/FashionMNIST, where the proposed training objective outperforms a beta-VAE objective and an objective that explicitly aims to maximize information on the global variable.\n\n---\n**Contributions, Novelty, Impact**\n\n1) Analysis of shortcomings of mutual-information maximization to regularize latent representations into capturing ‘global’ features. This topic has been widely discussed in the literature before, typically in the context separating nuisance factors from relevant variations in the data, or more broadly: separating relevant from irrelevant information (which is canonically addressed in the information bottleneck framework of course). Most of this previous discussion was aimed at supervised learning ([2]), but there is a considerable body of work in unsupervised learning as well ([1] discusses the same issue but with more clarity), and some recent, very relevant work targeting VAEs with autoregressive decoders as well ([3] is among the state-of-the-art models). The paper provides a recap of this literature, but misses some key references, and the clarity of the writing (pages 1-4) could be improved (see my comments on clarity below). Therefore I would rate the impact of this contribution as low.\n\n2) Proposal of a novel regularizer. The main idea behind the regularizer Eq. (8) is good, but certainly not novel - it has been broadly discussed in the literature and implemented in various ways. The merit is thus in the particular derivation and approximations that lead to the objective in Eq. (13) and (14). To me the derivation seems correct, though the precise motivation is somewhat unclear (what shortcomings of alternative approaches are addressed here, e.g. using the density ration trick?). I personally think that there is sufficient novelty, but in the current manuscript it is hard to assess whether the novel method has benefits compared to strong competitor methods (which are unfortunately missing from the experiments).\n\n3) Experiments on a speech dataset (using a state-space-model decoder), and MNIST/FashionMNIST (using a PixelCNN). Results indicate that the extracted latent space does capture global features slightly better than a beta-VAE, or (quite a bit better than) a MI-maximizing VAE. There is also some indication that local features capture less global information with the proposed method compared to a beta-VAE. These results are promising, but not surprising since beta-VAE and MI-VAE were not designed to solve the shortcomings that the method is trying to address. For results to be more convincing and stronger, it would be good to compare against alternative approaches that have the same objective, such as e.g. [1] and [3]. Additionally more control experiments and ablations as well as reporting more metrics (l(x;z) and I(z;s), or proxies/approximations thereof) would strengthen the findings and thus the potential impact (see my comments on improvements below).\n\n(I am not an author of any of these)\n[1] Moyer et al. Invariant Representations without Adversarial Training, 2018\n[2] Jaiswal et al. Discovery and Separation of Features for Invariant Representation Learning, 2019\n[3] Razavi et al. Generating Diverse High-Fidelity Images with VQ-VAE-2, 2019\n\n---\n**Score and reasons for score**\nThe paper addresses an important problem that has received attention in the literature for at least two decades (the InfoBottleneck framework lays the theoretical foundations here). The particular application to: (i) unsupervised learning, and (ii) global-conditioned autoregressive (VAE) models is very timely and has received less attention in the literature (but there are some papers). \n\nMy main issue is that the paper addresses two problems: (A) separating global from local information, (B) avoiding that autoregressive decoders ignore the global latent variable. Clearly stating both problems, reviewing the literature for each of them, and then showing how the paper solves both of them (and showing experimental results for both of them) would really help with clarity and readability of the paper. It would also help flesh out the novel contributions made by the paper. Additionally, it is not entirely clear how well the proposed objective actually addresses (A) and (B) in the experiments. There is some good indication for (A), but it is not directly measured (e.g. by estimating I(x:z) and I(s;z)), the effect is only shown indirectly via AoLR and mCAS (or Err(z) and Err(s)). The same is true for (B): there is some that the generative model does not ignore the global latent code via the mCAS experiments, but it is quite indirect (also looking at the generated examples in appendix K raises some doubts about diversity of the generative model). \n\nOverall I think the ingredients for a good paper are there, but they are not quite coming together yet. A deeper look into the empirical results (control experiments, additional metrics), and a comparison against strong competitor methods are needed. My recommendation would also be to really focus on the new objectives (Eq. 13 and 14) and discuss in more detail how they differ from competitor approaches and what the theoretical/empirical advantages of these differences are (for instance I am personally not yet fully convinced that using the density-ratio-trick with a neural network classifier will always work well in practice). If all of that were in place, I think the paper would be significantly stronger and could potentially have wide impact. I thus recommend a major revision of the work, which is not possible within the rebuttal period. Below are concrete suggestions for improvements, and I will of course take into account the other reviews and authors’ response.\n\n---\n**Strengths**\n\n1) The problem (separation of global and local info in variational unsupervised representation learning with autoregressive decoders) is timely and important, and has been somewhat neglected in the representation learning community (though there is work out there, and the same problem has been discussed extensively in a related context, such as e.g. supervised learning).\n\n2) The Method builds on a body of previous great work and applies it in an interesting context (global vs. local features).\n\n---\n**Weaknesses**\n\n1) Merits of the method somewhat unclear - the motivation/derivation when going from Eq. 8 to Eq. 13, 14  is a bit ad-hoc. What alternatives are there to the choices/approximations made? What are the advantages/disadvantages of these? Answering this might also involve some control experiments and ablations.\n\n2) The experiments show somewhat indirectly that the goals were achieved. There is some empirical evidence that the method is working to some degree, but it remains unclear whether e.g. the learned z capture only global information (and all of it), and whether s captures only local information (and how much of it). What’s needed here are additional results/experiments.\n\n3) The current writing is ok but could be improved. I think it would be helpful to clearly state the problems and previous approaches of solving them (and the issues with these previous approaches). This would make it easier to see how the proposed method fits into the wider picture and which specific problem it addresses/improves upon. I think Alemi 2018 (cited in the paper) and [1] mentioned above do a very good job of describing the overall problem..\n\n---\n**Correctness**\nThe derivation of the method looks ok to me, though it would be nice to justify the approximations made and attempt to empirically verify that they do not have a severe impact on the solution. The conclusions drawn from the experiments are broadly ok, but since the evaluation measures the desired properties in a quite indirect way, the generality of the findings and the extent to which the method solves the problem (quantitatively) remain somewhat unclear.\n\n---\n**Clarity**\nIt took me a bit longer to follow the main line of arguments than it should have (which might of course be my fault). It’s a bit hard to pinpoint to a specific paragraph, but perhaps the following suggestions are helpful for improving readability. It might be worth clearly stating the main problems (denoted (A) and (B) further up in ‘Score and Reasons for Score’) and separately discussion how they have been addressed (and what the remaining issues are) and how the paper addresses them. Currently this is entangled in the derivation of the method.\n\nIt would probably also help to have a short paragraph that summarises the novel contributions clearly (which makes it clear what’s novel and what’s been proposed before).\n\n---\n**Improvements (that would make me raise my score) / major issues**\n\n1) Comment on all assumptions made when going from Eq (8) to (13), (14). Are these assumptions justified in practice? Would there be alternative choices, and if yes what are the downsides of these alternative choices? Some of the assumptions will then lead to further control experiments that would ideally be included in the paper. One example (perhaps the most important one) is below in 2)\n\n2) Neural network classifiers are notoriously known to be ill-calibrated (typically having over-confident probability estimates). This could be problematic in the DRT approximation since the discriminator’s output probability crucially matters! Is the discriminator well calibrated in practice? How robust is the method against calibration issues? Is the problem expected to get worse when scaling to more complex data and bigger network architectures? This needs to be discussed, but ideally some points are also verified empirically.\n\n3) beta-VAE and MI-VAE are ok baselines, but are not sufficient to show that the method performs very well. These two baseline methods have not been designed to address the main issue (separating global from local information) - it is thus not too surprising that the proposed method performs well. What’s needed is comparisons against strong baselines, e.g. a (hierarchical) VQ-VAE2 (ref [3] further up). Given that the method only slightly outperforms beta-VAE on the metrics shown (which has no explicit incentive to capture global information) this is important.\n\n4) Report additional metrics (for each experiment it would be good to also report: reconstruction error, estimates of I(x;z) and I(z;s)). As \\gamma is varied, does the method lead to consistent increase in I(x;z) and decrease in I(z;s)? Are the values for the latter two significantly better than when using beta-VAE/MI-VAE?\n\n5) Reporting AoLR and mCAS with a logistic regressor/classifier is ok, because it says something about latent-space geometry which could be interesting. But for the paper it is more important to capture the exact amount of global information captured by z and s. Therefore it would be good to show additional results for AoLR and mCAS where the regressor/classifier is a powerful nonlinear model (a deep neural net).\n\n6a) Alemi et al. 2018 gets cited quite a bit in the paper, but is not very well represented in the paper. In particular: the paper proposes a quantitative measure as a diagnostic to see how much information is captured by the latent variable and how much of that is used/ignored by the decoder (which leads to the definition of several operating regimes, such as the “auto-decoding” regime). Why not report the same measure in the current method?\n\n6b) Alemi et al. 2018 actually propose a modified objective to target a specific rate. They empirically observe that a beta-VAE with beta<1 *in their experiments* leads to the VAE operating in the desired regime. As far as I understand they do not propose this as a general solution to fix decoders ignoring latent codes. This should be mentioned in the paper. As a consequence 6a) becomes even more important, or without any verification beta-VAE becomes an even weaker baseline, meaning that comparison against strong methods becomes more important.\n\n---\n**Minor comments**\n\na) Eq. (10) should be an inequality, because I(z;s) is upper bounded on r.h.s.?\n\nb) How was it determined that “alpha=1 works reasonably”, is this based on some control experiments?\n\nc)  Eq (13). Why this particular mixing in of the KL-term, why not multiply KL(s) with (1-\\gamma) as well?\n\nd) Table 1: report the reconstruction error. In particular, for high \\gamma is there still reasonable reconstruction performance (and thus separation into global z and local s), or is all information except global information discarded and s essentially does not capture much meaningful information anymore, making good reconstruction impossible?\n\ne) Fig 3a - is the x-axis ELBO or KL?\n\nf) Fig 3, Table 1: ideally report multiple repetitions with error bars.\n\ng) For \\gamma=0.6 in appendix K, there seems to be very little diversity in samples drawn from either model. This should be mentioned more clearly in the main text.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ytlnHIEGln", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after rebuttal:\nI agree with Reviewer 5 that this paper has good ingredients, and the discussion and update of the draft clarifies the novelty and provides better review on the related work. However, the experiments presented in this paper are not very comprehensive, particularly the baselines and the ablation/alternative studies. I am not fully convinced by the authors response of \"because MI-VAE performed worse than $\\beta$-VAE in PixelCNN-VAEs ... we expected that a similar tendency would be observed.\" PixelCNN-VAE uses an autoregressive decoder, which are known to exhibit issues that are not observed from non-autoregressive ones like DSAE. This explanation of why MI-VAE was emitted seems slightly hand-wavy. I decided to decrease the rating to 6 to reflect the insufficiency in experiments, but hope that this experiment can be added if the paper is accepted.\n\nSummary:\nThis paper aims to learn representations that capture global features in structured data, such as the speaker information within speech or the digit class in an image. The authors argue that previous work regularizing the representation $z$ by maximizing its mutual information $I(x; z)$ with the data $x$ has the side effect of simultaneously maximizing the mutual information between $z$ and local features $s$. This may cause the global feature $z$ to encode unwanted local information or vice versa. To address this issue, the authors propose to regularize $z$ through maximizing $I(x;z) - I(z;s)$, which is a lower bound of the conditional MI $I(x;z|s)$. The proposed regularization is further estimated using the density ratio trick, which employs a discriminator to estimate the ratio $\\dfrac{q(z,s)}{p(z)q(s)}$ via a binary classification task and provide training signals to the encoder. The proposed regularization is applied to DSAE and PixcelCNN-VAE to demonstrate its effectiveness.\n\nPros:\n- The paper is well written and easy to follow. Motivations of this work are clear, and related studies are also properly described to contrast the difference from this work.\n- The formulation of the regularization seems novel to me. Derivation and approximations also make sense.\n- Experiments conducted on multiple domains (speech and images) demonstrate superior performance compared to the baseline methods (no regularization or MI-based regularization). Multiple metrics are adopted for evaluation of different aspects (e.g., AoLR/EER evaluates linear separability, mCAS indirectly evaluates diversity)\n\nCons/Questions:\n- The experiments on images compared the proposed method with both beta-VAE and MI-VAE. I am curious why the authors only compare with beta-VAE for the speech experiments but not MI-VAE. The EER reported in Hsu et al. (2017) is much lower than the results in this paper, and as mentioned by the authors, that work regularizes the representations with a discriminative objective that approximates $H(x|z)$ and therefore can be seen as a form of MI-VAE. The authors should also compare with such regularization for the speech experiments since it’s shown effective in the previous work.\n- The authors state that approximating both $I(x;z)$ and $I(z;s)$ may complicate optimization, and avoid doing so by forcing alpha in Eq.8 to be 1 to enable rearrangement in Eq.10. However, it would be informative to show the results of approximating both terms with the density ratio trick and see how much it would affect the performance. By doing so, the strength of regularizing $I(z;s)$ can be independently tuned.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a representation method for time series data in the sequential VAE, where the global feature z and local features  s are better disentangled. The intuition behind learning z is to maximize the mutual information between z and input x, while minimizing the mutual information between z and s. The second mutual information is estimated with a discriminator in the DRT framework. Overall, the methodology can be seen as reasonable applications of the disentanglement principle to sequential data. The authors have shown that z and s learned in this way is better disentangled as compared to beta-VAE. In the end, the reviewers feel that while there is good intuitions/technical ingredients, the derivations in Section 3 are not very smooth, and several approximations/choices are not very carefully justified (e.g., choice of alpha, choice of DRT vs. other MI estimators), and perhaps stronger baselines than beta-VAE can be used.

The reviewers rate this paper to be borderline.
Decision: Reject