SlowGuess/ACL_2020_OCR huggingface dataset:
The second block in Table 3 shows the role of  $\lambda_r$ , which we interpret as follows. When  $\lambda_r$  is too small (e.g., 0.5), the learned parameterizations are still inadequate for a smooth transition map; when  $\lambda_r$  is too large (e.g., 5.0), it distracts the optimization too far away from the original objective (i.e.,  $\mathcal{L}_{rec} + \mathcal{L}_{reg}$ ). Note that  $\lambda_r = 0.0$  is equivalent to removing the coupled reconstruction loss  $\mathcal{L}_{rec}^c$  in Eq. (4).
5.6 The Heterogeneous Effect of Signal Matching on Probability Estimation
In Section 5.5 we observe richer latent space (i.e., larger MI and BLEU scores) with larger  $\lambda_{m}$ . However, a richer latent space does not guarantee a better probability estimation result. Thus, in this
Dist	λm	λr	PTB	Yelp
NLL (KL)	PPL	MI	BLEU-1/2	NLL (KL)	PPL	MI	BLEU-1/2
RQ	0.1*	103.1 (9.5)	110.5	11.99	23.4 / 4.5	191.2 (8.0)	51.6	9.65	30.4 / 5.8
1.0	1.0	103.3 (10.7)	111.4	14.32	24.0 / 4.8	191.1 (8.1)	51.5	9.92	30.5 / 5.8
5.0	103.7 (16.1)	113.2	32.78	26.5 / 5.8	191.5 (12.8)	51.9	19.77	32.8 / 6.5
RQ	0.1	0.0	104.1 (7.3)	115.3	8.60	21.0 / 3.7	191.7 (5.8)	52.1	6.40	27.7 / 5.0
0.5	103.4 (9.2)	111.8	11.58	23.1 / 4.3	191.3 (7.8)	51.7	9.32	29.8 / 5.7
1.0*	103.1 (9.5)	110.5	11.99	23.4 / 4.5	191.2 (8.0)	51.6	9.65	30.4 / 5.8
5.0	103.1 (9.1)	110.6	11.15	22.9 / 4.4	192.9 (8.0)	53.4	9.53	30.0 / 5.8
Eucl	0.1	103.3 (10.1)	111.5	13.25	23.4 / 4.7	191.2 (9.2)	51.6	11.69	31.1 / 6.0
1.0	1.0	103.9 (17.4)	114.5	30.52	27.7 / 6.1	192.1 (14.3)	52.5	23.14	33.8 / 6.9
5.0	108.9 (33.3)	144.0	98.02	32.0 / 8.5	194.4 (25.0)	55.1	61.62	36.8 / 8.2
VAE	103.6 (8.6)	112.9	10.48	23.2 / 4.4	193.7 (7.2)	54.3	8.28	28.7 / 5.3
Table 3: Hyperparameter analysis. The best results in each block are shown in **bold**. *Reported in Table 1 and 2.
PTB	Yelp
NLL	PPL	NLL	PPL
Coupled-VAE*	103.1	110.5	191.2	51.6
Coupled-VAE (λm=0)	103.1	110.3	190.7	51.1
Coupled-VAE-NF*	102.6	108.1	191.8	52.2
Coupled-VAE-NF (λm=0)	102.8	109.1	192.7	53.2
Coupled-vMF-VAE*	103.0	110.1	191.2	51.6
Coupled-vMF-VAE (λm=0)	104.4	117.1	193.5	54.1
Table 4: The effect of signal matching on probability estimation. * Reported in Table 1.
part, we delve deeper into whether the decoder signal matching mechanism helps improve probability estimation. We study three models of different posterior families (i.e., Coupled-VAE, Coupled-VAENF, and Coupled-vMF-VAE). Results are shown in Table 4, where we do not report the KL, MI, and BLEU scores because they have been shown to be improved with larger  $\lambda_{m}$  in Table 3. We observe that the effects of signal matching on probability estimation vary in different posterior families.
5.7 Is the Incompatibility Mitigated?
We study the three gradient norms defined in Section 3 on the test sets, displayed in Table 5 (for Coupled-VAE,  $\lambda_{m} = 0.1$ ). Notably,  $\| \partial \mathcal{L}_{rec}^{c} / \partial e\|_{2}$  in Coupled-VAE is even larger than  $\| \partial \mathcal{L}_{rec} / \partial e\|_{2}$  in DAE. It has two indications. First, the encoder indeed encodes rich information of the text. Second, compared with DAE, Coupled-VAE better generalizes to the test sets, which we conjecture is due to the regularization on the posterior. Coupled-VAE also has a larger  $\| \partial \mathcal{L}_{reg} / \partial e\|_{2}$  compared with VAE, which based on the argument in Section 3.1 indicates that, in Coupled-VAE, the posterior of each
instance is not similar to the prior. We also observe larger  $\| \partial h / \partial e\| _F / \| h\| _2$  in Coupled-VAE, which indicates a better transition map between the two parameterizations in Coupled-VAE than in VAE.
We also track the gradient norms of Coupled-VAE  $(\lambda_{m} = 10.0$  for a clearer comparison), plotted along with VAE and DAE in Figure 2. The curve for Coupled-VAE in Figure 2(a) stands for  $\| \partial (\mathcal{L}_{rec} + \mathcal{L}_{rec}^{c}) / \partial e\|_{2}$ . We observe that Coupled-VAE receives constantly increasing backpropagated gradients from the reconstruction. In contrast to VAE, the  $\| \partial \mathcal{L}_{reg} / \partial e\|_{2}$  in Coupled-VAE does not decrease significantly as the KL weight increases. The decrease of  $\| \partial h / \partial e\|_{F} / \| h\|_{2}$ , which VAE suffers from, is not observed in Coupled-VAE. Plots on more datasets are in Appendix F.
5.8 Sample Diversity
We evaluate the diversity of the samples from the prior distribution. We sample 3200 texts from the prior distribution and report the Dist-1 and Dist-2 metrics (Li et al., 2016), which are the ratios of distinct unigrams and bigrams over all generated unigrams and bigrams. Distinct-1 and Distinct-2 in Table 6 show that texts sampled from CoupledVAE  $(\lambda_{m} = 10.0)$  are more diverse than those from VAE. Given limited space, we put several samples in Appendix G for qualitative analysis.
5.9 Interpolation
A property of VAE is to match the interpolation in the latent space with the smooth transition in the data space (Bowman et al., 2016). In Table 7, we show the interpolation of VAE and Coupled-VAE on PTB. It shows that compared with VAE, Coupled-VAE has smoother transitions of subjects
||∂Lrec/∂e||2	||∂LCrec/∂e||2	||(∂Lrec+LCrec)/∂e||2	||∂Lreg/∂e||2	||∂h/∂e||F/||h||2
PTB	DAE	1719.8	-	-	-	3.14
VAE	112.5	-	-	19.4	2.05
Coupled-VAE	148.5	2109.6	2320.2	27.7	2.12
Yelp	DAE	2443.6	-	-	-	2.55
VAE	59.7	-	-	18.8	1.62
Coupled-VAE	84.8	3640.8	3764.7	25.0	2.25
Yahoo	DAE	4104.6	-	-	-	3.39
VAE	257.9	-	-	52.8	2.92
Coupled-VAE	335.3	5105.0	5615.0	65.0	3.91
Table 5: Gradient norms defined in Section 3.1 on each test set.  ${\lambda }_{m} = {0.1}$  .
PTB	Yelp	Yahoo
D-1	D-2	D-1	D-2	D-1	D-2
VAE	4.61	16.36	0.62	2.48	0.44	2.11
Coupled-VAE	5.51	24.46	1.15	5.93	0.75	3.97
Table 6: Diversity of samples from the prior.
Figure 4: A graphical overview of the generalization to Coupled-CVAE.  $u$  is the condition, encoded as  $e_{u}$ .
(both sides  $\rightarrow$  it) and verbs (are expected  $\rightarrow$  have been  $\rightarrow$  has been  $\rightarrow$  has), indicating that the linguistic information is more smoothly encoded in the latent space of Coupled-VAE.
5.10 Generalization to Conditional Language Modeling: Coupled-CVAE
To generalize our approach to conditional language modeling, we propose Coupled-CVAE. A graphical overview is displayed in Figure 4. Specifically, the (coupled) posterior network and the (coupled) decoder are additionally conditioned. The objective of Coupled-CVAE is identical to Eq. (4).
We compare Couple-CVAE with GRU encoder-decoder (Cho et al., 2014) and CVAE (Zhao et al., 2017) for dialogue generation. We use the Switchboard dataset (John and Holliman, 1993), whose training/validation/test splits are  $203\mathrm{K} / 5\mathrm{K} / 5\mathrm{K}$ , and the vocabulary size is  $13\mathrm{K}$ . For probability estimation, we report the NLL, KL, and PPL based on the gold responses. Since the key motivation of using
CVAE in Zhao et al. (2017) is the diversity of responses, we sample one response for each post and report the Distinct-1 and Distinct-2 metrics over all samples. Please find more details in Appendix I.
Table 8 shows that Coupled-CVAE greatly increases the diversity of dialogue modeling, while it only slightly harms the probability estimation capability. It indicates that Coupled-CVAE better captures the one-to-many nature of conversations than CVAE and GRU encoder-decoder. We also observe that the diversity is improved with increasing  $\lambda_{m}$ , which shows that  $\lambda_{m}$  can control diversity via specifying the richness of the latent space.
6 Relation to Related Work
Bowman et al. (2016) identify the posterior collapse problem of text VAE and propose KL annealing and word drop to handle the problem. Zhao et al. (2017) propose the bag-of-words loss to mitigate this issue. Later work on this problem focuses on less powerful decoders (Yang et al., 2017; Semeniuta et al., 2017), modified regularization objective (Higgins et al., 2017; Bahuleyan et al., 2019; Wang and Wang, 2019), alternative posterior families (Rezende and Mohamed, 2015; Xu and Durrett, 2018; Davidson et al., 2018; Xiao et al., 2018), richer prior distributions (Tomczak and Welling, 2018), improved optimization (He et al., 2019) or KL annealing strategy (Fu et al., 2019), the use of skip connections (Dieng et al., 2019), hierarchical or autoregressive posterior distributions (Park et al., 2018; Du et al., 2018), and narrowing the amortization gap (Hjelm et al., 2016; Kim et al., 2018; Marino et al., 2018). We provide the encoder-decoder incompatibility as a new perspective on the posterior collapse problem. Empirically, our approach can be combined with the above ones to alleviate the problem further.
VAE	Coupled-VAE (λm=10.0)
Text A (sampled from PTB): now those routes are n’t expected to begin until jan
they are n’t expected to be completed	both sides are expected to be delivered at their contract
the new york stock exchange is scheduled to resume today	both sides are expected to be delivered at least
the new york stock exchange is scheduled to resume	both sides have been able to produce up with the current level
it is n’t clear that it will be sold through its own account	it also has been used for comment
it is n’t a major source of credit	it also has been working for the first time
it also has a major chunk of its assets	it also has a new drug for two years
it also has a major pharmaceutical company	it also has a $ N million defense initiative
Text B (sampled from PTB): it also has a _unk_ facility in california
Table 7: Latent space interpolation.
NLL (KL)	PPL	D-1	D-2
GRU Encoder-Decoder*	53.9 (-)	41.6	0.33	0.80
CVAE	54.0 (3.8)	41.8	0.61	2.60
Coupled-CVAE (λm=0.1)	54.1 (4.6)	42.2	0.71	3.18
Coupled-CVAE (λm=0.5)	54.2 (5.3)	42.5	0.78	3.63
Coupled-CVAE (λm=1.0)	54.3 (6.1)	42.7	0.86	4.10
Coupled-CVAE (λm=2.0)	54.6 (7.8)	43.6	0.99	5.16
Table 8: Dialogue generation. *Exact NLL is reported.
A model to be noted is  $\beta$ -VAE (Higgins et al., 2017), in which the reconstruction and regularization are modeled as a hyperparameterized trade-off, i.e., the improvement of one term compromises the other. Different from  $\beta$ -VAE, we adopt the idea of multi-task learning, i.e., the coupled reconstruction task helps improve the encoder chart map and the signal matching task helps improve the decoder chart map. Both our analysis in Section 3.2 and the empirical results show that the modeling of posterior distribution can be improved (but not necessarily compromised) with the additional tasks.
Ghosh et al. (2020) propose to substitute stochasticity with explicit and implicit regularizations, which is easier to train and empirically improves the quality of generated outputs. Different from their work, we still strictly follow the generative nature (i.e., data density estimation) of VAE, and the deterministic network in our approach serves as an auxiliary to aid the optimization.
Encoder pretraining (Li et al., 2019) initializes the text encoder and the posterior network with an autoencoding objective. Li et al. (2019) shows that encoder pretraining itself does not improve the performance of VAE, which indicates that initialization is not strong enough as an inductive bias to learn a meaningful latent space.
Given the discrete nature of text data, we highlight the two-level representation learning for text
modeling: 1) the encoder and decoder parameterizations via autoencoding and 2) a transition map between the parameterizations. Notably, the transition map has large freedom. In our case, the transition map decides the amount and type of information encoded in the variational posterior, and there are other possible instances of the transition map, e.g., flow-based models (Dinh et al., 2015).
In this paper, we observe the encode-decoder incompatibility of VAE for text modeling. We bridge the incompatibility and the posterior collapse problem by viewing the encoder and the decoder as two inadequately learned chart maps from the data manifold to the parameterizations, and the posterior network as a part of the transition map between them. We couple the VAE model with a deterministic network and improve the parameterizations via encoder weight sharing and decoder signal matching. Our approach is model-agnostic and can be applied to a wide range of models in the VAE family. Experiments on benchmark datasets, i.e., PTB, Yelp, and Yahoo, show that our approach improves various VAE models in terms of probability estimation and the richness of the latent space. We also generalize Coupled-VAE to conditional language modeling and propose Coupled-CVAE. Results on Switchboard show that Coupled-CVAE largely improves diversity in dialogue generation.
We would like to thank the anonymous reviewers for their thorough and helpful comments.
Hareesh Bahuleyan, Lili Mou, Hao Zhou, and Olga Vechtomova. 2019. Stochastic Wasserstein autoencoder for probabilistic sentence generation. In Proceedings of NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4068-4076.
Dana H. Ballard. 1987. Modular learning in neural networks. In Proceedings of the 6th National Conference on Artificial Intelligence. Seattle, WA, USA, July 1987., pages 279-284.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-1828.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 10-21.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724-1734.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak. 2018. Hyperspherical variational auto-encoders. In Proceedings of UAI 2018, Monterey, California, USA, August 6-10, 2018, pages 856-865.
Adji B. Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. 2019. Avoiding latent variable collapse with generative skip models. In AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 2397-2405.
Laurent Dinh, David Krueger, and Yoshua Bengio. 2015. NICE: non-linear independent components estimation. In ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings.
Jiachen Du, Wenjie Li, Yulan He, Ruifeng Xu, Lidong Bing, and Xuan Wang. 2018. Variational autoregressive decoder for neural response generation. In Proceedings of EMNLP 2018, Brussels, Belgium, October 31 - November 4, 2018, pages 3154-3163.
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclic annealing schedule: A simple approach to mitigating KL vanishing. In Proceedings of NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 240-250.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. 2020. From variational to deterministic autoencoders. In ICLR 2020, Addis Ababa, Ethiopia, April 30, 2020.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Scholkopf, and Alexander J. Smola. 2012. A kernel two-sample test. J. Mach. Learn. Res., 13:723-773.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoencoders. In *ICLR* 2019, New Orleans, LA, USA, May 6-9, 2019.
Irina Higgins, Loic Matthews, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017.  $\beta$ -VAE: Learning basic visual concepts with a constrained variational framework. In ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
R. Devon Hjelm, Ruslan Salakhutdinov, Kyunghyun Cho, Nebojsa Jojic, Vince D. Calhoun, and Junyoung Chung. 2016. Iterative refinement of the approximate posterior for directed belief networks. In NIPS 2016, December 5-10, 2016, Barcelona, Spain, pages 4691-4699.
Kurt Hornik, Maxwell B. Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-366.
Godfrey John and Edward Holliman. 1993. Switchboard-1 release 2 ldc97s62. Web Download. Philadelphia: Linguistic Data Consortium.
Yoon Kim, Sam Wiseman, Andrew C. Miller, David Sontag, and Alexander M. Rush. 2018. Semi-amortized variational autoencoders. In ICML, volume 80 of Proceedings of Machine Learning Research, pages 2683-2692.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2019. A surprisingly effective fix for deep latent variable modeling of text. In Proceedings of EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3601-3612. Association for Computational Linguistics.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of NAACL-HLT 2016, San Diego California, USA, June 12-17, 2016, pages 110-119.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313-330.
Joseph Marino, Yisong Yue, and Stephan Mandt. 2018. Iterative amortized inference. In Proceedings of ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3400-3409.
Hariharan Narayanan and Sanjoy K. Mitter. 2010. Sample complexity of testing the manifold hypothesis. In NIPS 2010, December 6-9, 2010, Vancouver, British Columbia, Canada., pages 1786-1794.
Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018. A hierarchical latent structure for variational conversation modeling. In Proceedings of NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1792-1801.
Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In Proceedings of ICML 2015, Lille, France, 6-11 July 2015, pages 1530-1538.
DE Rumelhart, GE Hinton, and RJ Williams. 1986. Learning internal representations by error propagation. In Parallel distributed processing: explorations in the microstructure of cognition, vol. 1, pages 318-362. MIT Press.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. 2017. A hybrid convolutional variational autoencoder for text generation. In Proceedings of EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 627-637.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1958.
Ilya O. Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schölkopf. 2018. Wasserstein autoencoders. In ICLR 2018.
Jakub M. Tomczak and Max Welling. 2018. VAE with a vampprior. In AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Machine Learning Research, pages 1214-1223.
Prince Zizhuang Wang and William Yang Wang. 2019. Riemannian normalizing flow on variational wasserstein autoencoder for text modeling. In Proceedings of NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 284-294.
Yijun Xiao, Tiancheng Zhao, and William Yang Wang. 2018. Dirichlet variational autoencoder for text modeling. CoRR, abs/1811.00135.
Jiacheng Xu, Danlu Chen, Xipeng Qiu, and Xuanjing Huang. 2016. Cached long short-term memory neural networks for document-level sentiment classification. In Proceedings of EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1660-1669.
Jiacheng Xu and Greg Durrett. 2018. Spherical latent spaces for stable variational autoencoders. In Proceedings of EMNLP 2018, Brussels, Belgium, October 31 - November 4, 2018, pages 4503-4513.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 3881-3890.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In Proceedings of ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 654-664.
We first introduce the notations used in the following parts. Calligraphic letters (e.g.,  $\mathcal{Q}_0$ ) denotes continuous distributions, and the corresponding lowercase letters (e.g.,  $q_0$ ) stands for probability density functions. The probability of the text is represented as  $P$ .
B Deterministic Networks for Different Posterior Families
In this part, we detail the forward computation of the deterministic networks for different posterior families, including multivariate Gaussian, Gaussian with normalizing flows, and von MisesFisher.
B.1 Multivariate Gaussian
For multivariate Gaussian, we compute the coupled latent code  $z^{c}$  as
$$
\boldsymbol {z} ^ {c} = \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {Q} ^ {c} (\boldsymbol {z} | x)} [ \boldsymbol {z} ] \tag {6}
$$
where  $\mathcal{Q}^c (z|x)$  is the posterior distribution learned by the coupled deterministic network. In effect,  $z$  is the mean vector predicted by the coupled posterior network Posterior $^c$ .
B.2 Gaussian with Normalizing Flows
We first review the background and notations of normalizing flows. An initial latent code is first sampled from an initial distribution, i.e.,  $z_0 \sim \mathcal{Q}_0(z_0|x)$ . The normalizing flow is defined as a series of reversible transformations  $f_1, \ldots, f_K$ , i.e.,
$$
\boldsymbol {z} _ {k} = f _ {k} \circ \dots \circ f _ {1} (\boldsymbol {z} _ {0}) \tag {7}
$$
where  $k = 1,\ldots ,K$ . The evidence lower bound (ELBO) for normalizing flows is derived as
$$
\begin{array}{l} \log P (x) \geq \mathbb {E} _ {\boldsymbol {z} _ {K} \sim \mathcal {Q} _ {K} (\boldsymbol {z} _ {K} | x)} [ \log P (x | \boldsymbol {z} _ {K}) ] \\ - \operatorname {K L} \left[ \mathcal {Q} _ {K} \left(\boldsymbol {z} _ {K} | x\right) \| \mathcal {P} _ {K} \left(\boldsymbol {z} _ {K}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {z} _ {0} \sim \mathcal {Q} _ {0} (\boldsymbol {z} _ {0} | x)} \left[ \log P (x | \boldsymbol {z} _ {K}) \right. \tag {8} \\ - \log q _ {0} (\boldsymbol {z} _ {0} | x) + \log p _ {K} (\boldsymbol {z} _ {K}) \\ \left. + \sum_ {k = 1} ^ {K} \log | \det \frac {\partial f _ {k}}{\partial \boldsymbol {z} _ {k - 1}} | \right] \\ \end{array}
$$
where  $\mathcal{P}_K(z_K)$  is the prior distribution of the transformed latent variable and the reversibility of the transformations guarantees non-zero determinants. Obviously, the optimization of the ELBO for normalizing flows requires sampling from the initial distribution; thus, we compute the coupled latent code  $z^c$  by transforming the predicted mean vector of the coupled initial distribution, i.e.,
$$
\boldsymbol {z} ^ {c} = f _ {k} ^ {c} \circ \dots \circ f _ {1} ^ {c} \left(\mathbb {E} _ {\boldsymbol {z} _ {0} \sim \mathcal {Q} _ {0} ^ {c} (\boldsymbol {z} _ {0} | x)} [ \boldsymbol {z} _ {0} ]\right) \tag {9}
$$