SlowGuess/ACL_2020_OCR huggingface dataset:
where  $\mathcal{Q}_0^c (z_0|x)$  is the coupled initial distribution and  $f_{1}^{c},\ldots ,f_{K}^{c}$  are the coupled transformations. Note that all modules in the deterministic network share the structure with those in the stochastic network. We do not use the posterior mean as the coupled latent code for two reasons. First, our interest is to acquire a deterministic representation that guides the stochastic network, but not necessarily the mean vector. Second, the computation of the posterior mean after the transformations is intractable.
B.3 Von Mises-Fisher
The von Mises-Fisher distribution is supported on a  $(d - 1)$ -dimensional sphere in  $\mathbb{R}^d$  and parameterized by a direction parameter  $\pmb{\mu} \in \mathbb{R}^d$  ( $\| \pmb{\mu} \| = 1$ ) and a concentration parameter  $\kappa$ , both of which are mapped from the encoded text by the posterior network. The probability density function is
$$
q (\boldsymbol {z} | \boldsymbol {\mu}, \kappa) = \frac {\kappa^ {d / 2 - 1} \cdot \exp \left(\kappa \boldsymbol {\mu} ^ {\mathrm {T}} \boldsymbol {z}\right)}{(2 \pi) ^ {d / 2} I _ {d / 2 - 1} (\kappa)} \tag {10}
$$
where  $I_{v}$  is the modified Bessel function of the first kind at order  $v$ . We use the direction parameter  $\mu$  as the coupled latent code  $z^{c}$ . Note that we do not use the posterior mean as the coupled latent code for two reasons. First, similar to normalizing flows, our interest is a deterministic representation rather than the mean vector. Second, the posterior mean of von Mises-Fisher never lies on the support of the distribution, which is suboptimal to guide the stochastic network.
C Details of the Experimental Setup
The dimension of latent vectors is 32. The dimension of word embeddings is 200. The encoder and the decoder are one-layer GRUs with the hidden state size of 128 for PTB and 256 for Yelp and Yahoo. For optimization, we use Adam (Kingma and Ba, 2015) with a learning rate of  $10^{-3}$  and  $\beta_{1} = 0.9$ ,  $\beta_{1} = 0.999$ . The decoding signal is viewed as the first word embedding and also concatenated to the word embedding in each decoding step. After 30K steps, the learning rate is decayed by half each 2K steps. Dropout (Srivastava et al., 2014) rate is 0.2. KL-annealing (Bowman et al., 2016) is applied from step 2K to 42K (on Yelp, it is applied from step 1K to 41K for VAE, Coupled-VAE,  $\beta$ -VAE, and Coupled- $\beta$ -VAE; otherwise, the KL divergence becomes very large in the early stage of training). For each 1K steps, we estimate the NLL for validation.
For normalizing flows (NF), we use planar flows (Rezende and Mohamed, 2015) with three contiguous transformations. For WAE and WAE-NF, we use Maximum Mean Discrepancy (MMD) (Gretton et al., 2012) as the regularization term. An additional KL regularization term with the weight  $\beta = 0.8$  (also with KL-annealing) is added to WAE and WAE-NF since MMD does not guarantee the convergence of the KL divergence.
D Estimation of Language Modeling Metrics
For language modeling, we report negative log-likelihood (NLL), KL divergence, and perplexity. To get more reliable results, we make the estimation of each metric explicit. For each test sample  $x$ , NLL is estimated by importance sampling, and KL
is approximated by its Monte Carlo estimate:
$$
\begin{array}{l} \mathrm {N L L} _ {x} = - \log P (x) \\ \approx - \log \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \frac {p \left(\boldsymbol {z} ^ {(i)}\right) P \left(x \mid \boldsymbol {z} ^ {(i)}\right)}{q \left(\boldsymbol {z} ^ {(i)} \mid x\right)}\right) \tag {11} \\ \end{array}
$$
$$
\begin{array}{l} \mathrm {K L} _ {x} = \mathrm {K L} [ \mathcal {Q} (z | x) \| \mathcal {P} (z) ] \\ \approx \frac {1}{N} \sum_ {i = 1} ^ {N} \log \frac {q (\boldsymbol {z} ^ {(i)} | x)}{p (\boldsymbol {z} ^ {(i)})} \\ \end{array}
$$
where  $z^{(i)}\sim \mathcal{Q}(z|x)$  are sampled latent codes and all notations follow Eq. (1) in the main text. We report the averaged NLL and KL on all test samples. Perplexity is computed based on the estimated NLL. For validation, the number of samples is  $N = 10$  for evaluation, the number of samples is  $N = 100$
E Estimation of Mutual Information and Reconstruction Metrics
We report the mutual information (MI) between the text  $x$  and the latent code  $z$  under  $\mathcal{Q}(z|x)$  to investigate how much useful information is encoded. The MI component of each test sample  $x$  is approximated by Monte Carlo estimation:
$$
\begin{array}{l} \mathbf {M I} _ {x} = \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {Q} (\boldsymbol {z} | x)} [ \log \frac {q (\boldsymbol {z} | x)}{q (\boldsymbol {z})} ] \\ \approx \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\log q \left(\boldsymbol {z} ^ {(i)} \mid x\right) - \log q \left(\boldsymbol {z} ^ {(i)}\right)\right) \tag {12} \\ \end{array}
$$
where the aggregated posterior density  $q(\boldsymbol{z}^{(i)})$  is approximated with its Monte Carlo estimate:
$$
q \left(\boldsymbol {z} ^ {(i)}\right) = \mathbb {E} _ {x} [ q \left(\boldsymbol {z} ^ {(i)} \mid x\right) ] \approx \frac {1}{M} \sum_ {j = 1} ^ {M} q \left(\boldsymbol {z} ^ {(i)} \mid x ^ {(j)}\right) \tag {13}
$$
where  $x^{(j)}$  are sampled from the test set. For convenience, most previous work uses the texts within each batch as the sampled  $x^{(j)}$ 's (which are supposed to be sampled from the entire test set). However, this convention results in a biased estimation since the  $q(z^{(i)}|x^{(i)})$  is computed when  $j = i$ , i.e., the text itself is always sampled when computing its MI component. We remedy it by skipping the term when  $j = i$ . The overall  $\mathrm{MI} = \mathbb{E}_x[\mathrm{MI}_x]$  is then estimated by averaging  $\mathrm{MI}_x$  over all test samples. We set the numbers of samples as  $N = 100$  and  $M = 512$ .
For reconstruction, we sample ten latent codes from the posterior of each text input and decode them with greedy search. We compute BLEU-1 and BLEU-2 between the reconstruction and the input with the Moses script.
F Training Dynamics of Gradient Norms
We show the tracked gradient norms on all datasets in Figure 5. The observations are consistent with those discussed in Section 5.7 in the main text.
G Diversity and Samples from the Prior Distribution
Given the limited space in the main text, we place the comprehensive evaluation of samples from the prior distribution in this part. Table 9 shows the diversity metrics and the first three (thus totally random) samples from each model. Qualitatively, samples from Coupled-VAE is more diverse than those from VAE. The long texts generated from VAE have more redundancies compared with Coupled-VAE. Given that both models have the same latent dimension, it indicates that Coupled-VAE is using the latent codes more efficiently.
H Interpolation
A property of VAE is to match the interpolation in the latent space with the smooth transition in the text space (Bowman et al., 2016). In Table 7, we show the interpolation of VAE and Coupled-VAE on PTB. It shows that compared with VAE, Coupled-VAE has smoother transitions of subjects (both sides  $\rightarrow$  it) and verbs (are expected  $\rightarrow$  have been  $\rightarrow$  has been  $\rightarrow$  has), indicating that the information about subjects and verbs is more smoothly encoded in the latent space of Coupled-VAE.
I Generalization to Conditional Generation: Coupled-CVAE
To generalize our approach to conditional generation, we focus on whether it can improve the CVAE model (Zhao et al., 2017) for dialogue generation. To this end, we propose the Coupled-CVAE model.
I.1 CVAE
CVAE adopts a two-step view of diverse dialogue generation. Let  $x$  be the response and  $y$  be the post (or the context). CVAE first samples the latent code  $z$  from the prior distribution  $\mathcal{P}(z|y)$  and then samples the response from the decoder  $P(x|z, y; \theta)$ . Given the post  $y$ , the marginal distribution of the response  $x$  is
$$
P (x \mid y; \theta) = \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {P} (\boldsymbol {z} \mid y)} [ P (x \mid \boldsymbol {z}, y; \theta) ] \tag {14}
$$
Similar to VAE, the exact marginalization is intractable, and we derive the evidence lower bound
(a) PTB
(b) PTB
(c) PTB
(e) Yelp
(f)Yelp
(g) Yahoo
(h) Yahoo
(i) Yahoo
Figure 5: Training dynamics of DAE, VAE, and Coupled-VAE ( $\lambda_{m} = 10.0$ ). (a), (d), and (g) are  $\| \partial \mathcal{L}_{rec} / \partial e \|_2$  for DAE and VAE, and  $\| \partial (\mathcal{L}_{rec} + \mathcal{L}_{rec}^c) / \partial e \|_2$  for Coupled-VAE. (b), (e), (h) denote  $\| \partial \mathcal{L}_{reg} / \partial e \|_2$ . (c), (f), (i) stand for  $\| \partial h / \partial e \|_F / \| h \|_2$ . Best viewed in color (yet the models are distinguished by line markers).
(ELBO) of CVAE as
$$
\begin{array}{l} \log P (x | y; \theta) \geq \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {Q} (\boldsymbol {z} | x, y; \phi)} [ \log P (x | \boldsymbol {z}, y; \theta) ] \\ - \operatorname {K L} \left[ \mathcal {Q} (z | x, y; \phi) \| \mathcal {P} (z | y) \right] \tag {15} \\ \end{array}
$$
During training, the response and the post are encoded as  $e_x$  and  $e_y$ , respectively. The two vectors are concatenated and transformed into the posterior via the posterior network. A latent code is then sampled and mapped to a higher-dimensional  $h$ . The decoding signal in CVAE is computed by  $h$  and  $e_y$  and utilized to infer the response. Similar to VAE, the objective of CVAE can also be viewed as a reconstruction loss and a regularization term in Eq. (15).
I.2 Coupled-CVAE
As observed in Zhao et al. (2017), the CVAE model also suffers from the posterior collapse problem. We generalize our approach to the conditional setting and arrive at Coupled-CVAE. A graphical overview is displayed in Figure 4. The difference from Coupled-VAE is shown in red. Specifically, the (coupled) posterior network and the (coupled) decoder are additionally conditioned on the post representation. The objective of Coupled-CVAE is identical to Eq. (4) in the main text.
The coupled reconstruction loss  $\mathcal{L}_{rec}^{c}$  in Coupled-CVAE has two functions. First, it improves the encoded response  $e_{x}$ , which is similar to Coupled-VAE. Second, it encourages  $h^c$  to encode more response information rather than the post information, which collaborates with  $\mathcal{L}_{match}$  to improve the parameterization  $h$ .
I.3 Dataset
We use the Switchboard dataset (John and Holliman, 1993). We split the dialogues into single-turn post-response pairs, and the number of pairs in the training/validation/test split is  $203\mathrm{K} / 5\mathrm{K} / 5\mathrm{K}$ . The vocabulary size is  $13\mathrm{K}$ .
I.4 Evaluation
For probability estimation, we report the NLL, KL, and PPL based on the gold responses. NLL, KL, and PPL are as computed in Appendix D except for the additional condition on the post. Since the key motivation of using CVAE in Zhao et al. (2017) is the response diversity, we sample one response for each post and report the Distinct-1 and Distinct-2 metrics over all test samples.
I.5 Experimental Setup
We compare our Coupled-CVAE model with two baselines: GRU encoder-decoder (Cho et al., 2014) and CVAE (Zhao et al., 2017). The detailed setup follows that of the PTB dataset in Appendix C. For each 1K steps, we estimate the NLL for validation.
I.6 Results
Experimental results of Coupled-CVAE are shown in the main text.
Table 9: Diversity metrics and the first three samples from each model. Redundancies (pieces of text that appeared before) are shown in red.