SlowGuess/NAACL_2022_OCR huggingface dataset:
Jinyi Hu $^{1,2,3}$ , Xiaoyuan Yi $^{5}$ , Wenhao Li $^{1,2,3}$ , Maosong Sun $^{1,2,3,4*}$ , Xing Xie $^{5}$
<sup>1</sup> Department of Computer Science and Technology, Tsinghua University, Beijing, China
$^{2}$  Beijing National Research Center for Information Science and Technology
<sup>3</sup> Institute for Artificial Intelligence, Tsinghua University, Beijing, China
$^{4}$  International Innovation Center of Tsinghua University, Shanghai, China
hu- jy21@mails.tsinghua.edu.cn, xiaoyuanyi@microsoft.com
The past several years have witnessed Variational Auto-Encoder's superiority in various text generation tasks. However, due to the sequential nature of the text, auto-regressive decoders tend to ignore latent variables and then reduce to simple language models, known as the KL vanishing problem, which would further deteriorate when VAE is combined with Transformer-based structures. To ameliorate this problem, we propose DELLA, a novel variational Transformer framework. DELLA learns a series of layer-wise latent variables with each inferred from those of lower layers and tightly coupled with the hidden states by low-rank tensor product. In this way, DELLA forces these posterior latent variables to be fused deeply with the whole computation path and hence incorporate more information. We theoretically demonstrate that our method can be regarded as entangling latent variables to avoid posterior information decrease through layers, enabling DELLA to get higher nonzero KL values even without any annealing or thresholding tricks. Experiments on four unconditional and three conditional generation tasks show that DELLA could better alleviate KL vanishing and improve both quality and diversity compared to several strong baselines.
Variational Autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014) has proven to be successful in generating various kinds of text, such as stylistic text (Hu et al., 2017; John et al., 2019), dialogue (Zhao et al., 2017), story (Yu et al., 2020) and poetry (Yi et al., 2020). The sequential nature of the text leads to typically used autoregressive decoders in VAE for language generation. However, such strong decoders tend to evade the difficulty of learning meaningful latent codes by heavily relying on previously generated words
Figure 1: Existing paradigms of Transformer VAE.
and hence ignore latent variables (Bowman et al., 2016), known as  $KL$  vanishing or posterior collapse. This problem causes two drawbacks: (a) the posterior distribution quickly turns into the prior one (usually standard Gaussian), falling to build expressive latent representations; (b) the decoder reduces to a naive language model, resulting in monotonous generated text (Fu et al., 2019).
To ameliorate this problem, researchers have designed various techniques. Among them, three broadly used methods include weakening decoders (Bowman et al., 2016; Semeniuta et al., 2017; Zhao et al., 2017), KL annealing (Bowman et al., 2016; Fu et al., 2019) and KL threshold (Kingma et al., 2016; Higgins et al., 2017; Li et al., 2019). Nonetheless, the weakening of decoders restrains models' language modelling capability; annealing hyperparameters are hard to tune; KL threshold introduces a non-smooth objective with some optimization difficulties.
In the era of RNN, VAE can be easily incorporated by using the latent variable as the initial decoder state, while how to combine VAE with recently prevalent Transformer (Vaswani et al., 2017) architectures, which have made a breakthrough in text generation, still remains an open challenge.
*Corresponding author. Email: sms@tsinghua.edu.cn
Human Language Technologies, pages 697 - 716
As shown in Fig.1, existing methods of integrating Transformer into VAE fall into three main paradigms: (a) directly adding latent variables to input token embeddings (abbr. Embedding) (Li et al., 2020a); (b) using latent variables as a separate memory token vector to be attended by self-attention in each layer (abbr. Memory) (Fang et al., 2021); (c) combining latent variables with the last-layer decoder states before output softmax (abbr. Softmax) (Wang and Wan, 2019). However, paradigm (a) brings noise for self-attention. In paradigm (b), memory vectors tend to be ignored by attention, even exacerbating KL vanishing. In paradigm (c), latent variables couldn't deeply interfere with the whole computation path. Sec.3.3 presents more detailed analyses.
To better incorporate Transformer into VAE and theoretically ameliorate the  $KL$  vanishing problem, we propose DELLA<sup>1</sup>, a novel variational transformer framework. DELLA learns a series of layerwise latent variables in a Transformer encoder, and each is inferred from those of lower layers and then tightly coupled with the hidden states in the corresponding decoder layer by low-rank tensor product. Our method theoretically stimulates the entanglement of latent variables and hence allows propagation of undiminished latent information through layers. As a result, DELLA forces posterior latent variables to be deeply fused with the entire computation path and encode richer information of input text, achieving higher KL values even without any annealing or threshold training tricks.
In summary, our contributions are as follow: (i) We are the first to propose layer-wise inferred latent variables in Transformer-based architecture to mitigate KL vanishing; We (ii) innovatively inject latent variables using low-rank tensor product, (iii) provide a theoretical validity of our method and (iv) demonstrate its effectiveness on four unconditional and three conditional generation tasks. Our codes are available at https://github.com/OpenVLG/DELLA.git.
Thanks to the representation capacity of latent space, VAE has been widely adopted for both image generation (van den Oord et al., 2017; Vahdat and Kautz, 2020) and text generation (Bowman et al., 2016; Hu et al., 2017). In the early stage, VAE was combined with RNN decoders for gener
ating a broad range of text, varying from dialogue (Serban et al., 2016), image caption (Wang et al., 2017), text summarization (Gupta et al., 2017) to story (Yu et al., 2020) and poetry (Yi et al., 2020). In this case, latent variables are usually utilized as either the initial decoder state (Li et al., 2018) or input at each time step (Gupta et al., 2017).
In spite of extensive applications, VAE suffered from KL vanishing in the scenario of text generation (Bowman et al., 2016). Several lines of techniques have been proposed to alleviate this problem. The first line is to avoid a too fast decrease of the KL divergence by re-weighting. KL annealing (Bowman et al., 2016) linearly increased the weight of KL term from 0 to 1 during the warm-up period. Fu et al. (2019) further proposed cyclical annealing, which repeats the warm-up process multiple times. The second line guarantees a positive lower bound of the KL term. KL thresholding (Kingma et al., 2016) achieved a fixed minimum by combining a hinge loss, while BN-VAE (Zhu et al., 2020) learned more flexible ones via batch normalization.  $\delta$ -VAE (Razavi et al., 2019) chose to restrain the family of posterior distributions. The third line aims to constraint decoders to force a more informative latent variable. Wang et al. (2017) introduced an auxiliary BOW (bag-of-words) loss. He et al. (2019) added additional training loops for the encoder. Yang et al. (2017) adopted dilated CNN as decoder, and Dieng et al. (2019) added skip connections to the decoder. Although the above methods mitigate KL vanishing to some extent, it is still challenging for either tuning or optimization.
In these years, the powerful Transformer has been integrated with VAE to benefit diverse tasks, including text classification (Gururangan et al., 2019), story generation (Wang and Wan, 2019; Fang et al., 2021) and dialogue generation (Lin et al., 2020). Optimus (Li et al., 2020a) further bridged the pre-trained BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019) with VAE for pre-training. Most existing works inject latent variables into the Transformer decoder by the three paradigms, Embedding (Li et al., 2020a), Memory (Li et al., 2020a; Fang et al., 2021) and Softmax(Wang and Wan, 2019), as discussed in Sec. 1, while these methods shallowly fuse the latent variables with hidden states. To achieve deeper fusion and ameliorate KL vanishing, we propose DELLA.
The most relevant architecture to our model is hierarchical VAE (Sønderby et al., 2016; Klushyn
1 DELLA: D'epply Fused Layer-wise LAtent Variables
et al., 2019; Vahdat and Kautz, 2020; Child, 2020), which is mainly designed for image generation and not suitable for text. For text generation, hierarchical latent variables are either independent of each other (Serban et al., 2016), or corresponding to different text granularities (sentence or word level), while our DELLA learns conditionally inferred and layer-wise latent variables based on Transformer.
Transformer (Vaswani et al., 2017) represents an input sequence  $\pmb{x} = \{x_{1},\dots,x_{i},\dots,x_{n}\}$  as contextualized distributed hidden states  $\pmb{h} = \{h_1,\dots,h_i,\dots,h_n\}$  by a series of stacked layers, and states in the  $l$ -th layer,  $\pmb{h}^{(l)}$ , are calculated with scaled dot-product attention:
$$
\operatorname {A t t e n t i o n} (Q, K, V) = \operatorname {s o f t m a x} \left(\frac {Q ^ {T} K}{\sqrt {d}}\right) V ^ {T}, \tag {1}
$$
where  $Q, K, V$  stand for Query, Key, Value, respectively, which are projected from outputs of the previous layer:  $Q = W^{q} \pmb{h}^{(l-1)}$ ,  $K = W^{k} \pmb{h}^{(l-1)}$ ,  $V = W^{v} \pmb{h}^{(l-1)}$ .  $d$  is the dimension of hidden states. In practice, multiple groups of states are calculated with different attention parameters and then concatenated, known as multi-head attention.
3.2 VAE
As a kind of generative model, VAE estimates the intractable data distribution  $p(x)$  by deriving and maximizing its lower bound as:
$$
\begin{array}{l} \log p (\boldsymbol {x}) \geq \mathcal {L} _ {E L B O} (\boldsymbol {x}; \boldsymbol {\theta}, \phi) = \\ \mathbb {E} _ {q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})} [ \log p _ {\boldsymbol {\theta}} (\boldsymbol {x} | \boldsymbol {z}) ] - \mathrm {K L} (q _ {\phi} (\boldsymbol {z} | \boldsymbol {x}) | | p (\boldsymbol {z})) \tag {2} \\ \end{array}
$$
where  $z$  is the latent variable and  $p(z)$  is the prior distribution of latent variable which is commonly assumed as standard Gaussian; the posterior distribution  $p(z|x)$  is approximated by an inference network (encoder)  $q_{\phi}(z|x); p_{\theta}(x|z)$  is a generator (decoder) to generate text  $x$  from the latent variable  $z$ ;  $\theta$  and  $\phi$  are corresponding parameters.
The whole lower bound in Eq.(2), called Evidence Lower BOund (ELBO), consists of two terms: the reconstruction loss,
$$
\mathcal {L} _ {E} = - \mathbb {E} _ {q _ {\phi} (z | x)} \left[ \log p _ {\boldsymbol {\theta}} (\boldsymbol {x} | z) \right], \tag {3}
$$
which helps reconstruct the input given the posterior latent variable  $z$ , and the KL divergence,
$$
\mathcal {L} _ {R} = \operatorname {K L} \left(q _ {\phi} (\boldsymbol {z} | \boldsymbol {x}) \| p (\boldsymbol {z})\right). \tag {4}
$$
In practice, VAE is considered as a regularized Auto-encoder, and a hyper-parameter  $\beta$  is introduced to control the strength of KL,  $\beta \mathcal{L}_R$ , usually used in KL annealing methods (Fu et al., 2019).
3.3 Incorporate Transformer into VAE
For Transformer encoder, the posterior  $z$  is mapped from the text representation, which can be the pooling of all hidden states in the last layer (Fang et al., 2021), or state of a special token (Li et al., 2020a), e.g., [CLS]. Then  $z$  is injected into Transformer decoder by the paradigms discussed in Sec. 1.
Now we take a further step and investigate why intrinsically these three paradigms, namely Embedding, Memory and Softmax, would perform poorly.
Embedding: Define  $\pmb{e}_i, \pmb{e}_j$  as two token embeddings and  $\alpha_{i,j}$  as the attention weight of  $i$ -th and  $j$ -th tokens. From Eq.(1), we have  $\alpha_{i,j} = (W^q \pmb{e}_i)^T (W^k \pmb{e}_j) = \pmb{e}_i^T (W^q)^T W^k \pmb{e}_j$ , which is further abbreviated as  $\langle \pmb{e}_i, \pmb{e}_j \rangle$ . Such Embedding paradigm directly adds  $\pmb{z}$  to token embeddings as:
$$
\begin{array}{l} \alpha_ {i, j} ^ {\prime} = \left[ W ^ {q} \left(\boldsymbol {e} _ {i} + \boldsymbol {z}\right) \right] ^ {T} \left[ W ^ {k} \left(\boldsymbol {e} _ {j} + \boldsymbol {z}\right) \right] \tag {5} \\ = \langle \boldsymbol {e} _ {i}, \boldsymbol {e} _ {j} \rangle + \langle \boldsymbol {e} _ {i}, \boldsymbol {z} \rangle + \langle \boldsymbol {z}, \boldsymbol {e} _ {j} \rangle + \langle \boldsymbol {z}, \boldsymbol {z} \rangle , \\ \end{array}
$$
where we can find that a redundant term,  $\langle z,z\rangle$ , is introduced, bringing extra noise for attention mechanism. Moreover, information in  $z$  could diminish with propagation through layers (Fig. 2), aggravating KL vanishing.
Memory: This paradigm treats  $z$  as an additional memory token and places it at the beginning of  $x$  to be attended by other tokens via attention. Nevertheless, as mentioned in Sec. 1, the powerful Transformer decoder may only rely on preceding decoded tokens. Consequently, with no explicit constraints (e.g., auxiliary loss), such a memory token is more likely to be ignored by self-attention (Fig. 6 & 7), even exacerbating KL vanishing.
Softmax: This paradigm first adds  $\mathbf{z}$  to the last-layer hidden states  $\mathbf{h}$ , and then projects  $\mathbf{z} + \mathbf{h}$  into a logit vector  $\mathbf{p} \in \mathbb{R}^v$  over the vocabulary, where  $v$  is vocab size. In this method, latent variables do not interact with hidden states until the last layer, which erodes the effect of latent variables (see Fig. 2).
As demonstrated in Sec. 3, existing three paradigms make latent variables gradually diminish through layers, be ignored by self-attention or inadequately interact with hidden states, which would not mitigate but even worsen the KL vanishing problem.
To deeply fuse latent variables with the whole computation path of Transformer, we propose DELLA to learn a series of layer-wise posterior latent variables which are conditionally inferred in encoder, and injected into hidden states in decoder by low-rank tensor product. We present layer-wise latent variables in Sec. 4.1, describe the tensor product fusion in Sec. 4.2, give the theoretical verification of DELLA's effectiveness for ameliorating KL vanishing in Sec. 4.3, and then extend DELLA to Conditional VAE (CVAE) in Sec. 4.4.
4.1 Layer-wise Latent Variables
Different from previous work where only one latent variable  $z$  is calculated and shared by (Li et al., 2020a) or projected to (Fang et al., 2021) decoder layers, we involve a series of latent variables  $z = \{z_{1}, z_{2}, \ldots, z_{L}\}$ , where  $L$  is the number of Transformer layers. Then we reformulate the prior and posterior distributions as  $p(z) = \prod_{l=1}^{L} p(z_{l} | z_{<l})$ ,  $q(z | x) = \prod_{l=1}^{L} q(z_{l} | z_{<l}, x)$ , respectively, with each  $z_{l}$  still following Gaussian distribution. Then we rewrite  $\mathcal{L}_{R}$  in Eq.(4) similar to Vahdat and Kautz (2020):
$$
\begin{array}{l} \mathcal {L} _ {R} = \mathrm {K L} (q (\boldsymbol {z} | \boldsymbol {x}) | | p (\boldsymbol {z})) \\ = \sum_ {l = 1} ^ {L} \mathbb {E} _ {q \left(\boldsymbol {z} _ {<   l} \mid \boldsymbol {x}\right)} \left[ \mathrm {K L} \left(q \left(\boldsymbol {z} _ {l} \mid \boldsymbol {x}, \boldsymbol {z} _ {<   l}\right) \mid p \left(\boldsymbol {z} _ {l} \mid \boldsymbol {z} _ {<   l}\right)\right) \right]. \tag {6} \\ \end{array}
$$
When  $l = 1$ ,  $p(\pmb{z}_1|\pmb{z}_{<1}) = p(\pmb{z}_1)$  is the standard Gaussian distribution,  $q(\pmb{z}_1|\pmb{x},\pmb{z}_{<1}) = q(\pmb{z}_1|\pmb{x})$ . We give detailed derivations in Appendix B.1.
These latent variables  $\mathbf{z}_l$  are calculated (inferred) layer by layer using representations of the corresponding layer. Concretely, in  $l$ -th layer, we use the hidden state of the first token in text  $\mathbf{x}$ , as its  $l$ -th-layer representation, denoted as  $\mathbf{x}^{(l)} \in \mathbb{R}^d$ , where  $d$  is hidden size. Then we represent latent variables in lower layers as  $\mathbf{z}_{<l}$  and obtain it by:
$$
\boldsymbol {z} _ {<   l} = \tanh (\boldsymbol {W} _ {h h} ^ {(l)} \boldsymbol {z} _ {<   l - 1} + \boldsymbol {W} _ {i h} ^ {(l)} \boldsymbol {z} _ {l - 1}), \quad (7)
$$
where  $W_{hh}, W_{ih} \in \mathbb{R}^{p \times p}$ , so  $\boldsymbol{z}_{<l} \in \mathbb{R}^p$  and  $p$  is the dimension of latent variable.  $\boldsymbol{z}_0$  and  $\boldsymbol{z}_{<0}$  are set as zero vector. We calculate the mean and variance vectors of  $p(\boldsymbol{z}_l | \boldsymbol{z}_{<l})$  and  $q(\boldsymbol{z}_l | \boldsymbol{z}_{<l}, \boldsymbol{x})$  by:
$$
\begin{array}{l} \binom {\boldsymbol {\mu} _ {p}} {\log \left(\boldsymbol {\sigma} _ {p} ^ {2}\right)} = \boldsymbol {W} _ {p} ^ {(l)} \boldsymbol {z} _ {<   l}, \tag {8} \\ \left( \begin{array}{c} \boldsymbol {\mu} _ {q} \\ \log (\boldsymbol {\sigma} _ {q} ^ {2}) \end{array} \right) = \boldsymbol {W} _ {q} ^ {(l)} \left( \begin{array}{c} \boldsymbol {z} _ {<   l} \\ \boldsymbol {x} ^ {(l)} \end{array} \right), \\ \end{array}
$$
where  $\pmb{W}_p\in \mathbb{R}^{p\times 2p}$ $\pmb {W}_p\in \mathbb{R}^{p\times 2p}$
The latent variable  $\boldsymbol{z}_l$  is sampled from the posterior distribution  $q(\boldsymbol{z}_l | \boldsymbol{z}_{<l}, \boldsymbol{x}) = \mathcal{N}(\boldsymbol{\mu}_q, \boldsymbol{\sigma}_q^2 \boldsymbol{I})$  for training, and from the prior one  $q(\boldsymbol{z}_l | \boldsymbol{z}_{<l}) = \mathcal{N}(\boldsymbol{\mu}_p, \boldsymbol{\sigma}_p^2 \boldsymbol{I})$  for testing. Since hidden states in each layer belong to different vector spaces, the parameters to calculate each  $\boldsymbol{z}_{<l}$ , e.g.,  $\boldsymbol{W}_p^{(l)}$  and  $\boldsymbol{W}_q^{(l)}$ , do not share throughout different layers.
4.2 Low-rank Tensor Product
We inject the latent variable  $z_{l}$ , which is obtained based on  $l$ -th encoder layer, into the corresponding  $l$ -th decoder layer. Instead of simply using  $z_{l}$  as a memory token as discussed in Sec. 3.3, we resort to low-rank tensor product, which has been successfully utilized for fusing multimodal representations (Liu et al., 2018), to deeply fuse latent variables with hidden states in the decoder.
In detail, we conduct low-rank tensor product on  $\mathbf{z}_l$  and  $\mathbf{x}_i$ 's  $l$ -th-layer value vector  $\mathbf{v}_i^{(l)}$  as:
$$
\widetilde {\boldsymbol {v}} _ {i} ^ {(l)} = \left(\sum_ {j = 1} ^ {r} \boldsymbol {W} _ {v} ^ {(l, j)} \boldsymbol {v} _ {i} ^ {(l)}\right) \circ \left(\sum_ {j = 1} ^ {r} \boldsymbol {W} _ {z} ^ {(l, j)} \boldsymbol {z} _ {l}\right), \tag {9}
$$
where  $r$  is a hyper-parameter,  $\circ$  means elementwise multiplication,  $\mathbf{W}_v\in \mathbb{R}^{d\times d}$ .  $\mathbf{W}_z\in \mathbb{R}^{p\times d}$  are learnable parameters which are shared across all positions  $(i)$  but not shared with layers  $(l)$ , considering distinct vector spaces in different layers, as mentioned in Sec. 4.1. Then the fused Value  $\widetilde{V}^{(l)} = \{\widetilde{\boldsymbol{v}}_1^{(l)},\dots,\widetilde{\boldsymbol{v}}_n^{(l)}\}$  is used in Eq.(1)
In this way, layer-wise  $\mathbf{z}_l$  is conditionally inferred from latent variables in previous encoder layers, together with  $l$ -th-layer text representation, and then explicitly fused with the corresponding decoder layer, yielding a deeper intervention throughout the whole computation path of Transformer.
4.3 Why Could DELLA Work Well?
To theoretically interpret the advantage of layerwise latent variables which contributes most to DELLA (Table 4), we give the following theorem:
Theorem 1 For an observation  $\pmb{x}$  and a sequence of latent variables  $\pmb{z}_1, \pmb{z}_2, \dots, \pmb{z}_L$ , satisfying  $p(\pmb{z}) = \prod_{l=1}^{L} p(\pmb{z}_l | \pmb{z}_{<l})$ , and  $q(\pmb{z} | \pmb{x}) = \prod_{l=1}^{L} q(\pmb{z}_l | \pmb{z}_{<l}, \pmb{x})$ , then the expectation of the KL term,  $\mathbb{E}_{p(\pmb{x})}[\mathcal{L}_R]$  is an upper bound of:
$$
- \sum_ {i = 2} ^ {L - 1} I \left(\boldsymbol {z} _ {L}; \dots ; \boldsymbol {z} _ {i} \mid \boldsymbol {z} _ {i - 1}\right) - I \left(\boldsymbol {z} _ {L}; \dots ; \boldsymbol {z} _ {1} \mid \boldsymbol {x}\right), \tag {10}
$$
where  $I$  is the interaction information<sup>2</sup>.
See Appendix B.2 for proof. Based on Theorem 1, minimizing  $\mathcal{L}_R$  approximatively means maximizing each interaction information term in Eq.(10), which forces the entanglement of all latent variables  $z_{1};\ldots ;z_{L}$  given the observation  $\pmb{x}$ , alleviating the diminishing of information encoded in latent variables when propagating through layers.
4.4 Extension to CVAE
DELLA could also be applied to CVAE for conditional generation tasks like storytelling. Given an observation  $\pmb{x}$  and its condition  $c$ , we can optimize:
$$
\begin{array}{l} \log p (\boldsymbol {x} | \boldsymbol {c}) \geq \mathbb {E} _ {q _ {\phi} (\boldsymbol {z} | \boldsymbol {x}, \boldsymbol {c})} [ \log p _ {\boldsymbol {\theta}} (\boldsymbol {x} | \boldsymbol {z}, \boldsymbol {c}) ] \tag {11} \\ - \operatorname {K L} \left(q _ {\phi} (\boldsymbol {z} | \boldsymbol {x}, \boldsymbol {c}) | | p (\boldsymbol {z} | \boldsymbol {c})\right), \\ \end{array}
$$
and then replace the prior distribution  $q(\pmb{z}_l|\pmb{x},\pmb{z}_{< l})$  and posterior distribution  $p(\pmb{z}_l|\pmb{z}_{< l})$  in Eq.(6) with  $q(\pmb{z}_l|\pmb{x},\pmb{c},\pmb{z}_{< l})$  and  $p(\pmb{z}_l|\pmb{z}_{< l},\pmb{c})$ , respectively.
In this case, we encode the condition  $\pmb{c}$  with the same encoder. Similarly, we can obtain the representation of  $\pmb{c}$  at  $l$ -th layer, denoted as  $\pmb{c}^{(l)} \in \mathbb{R}^d$ , and then calculate the mean and log variance of  $p(\pmb{z}_l | \pmb{z}_{<l}, \pmb{c})$  and  $q(\pmb{z}_l | \pmb{z}_{<l}, \pmb{x}, \pmb{c})$  by:
$$
\begin{array}{l} \left( \begin{array}{c} \boldsymbol {\mu} _ {p} \\ \log (\boldsymbol {\sigma} _ {p} ^ {2}) \end{array} \right) = \hat {\boldsymbol {W}} _ {p} ^ {(l)} \left( \begin{array}{c} \boldsymbol {z} _ {<   l} \\ \boldsymbol {c} ^ {(l)} \end{array} \right), \\ \binom {\boldsymbol {\mu} _ {q}} {\log \left(\boldsymbol {\sigma} _ {q} ^ {2}\right)} = \hat {\boldsymbol {W}} _ {q} ^ {(l)} \binom {\boldsymbol {z} _ {<   l}} {\boldsymbol {x} ^ {(l)}}, \tag {12} \\ \end{array}
$$
where  $\hat{\pmb{W}}_p^{(l)}\in \mathbb{R}^{(p + d)\times 2p}$ $\hat{\pmb{W}}_q^{(l)}\mathbb{R}^{(p + 2d)\times 2p}$
We consider four datasets for language modelling and unconditional generation, including the Yelp, and Yahoo (Yang et al., 2017; He et al., 2019), Penn Treebank (PTB) (Marcus et al., 1993), and SNLI (Bowman et al., 2015), and three datasets for conditional generation tasks, including summarization generation with CNN/DailyMail (CNN/DM) (See et al., 2017), story generation with WritingPrompts (WP) (Fan et al., 2018) and paraphrase generation with Quora  $^{3}$ . Detailed data statistics are listed in Table 7. Due to the limited computation capability, we use 165,157 samples in CNN/DM and 22,2614 in WP with the max length of 900 for training.
We use pretrained language models as the backbone and fine-tune them on each task mentioned above with our DELLA as in (Li et al., 2020a). For unconditional generation and story generation, encoder and decoder shared the same parameters initialized with 12-layer GPT-2 (Radford et al., 2019). For summarization and paraphrase generation, parameters are not shared and initialized with BART-base (Lewis et al., 2020). We set the dimension of latent variable as 32 for all VAE-based models and use cyclical annealing for training, following (Li et al., 2020a). More details are given in Appendix A.1.
We make a comprehensive comparison with strong Transformer-based baselines. We do not consider RNN-based models that are inferior to Transformer for text generation as shown in (Li et al., 2020a).