ON VARIATIONAL LEARNING OF CONTROLLABLE REPRESENTATIONS FOR TEXT WITHOUT SUPERVISION
Year: 2019

Abstract:
The variational autoencoder (VAE) has found success in modelling the manifold of natural images on certain datasets, allowing meaningful images to be generated while interpolating or extrapolating in the latent code space, but it is unclear whether similar capabilities are feasible for text considering its discrete nature. In this work, we investigate the reason why unsupervised learning of controllable representations fails for text. We find that traditional sequence VAEs can learn disentangled representations through their latent codes to some extent, but they often fail to properly decode when the latent factor is being manipulated, because the manipulated codes often land in holes or vacant regions in the aggregated posterior latent space, which the decoding network is not trained to process. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method significantly outperforms unsupervised baselines and is competitive with strong supervised approaches on text style transfer. Furthermore, when switching the latent factor (e.g., topic) during a long sentence generation, our proposed framework can often complete the sentence in a seemingly natural way – a capability that has never been attempted by previous methods.

heading: 1 INTRODUCTION
text: High-dimensional data, such as images and text, are often causally generated through the interaction of many complex factors, such as lighting and pose in images or style and content in texts. Recently, VAEs and other unsupervised generative models have found successes in modelling the manifold of natural images (Higgins et al., 2017; Kumar et al., 2017; Chen et al., 2016). These models often discover controllable latent factors that allow manipulation of the images through conditional generation from interpolated or extrapolated latent codes, often with impressive quality. On the other hand, while various attributes of text such as sentiment and topic can be discovered in an unsupervised way, manipulating the text by changing these learned factors have not been possible with unsupervised generative models to the best of our knowledge. Cı́fka et al. (2018); Zhao et al. (2018) observed that text manipulation is generally more challenging compared to images, and the successes of these models cannot be directly transferred to texts.
Controllable text generation aims at generating realistic text with control over various attributes including sentiment, topic and other high-level properties. Besides being a scientific curiosity, the possibility of unsupervised controllable text generation could help in a wide range of application, e.g., dialogues systems (Wen et al., 2016). Existing promising progress (Shen et al., 2017; Fu et al., 2018; Li et al., 2018; Sudhakar et al., 2019) all relies on supervised learning from annotated attributes to generate the text in a controllable fashion. The high cost of labelling large training corpora with attributes of interest limits the usage of these models, as pre-existing annotations often do not align with some downstream goal. Even if cheap labels are available, for example, review scores as a proxy for sentiment, the control is limited to the variation defined by the attributes.
In this work, we examine the obstacles that prevent sequence VAEs from performing well in unsupervised controllable text generation. We empirically discover that manipulating the latent factors for typical semantic variations often leads to latent codes that reside in some low-density region of the
aggregated posterior distribution. In other words, there are vacant regions in the latent code space (Makhzani et al., 2015; Rezende & Viola, 2018) not being considered by the decoding network, at least not at convergence. As a result, the decoding network is unable to process such manipulated latent codes, yielding unpredictable generation results of low quality.
In order to mitigate the latent vacancy problem, we propose to constrain the posterior mean to a learned probability simplex and only perform manipulation within the probability simplex. Two regularizers are added to the original objective of VAE. The first enforces an orthogonal structure of the learned probability simplex; the other encourages this simplex to be filled without holes. Besides confirming that latent vacancy is indeed a cause of failure in previous sequence VAEs’, it is also the first successful attempt towards unsupervised learning of controllable representations for text to the best of our knowledge. Experimental results on text style transfer show that our approach significantly outperforms unsupervised baselines, and is competitive with strong supervised approaches across a wide range of evaluation metrics. Our proposed framework also enables finer-grained and more flexible control over text generation. In particular, we can switch the topic in the middle of sentence generation, and the model will often still find a way to complete the sentence in a natural way.

heading: 2 BACKGROUND: VARIATIONAL AUTOENCODERS
text: The variational autoencoder (VAE) (Kingma & Welling, 2013) is a generative model defined by a prior p(z) and a conditional distribution pθ(x|z). The VAE is trained to optimize a tractable variational lower bound of log pθ(x):
LVAE(x;θ,φ) = Ez∼qφ(z|x)[log pθ(x|z)]−DKL(qφ(z |x)||p(z)), (1)
where qφ(z |x) is a variational distribution parameterized by an encoding network with parameters φ, and pθ(x|z) denotes the decoding network with parameters θ. This objective tries to minimize the reconstruction error to generate the data, and at the same time regularizes qφ(z |x) towards the prior p(z). In this paper, p(z) is chosen as N (0, I ). For text modelling, the input x is some observed text. Both the encoding and decoding network are usually recurrent neural networks.
Note that during learning, the decoding network pθ only learns to decode conditioned on z that are sampled from qφ(z |x). In other words, the decoding network only learns to process z sampled from the aggregated posterior distribution qφ(z) = Ex∼pd(x)qφ(z |x), where pd(x) is the data distribution. If qφ(z) has regions of low density, there is no guarantee that pθ would decode well in such regions. This is an important intuition that will become central to our analysis in Sec. 3.

heading: 3 LATENT VACANCY PREVENTS EFFECTIVE MANIPULATION
text: In this section, we take a deeper look into the aggregated posterior latent space of sequence VAE trained on text, and provide justification for the alternative solution we propose in Section 4.

heading: 3.1 OBSERVATIONS FROM UNSUPERVISED SENTIMENT MANIPULATION
text: As pointed out by Bowman et al. (2015), one of the motivations to apply VAEs on text is to allow generation of the sentences conditioned on extrinsic features by controlling the latent codes. Without annotated labels, no previous methods have successfully learned controllable latent factors as mentioned in Sec. 1. To understand what is missing, we conduct exploratory experiments to use VAE for unsupervised sentiment manipulation.
We use the Yelp restaurant reviews dataset and the same data split following Li et al. (2018). We train a β-VAE (Higgins et al., 2017)1 with a latent space of 80 dimensions, an LSTM encoder, and an LSTM decoder. Details about this experiment are described in Appendix A.1.
By inspecting the accuracy on the validation set, we find that there exists one dimension of latent code achieving higher than 90% sentiment classification accuracy by its value alone, while other latent codes get accuracy around 50%. Further details can be found in Appendix A.2. It means that this latent dimension is an effective sentiment indicator. Similar phenomena have been observed in
1We also try state-of-the-art techniques (He et al., 2019) on VAE w.r.t. optimizing ELBO but the KL term trained with those techniques are too small to capture the details of the source sentence.
large-scale language models (Radford et al., 2017). However, the direct influence on the generative process of the model observed in Radford et al. (2017) does not apply on the VAE. When we try to perform sentiment manipulation by modifying this latent dimension2, the decoding network fails to generate the desired outputs most of the time, as evidenced by the poor quantitative evaluation in Table. 1, and poor samples shown in Appendix A.3.

heading: 3.2 LATENT VACANCY IN TEXT MODELLING
text: One possible reason for the failure is that the decoding network is never trained on codes like the manipulated ones. This is the case if the aggregated posterior has holes or regions of low density, and the manipulated codes fall into such vacant regions. Supposing the aggregated posterior latent space possesses a shape as shown in Fig. 1, the direct manipulated latent codes will fall out of the aggregated posterior latent space for most input samples. Such latent codes are never seen by the model during training and possess a low density under the aggregated posterior distribution, leading to unpredictable behaviours during decoding.
To verify our hypothesis demonstrated in Fig. 1, we empirically estimate the density of sentimentmanipulated codes under the aggregated posterior distribution of our trained VAE. Here, we approximate the data distribution pd(x) with the empirical distribution over all the training samples. As a result, the estimated aggregated posterior distribution is a large mixture of Gaussian distribution. For all 1000 test samples, we move the dimension of code capturing sentiment from µ− 2σ to µ+ 2σ where µ and σ are the mean and the standard deviation estimated on all the training samples and
2Different strategies are attempted, see Appendix A.4 for details.
measure the averaged negative log-likelihood (NLL) under the aggregated posterior distribution. As depicted in Fig. 3 (a), the NLL plotted in blue dot curve rises sharply when moving away from µ even if there is only one dimension of code is changing, indicating the existence of the vacancy in the aggregated posterior latent space. In addition, we draw the histogram of all the test samples’ NLL considering their original latent codes and modified ones in Fig. 3 (b). The histogram shows that there is a large divergence in NLL between the original latent codes and the modified ones. Also, the modified latent codes have two separate modes, confirming the irregular shape of the aggregated posterior latent space. In order to resolve this issue, the approach proposed in this work is to constrain the posterior in a way that the manipulation only happens in a learned simplex, as depicted in Fig. 2. In this constrained subspace, the phenomenom of low density holes of aggregated posterior is significantly reduced, as Fig. 3 (a) and (c) empirically show that there is little change in NLL of original versus modified codes. The details of our method is presented in the next section.

heading: 4 METHOD

heading: 4.1 OVERVIEW
text: The experiments conducted in Sec. 3 validates the existence of vacancy in the aggregated posterior latent space. One potential way to resolve the problem is to better match the aggregated posterior with the prior (Makhzani et al., 2015; Tomczak & Welling, 2017; Zhao et al., 2018). However, in terms of unsupervised learning of controllable representation for text, these previous methods have not shown successes; Zhao et al. (2018) only attempted supervised text style transfer, and also reported negative results from the AAE (Makhzani et al., 2015). Another way to resolve the vacancy issue is to directly enforce that the aggregated posterior itself has no vacant region anywhere where we would like to perform latent code manipulation. We propose to map the posterior Gaussian mean to a constrained space, more specifically a learned probability simplex, where we can encourage the constrained latent space to be filled without vacancy, and perform manipulation to be within this simplex. As illustrated in Fig. 2, we add an additional mapping function as part of the encoding network which maps the mean of the Gaussian posterior to a constrained space. Two regularization terms are introduced later to ensure the learned simplex is not degenerate and that this subspace is well filled.
In addition, we separately model the relevant factors that we wish to control and the irrelevant factors by splitting z into two parts, z(1) and z(2), following prior work (Bao et al., 2019). The first part captures the relevant factors that are dominant in the data without an inductive bias from external signals, while the second part learns to encode the remaining local information that is useful for reconstructing the source sentences. As a result, qφ(z |x) is decomposed into qφ1(z(1)|x)qφ2(z(2)|x) where φ = φ1 ∪ φ2. With diagonal covariances the KL divergence term in Eq. 1 splits into two separate KL terms. In practice, we use a MLP encoding network to parametrize z(1) with some sentence representations as the input (e.g., averaging GloVe embeddings (Pennington et al., 2014) over the input tokens) and a LSTM encoding network to parametrize z(2). We only constrain the posterior of z(1) and z(2) is optimized the same way as the traditional VAE.

heading: 4.2 CONSTRAINING THE POSTERIOR
text: We now describe how to map the mean µ of the Gaussian posterior for z(1) ∈ RN to a constrained latent space. We would like to constrain the mean µ to have a structure as follows:
µ = K∑ i=1 piei, K∑ i=1 pi = 1, 〈ei, ej〉 = 0, i 6= j, K ≤ N (2)
where ei are vectors representing the relevant factors, pi is the proportion of ith relevant factor encoded in z(1) andK is a hyperparameter indicating the number of relevant factors to discover. In other words, the mean of the Gaussian posterior of z(1) is constrained to be inside a K-dimension probability simplex in RN whose vertices are represented by the orthogonal basis vectors ei, i = 1, . . . ,K. Given the outputs of the MLP encoder h and logσ2, we learn an additional mapping function π which maps h to the constrained posterior space, which can be treated as part of the encoding network:
µ = π(h) = E · softmax(Wh + b), (3) whereE = [e1, . . . , eK ] is a learnable embedding matrix representing the bases,W is the learnable weight matrix, and b is the learnable bias vector. As a result, the constrained posterior is parametrized by µ and logσ2 as a Gaussian distribution N (µ, diag(σ2)).
With the mapping function alone, the proposed VAE suffers from posterior collapse (Bowman et al., 2015), a well-known problem where the model ignores the latent code z during the training. Further complicating matters is the fact that there is an abundance of signals for predicting the next token in the text, but the signals indicating high-level semantics are quite sparse. It is thus unlikely that the VAEs can capture useful relevant factors from raw text without collapse. For these reasons, we enforce orthogonality in the learnt basis vectors as defined in Eq. 2, which introduces a natural recipe to prevent posterior collapse for z(1). Note that the KL divergence between qφ1(z (1)|x) and p(z(1)) is
DKL(qφ1(z (1)|x)‖p(z(1))) = 1 2 µ>µ + 1 2
( σ>σ − logσ>σ − 1 ) . (4)
With orthogonality in the basis vectors, the first term in the above equation can be factorized into µ>µ = ( ∑ i piei) >( ∑ i piei) = ∑ i p2ie > i ei. (5)
To encourage orthogonality in the basis vectors, a regularization term is added to the objective function:
LREG(x;φ1) = ‖E>E − αI‖, (6)
where I is the identity matrix and α is a hyperparamter. When LREG = 0, e>i ei = α. In this case, µ>µ = α ∑ i p 2 i reaches its minimum α K when p is a uniform distribution. The proof can be found in Appendix C. In practice, LREG will quickly decrease to around 0, ensuring that the KL term will never fully collapse with the structural constraint. When it comes to controlled generation, one can choose a vertex or any desired point in the probability simplex, as illustrated in Fig. 2.
Note that the constrained posterior also means that the aggregated posterior can never match the isotropic Gaussian prior. In other word, we achieve good controlled text generation potentially at the cost of poor uncontrolled generation from the prior, but such is not the focus of this current work, and could potentially be resolved by selecting or learning a better prior as in Tomczak & Welling (2017).

heading: 4.3 FILLING THE CONSTRAINED SPACE
text: Constraining the posterior inside a certain space does not guarantee that this space will be filled after training. In order to prevent this, we want the probability distribution over the relevant factors p to cover as much of the constrained latent space as possible. We introduce a reconstruction error of the structured latent code in order to push p away from a uniform distribution. For each input sentence, we randomly sample m sentences from the training data as negative samples. By applying the same encoding process, we get the structured latent code µ(−)i for each negative sample. Our goal is to make the raw latent code h similar to the restructured latent code µ while different from latent codes µ (−) i of the negative samples, so that p is generally different for each input sample. The structured reconstruction loss is formulated as a margin loss as follows:
LS-REC(x;φ1) = Ez(1)∼qφ1 (z(1)|x)
[ 1
m m∑ i=1 max(0, 1− h ·µ + h ·µ(−)i )
] . (7)
Our final objective function is defined as follows:
L(x;θ,φ) = LVAE + LREG + LS-REC. (8)

heading: 5 RELATED WORK

heading: 5.1 UNSUPERVISED LEARNING OF DISENTANGLED REPRESENTATIONS
text: Learning disentangled representations is an important step towards better representation learning (Bengio et al., 2013) which can be useful for (semi-)supervised learning of downstream tasks, transfer and few-shot learning (Peters et al., 2017). VAEs have achieved promising results for unsupervised learning of disentangled representations. Several variations of VAEs have been proposed to achieve better disentanglement (Higgins et al., 2017; Kumar et al., 2017; Chen et al., 2016; Razavi et al., 2019). However, most recent progress in this direction has been restricted to the domain of images.

heading: 5.2 CONTROLLED TEXT GENERATION
text: In order to perform controllable text generation, previous methods either assume annotated attributes or multiple text datasets with different known styles (Hu et al., 2017; Shen et al., 2017; Zhao et al., 2018; Fu et al., 2018; Li et al., 2018; Sudhakar et al., 2019; Logeswaran et al., 2018; Lample et al., 2018). The requirement of labelled data largely restricts the capabilities and the applications of these models. Instead, all our proposed framework needs is raw text without any annotated attribute. The dominant underlying relevant factors in the given corpus will be discovered and disentangled by our unsupervised method, which can in turn be used for controlled generation.

heading: 6 EXPERIMENTS
text: To demonstrate the effectiveness of our approach, we compare it to unsupervised baselines with traditional VAEs, considering the density under the aggregated posterior distribution and the performance on sentiment manipulation. Following evaluation protocols in text style transfer, we also compare our method to strong supervised approaches. Furthermore, we showcase the ability of finer-grained style discovery and transition possessed by our system, which has not been attempted in the literature.
In this section, our proposed framework is referred as CP-VAE (Constrained Posterior VAE). Detailed configurations including the hyperparameters, model architecture, training regimes, and decoding strategy are found in Appendix B.

heading: 6.1 COMPARISONS WITH UNSUPERVISED BASELINES
text: Experimental setup: We use the same experimental setting and dataset as mentioned in Sec. 3. The 80D latent code is split into 16 and 64 dimensions for z(1) and z(2) respectively. The sentence representations used for z(1) is the averaged GloVe embeddings over the input tokens andK is chosen as 3. To decide which basis vector corresponds to which sentiment, we sample 10 positive and 10 negative sentences respectively in the development set, pass them to the encoder, and choose the basis vector with the highest average pi in p = softmax(Wh + b), yielding vp as the positive basis and vn as the negative basis. If vp and vn are chosen to be the same vector, we choose the index with the second highest pi for vp. To perform sentiment manipulation, we fix z(1) to be the chosen basis vector; that is, vp or vn.
Comparisons on density under the aggregated posterior distribution: First, we do linear interpolation between the two discovered basis vectors vp and vn and estimate the averaged NLL under the aggregated posterior distribution the same way as introduced in Sec. 3. The green solid curve in Fig. 3 (a) shows that the NLL of CP-VAE is relatively stable for the whole range of the interpolation. In Fig. 3 (c), the original latent codes and the modified ones largely overlap with each other. Both observations validate the effectiveness of CP-VAE in resolving the latent vacancy problem, leading to significant improvements on unsupervised sentiment manipulation, as seen later.
Comparsions with metrics on text style transfer: For quantitative evaluation, we adopt automatic evaluation metrics used in text style transfer (Sudhakar et al., 2019) including classification accuracy
(AC), BLEU score (BL), GLEU score (GL) and language model perplexity (PL), whose definitions are elaborated in the next section. We also report DKL(qφ1(z
(1)|x)‖p(z)) (KL) for z(1) of CP-VAE. As shown in Tab. 1, CP-VAE performs significantly better than β-VAE in terms of accuracy, BLEU and GLEU. The lower perplexity of β-VAE is due to mode collapse, which produces very short pivot sentences such as “great !”. The results match our observations from the experiments on density under the aggregated posterior distribution, confirming that latent vacancy prevents effective manipulation of the latent codes. We also conduct an ablation study by removing LREG and LS-REC from the objective. The results demonstrate that both terms are crucial to the success of CP-VAE. Without LREG, CP-VAE experiences posterior collapse for z(1). As a result, vp and vn collide with each other, leading to failure in disentangled representation learning. Since we choose K as 3, it is convenient to visualize the samples during training with p in the learnt probability simplex, as shown in Fig. 4. We can see that the whole simplex is mostly covered with samples with the help of LS-REC. Without LS-REC, the decoding network fails to recognize the basis vectors due to the poor coverage of the probability simplex, causing the model to lose most of its transferring ability.

heading: 6.2 COMPARISONS TO SUPERVISED APPROACHES ON TEXT STYLE TRANSFER
text: Experimental setup: We choose two datasets, Yelp and Amazon, used in works (Li et al., 2018; Sudhakar et al., 2019) on text style transfer which provide human gold-standard references for the test set. The same train-dev-test splits are used in our experiments. Two different sentence representations are used in this experiment, averaged GloVe and BERT (Devlin et al., 2018), denoted as CP-G(loVe) and CP-B(ert) respectively. The remaining settings are as described in the above section.
Compared supervised approaches: On the two datasets, we compare to three adversarially trained models: StyleEmbedding (SE) (Fu et al., 2018), MultiDecoder (MD) (Fu et al., 2018), CrossAligned (CA) (Shen et al., 2017) and two state-of-the-art models based on a “delete, transform, and generate” framework: DeleteAndRetrieve (D&R) (Li et al., 2018) and Blind-GenerativeStyleTransformer (B-GST) (Sudhakar et al., 2019).
Evaluation protocols: Four different automatic evaluation metrics are used to measure the different perspectives of the transferring quality, following Sudhakar et al. (2019). To measure transferring ability, we use pre-trained CNN based classifiers achieving 98% and 84% accuracies on the test sets of Yelp and Amazon respectively. To measure content preservation, we use the BLEU (Papineni et al., 2002) score between the transferred sentences and the source sentences. To measure fluency, we finetune OpenAI GPT-2 (Radford et al., 2019) with 345 million parameters on the same trainingdev-test split to obtain the perplexity of generated sentences. The fine-tuned language models achieve perplexities of 26.6 and 34.5 on the test sets of Yelp and Amazon respectively. In addition, Sudhakar et al. (2019) argued that the Generalized Language Evaluation Understanding Metric (GLEU) has a better correlation with the human judgement. Here, we use the implementation of GLEU3 provided by Napoles et al. (2015) to calculate the GLEU score. Result Analysis: As observed by Li et al. (2018) and Sudhakar et al. (2019), accuracy, BLEU score and perplexity do not correlate well with human evaluations. Therefore, it is important to not consider them in isolation. Tab. 2 shows that our proposed approaches get similar scores on these metrics with human reference sentences on the second row, indicating that the generated sentences of our proposed approaches is reasonable considering the combination of these metrics. As seen by Sudhakar et al. (2019) and verified in Sec. 6.1, GLEU strike a balance between target style match and content retention and correlate well with the human evaluations. From Tab. 2, CP-VAE consistently outperforms the three adversarially trained models on GLEU by a noticeable margin and achieve competitive results as compared to the recent state-of-the-art models. By checking the samples generated from the models as shown in Tab. 3, B-GST, the current state-of-the-art, is more
3https://github.com/cnap/gec-ranking
consistent to the source sentence, which can be expected, since it only makes necessary edits to flip the sentiment. CP-VAE tends to generate more diverse contents which may not be relevant sometimes, but the overall quality is reasonable considering it is trained without the label information. More samples can be found in Appendix E.

heading: 6.3 FINER-GRAINED STYLE DISCOVERY AND TRANSITION
text: To further explore the potential of CP-VAE, we conduct the following exploratory experiments. We use the AG news dataset constructed by (Zhang et al., 2015), which contains four topic categories which are World, Sports, Business and Sci/Tech, with the title and description fields. Here, we drop the title and just use the description field to train CP-VAE and set K = 10. All four topics are automatically discovered by CP-VAE and identified as described in Sec. 6.1. We also compare the results of our identified topics to standard baselines for unsupervised topic modelling, the details
can be found in Appendix D. We choose a basis vector discovered by our model and generate a few tokens. Then, we switch the basis vector and continue the generation until the end-of-seq token is generated. Generated samples are shown in Table 4. We see that our model learns to transition from one topic to another in a natural and fluent way within the same sentence. Several observations can be made based on these samples: (1) it is good at detecting name entities and replacing them with the name entities related to the chosen topic; (2) there is no hard restriction on when to switch the topic; the model will determine an appropriate way to do the transition by itself. Such observations confirm that CP-VAE possesses a filled constrained latent space which make the latent code robust to manipulation across different time steps, which can be effectively reflected in the generation process. Due to space limitations, we put more samples in Appendix F.

heading: 7 CONCLUSION
text: In this work, we investigate latent vacancy as an important problem in unsupervised learning of controllable representations when modelling text with VAEs. To mitigate this, we propose to constrain the posterior within a learned probability simplex, achieving the first success towards controlled text generation without supervision.

heading: A DETAILS ABOUT EXPLORATORY EXPERIMENTS
text: A.1 MODEL DETAILS
For the β-VAE we used for the exploratory experiments, we use a LSTM encoding network and a LSTM decoding network. For the encoding network, the input size is 256, and the hidden size is 1,024. For the decoding network, the input size is 256, the hidden size is 1,024, and dropouts with probability 0.5 are applied on after the embedding layer and the LSTM layer in the decoding network. β is chosen as 0.35, the dimension for the latent code is 80, and the batch size is 32. We use SGD with learning rate 1.0 to update the parameters for both the encoding and the decoding network. We train the model until the reconstruction loss stops decreasing.
A.2 IDENTIFYING THE LATENT FACTOR INDICATING THE SENTIMENT
First, we normalize the value of each latent code by subtracting the mean estimated over all the training samples. Then we use the polarity of each latent code to classify the sentiment in the validation set. The one with the highest accuracy is identified as the latent factor indicating the sentiment.
A.3 SAMPLES GENERATED FROM β-VAE
A.4 MANIPULATION STRATEGIES
Following manipulation strategies have been attempted: (1) fixing the relevant factor to µ+ 2σ and µ−2σ; (2) fixing the relevant factor to µ−σ and µ−σ; (3) fixing the relevant factor to the maximum value and the minimum value of the relevant factor appearing in the training samples; (4) calculating a latent vector based on 10 manually constructed parallel sentences with opposite sentiment while keeping other factors unchanged. However, none of these four strategies is effective considering the generation results. We report the result with the first strategy in the paper, since it performs the best considering the accuracy and the BLEU score.

heading: B DETAILS ABOUT EXPERIMENTS ON TEXT STYLE TRANSFER
text: B.1 TRAINING REGIMES
Across all the datasets, we use Adam with learning rate 0.001 to update the parameters for the encoding network, while SGD with learning rate 1.0 to update the parameters for the decoding network. The batch size is chosen to be 32. Dropouts with drop probability 0.5 are applied on applied on after the embedding layer and the LSTM layer in the decoding network. We train the model until the reconstruction loss stops decreasing.
B.2 MITIGATING POSTERIOR COLLAPSE
For the structured part z(1), we use β-VAE setting β as 0.2 across all the datasets. For the unstructured part z(2), different strategies are employed for each dataset:
• Yelp: β-VAE setting β as 0.35. • Amazon: β-VAE setting β as 0.35. • AG-News: KL annealing, from 0.1 to 1.0 in 10 epochs.
B.3 HYPERPARAMETER SETTINGS
The hyperparameters are chosen by checkingLVAE, KL, and the generated outputs on the development set for Yelp and AG-News. Amazon follows the same setting as Yelp without extra tuning.
B.4 DECODING STRATEGY
For decoding, we use beam search with a beam size of 5.

heading: C PROOF OF MINIMALIZATION OF EQ. 5
text: The problem can be formulated as an optimization problem as follows:
maximize K∑ i=1 p2i , subject to K∑ i=1 pi = 1.
By introducing a Lagrange multiplier λ, the Lagrange function is defined as
L(p1, p2, . . . , pK , λ) = K∑ i=1 p2i − λ( K∑ i=1 pi − 1).
In order to find the optimal point, we require that
∂
∂pi ( K∑ i=1 p2i − λ( K∑ i=1 pi − 1) ) = 2pi − λ = 0, i = 1, 2, . . . ,K,
which shows that all pi are equal. By using the constraint ∑ i pi = 1, we find pi = 1 K , i =
1, 2, . . . ,K. By plugging into the results, µ>µ = α ∑ i p 2 i reaches its minimum α K .

heading: D COMPARISONS WITH BASELINES ON TOPIC MODELLING
text: Experimental setup: We use the AG news dataset for this task constructed by (Zhang et al., 2015). It contains four topic categories which are World, Sports, Business and Sci/Tech, with the title and description fields. For each category, there are 30, 000 training samples and 1, 900 test samples. In this paper, we drop the title and just use the description field. We compare our approach to two standard baselines for unsupervised topic modelling: (1) LDA (Blei et al., 2003), a standard implementation of LDA is used for this baseline4; (2) k-means. To show the power of our approach
4https://radimrehurek.com/gensim/
beyond the pre-trained sentence representations, we perform k-means clustering directly on the sentence representations. Following (Manning et al., 2010), we assign each inferred topic to one of the gold-standard topics with the optimal mapping and report the precision (a.k.a. purity), recall (a.k.a. collocation) and F1 score. The number of topics is chosen to be 10. The results reported for the baselines and our model are the average over 10 runs.
Quantitative results: The results are shown in Table 7. We can see that our approach achieves comparable results to LDA while significantly outperforming k-means in all four categories, indicating that our approach can go beyond just clustering on pre-trained sentence representations.

heading: E TEXT TRANSFER EXAMPLES
text: E.1 SENTIMENT MANIPULATION ON YELP DATASET
E.2 SENTIMENT MANIPULATION ON AMAZON DATASET

heading: F TEXT TRANSITION EXAMPLES ON AG NEWS

Review summaries:
The paper "On Variational Learning of Controllable Representations for Text without Supervision" tackles the problem of latent vacancy of text representation via variational text auto-encoders. Based on the observation that a single factor of the sentence encoding gathers most of relevant information for classifying the sentence as positive or negative (sentiment classification), authors study the impact of manipulating this factor in term of the corresponding decoded sentences. They reasonnably claim that if such a manipulation fails at decoding accurate sentences, it is because we fall in representation areas that the decoder never seen during training. Thus they propose a way to constrain the posterior mean to a

This paper presents a method for controlled text generation by using a new loss function (standard VAE loss with auxiliary losses added on). The method is tested on style transfer datasets: Yelp and Amazon. The central hypothesis is that when manipulating latent codes of a VAE, you can end up in low-density regions of the aggregated posterior. Such latent codes are rarely seen by the decoder so that quality of generation is low. To address this problem, they constrain the posterior mean to a learnt probability simplex and try to ensure that the simplex is densely filled. They do this by adding 2 regularizing losses to the VAE loss.