On Variational Learning of Controllable Representations for Text without Supervision
Authors: Peng Xu
Yanshuai Cao
Jackie Chi Kit Cheung
Venue:  (2020)
Keywords: sequence variational autoencoders
unsupervised learning
controllable text generation
text style transfer

Abstract: The variational autoencoder (VAE) has found success in modelling the manifold of natural images on certain datasets, allowing meaningful images to be generated while interpolating or extrapolating in the latent code space, but it is unclear whether similar capabilities are feasible for text considering its discrete nature. In this work, we investigate the reason why unsupervised learning of controllable representations fails for text. We find that traditional sequence VAEs can learn disentangled representations through their latent codes to some extent, but they often fail to properly decode when the latent factor is being manipulated, because the manipulated codes often land in holes or vacant regions in the aggregated posterior latent space, which the decoding network is not trained to process. Both as a validation of the explanation and as a fix to the problem, we propose to constrain the posterior mean to a learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method significantly outperforms unsupervised baselines and is competitive with strong supervised approaches on text style transfer. Furthermore, when switching the latent factor (e.g., topic) during a long sentence generation, our proposed framework can often complete the sentence in a seemingly natural way -- a capability that has never been attempted by previous methods. 

Under review as a conference paper at ICLR 2020
ON VARIATIONAL LEARNING OF CONTROLLABLE REPRESENTATIONS FOR TEXT WITHOUT SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
The variational autoencoder (VAE) has found success in modelling the manifold of natural images on certain datasets, allowing meaningful images to be gener- ated while interpolating or extrapolating in the latent code space, but it is unclear whether similar capabilities are feasible for text considering its discrete nature. this work, we investigate the reason why unsupervised learning of controllable representations fails for text. We find that traditional sequence VAEs can learn disentangled representations through their latent codes some extent, but they often fail to properly decode when the latent factor is being manipulated, because the manipulated codes often land in holes vacant regions in the aggregated pos- terior latent space, which the decoding network is not trained to process. Both as a validation of the explanation and fix the problem, we propose constrain the posterior mean to learned probability simplex, and performs manipulation within this simplex. Our proposed method mitigates the latent vacancy problem and achieves the first success in unsupervised learning of controllable representations for text. Empirically, our method significantly outperforms unsupervised base- lines and is competitive with strong supervised approaches text style transfer. Furthermore, when switching the latent factor (e.g., topic) during long sentence generation, our proposed framework can often complete the sentence in seemingly natural way a capability that has never been attempted by previous methods.
INTRODUCTION
High-dimensional data, such as images and text, are often causally generated through the interaction of many complex factors, such as lighting and pose in images or style and content in texts. Recently,
VAEs and other unsupervised generative models have found successes in modelling the manifold of natural images (Higgins et al., 2017 Kumar et al. 2017, Chen et al. 2016). These models often discover controllable latent factors that allow manipulation of the images through conditional generation from interpolated or extrapolated latent codes, often with impressive quality. On the other hand, while various attributes of text such as sentiment and topic can be discovered in an unsupervised way, manipulating the text by changing these learned factors have not been possible with unsupervised generative models to the best of our knowledge. Cifka et al. (2018): Zhao et al. (0018) observed that text manipulation is generally more challenging compared to images, and the successes of these models cannot be directly transferred to texts.
Controllable text generation aims at generating realistic text with control over various attributes including sentiment, topic and other high-level properties. Besides being scientific curiosity, the possibility of unsupervised controllable text generation could help in a wide range of application. e.g., dialogues systems (Wen et al. 2016). Existing promising progress (Shen et al. 2017 Fu et al.) 2018, et al. 2018 (Sudhakar et al. 2019) all relies on supervised learning from annotated attributes to generate the text in a controllable fashion. The high cost of labelling large training corpora with attributes of interest limits the usage these models, as pre-existing annotations often do not align with some downstream goal. Even if cheap labels available, for example, review scores as proxy for sentiment, the control is limited to the variation defined by the attributes.
In this work, we examine the obstacles that prevent sequence VAEs from performing well in unsuper- vised controllable text generation. We empirically discover that manipulating the latent factors for typical semantic variations often leads to latent codes that reside in some low-density region of the

---

Under review as a conference paper at ICLR 2020
aggregated posterior distribution. In other words, there are vacant regions in the latent code space (Makhzami et al., 2015 Rezende & Viola 2018) not being considered by the decoding network, at least not convergence. As a result, the decoding network is unable to process such manipulated latent codes, yielding unpredictable generation results of low quality.
In order mitigate the latent vacancy problem, we propose constrain the posterior mean learned probability simplex and only perform manipulation within the probability simplex. Two regularizers are added to the original objective of VAE. The first enforces an orthogonal structure of the learned probability simplex; the other encourages this simplex to be filled without holes, Besides confirming that latent vacancy is indeed a cause of failure previous sequence VAEs', it also the first successful attempt towards unsupervised learning of controllable representations for text to the best of our knowledge, Experimental results on text style transfer show that our approach significantly outperforms unsupervised baselines, and is competitive with strong supervised approaches across a wide range of evaluation metrics. Our proposed framework also enables finer-grained and more flexible control over text generation. In particular, we can switch the topic in the middle of sentence generation, and the model will often still find a way to complete the sentence a natural way.
2 BACKGROUND: VARIATIONAL AUTOENCODERS
The variational autoencoder (VAE) (Kingma & Welling 2013 a generative model defined by a prior p(z) and conditional distribution pe (z) The VAE is trained to optimize a tractable variational lower bound of log po (z):
vAAE x 0) Eg x) [log po (z) 2)] DKL (g6 (z(=)))(((
(1)
where qo (z(x) is variational distribution parameterized by an encoding network with parameters ø. and po (z|z) denotes the decoding network with parameters 0. This objective tries to minimize the reconstruction error to generate the data, and at the same time regularizes (2(2)²) towards the prior p(z). In this paper, p(z) is chosen as (0, I). For text modelling, the input x some observed text. Both the encoding and decoding network are usually recurrent neural networks.
Note that during learning. the decoding network Pe only learns to decode conditioned on z that are sampled from % (2)x). In other words, the decoding network only learns to process z sampled from the aggregated posterior distribution qs (z) = Ex>pa(x)( ls (z x), where pa(x) is the data distribution.
(z) has regions of low density, there is no guarantee that po would decode well in such regions.
This is an important intuition that will become central to our analysis in Sec. 3
LATENT VACANCY PREVENTS EFFECTIVE MANIPULATION
In this section, we take deeper look into the aggregated posterior latent space of sequence VAE trained on text, and provide justification for the alternative solution we propose in Section 4
3.1 OBSERVATIONS FROM UNSUPERVISED SENTIMENT MANIPULATION
As pointed out by Bowman et al. (2015), one of the motivations to apply VAEs on text to allow generation of the sentences conditioned extrinsic features by controlling the latent codes. Without annotated labels, no previous methods have successfully learned controllable latent factors mentioned in Sec. T To understand what is missing, we conduct exploratory experiments to VAE for unsupervised sentiment manipulation.
We use the Yelp restaurant reviews dataset and the same data split following Li et (2018). We train a 8-V (Higgins et al. 2017 with a latent space of 80 dimensions, an LSTM encoder, and an L.STM decoder. Details about this experiment are described in Appendix AA
By inspecting the accuracy on the validation set, we find that there exists one dimension of latent code achieving higher than 90% sentiment classification accuracy by its value alone, while other latent codes get accuracy around 50%. Further details can be found in Appendix A.2 It means that this latent dimension is an effective sentiment indicator, Similar phenomena have been observed in
We also try state-of-the-ar techniques (He al. 2019 VAE w.r.t. optimizing EI but the KI. term trained with those techniques are too small to capture the details of the source sentence.
2

---

Under review as a conference paper at ICLR 2020
large-scale language models (Radford al. 201 7). However, the direct influence on the generative process of the model observed in Radford et al. (2017) does not apply on the VAE. When we try to perform sentiment manipulation by moditying this latent dimension the decoding network fails to generate the desired outputs most of the time, as evidenced by the poor quantitative evaluation in Table. T and poor samples shown in Appendix A.3
Irrelvann
Irrelvannt Factor
constrained manipulation
Aggregate Posterior
manipulation
mapping function
Un
Constrained Posterior
Up
Prior
Relevant Factor
Relevant Factor
Prior
11 + 20
J 20
μ
Figure 1: Illustration of why latent vacancy
Figure 2: CP-VAE, mapping the posterior to a proba- bility simplex with orthogonal basis vectors.
prevents effective manipulation in VAEs.
B-VAE
20
1+
u+
(b) B-VAE
80
original modified
JAVAE
Sample Number
1200
60
20
400
600
1200
1400
1600
800
original modified
Sample Number
10
0.5Mg
(a)
(c) CP-VAE
NLL
Figure 3: (a) Comparisons between 3-VAE and CP-VAE considering density under the aggregated posterior distribution, the blue and green solid arrows match with the manipulation illustrated in Fig and (b) Histogram of original and modified latent codes' NLL in 3-VAE; (c) Histogram of original and modified latent codes' NLL CP-VAE.
3.2 LATENT VACANCY IN TEXT MODELLING
One possible reason for the failure is that the decoding network is never trained codes like the manipulated ones. This is the case if the aggregated posterior has holes or regions of low density, and the manipulated codes fall into such vacant regions. Supposing the aggregated posterior latent space possesses a shape as shown in Fig. π the direct manipulated latent codes will fall out of the aggregated posterior latent space for most input samples. Such latent codes are never seen by the model during training and possess a low density under the aggregated posterior distribution, leading unpredictable behaviours during decoding.
To verify our hypothesis demonstrated in Fig. | we empirically estimate the density of sentiment- manipulated codes under the aggregated posterior distribution of our trained VAE. Here, we approxi- mate the data distribution (z(z) with the empirical distribution over all the training samples. As result, the estimated aggregated posterior distribution is large mixture of Gaussian distribution. For all 1000 test samples, we move the dimension of code capturing sentiment from 2x to y + 2o where μ and a are the mean and the standard deviation estimated on the training samples and
Different strategies attempted, Appendix A.4 for details.

---

Under review as a conference paper at ICLR 2020
measure the averaged negative log- log-likelihood (NLL) under the aggregated posterior distribution. As depicted in Fig. (a), the NLL plotted in blue dot curve rises sharply when moving away from ja even if there is only one dimension of code is changing, indicating the existence of the vacancy in the aggregated posterior latent space. In addition, we draw the histogram of all the test samples' NILL considering their original latent codes and modified ones in Fig. 3 (b). The histogram shows that there is a large divergence in NLL between the original latent codes and the modified ones. Also, the modified latent codes have two separate modes, confirming the irregular shape of the aggregated posterior latent space. In order to resolve this issue, the approach proposed in this work to constrain the posterior in a way that the manipulation only happens in learned simplex, as depicted in Fig. 2 In this constrained subspace, the phenomenom of low density holes of aggregated posterior is significantly reduced, as Fig. (a) and (c) empirically show that there is little change in NLL of original versus modified codes. The details of our method is presented in the next section.
4
METHOD
4.1
OVERVIEW
The experiments conducted in Sec. validates the existence of vacancy in the aggregated posterior latent space. One potential way to resolve the problem is to better match the aggregated posterior with the prior (Makhzani et al. 2015 Tomezak & Welling 2017: Zhao et al. 2018). However, in terms of unsupervised learning of controllable representation for text, these previous methods have not shown successes: Zhao et al. (2018) only attempted supervised text style transfer, and also reported negative results from the AAE (Makhzani et al. (2015). Another way to resolve the vacancy issue is to directly enforce that the aggregated posterior itself has no vacant region anywhere where we would like to perform latent code manipulation. We propose to map the posterior Gaussian mean to a constrained space, more specifically& a learned probability simplex, where we can encourage the constrained latent space to be filled without vacancy, and perform manipulation to be within this simplex. As illustrated in Fig. 2 we add an additional mapping function as part of the encoding network which maps the mean of the Gaussian posterior to a constrained space. Two regularization terms are introduced later to ensure the learned simplex is not degenerate and that this subspace is well filled.
In addition, we separately model the relevant factors that we wish to control and the irrelevant factors
by splitting z into two parts, (1)) and 2(2), following prior work (Bao et al. 2019). The first part captures the relevant factors that are dominant in the data without an inductive bias from external
signals, while the second part learns to encode the remaining local information that is useful for
reconstructing the source sentences, As a result, 96 (z|z²) is decomposed into g (z [x) ( (x) (x)
where o- = Φ1 Φ2. With diagonal covariances the KL divergence term in Eq. [ (pllts into two separate KI. terms. In practice, we use MI. encoding network to parametrize 21)) with some sentence representations as the input (e.g., averaging GloVe embeddings (Pennington et al.) 2014)
over the input tokens) and a LSTM encoding network to parametrize 22)) We only constrain the posterior of (2)) and x(2) optimized the same way as the traditional VAE.
4.2 CONSTRAINING THE POSTERIOR
We now describe how to map the mean μ of the Gaussian posterior for 2(1) E R to a constrained latent space. We would like to constrain the mean μ to have a structure as follows:
K
K Σp=1, (e,,e,)=
1 Σ P.C.,
KSN
where e, are vectors representing the relevant factors. Pt is the proportion of ith relevant factor encoded in (1) and K is hyperparameter indicating the number of relevant factors to discover. In other words, the mean of the Gaussian posterior of 2(1) is constrained to be inside a K K-dimension probability simplex in N whose vertices are represented by the orthogonal basis vectors e,, = 1,..., K. Given the outputs of the MLP encoder h and o ; we learn additional mapping function π which maps h to the constrained posterior space, which can be treated as part of the encoding network:
p π (h) E softmax (Wh + b),
(3)
where E = fei.............. ... ex] is learnable embedding matrix representing the bases, W is the learnable weight matrix, and is the learnable bias vector. As a result, the constrained posterior is parametrized by a and log 2 as Gaussian distribution A N(4,, diag(a2))
4

---

Under review as a conference paper at ICLR 2020
With the mapping function alone, the proposed VAE suffers from posterior collapse (Bowman et al. 2013), a well-known problem where the model ignores the latent code 2 during the training. Further complicating matters is the fact that there is abundance of signals for predicting the next token in the text, but the signals indicating high-level semantics are quite sparse. It thus unlikely that the VAEs can capture useful relevant factors from text without collapse. For these reasons, enforce orthogonality in the learnt basis vectors as defined in Eq. 2 which introduces natural recipe to prevent posterior collapse for z!! 2(1, Note that the KL divergence between qΦ, (z(1) (x) and p(z(1) is
1
DKL (9p, (z(4) (2) ())((((() 24 # + lo lo
log
1).
(4)
With orthogonality in the basis vectors, the first term in the above equation can be factorized into
F Epee)) ( p.es) = Σ pie
(5)
To encourage orthogonality the basis vectors, a regularization term added to the objective function:
CZEG(Z; = |E E --
(6)
where I is the identity matrix and a a hyperparamter. When LRGG = 0, e, e; = a. In this case, y μ = a Σ pf reaches its minimum K when p is a uniform distribution. The proof can be found in Appendix C In practice, LREG will quickly decrease to around 0 ensuring that the KI term will never fully collapse with the structural constraint. When it comes to controlled generation, one can choose vertex any desired point in the probability simplex, illustrated in Fig. [
&
Note that the constrained posterior also means that the aggregated posterior can never match the isotropic Gaussian prior. In other word, achieve good controlled text generation potentially the cost of poor uncontrolled generation from the prior, but such not the focus of this current work, and could potentially be resolved by selecting or learning better prior in Tomezak & Welling (0117).
4.3 FILLING THE CONSTRAINED SPACE
Constraining the posterior inside certain space does not guarantee that this space will be filled after training. In order to prevent this, we want the probability distribution over the relevant factors p to cover much of the constrained latent space possible. We introduce reconstruction error of the structured latent code order to push p away from a uniform distribution. For each input sentence, randomly sample sentences from the training data negative samples. By applying the same encoding process, we get the structured latent code μ for each negative sample. Our goal is to make the raw latent code h similar to the restructured latent code μ while different from latent codes ([() of the negative samples, so that p generally different for each input sample. The structured reconstruction loss is formulated as margin loss as follows:
1 m
p (6()))
Ls REC Φ1) Eg00 (=(1)
m Σ
max(0, (0. h-
Our final objective function is defined as follows:
L(:; 0, LVAR + LREG + LS.REC.
(8)
RELATED WORK
5.1 UNSUPERVISED LEARNING OF DISENTANGLED REPRESENTATIONS
Learning disentangled representations an important step towards better representation learning (Bengio et al.) 2013) which can be useful for and few-shot learning (Peters et al. 2017). VAES have achieved promising results for unsupervised learning of disentangled representations, Several variations of VAEs have been proposed to achieve
learning of downstream tasks, transfer
better disentanglement (Higgins et al. 2017; Kumar et al., 2017 Chen et al. 2016 Razavi et al. 2019). However, most recent progress this direction has been restricted to the domain of images.
5

---

Under review as a conference paper at ICLR 2020
5.2
CONTROLLED TEXT GENERATION
In order to perform controllable text generation, previous methods either assume annotated attributes or multiple text datasets with different known styles (Hu et al. 2017) et al., 2017] Zhao et al.
2018 Fu et al. 2018 Li et al. 2018 Sudhakar et al.. 2019 Logeswaran et al.) 2018| Lample et al. 2018 The requirement of labelled data largely restricts the capabilities and the applications of these models. Instead, all our proposed framework needs is raw text without any annotated attribute. The dominant underlying relevant factors in the given corpus will be discovered and disentangled by our unsupervised method, which can in turn be used for controlled generation.
EXPERIMENTS
To demonstrate the effectiveness of our approach, we compare it to unsupervised baselines with tradi- tional VAEs, considering the density under the aggregated posterior distribution and the performance on sentiment manipulation. Following evaluation protocols in text style transfer, we also compare our method to strong supervised approaches. Furthermore, we showcase the ability of finerggraing style discovery and transition possessed by our system, which has not been attempted the literature.
In this section, our proposed framework is referred as CP-VAE (Constrained Posterior VAE). Detailed configurations including the hyperparameters, model architecture, training regimes, and decoding strategy are found in Appendix B.
6.1 COMPARISONS WITH UNSUPERVISED BASELINES
Experimental setup: We use the same experimental setting and dataset mentioned in Sec.
The 80D latent code is split into 16 and 64 dimensions for z(1) and 22)) respectively. The sentence representations used for z (1) is the averaged Glo embeddings over the input tokens and K is chosen sample 10 positive and 10 negative sentences respectively in the development set, pass them to the encoder, and choose the basis vector with the highest average pt in p softmax (Wh b), yielding Up as the positive basis and (Wot
as 3. To decide which basis vector corresponds to which sentiment,
on as the negative basis. If Up and vn are chosen to be the same vector, we choose the index with the second highest Pt for Up. To perform sentiment manipulation, we fix (!)) to be the chosen basis vector; that is, Up or Un-
Comparisons density under the aggregated posterior distribution: First, do linear inter- polation between the two discovered basis vectors up and U, and estimate the averaged NLL under the aggregated posterior distribution the same way introduced in Sec. 3 The green solid curve in Fig. 3 (a) shows that the NLL of CP-VAE is relatively stable for the whole range of the interpolation. In Fig. B (c), the original latent codes and the modified ones largely overlap with each other. Both observations validate the effectiveness of CP- VAE in resolving the latent vacancy problem, leading to significant improvements on unsupervised sentiment manipulation, as seen later.
(A]
1.
0.8
0.8
0.6
022
8.2
0.0
2,0
0.5
10
10
Figure 4: Visualization of all training samples in the probability simplex: (A) With Lsaie (B) Without LsREc-
Comparsions with metrics on text style transfer: For quantitative evaluation, we adopt automatic evaluation metries used in text style transfer (Sudhakar et al. 2019) including classification accuracy

---

Under review as a conference paper at ICLR 2020
(AC), BLEU score (BL), GLEU score (GL) and language model perplexity (PL), whose definitions are elaborated in the next section. We also report (1 (z))(())() (KI.) for 211) of CP-VAE. As shown in Tab. [ CP-VAE performs significantly better than 3-VAR in terms of accuracy, BLEU and GLEU. The lower perplexity of B-VAE is due to mode collapse, which produces very short pivot sentences such as "great !". The results match our observations from the experiments on density under the aggregated posterior distribution, confirming that latent vacancy prevents effective manipulation of the latent codes. We also conduct an ablation study by removing CREO and LS-REC from the objective. The results demonstrate that both terms are crucial the success of CP-VAE. Without CREG. CP-VAE experiences posterior collapse for 2 As a result, Up and Un collide with each other, leading to failure in disentangled representation learning. Since we choose K as 3, is convenient to visualize the samples during training with p in the learnt probability simplex, as shown in Fig. 4 We can see that the whole simplex is mostly covered with samples with the help of REC Without LS-REC. the decoding network fails to recognize the basis vectors due the poor coverage of the probability simplex, causing the model to lose most of its transferring ability.
Table 1: Comparisons with unsupervised baselines on Yelp dataset.
Model
AC 1
BI.
GI.
P
KI. for 2
VA
50.44 + 2.04
7.68
+ 0.33
2.97 + 0.14
24.63
1.85
NA
CP-G
60.22 + 4.57
33.69 1 1.47
6,78 +: 0.44
13.12:2.41
18.35 + 0.15
LARG
10.82 + 0.91
33.27
+: 2.84
5.04
+ 0.27
37.25
- 2.40
0.04 + 0.01
12.28 + 3.69
49.34 2.65
6,22 + 0.40
55.26 + 2.46
17.57 + 0.12
6.2 COMPARISONS TO SUPERVISED APPROACHES ON TEXT STYLE TRANSFER
Experimental setup: We choose two datasets, Yelp and Amazon, used in works (Li et al. 2018
Sudhakar et al. 2019) on text style transfer which provide human gold-standan references for the test set. The same train-dev-test splits are used in our experiments. Two different sentence representations are used in this experiment, averaged Glo and BERT (Devlin et al. 2018), denoted as CP-GG(oVe) and CP-B(ert) respectively. The remaining settings are as described in the above section.
Compared supervised approaches: On the two datasets, we compare to three adversarially trained models: StyleEmbedding (SE) (Fu et al. 2018), MultiDecoder (MD) (Fu et al., 2018). CrossAligned (CA) (Shen et al.) 2017) and two state-of-the-art models based on "delete, transform, and generate" framework: Delete
(D&R) (Li et al. 2018) and BiinddGenerrtiivSStyeeTTee Trrnnforrreee
(B-GST) (Sudhakar et al. 2019).
Four different automatic evaluation metrics are used to measure the different perspectives of the transferring quality, following Sudhakar et al. 2019). To measure transferring ability, we use pre-trained CNN based classifiers achieving 98% and 84% accuracies on the test sets of Yelp and Amazon respectively. To measure content preservation, we use the BLEU (Papineni) jet al., 22022) score between the transferred sentences and the source sentences. To measure fluency, we finetune OpenAl GPT-2 (Radford et al. 2019) with 345 million parameters on the same training- dev-test split to obtain the perplexity of generated sentences. The fine-tuned language models achieve perplexities of 26.6 and 34.5 the test sets of Yclp and Amazon respectively. In addition, Sudhakar et al. (2019) argued that the Generalized Language Evaluation Understanding Metric (GLEU) has a better correlation with the human judgement. Here, use the implementation of GLEU provided by Napoles et al. (2015) to calculate the GLEU score.
Evaluation protocols:
Result Analysis: As observed by Li et al. (2018) and (Suhaaaar al. (019). accuracy, BLEU score and perplexity do not correlate well with human evaluations. Therefore, it is important to not consider them in isolation. Tab. 2 shows that our proposed approaches get similar scores on these metrics with human reference sentences on the second row, indicating that the generated sentences of our proposed approaches is reasonable considering the combination of these metries. As seen by (Sudhakar et al. (2019) and verified in Sec. 6. GLEU strike balance between target style match and content retention and correlate well with the human evaluations. From Tab. 2, CP-VAE consistently outperforms the three adversarially trained models on GLEU by noticeable margin and achieve competitive results as compared to the recent state-of-the-an models. By checking the samples generated from the models shown in Tab. 3 B-GST, the current -ate-of-the-art, is more

---

Under review as a conference paper at ICLR 2020
Table Comparisons with supervised approaches on Yelp and Amazon dataset.
Yelp
Amazon
Model
A
BL 1
GL
PL
AC
BL
GL
PL
Source
1.8
100.0
8.4
26.6
16.3
100.0
22.8
34.5
Human
70.1
25.3
100.0
63.7
41.2
45.7
100.0
68.6
CA
74.0
20.7
6.0
103.6
75.5
0.0
0.0
39,3
SE
8.2
67.4
6.9
65.4
40.2
0.4
0.0
125.0
MD
49.5
40.1
6.6
164.1
70.1
0.3
0.0
138.8
D&R
88,1
36.7
7.9
85.5
49.2
0.6
0.0
46.3
B-GST
85.6
45.2
12.7
49.6
55.2
52.3
18.1
48,2
CP-G
66.7
35.5
7.5
67.8
35.4
11.5
109.1
CP-B
55.4
48.4
9.6
47.6
40.0
39.7
12.7
97.3
Table 3: Samples of generated sentences. SRC is the imput sentence and HUMAN is the human references.
Yelp
Positive to Negative
Negative to Positive
SRC
this place is super yummy
but probably sucks too
HUMAN
this place is super yucky
but probably doesn't 't suck too
B-GST
this place is super bad
but tastes great too!
CP-G
this place is super slow and watered down.
but it 's truly fun and insanely delicious.
CP-B
this place is super greasy and gross
but probably wonderful when you!
Amazon
Positive to Negative
Negative Poxitive
SRC
becausc it s made of cast iron, scorching is
they are cheerios, afterall, and we love the
minimized,
original kind
HUMAN
because made cast iron, scorching maximized
they are cheerios, we love them.
B-GST
because it cheaply made of cast iron is useless
they are sturdy, afterall, sturdy and we love the original.
CP-G
because s made cast iron, , vomitting.
they are ripe, tastier, and we love them they are divine, fluffier and we love them
CP-B
because it s made of cast iron, limp.
consistent to the source sentence, which can be expected, since it only makes necessary edits to flip the sentiment. CP- VAE tends to generate more diverse contents which may not be relevant sometimes, but the overall quality reasonable considering trained without the label information. More samples can be found in Appendix
6.3 FINER-GRAINED STYLE DISCOVERY AND TRANSITION
Table 4: Two pairs of samples generated without and with topic transition. The first sentence in the pair generated with topic fixed throughout the generation; while the second sentence is generated with topic transition, the generated oulputs after switching are marked as bold.
World throughout
A federal judge Friday ordered a federal appeals court to overturn federal appeals court ruling that the Visa and MasterCard credit card associations violated federal antitrust law by barring the names of the state
World to Sci/Tech
A federal judge on Friday ordered a federal appeals court to overturn a decision by the Supreme Court overturn decision by the Federal Communications Commission to block the company's antitrust case against Microsoft Corp
Sports throughout
NEW YORK (Reuters) Roger Federer, the world's No. player, will miss the rest of the season because of sore quadriceps.
Sports to Business
NEW YORK (Reuters) Roger Federer, the world's ' No. player, will miss the rest of the year because of bid-rigging scandal.
To further explore the potential of CP- VAE, we conduct the following exploratory experiments. We use the AG news dataset constructed by (Zhang et al. 2015), which contains four topic categories which are World, Sports, Business and Sci/Tech, with the title and description fields. Here, we drop the title and just the description field train CP-VAL and set K 10. All four topics are automatically discovered by CP- VAE and identified as described in Sec. 6. We also compare the results of our identified topies to standard baselines for unsupervised topic modelling, the details
8

---

Under review as a conference paper at ICLR 2020
can be found in Appendix D We choose basis vector discovered by model and generate few tokens. Then, we switch the basis vector and continue the generation until the end-of-seq token is generated. Generated samples shown in Table 4 one topic to another in a natural and fluent way within the same sentence. Several observations can be made based these samples: (1) it is good detecting name entities and replacing them with the name entities related to the chosen topic; (2) there is no hard restriction on when to switch the topic; the model will determine an appropriate way do the transition by itself. Such observations confirm that EA- possesses a filled constrained latent space which make the latent code robust to manipulation across different time steps, which can be effectively reflected in the generation process. Due to space limitations, we put more samples in Appendix
We see that our model learns to transition from
7 CONCLUSION
In this work, investigate latent vacancy important problem in unsupervised learning of controllable representations when modelling text with VAFs. To mitigate this, we propose to constrain the posterior within a learned probability simplex, achieving the first success towards controlled text generation without supervision.
REFERENCES
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, and Jiajun Chen. Generating sentences from disentangled syntactic and semantic spaces. arXiv preprint arXiv:1907.05789, 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1788-1825 2013.
David M Blei, Andrew Y Ng. and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 393n 2003.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. arXiv preprint 1rXfv:1511.06349, 2015.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Ondřej Cifka, Aliaksei Severyn, Enrique Alfonseca, and Katja Filippova. Eval all, trust few, do wrong to none: Comparing sentence generation models. arXiv preprint T.XXvv/804007972 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 180.001810004805, 2018.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Explo- ration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Jumxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg- Beng-Kirkpatrick. Lagging inference networks and posterior collapse variational autoencoders. arXiv preprint N.ki::9901.05500 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, volume 3, 2017.
Zhiting Hu, Zichao Yang, Xiaodan Liang. Ruslan Salakhutdinov, and Eric Xing. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1587-1596. JMLR. org, 2017.
Diederik P Kingma and Max Welling. 13Xiv 2013.
Auto-encoding variational bayes. arXiv preprint

---

Under review as a conference paper at ICLR 2020
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848,, 2017.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc' Aurelio Ranzato, and Y-Lan Boureau. Multiple-atrribute text rewriting. 2018.
Juncen Li, Robin Jia, He He, and Perey Liang. Delete, retrieve, generate: A simple approach to sentiment and style transfer. arXiv preprint arXiv: I 1800.064337 2018.
Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. Content preserving text generation with attribute controls. In Advances in Neural Information Processing Systems, pp. 5103-5113, 2018.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:15/ 2015.
Christopher Manning, Prabhakar Raghavan, and Hinrich Schüize. Introduction to information retrieval. Natural Language Engineering. 16(1):100-105 103, 2010.
Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. Ground truth for grammatical error correction metrics, In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 588-588-588 188-593,20 2015.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 31 311-318. Association for Computational Linguistics, 2002.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017.
Alec Radford, Rafal Jozefowicz. and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.0144 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1:8, 2019.
Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with daesa-vaes. arXiv preprint arXiv: 1901,030030 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint arXiv:1810.0055 2018.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non non-parallel text by cross-aligment. Advances in neural information processing systems, pp. 6830-6841, 2017.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. Transforming delete, retrieve, generate approach for controlled text style transfer. arXiv preprint arXin:1908.093668 2019.
Jakub M Tomezak and Max Welling. Vae with vampprior. arXiv preprint arXiv: 2017.
Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. A network- based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv: 2016.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pp. 649-657, 2015.
Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, Yann LeCun, al. Adversarially regularized autoencoders. Proceedings of the 35th International Conference on Machine Learning, 2018
10

---

Under review as a conference paper at ICLR 2020
A DETAILS ABOUT EXPLORATORY EXPERIMENTS
A.1 MODEL DETAILS
For the 3-VAE we used for the exploratory experiments, we use a LSTM encoding network and LSTM decoding network. For the encoding network, the imput size is 256, and the hidden size is 1,024. For the decoding network, the input size is 256, the hidden size is 1,024, and dropouts with probability 0.5 are applied on after the embedding layer and the LSTM layer in the decoding network. 3 is chosen as 0.35, the dimension for the latent code is 80, and the batch size is 32. We use SGD with learning rate 1.0 to update the parameters for both the encoding and the decoding network. We train the model until the reconstruction loss stops decreasing.
A.2 IDENTIFYING THE LATENT FACTOR INDICATING THE SENTIMENT
First, we normalize the value of each latent code by subtracting the mean estimated over all the training samples. Then we use the polarity of each latent code to classify the sentiment in the validation set. The one with the highest accuracy is identified the latent factor indicating the sentiment.
A.3 SAMPLES GENERATED FROM 3- VAE
Table 5: Samples of generated sentences from 3-VAE on Yelp.
Positive to Negative
Negative to Positive
SRC
this place is super yummy
but it probably sucks too thank you?
B-VAE
this place perfect for all of or long and over priced!
SRC
will going back and enjoying this great place
there is definitely not enough room that part of the venue.. there is great.
8-VAE
will be going back and recommending this
place anyone who lives in the valley
A.4 MANIPULATION STRATEGIES
Following manipulation strategies have been attempted: (1) fixing the relevant factor μ 2a and - 2o2o; (2) fixing the relevant factor to µ and μ a;; (3) fixing the relevant factor to the maximum value and the minimum value of the relevant factor appearing
the training samples; (4) calculating a latent vector based on 10 manually constructed parallel sentences with opposite sentiment while keeping other factors unchanged. However, none of these four strategies is effective considering the generation results. We report the result with the first strategy in the paper, since it performs the best considering the accuracy and the BLEU score.
B DETAILS ABOUT EXPERIMENTS ON TEXT STYLE TRANSFER
B. TRAINING REGIMES
Across all the datasets, we use Adam with learning rate 0.001 to update the parameters for the encoding network, while SGD with learning rate 1.0 update the parameters for the decoding network. The batch size is chosen to be 32. Dropouts with drop probability 0.5 are applied on applied on after the embedding layer and the LSTM layer in the decoding network. We train the model until the reconstruction loss stops decreasing.
B.2 MITIGATING POSTERIOR COLLAPSE
For the structured part 2(), we use B- VAE setting 0.2 across all the datasets. For the unstructured
part 22) different strategies employed for each dataset:
11

---

Under review as a conference paper at ICLR 2020
Yelp: 3-VAE setting as 0.35.
Amazon: 3-VAE setting 3 as 0.35.
AG-News: KL annealing, from 0.1 to 1.0 10 epochs.
B.3 HYPERPARAMETER SETTINGS
Table 6: Hyperparameter settings.
Yelp
Amnzon
AG News
Nomber variations K
Parameter control the KL
100
100
10
Input dimension LSTM encoder
256
Hiddlen dimension LSTM encoder
1024
1024
1024
Dimension for (2)
64
64
96
Dimension for
16
32
Input dimension LSTM decoder
128
128
Hidleen dimension for LSTM decnder
1024
1024
1024
The hyperparameters chosen by checking LVVE, KL,, and the generated outputs the development set for Yelp and AG-News. Amazon follows the same setting as Yelp without extra tuning.
B.4 DECODING STRATEGY
For decoding, we use beam search with a beam size of
C PROOF OF MINIMALIZATION OF EQ.
The problem can be formulated as an optimization problem as follows:
K
K
maximize
Σ
2 pi
subject to
Σ
=1
By introducing Lagrange multiplier A, the Lagrange function is defined as
K
K
C(p1. P2.... PK. A)
Σ
p
pt
1).
1=1
In order to find the optimal point, we require that
i
Σ
K
Pi 11 2p: >= i= 1=1,2....,
p
Op:
which shows that all ps are equal. By using the constraint E,p, 1, we find pt A 1, 2...., K. By plugging into the results, 4 E, oi reaches its minimum x.
D COMPARISONS WITH BASELINES ON TOPIC MODELLING
Experimental setup: We use the AG news dataset for this task constructed by (Zhang et 2015). It contains four topic categories which are World, Sports, Business and Sci/Tech, with the title and description fields. For each category, there are 30, 000 training samples and 1, 900 test samples. In this paper, drop the title and just use the description field. We compare two standard basclines for unsupervised topic modelling: (1) LDA (Bli et al. 2003). standard implementation of LDA is used for this baseling") (2) k-means. To show the power of our approach
approach to
hhan:://tadmmmchurkk..om////
12

---

Under review as a conference paper at ICLR 2020
beyond the pre-trained sentence representations, perform k-means clustering directly the sentence representations. Following (Manning et al. 2010). we assign each inferred topic to one of the gold-standard topics with the optimal mapping and report the precision (a.ka. purity), recall (a.k.a. collocation) and F score. The number of topics is chosen to be 10. The results reported for the baselines and model are the average over 10 runs.
Quantitative results: The results are shown in 7 We can see that our approach achieves com- parable results to LDA while significantly outperforming k-means all four categories, indicating that our approach can go beyond just clustering on pre-trained sentence representations.
Table 7: Results for topic identification.
Topic
Model
Precision
Recall
F
LDA
69.73
75.32
72.14
World
k-means
67.64
47.63
55.90
Ours
80.83
70.55
74.59
LDA
79.17
82.50
80.22
Sports
k-means
47.66
89.50
62.04
Ours
81.14
78.88
79.49
LDA
72.10
66.45
68,46
Business
k-means
53.06
53.16
53.11
Ours
64.04
64.53
63.97
LDA
66.55
59.77
61.60
Sci/Tech
k-means
81.32
31.59
44.67
Ours
65.20
71.74
66.77
13

---

Under review as a conference paper at ICLR 2020
E
TEXT TRANSFER EXAMPLES
E.1
SENTIMENT MANIPULATION ON YELP DATASET
Table 8: Sentiment manipulation results from positive negative
SRC
this was the best have ever had
B-GST
this was the worst place have ever had
CP-G
this was the worst pizza have ever had
CP-B
this was the worst have ever had
SRC
friendly and welcoming with fun atmosphere and terrific food. the hummus is ridiculously bland and bland
B-GST
CP-G
rude and unorganized with terrible atmosphere and coffee.
CP-B
the hummus is ridiculously greasy and tasteless.
SRC
i ordered the carne asada steak and it was cooked perfectly! !
B-GST
ordered the carne asada steak and it was just as bad
CP-G
i ordered the carne asada steak and it
n't cooked and it was lacking.
CP-B
ordered the carne asada burrito and it was mediocre.
SRC
the owner is hoot and the facility is very accommodating.
B-GST
the owner is a jerk and the facility is very outdated.
CP-G
the owner is a hoot and the facility is empty and the layout is empty
CP-B
the owner is riot and the facility is very clean.
SRC
i will be going back and enjoying this great place
B-GST
i wo n't be going back and this place is horrible !
CP-G
wwll be going back and eat this pizza hut elsewhere.
CP-B
will be going back and hated the worst dining experience.
Table 9: Sentiment manipulation results from negative to positive
SRC
there is definitely not enough room in that part of the venue. there is plenty enough seating in that part of the venue
B-GST
CP-G
there is definitely authentic dinner in that part.
CP-B
there is definitely nice theatre in that part.
SRC
but it probably sucks too
B-GST
but it tastes great too
CP-G
but it truly fun and insanely delicious.
CP-B
but it 's probably wonderful when u
SRC
always rude in their tone and always have shitty customer service always in tune with their tone and have great customer service.
B-GST
CP-G
always great with their birthdays and always excellent music.
CP-B
always accommodating and my dog is always on family
SRC
i was very sick the night after.
B-GST
i was very happy the night after.
CP-G
i was very pleased with the night.
CP-B
was very happy with the night.
SRC
this is a horrible venue..
B-GST
this is a wonderful venue..
CP-G
this is great place for celebrating friends.
CP-B
this is great place for beginners.
14

---

Under review as a conference paper at ICLR 2020
E.2
SENTIMENT MANIPULATION ON AMAZON DATASET
Table 10: Sentiment manipulation results from positive to negative
SRC
most pizza wheels that i ve seen are much smaller.
B-GST
most pizza dough that ve seen much better.
CP-G
most pizza wheels that i ve seen are much more good and are much quality
CP-B
most pizza wheels that ve seen are much better than are much better
SRC
however, this is an example of how rosle got it right.
B-GST
however, this game is an example of how rosle loves it
CP-G
however, this is example of how toxic sad obviously.
CP-B
however this is an example of how cheap similar cheap advice. cheap advice.simme
SRC
auto shut off after num num hours, which is good feature.
B-GST
auto shuts off after num num hours, which is shame
CP-G
whipped mask off after num num hours, which is slimy, which is disgusting. auto shut off after num num hours, which is a stupid idea, which seems to be bad
CP-B
SRC
that said, the mic did pic up everything it could.
B-GST
that said, the game took up everything could.
CP-G
that said, the shampoo did nt smell him well stopped cleaning everything ended up smelling sick
CP-B
that said, the mic did not fit everything
well, let me down it weren cleaning
SRC
also prefered tha blade weight and thickness of the wustof
B-GST
also like the blade weight and of the wustof"
CP-G
also disliked the blade weight and thickness of the materials.
CP-B
also slammed the blade weight and thickness the wide
Table 11: Sentiment manipulation results from negative to positive
SRC
the quality is declined quickly by heat exposure..
B-GST
the water quickly drained by head exposure
CP-G
the quality is utilitarian grinding sandwiches.
CP-B
the quality is priceless quickly by heat rises
SRC
the directions were easy to follow but the quality of the casel was pathetic.. the directions were easy to follow but the quality of the product was excellent. the directions were easy to follow but the quality is good for the quality and is the directions were easy to follow but the quality is what the quality is like the best quality of
B-GST
CP-G
CP-B
SRC
multiplayer is just as bad, though thankfully not worse..
B-GST
quality is just as good, though thankfully not perfect.
CP-G
besides it just good though those usually usually... ...
CP-B
multiplayer is just as bad, though somebody s also so far not so far but no problem
SRC
another energy product that simply wastes our money... another energy product that simply saves our money... another energy product that simply glides your pasta.
B-GST
CP-G
CP-B
another energy product that simply wastes this money.
SRC
i received the wrong color and shreds easily
B-GST
received the color and works easily.
CP-G
low the new color and closes easily. received the wrong color and it pours easily from dishwasher and dries easily on garlic easily.
CP-B
15

---

Under review as a conference paper at ICLR 2020
F
TEXT TRANSITION EXAMPLES ON AG NEWS
Table 12: Topic transition examples.
World throughout
BAGHDAD (Reuters) Iraq interim prime minister. lyad Allawi, said on Monday that the United States had no intention of withdrawing from the country to end the violence in Iraq-
World Sports
BAGHDAD (Reuters) Iraq 's interim prime minister Iyad Allawi, said on Monday that the United States had no intention of withdrawing its troops from the country to the end of the year.
World to Business
BAGHDAD (Reuters)- Iraq 's interim prime minister. lyad Allawi, said on Monday that the United States had no intention of withdrawing its troops from the country to the country.
World to Sci/Tech
BAGHDAD (Reuters) Iraq 's interim prime minister, Iyad Allawi, said on Monday that the United States had no intention of withdrawing its uranium enrichment program to the United States.
Sports throughout
For the first time in four years, the US men 's baskethall team won the gold medal in the men 's 400-meter medley relay.
Sports to World
For the first time in four years, the US men 's basketball team won the gold medal at the Athens Olympics in Athens, where the United States and the United States have agreed to a peace deal.
Sports to Business
For the first time in four years, the US men 's basketball team won the gold medal at the Athens Olympics on Wednesday, with surge in crude oil prices.
Sports to Sci/Tech
For the first time in four years, the US men 's basketball team won the gold medal in the men Olympic basketball tournament in Beijing on Tuesday.
Business throughout
NEW YORK (Reuters) U.S. stocks opened higher Friday oil prices climbed above $48 a barrel and the Federal Reserve raised interest rates by quarter percentage point.
Business to World
NEW YORK (Reuters) U.S. stocks opened higher on Friday oil prices climbed above $48 a barrel and the Federal Reserve raised interest rates by quarter percentage point.
Business to Sports
NEW YORK (Reuters)- - U.S. stocks opened higher on Friday as oil prices climbed above S48 barrel and the Federal Reserve raised interest rates by quarter percentage point.
Business to Sci/Tech
NEW YORK (Reuters) U.S. stocks opened higher on Friday as oil prices climbed above $48 a barrel and the Federal Communications Commission said it would allow the companies to use mobile phones
Sci/Tech throughout
SINGAPORE (Reuters) - South Korea 's Hynix Semiconductor Inc. said on Tuesday it had developed a prototype micro fuel cell recharger for range of security vulnerabilities India
Sci/Tech World
SINGAPORE (Reuters)- South Korea 's Hynix Semiconductor Inc. said on Tuesday it had developed prototype micro fuel cell aimed at ending standoff with North Korea.
ScifTech to Sports
SINGAPORE (Reuters) South Korea 's Hynix Semiconductor Inc. said on Tuesday had developed a prototype micro fuel cell aimed at protecting the world 's biggest gold medal.
Sci/Tech to Business
SINGAPORE (Reuters) South Korea 's Hynix Semiconductor Inc. said on Tuesday had developed prototype micro fuel cell aimed at protecting the world 's largest oil producer.
16


Reviews:
[{"review_id": "r1gTTOdaFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to pinpoint why sequence VAEs haven't worked well to provide disentangled representations. The authors posits that this happens due to the fact that perturbing the intermediate representation (codes) pushes them in regions which are not seen in training, and hence the model is not well equipped to perform well for those codes. To address this, they augment the VAE objective with terms to ensure that codes are present in a probability simplex and the entire simplex is uniformly filled by codes. I thought this was a very good paper. I found the observation very interesting, and the supporting experiments confirm the hypothesis. However, since I do not actively work in this area, I am not sure how exciting the result will be to other researchers. \n\nThere's some related work on controlled text generation.\nhttps://arxiv.org/pdf/1811.00552.pdf\nhttps://arxiv.org/pdf/1811.01135.pdf\nIt will be good to mention them and possibly compare and contrast with the above work. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkeZKd3atr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper \"On Variational Learning of Controllable Representations for Text without Supervision\" tackles the problem of latent vacancy of text representation via variational text auto-encoders. Based on the observation that a single factor of the sentence encoding gathers most of relevant information for classifying the sentence as positive or negative (sentiment classification), authors study the impact of manipulating this factor in term of the corresponding decoded sentences. They reasonnably claim that if such a manipulation fails at decoding accurate sentences, it is because we fall in representation areas that the decoder never seen during training. Thus they propose a way to constrain the posterior mean to a\nlearned probability simplex and only perform manipulation within the probability simplex. \n\nThe tackled problem is important. Variationnal text auto-encoding is a very challenging task, for which no perfect solution has been proposed yet. A important issue is that usually the posterior collapses, with the auto-regressive decoder eventually ignoring the codes. Another problem is that indeed the representation space can contain many holes, in codes have never been seen during training. The authors propose to cope with both problems by encouraging a part of the code mean to live in a simplex, which prevents from posterior collapsing. Next, to ensure that information is filled in this constrained part, they define a pairwise ranking loss which enforce the mean of each sentence to be more similar to the output of the encoding lstm than the mean of other sentences. For this part, more intuition justification is needed to well understand the effect of this additional loss. In what sense does it ensure that the space does not contain holes ? What ensures that the constrained part of the code is actually used by the decoder ? \n\nMy main concern is with regards to the experiments, which are clearly not enough detailled. First, I cannot understand what NLL is considered in the preliminary experiments. Authors study the effect of code manipulation on an NLL. But the likelihood of what ? Of the encoded sentence ? If yes it is natural that the NLL is impacted since we move the representation so the manipulated representation encodes another sentence... Or maybe it is w.r.t. a generated sentence from the obtained code ? But what sense does it make to assess the nll of the generated sentence ? Ok if the distribution is to flat, the NLL would not be good, but is it really what we want to observe ? Also, authors compare the impact of modifications on the representations of $\\beta$-VAE with modifications on their model, but these are not the same modifications. What ensure that they have the same magnitude ? I cannot understand the paragraph on vp and vn in the experimental setup. Comparisons with metrics on text style transfer are also difficult to understand to me. What is the reference sentence ? \n\nMinor questions:\n    - z(1) is said to be parametrized by a MLP with sentences representations before eq2 and is said to be encoded by a LSTM after eq 2. What am I missing ? \n     - Please better detail fig2      \n\n    ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxK6Ipc9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for controlled text generation by using a new loss function (standard VAE loss with auxiliary losses added on). The method is tested on style transfer datasets: Yelp and Amazon. The central hypothesis is that when manipulating latent codes of a VAE, you can end up in low-density regions of the aggregated posterior. Such latent codes are rarely seen by the decoder so that quality of generation is low. To address this problem, they constrain the posterior mean to a learnt probability simplex and try to ensure that the simplex is densely filled. They do this by adding 2 regularizing losses to the VAE loss.\n\nSome questions:-\n- I had some trouble following section 4.2 where the derivation for L-reg is explained, this is the loss that constrains the posterior to a simplex. Is E = [e_1, ..., e_k] learnt or a hyperparameter? Most of the text seems to indicate that it is learnt, but you mention that alpha is a tunable hyperparameter, where alpha = e_i^2. Could you elaborate on how e_i^2 can be fixed to alpha is E is learnt?\n- You state that mu^2 reaches a minimum at alpha/K. Could you please provide a proof? This could be placed in an Appendix.\n- In section 6.3, is K still 3 or is it now >=4?\n\nSome citations issues \n- You're missing a couple key citations. I think your work should cite Hu et al.'s (2018) Toward Controlled Generation of Text. \n- You should definitely cite Lample et al.'s (2019)  Multiple-Attribute Text Rewriting. The results in Lample et al. are definitely comparable, and often better than the results presented in this paper. While I think this paper makes an independent and useful contribution, some of the claims are overblown since the results from Lample et al. are not presented or discussed. For example, the Lample's model on Yelp achieves a higher accuracy and a lower perplexity than the CP-VAE model presented in this paper.\n- Correction of citations: Kim et al. (2018), Adversarially regularized autoencoders, is in fact Zhao et al. (2018), you're just missing the first author.\n\n\nAdditionally, I'd like to see some more detail about the robustness of the method. Is the model robust to initialization and hyperparameter settings? Is posterior collapse always avoided? In Table 1 you show that without L-Reg the CP-VAE suffer with posterior collapse, but this would carry more weight if you showed some statistics on how often using L-Reg helps avoid posterior collapse. In general it looks like we're mostly presented with the best results, I'd like to know a little more about the mean and standard deviation of the model runs as well.\n\nOverall, I think this paper makes a worthwhile contribution. I was unclear on a few parts of the derivation for the regularizing losses, but the intuition seems sensible. The presented results are impressive but have a few missing pieces: citing some key results (Lample et al. 2019) and including results about robustness. I think the questions I have, and any shortcomings this paper has, could be addressed in a camera-ready version.\n\n\nOther bits,\n- I do not understand figure 2. It's also never referred back to once we learn what v_p and v_n are. Some clarification here would be helpful.\n- Section 5.1, line 3: achieve -> achieved", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the behavior of VAE for learning controllable text representations and uses this insight to introduce a method to constrain the posterior space by introducing a regularization term and a structured reconstruction term to the standard VAE loss. Experiments show the proposed method improves over unsupervised baselines, although it still underperforms supervised approaches in text style transfer.

The paper had some issues with presentation, as pointed out by R1 and R3. In addition, it missed citations to many prior work. Some of these issues had been addressed after the rebuttal, but I still think it needs to be more self contained (e.g., include details of evaluation protocols in the appendix, instead of citing another paper). 

In an internal discussion, R1 still has some concerns regarding whether the negative log likelihood is less affected by manipulations in the constrained space compared to beta-VAE. In particular, the concern is about whether the magnitude of the manipulation is comparable across models, which is also shared by R3. R1 also think some of the generated samples are not very convincing. 

This is a borderline paper with some interesting insights that tackles an important problem. However, due to its shortcoming in the current state, I recommend to reject the paper.
Decision: Reject