Venue: NAACL 2022

Towards Robust and Semantically Organised Latent Representations for Unsupervised Text Style Transfer

Sharan Narasimhan Suvodip Dey Maunendra Sankar Desarkar

###### Abstract

Recent studies show that auto-encoder based approaches successfully perform language generation, smooth sentence interpolation, and style transfer over unseen attributes using unlabelled datasets in a zero-shot manner. The latent space geometry of such models is organised well enough to perform on datasets where the style is "coarse-grained" i.e. a small fraction of words alone in a sentence are enough to determine the overall style label. A recent study uses a discrete token-based perturbation approach to map "similar" sentences ("similar" defined by low Levenshtein distance/ high word overlap) close by in latent space. This definition of "similarity" does not look into the underlying nuances of the constituent words while mapping latent space neighbourhoods and therefore fails to recognise sentences with different style-based semantics while mapping latent neighbourhoods. We introduce EPAAEs (Embedding Perturbed Adversarial AutoEncoders) which completes this perturbation model, by adding a finely adjustable noise component on the continuous embeddings space. We empirically show that this (a) produces a better organised latent space that clusters stylistically similar sentences together, (b) performs best on a diverse set of text style transfer tasks than similar denoising-inspired baselines, and (c) is capable of fine-grained control of Style Transfer strength. We also extend the text style transfer tasks to NLI datasets and show that these more complex definitions of style are learned best by EPAAE. To the best of our knowledge, extending style transfer to NLI tasks has not been explored before. +
Footnote †: Our code, data and outputs are available at [https://github.com/sharan21/EPAAE](https://github.com/sharan21/EPAAE)

Indian Institute of Technology Hyderabad, India

sharan.n21@gmail.com suvodip15@gmail.com maunendra@cse.iith.ac.in

## 1 Introduction

The Text Style transfer (TST) task is a form of controlled language generation. The goal is to produce fluent style-altered sentences from a given base sentence, while also preserving its style-independent content. The definition of "style" depends on the class labels of the end task. Our work focuses on the unsupervised scenario i.e. performing the training on completely unlabelled corpora. By inducing the latent space organization through input perturbation, TST can be performed using a simple vector arithmetic method (discussed in Section 6).

**Background.** Several well-known architectures use auxiliary objectives that serve as regularizers to ensure that the latent space geometry is smoothly interpolatable and learns high-level semantic features (refer Appendix D). Inspired by successes in denoising approaches in vision Creswell and Bharath (2019); Vincent et al. (2008), we look at input perturbation based approaches for unsupervised style transfer. Unlike text, in vision, freedom exists to finely control the degree of Gaussian "blur" on the continuous input image space. We conjecture that this notion of "controlling blur" may be useful in the text domain as well, serving as our motivation for our chosen model for Input Perturbation. Shen et al. (2020) map "similar" (defined by low Levenshtein distance/ high word overlap) sentences together in the latent space by introducing a simple denoising objective over an underlying Adversarial Autoencoder (AAE) Makhizani et al. (2015). This noise model includes simple token manipulation

Figure 1: A TSNE plot of encoded latent vectors for all 1950 sentences in the Toy Dataset. EPAAE shows tighter and more organised clustering. Refer to B.1 and B.2 for larger plots with legend.

 

(token dropout and substitution) with some probability \(p\), to reconstruct the original sentences from the perturbed inputs. We can reason intuitively that discrete word dropout allows sentences with high word overlap (or low Levenshtein distance) to have a higher chance of being perturbed into one another, thereby mapping them close in the latent space during the training time. However, this method allows for stylistically dissimilar sentences, albeit with high word overlap, to be mapped together in the latent space.

**Idea.** We argue that this negatively impacts the quality of final latent geometry and the results of the subsequent style transfer task. As an alternative, we explore Embedding Perturbation, where a noise vector is sampled, appropriately scaled and added to the embeddings of each input sentence, such that the resultant embeddings are constrained to live inside an E-dimensional hypersphere. The radius of this hypersphere is controllable using a tune-able hyper-parameter \(\zeta\). This hypersphere constraint is partially inspired by concepts in adversarial robustness and ensures that each resultant noised word embedding is not altered to the extent of causing the underlying semantics of the sentence to change. We argue that this allows sentences with stylistically similar constituent embeddings to be mapped together, also encouraging the formation of style-preserving latent neighbourhoods (more on this in Section 4). The resulting latent representation from "embedding-perturbed" autoencoders consequently are better semantically organized and stylistically robust, enabling us to perform TST using an inductive method i.e. simple vector arithmetic on latent vectors.

**Contributions.** We show that this extended model of input perturbation with both discrete and continuous components, allows for overall better quality text style transfer, particularly in its ability to preserve style-independent "content" information. To expand the traditional definitions of styles such as Polarity, Formality, which are based on simple "intra-sentence" attributes, we introduce the "Discourse Style Transfer Task" by salvaging NLI datasets in which the flow of logic between sentences are captured using "Entailment", "Contradiction", and "Neutral" labels. This enables interesting applications such as discourse manipulation in which a pair of sentences agreeing with each other can be made to contradict, and vice versa. We also test our model on fine-grained styles present in the Style-PTB dataset. We empirically show that our model performs the best on a diverse set of datasets with styles ranging from coarse-grained styles (like sentiment) to fine-grained styles (likeness) and complex inter-sentence styles (like discourse or flow of logic).

## 2 Related work

**Seminal work in TST.** Autoencoder based approaches for TST on labelled non-parallel datasets are quite well explored (Shen et al., 2017; Fu et al., 2018). Some techniques involve implicit Style-Content disentanglement of the latent space using Back Translation (Prabhumoye et al., 2018) and adversarial learning (John et al., 2019). Li et al. (2018) achieve disentanglement using simple keyword replacement. Most studies look at simple non-parallel classification datasets, defining their style to be the class label. Studies also look at Syntax-Semantic disentanglement of the latent space (Chen et al., 2019; Bao et al., 2019). A \(\lambda_{1}\) penalty is imposed on the log variance of the perturbations to prevent it from vanishing. The latent vacancy problem (Xu et al., 2020) of the \(\beta-\)VAE is mitigated by introducing auxiliary losses and provided for one of the earliest methods for unsupervised TST. Similar to our work, Rubenstein et al. (2018) introduces the Latent noised AAE (LAAE), Gaussian perturbation is instead added to latent encodings to promote organization. Unsupervised work includes using a language model as a discriminator for a richer feedback mechanism (Yang et al., 2018), allowing it to increase performance in word substitution decipherment, sentiment modification, and related language translation. More seminal work related to autoencoders in the context of Style Transfer is mentioned in Appendix D.

**Contemporary work in TST.** More recent work, treat the style transfer problem as paraphrase generation and fine-tune pre-trained language models (Krishna et al., 2020). Malmi et al. (2020) trains masked language models or MLMs on the source and target domains to identify input tokens to be removed and replace them with tokens from the target MLM in an unsupervised manner. Liu et al. (2020) uses gradient-based update rules in the continuous latent space \(z\) from style and content predictor networks, enabling the transfer of fine-grained styles without using an adversarial approach. Reid and Zhong (2021) performs TST on sentiment and politeness datasets using token editing methods 

(similar to Li et al. (2018)) using Levenshtein editing operations. Lee et al. (2021) also focuses on enhancing content preservation by introducing a method to remove style at the token level using reverse attention and fuse this content representation with style using a conditional layer normalization technique. Riley et al. (2021) adapts the T5 model Raffel et al. (2020) for few shot text style transfer by extracting a style representation and performing conditioned decoding, using only a handful of examples at inference time.

## 3 Method

The underlying language model is a generative auto-encoder which models an input distribution assumed to be from an underlying latent distribution . A deterministic encoder representing the distribution , in the form of an RNN, whose output is reparameterized by another distribution to give the aggregated posterior distribution . Various auxiliary loss functions are used to enforce the learned prior to match . The Generator representing , also in the form of an RNN, decodes back the sample from the learnt prior into its corresponding input from . Gradient descent is applied on the reconstruction loss of the autoencoder given by:

(1)

We use the AAE Makhzani et al. (2015) as our choice for the underlying generative autoencoder over which the input perturbation techniques were applied. AAE uses a discriminator to enforce to match , a standard Gaussian, by learning to distinguish between samples from the two different distributions. This adversarial loss serves as a regularizer for the latent space, giving it the ability the organize itself better for smooth sentence interpolation.

(2)

The final min-max objective is a sum of the reconstruction loss (given below) and weighted adversarial loss:

(3)

We found empirically that AAEs performed well and were stable during training. On the other hand, VAEs required careful tuning of the hyperparameter to prevent posterior collapse and did not perform as well.

### Finely-controlled continuous noise on embedding space

To further organize the latent geometry of the underlying AAE to encode style-based semantic similarity among sentences, we propose a perturbation-based approach on the continuous embedding space. Word embeddings of dimensionality , of each input token are denoted as . Consider an input sentence of length containing tokens having embeddings respectively , respectively. Our objective is to blur every embedding vector by adding some appropriately scaled noise vector to produce a resultant noised embedding vector , such that does not lie too far away from to not change the underlying semantics of the word completely. We do this by ensuring that each new lives inside an -dimensional hypersphere. The center of the hypersphere is the original embedding and its radius is defined as , where is a random variable sampled from a distribution . is probability distribution of the form where is some arbitrarily chosen distribution and as function of hyperparameter . This distribution parameterized by models the probability density cloud inside the embedding hypersphere, controlling its radius and interior densities. Figure 1 neatly summarizes the aforementioned embedding perturbation mechanism for a simple example with only two individual word embeddings. In

Figure 2: An abstract representation of the continuous embedding noise approach for two arbitrary embeddings and on an embedding space



practice, we use a vectorized representation of the above mechanism to blur the embeddings of a mini-batch of sentences of size \(L\) in constant O(1) time. The embedding perturbation method is summarized below in a vectorized form:

Footnote †: We also tried the uniform distribution with varying values for \(\zeta\)

\[z \sim\mathcal{N}(\mu,\,\sigma^{2}) \tag{4}\] \[\hat{z}=z/|z|\] \[s \sim Y(\mu=0,\sigma^{2}=(\zeta/3)^{2})\] \[n =(s\odot|e|)^{T}\hat{z}\] \[e^{\prime} =e+n\] \[r =\zeta*|e|\]

where \(z\), \(\hat{z}\), \(s\), \(n\) and \(e\) are vectorized representations of \(z_{i}\), \(\hat{z}_{i}\), \(s_{i}\), \(n_{i}\) and \(e_{i}\) respectively for a mini-batch of sentences of size \(l\). \(\odot\) and * denote element wise and scalar multiplication respectively. \(|x|\) denotes the magnitude of a vector/batch of vectors \(x\). \(r\) is the vectorized mini-batch representation of \(r_{i}\), the expected radius of \(hypersphere_{i}\). We choose the Gaussian distribution as our choice of the probability distribution \(\mathcal{Y}\), as on testing it produced the best results+. To couple the radius \(r_{i}\) of \(hypersphere_{i}\) to our hyperparameter \(\zeta\) to enable fine-grained control using \(\zeta\), we constrain the probability density of \(Y\) to live inside the hypersphere within three standard deviations. To achieve this, we equate \(3\sigma\) to \(\zeta\), and consequently set the variance \(\sigma^{2}\) of \(\mathcal{Y}\) to be \((\zeta/3)^{2}\).

Footnote †: We also tried the uniform distribution with varying values for \(\zeta\)

### Discrete word dropout probability

The "Denoising Autoencoder" or DAAE Shen et al. (2020) considered discrete token-level perturbations such as token masking, substitution and deletion. We consider token deletion as discrete noise. Drop probability \(p=0.3\) is found to be the best in both their experiments and ours.

Token deletion is the only type of input perturbation that can alter the number of tokens in a sentence. Any continuous model for input noise cannot mathematically generalize the effects of this kind of discrete word dropout. Furthermore, during our experiments, we find that for some datasets, both discrete and continuous noise components are required to produce the overall best model. In such a case, we first perform token deletion and then subject the leftover token embeddings to perturbation. We refer to this generalized noise model as "Embeddings-Perturbed Adversarial Autoencoder" or "EPAAE" parameterized by \(\zeta\) and \(p\).

## 4 Semantic Similarity in Latent Space Neighbourhoods

We contrast and compare the resultant latent space neighbourhoods of DAAEs and EPAAEs.

### Preliminary reasoning

Here, we first investigate the question - _Does token deletion during training group truly put semantically similar sentences together in latent space \(Z\)?_. Intuitively we can reason that the answer may be in the affirmative if the drop probability is small. For example, the sentences "The food was good" and "The food was superb" might get perturbed into a common version, i.e. "The food was" and therefore be mapped nearby in \(Z\) during training. As analysed and concluded in Shen et al. (2020), latent neighbourhoods in the latent space of DAAEs successfully cluster sentences with high word overlap (low Levenshtein/Edit distance) together. However, the Levenshtein distance metric is not an accurate measure of the true semantic similarity between sentences. A pair of sentences with high word overlap might convey different ideas w.r.t the underlying style-based semantics of the dataset. For example, in the context of the Yelp dataset, the sentences "The food was good" and "The food was bad" are stylistically opposing (style being polarity in this case) and yet still get mapped close by in Z for DAAEs.

### Testing the hypothesis with a Toy Dataset

We conduct specifically curated experiments on a synthetic dataset to verify our hypothesis that EPAAEs map stylistically similar sentences together.

**Details of Toy dataset:** Inspired by Yelp, each sentence in this dataset either represents a positive or negative sentiment. It also contains different sentiment independent components, such as the identity of the person and the subject of the review. Each sentence is of the format: "The <_identity_token>_ said the <_subject_token>_ is <_decision_token>_ where _identity_, _subject_ and _decision_ token classes are the only variable parameters in each sentence. The entire set of all permutations of these token classes forms the dataset. For example, the decision class is further subdivided into two subclasses, i.e. positive/ negative sentiment, 

each subclass containing 7 and 8 choosable tokens, respectively. The resultant dataset consisted of 1950 sentences. The details of the subclasses and the representative tokens inside each subclass are shown in Table 1. We produce output labels for each sentence by using a 3-bit representation, one bit for each of three token classes, where the values of each bit represent the token label subclass within that class. For example, a label of 5 is encoded as 101 corresponding to a sentence with _Female_, _Food_ and _Negative_ labels. There are \(2^{3}\) = 8 labels in total labelled 0-7.

**Qual. Analysis of Latent space.** We consider two models, DAAE with token deletion probability \(p=0.3\) and EPAAE with \(\zeta=2.5\). Both models are initially pre-trained over the unlabelled Yelp dataset. The synthetic dataset is used during inference time only. We encode all 1950 sentences into their respective latent space vectors and use TSNE plots to visualise the latent space of each model (Figure 1). We pick a random _query_ sentence, e.g. "The man said the pasta is spicy", and encode it into \(Z\). We then retrieve the top five nearest encoding to the query and observe their decoded outputs to check for the preservation of style-based semantics (Table 2) around the query. We find that the EPAAE maps the latent neighbourhood such that stylistically similar (positive/negative sentiment in this context) are grouped. This is not the case with the DAAE, evident from Table 2 in which neighbours 1 and 3 have a differing sentiment from the query. This offers an explanation as to why the DAAE is not able to produce tightly confined clusters.

**Quant. Analysis of Latent space.** To further validate our hypothesis that EPAAEs better preserve style-based semantics in latent neighbours, we generate useful quantitative metrics over a k-nearest neighbours experiment, done on the test split across all datasets. We document these metrics in Table 3. Column 3 conveys the mean distance to the closest neighbour with the opposite label. Column 4 conveys the mean number of hops to reach to the closest neighbour with opposite labels. In all but a few datasets, the DAAE reports a smaller "mean hops for label flip", supporting our hypothesis that stylistically dissimilar sentences are mapped closer together in the latent neighbour than our proposed model. We also see this trend mostly holds true for the "Mean L2 Norm for Label Flip" metric in Column 4 as well, providing further evidence to our hypothesis.

\begin{table}
\begin{tabular}{c c c} \hline
**Token Class** & **Sub Classes** & **Token Options** \\ \hline \multirow{2}{*}{**(0) Identity**} & (0) Male & boy, man, husband, boyfriend, waiter girl, woman, wife, girlfriend, waitress \\  & (1) Female & food, meal, dimmer, breakfast, lunch, pasta, chicken service, staff, experience, atmosphere, location, place \\ \hline \multirow{2}{*}{**(1) Subject**} & (0) Food & food, meal, dimmer, breakfast, lunch, pasta, chicken service, staff, experience, atmosphere, location, place \\ \hline \multirow{2}{*}{**(2) Decision**} & (0) Positive & good, great, excellent, decent, amazing, wonderful, reasonable bad, worst, horrible, spicy, bland, expensive, disgusting, medicore \\ \cline{2-3}  & (1) Negative & good, great, excellent, dream, amazing, wonderful, reasonable bad, worst, horrible, spicy, bland, expensive, disgusting, medicore \\ \hline \end{tabular}
\end{table}
Table 1: Structure of token classes and division of subclasses with token options used to create Toy Dataset

\begin{table}
\begin{tabular}{l l r p{160.7pt}} \hline
**DAAE** (\(p=0.3\)) & **L2 Norm** & **EPAAE Norm** & **L2 Norm** \\ \hline the man said the pasta is spicy & 0 & the man snail the pasta is spicy & 0 \\ \hline the man said the pasta is good & 2.79 & the man said the staff is spicy & 2.62 \\ \hline the girl saled the pasta is spicy & 2.89 & the man said the pasta is band & 2.69 \\ \hline the non said the pasta is spicy & 2.89 & the man said the pasta is bland & 2.77 \\ \hline the man said the service is spicy & 3 & the girl said the pasta is spicy & 2.84 \\ \hline the man said the pasta is bad & 3.02 & the husband said the pasta is spicy & 2.89 \\ \hline \end{tabular}
\end{table}
Table 2: Nearest Neighbour experiment performed on EPAAE and DAAE. Sentences in red indicate dissimilarity in semantics with respect to cluster center

\begin{table}
\begin{tabular}{l l r p{160.7pt}} \hline
**Dataset** & **Model** & \multicolumn{1}{r}{**Mean L2 Norm for Label Flip Label Flip**} \\ \hline Toy Dataset & \multicolumn{1}{r}{DAAE} & **5.725** & 14.979 \\ EPAAE & 4.324 & **19.587** \\ \hline \multirow{2}{*}{Yelp} & DAAE & **9.386** & 5.850 \\  & EPAAE & 8.211 & **6.513** \\ \hline SNLI & \multicolumn{1}{r}{DAAE} & 9.794 & 3.185 \\  & EPAAE & **10.373** & **3.528** \\ \hline DNLI & \multicolumn{1}{r}{DAAE} & 9.320 & **6.027** \\  & EPAAE & **11.681** & 4.882 \\ \hline SciTail & \multicolumn{1}{r}{DAAE} & 10.224 & 5.750 \\  & EPAAE & **10.676** & **6.253** \\ \hline PP Removal & \multicolumn{1}{r}{DAAE} & 8.195 & 6.722 \\  & EPAAE & **8.774** & **7.409** \\ \hline \multirow{2}{*}{Tenses} & DAAE & 7.686 & **8.802** \\  & EPAAE & **10.924** & 5.426 \\ \hline Voices & \multicolumn{1}{r}{DAAE} & 6.805 & 8.367 \\  & EPAAE & **9.657** & **8.726** \\ \hline \end{tabular}
\end{table}
Table 3: Quantitative metrics to capture model’s ability to preserve style-based semantics inside Latent Neighbourhoods. Bold indicates model with higher value for that corresponding metric in that specific dataset, which is suggestive of ability to better preserve style-based semantics.

 

Setup, Datasets and Metrics

### Experimental Setup

**Baselines.** This work focuses on simple denoising approaches for Unsupervised TST and subsequently constrains our choice of baselines to follow these criteria. We consider three other autoencoder based models for our experimentation, i.e. Denoising AAE (DAAE), Latent-Noised AAE (LAAE) and the \(\beta\)-VAE.

**Hyperparameters and Setup.** Details on hyperparameter selection can be found in A.2. Other common hyperparameters (detailed in A.1) related to encoder/decoder architectures remained identical across all models. Training is completely unsupervised, and labels are only used during inference time. Details on computation time, number of parameters and infrastructure used can be found in Appendix. E.

### Datasets

In this section, we briefly describe the different kinds of datasets used for experimentation. Further details is provided in in Appendix C.

**Complexity of Styles in Datasets:** Current studies mainly focus on high-level styles to validate the approaches. To validate our hypothesis that the EPAAE can perform fine-grained style transfer due to semantics learnt from embeddings, we consider three tasks: sentiment, discourse and fine-grained text style transfer. As prepossessing, we remove non-essential special characters and lowercase all sentences. Except for the Yelp dataset, no pruning is done based on sentence length.

**Sentiment Style Datasets:** We use the preprocessed version from Shen et al. (2017) of the Yelp dataset. The sentiment labels (positive, negative) were considered as style.

**Discourse Style Datasets:** To check for the model's ability to alter the discourse or _flow of logic_ between two sentences we make use of NLI datasets such as SNLI, DNLI, and Scitail. Each instance in the SNLI dataset Bowman et al. (2015) consists of two sentences that either contradict, entail (agree) or are neutral towards each other. Similarly, the DNLI dataset Welleck et al. (2019) consists of contradiction, entailment and neutrality labelled instances. Scitail Khot et al. (2018) is an entailment dataset created from multiple-choice science exams and the web, in a two-sentence format similar to SNLI and DNLI. The first sentence is formed from a question and answer pair from science exams and the second sentence is either a supporting (enttailment) or non-supporting (neutrality) premise.

**Fine-grained Style Datasets:** The Style-PTB dataset Lyu et al. (2021) consists of 21 styles/labels with themes ranging from syntax, lexical, semantic and thematic transfers as well as compositional transfers which consist of transferring more than one of the aforementioned fine-grained styles. To check whether the EPAAE can capture fine-grained styles better by leveraging its better organised latent space, we make use of three styles i.e. Tenses, Voices (Active or Passive) and Syntactic PP tag removal (PPR). In the Tenses dataset, each sentence is labelled with "Present", "Past", and "Future". The Voices dataset contains "Active" and "Passive" voices labels and the PPR dataset contains "PP removed" and "PP not removed" labels.

### Automatic Evaluation metrics

Evaluation for text style transfer includes checking for a) Style Transfer accuracy, b) Content preservation metrics and c) Fluency of output sentences. Recent studies show that automatic Evaluation metrics are still an open problem and can be gamed Xu et al. (2020).

**Style Transfer Measure:** A pre-trained classifier is used to check the presence of the target label in the output sentence. Mir et al. (2019) introduces the notion of checking the style transfer intensity apart from just the presence of the target label. While we find this notion intriguing, we wish to first accomplish the style transfer task convincingly for the current set of tasks before assuming a more complex metric.

**Content Preservation:** In recent work, we observe that models typically struggle more in content preservation and the ability to preserve the contextual meaning of the base sentence. The BLEU score alone does not suffice to correlate strongly with actual qualitative results. To truly validate our hypothesis that EPAAE's are better able to preserve content better, we augment the bucket of standard content preservation metrics ++. Apart from the standard of using BLEU between sentences of source and target styles, we borrow evaluation techniques from fields similar to Text Style Transfer such as Machine Translation and Text Summarization, such as METERO Banerjee and Lavie (2005), 

ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015) which have been shown to correlate more strongly with human judgement. Following the study of Sharma et al. (2017), in which they show BLEU does not necessarily correlate with human evaluations in dialogue response generation, we also adopt Embedding Average, Vector Extrema (Forgues et al., 2014) and Greedy matching score Rus and Lintean (2012).

**Fluency of generations:** Past work measures the perplexity using a pre-trained language model to gauge the fluency or grammatical correctness of the style transferred outputs. Mir et al. (2019) argues such perplexity calculations for style transfer tasks may not necessarily correlate with a human judgement of fluency. Adversarial classifiers in the form of logistic regression networks are trained with the goal of distinguishing between human-produced and machine produced sentences. These classifiers are then used to score the naturalness of the output sentences. We follow this metric during our evaluation of fluency or naturalness 8

Footnote 8: We use code and pretrained models from [https://github.com/passeul/style-transfer-model-evaluation](https://github.com/passeul/style-transfer-model-evaluation) to measure naturalness.

## 6 Experiments

In this section, we look at the quantitative and quantitative results for the text style transfer task for four autoencoder models in seven datasets. We use the vector arithmetic method on latent space representations, inspired by Mikolov et al. (2013) where it showed that word embeddings learnt can capture linguistic relationships using simple vector arithmetic. Analogous to the standard example where "King" - "Man" + "Woman" \(\approx\) "Queen", we manipulate an arbitrary sentence encoding \(z_{x}\) of Style X to Style Y !

\[z_{x}=~{}z_{y}~{}+k(\frac{1}{N_{y}}\sum_{i=0}^{N_{y}}z_{y}^{i}-\frac{1}{N_{x} }\sum_{i=0}^{N_{x}}z_{x}^{i}) \tag{5}\]

where \(z_{y}^{i}\), \(z_{y}^{i}\) denotes the latent vector of the \(i^{th}\) sentence in style \(y\) and \(x\) respectively and \(N_{x}\) and \(N_{y}\) represent the number of encoding present in the corpus for style \(x\) and \(y\) respectively. \(k\) is a scaling parameter used to control the style transfer strength. The resultant latent vector is passed through the decoder to produce the output sentence.

### Quantitative Analysis

TST accuracy, content preservation and naturalness were computed on the converted sentences (shown in Table 4, 5, 6). Content preservation metrics can be found from Column 4 onwards. We consider two versions of the EPAAE i.e. only continuous embedding noise, continuous embedding noise + token deletion, and find that in some cases a mixture of both is required for optimal performance. We find that a slightly lowered value \(p\) for the EPAAE combined with its optimal \(\zeta\) parameter outperforms other models as well.

#### 6.1.1 Sentiment TST

Table 4 summarises the results of TST on the Yelp dataset. On visual inspection, there appears to be a general tradeoff between TST% and content preservation metrics. For example, the \(\beta\)-VAE achieves the best TST% but suffers from bad content preservation capability. In this case, we observe that EPAAE (\(\zeta=2.0,p=0.1\)) has the best content preservation capabilities across all metrics and achieves a reasonable tradeoff of TST%=77.1 as well.

#### 6.1.2 Discourse TST

Table 5 summarises the results on four NLI datasets. Similar to the sentiment task, we see that EPAAEs have the best content preservation capabilities as well as the naturalness metric. It achieves this while achieving a comparable TST% as well. \(\beta\)-VAE shows best TST% but again suffers in content preservation. DAAE also display reasonable TST% vs Content preservation tradeoffs but overall cannot match the tradeoffs achieved by EPAAEs. Human evaluations on the SNLI dataset (Table 8) confirm this as well. The TST% achieved by any model in any task peaks at only 60.4% compared to 81.9% in sentiment task, a significant difference that hints at the fact that the Discourse TST task might be intrinsically more complex than sentiment style. Future work that aims to increase performance on the NLI task will be beneficial.

#### 6.1.3 Style-PTB TST

Table 6 summarises the results on three datasets from the Style-PTB benchmark. Lyu et al. (2021) produces a hierarchy of styles based on transfer difficulty measured by the average token-level hamm



ming distance between the base and converted sentence. According to this hierarchy Tense inversion, PP removal/addition and Voice change are labelled in ascending order as easy, medium and hard respectively. Our results seem to partially validate this observation, in that the max TST% is obtained on the Tenses dataset (100%). Generally speaking, we also observe that TST has much better performance on fine-grained Style-PTB datasets than sentiment and discourse styles. Similar to before, the EPAAE shows best content preservation at competitive values of TST% as well.

It is also noteworthy to consider the direction of style transfer, particularly in the case of complex styles such as Discourse styles present in NLI datasets. Results and analysis on direction-specific metrics for Discourse TST are presented in Appendix B.2.2.

### Qualitative Analysis

**Samples of Output.** For qualitative analysis, sample outputs by DAAE and EPAAE for the Yelp, SNLI and Tenses dataset are given in Table B.4. For a full list of qualitative examples on all datasets along with setup details, please refer to Appendix B.2.1.

**Varying \(k\) for Fine-Grained TST.** By varying the value of \(k\) in Equation 5, it is possible to finely control the strength of Text Style transfer. We show examples of this for the Yelp (Table B.9) and the SNLI (Table 7) dataset, for the baseline DAAE (\(p=0.3\)) and the proposed model EPAAE. For the proposed EPAAE model, the best performing models (specifically in content preservation metrics) were chosen i.e. \(zeta=2.0\), \(p=0.1\) for the Yelp dataset and \(zeta=2.5\) for the SNLI dataset. The chosen qualitative examples highlight the EPAAEs slight superiority in performing fine-grained TST compared to the baseline.

**Smooth Interpolation.** Sentence interpolation experiments are reported in Appendix B.1, in which latent space points in an interpolation along a specific direction are decoded to gauge the smoothness of the space and its ability to generate coherent sentences

### Human Evaluations

Each human annotator was given a set of base sentences and asked to vote for which model produced the most appropriate corresponding style inverted sentences. Please refer to Appendix A.3 for full details on the setup. Results are shown in Table 8. We observe that the proposed EPAAE model was overall more preferred across all three chosen datasets. This margin was most significant in the case of the SNLI dataset.

\begin{table}
\begin{tabular}{l c c c c c c c c c} \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Naturalness**} & \multirow{2}{*}{**TST Acc.**} & \multicolumn{1}{c|}{**BLEU-2**} & \multicolumn{1}{c}{**METFOR**} & \multicolumn{1}{c}{**ROUGLE-L**} & \multicolumn{1}{c}{**CIDER**} & \multicolumn{1}{c}{**Embedding Average**} & \multicolumn{1}{c}{**Vector Extreme**} & \multicolumn{1}{c}{**Greedy Matching Score**} \\  & & & & & & & & & & \\ \hline \(\beta\)-VAE (\(\beta=0.15\)) & 0.792 & **0.819** & 0.064 & 0.079 & 0.212 & 0.271 & 0.773 & 0.445 & 0.600 \\ LAE (\(\lambda_{1}=0.05\)) & 0.763 & 0.782 & 0.056 & 0.072 & 0.182 & 0.271 & 0.756 & 0.436 & 0.583 \\ DCAE (\(\alpha=0.3\)) & 0.711 & 0.812 & 0.167 & 0.134 & 0.339 & 0.775 & 0.810 & 0.551 & 0.674 \\ EPAEAE (\(\zeta=2.0\)) & **0.832** & 0.749 & 0.142 & 0.125 & 0.320 & 0.609 & 0.800 & 0.491 & 0.681 \\ EPAEAE (\(\zeta=2.0\), \(p=0.3\)) & 0.708 & 0.808 & 0.193 & 0.145 & 0.368 & 0.838 & 0.824 & 0.542 & 0.688 \\ EPAEE (\(\zeta=2.0\), \(p=0.1\)) & 0.718 & 0.771 & **0.218** & **0.170** & **0.396** & **1.017** & **0.827** & **0.571** & **0.797** \\ \hline \end{tabular}
\end{table}
Table 4: Quantitative results of TST experiments on Yelp Dataset

\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Naturalness**} & \multirow{2}{*}{**TSI Acc.**} & \multirow{2}{*}{**BLEU-2**} & \multirow{2}{*}{**METFOR**} & \multirow{2}{*}{**ROUGLE-L**} & \multirow{2}{*}{**CODER**} & \multicolumn{1}{c}{**Embedding Average**} & \multicolumn{1}{c}{**Vector Encoding Greedy Extracting Score**} \\  & & & & & & & & & & \\ \hline
**SNLI** & \(\beta\)-VAE (\(\beta=0.15\)) & 0.983 & **0.523** & 0.424 & 0.257 & 0.541 & 2.288 & 0.948 & 0.615 & 0.814 \\  & LAE (\(\lambda_{1}=0.05\)) & 0.946 & 0.522 & 0.036 & 0.051 & 0.158 & 0.026 & 0.824 & 0.318 & 0.593 \\  & DAAE (\(p=0.3\)) & 0.980 & 0.519 & 0.415 & 0.254 & 0.534 & 2.062 & 0.950 & 0.650 & 0.838 \\  & EPAAE (\(\zeta=2.5\)) & 0.979 & 0.480 & **0.534** & **0.316** & **0.655** & **3.615** & **0.958** & **0.688** & **0.857** \\  & EAE (\(\zeta=2.5\), \(p=0.3\)) & **0.983** & 0.513 & 0.388 & 0.240 & 0.504 & 1.783 & 0.948 & 0.626 & 0.826 \\  & EPAAE (\(\zeta=2.5\), \(p=0.1\)) & 0.978 & 0.511 & 0.461 & 0.281 & 0.582 & 2.492 & 0.953 & 0.674 & 0.854 \\ \hline
**DNL** & \(\beta\)-VAE (\(\beta=0.15\)) & 0.927 & 0.578 & 0.243 & 0.144 & 0.356 & 0.654 & 0.905 & 0.503 & 0.722 \\  & LAE (\(\lambda_{1}=0.05\)) & 0.935 & **0.604** & 0.205 & 0.128 & 0.319 & 0.384 & 0.897 & 0.475 & 0.704 \\  & DAAE (\(\rho=0.3\)) & 0.933 & 0.597 & 0.416 & 0.237 & 0.522 & 2.269 & 0.934 & 0.616 & 0.803 \\  & PAPAE (\(\zeta=2.5\)) & **0.939** & 0.513 & 0.440 & 0.253 & 0.557 & 2.573 & 0.943 & 0.635 & **0.818** \\  & EAEAE (\(\zeta=2.5\), \(p=0.3\)) & 0.927 & 0.574 & 0.397 & 0.225 & 0.506 & 2.064 & 0.933 & 0.606 & 0.798 \\  & EPAAE (\(\zeta=2.5\), \(p=0.1\)) & 0.934 & 0.579 & **0.457** & **0.260** & **0.559** & **2.912** & **0.941** & **0.636** & 0.636 \\ \hline
**Sotail** & \(\beta\)-VAE (\(\beta=0.15\)) & 0.770 & **0.570** & 0.095 & 0.065 & 0.173 & 0.142 & 0.796 & 0.387 & 0.643 \\  & LAE (\(\lambda_{1}=0.05\)) & 0.832 & 0.497 & 0.196 & 0.124 & 0.287 & 0.752 & 0.874 & 0.489 & 0.712 \\  & DAAE (\(p=0.3\)) & 0.792 & 0.454 & 0.325 & 0.199 & 0.413 & 1.560 & 0.913 & 0.611 & 0.805 \\  & EPAAE (\(\zeta=2.5\)) & **0.839** & 0.422 & **0.367** & **0.222** & **0.471** & **2.022** & 0.930 & **0.647** & **0.827** \\  & EPAAE (\(\zeta=2.5\), \(p=0.3\)) & 0.813 & 0.495 & 0.276 & 0.171 & 0.355 & 1.241 & 0.906 & 0.560 & 0.770 \\  & EPAAE (\(\zeta=2.5\), \(p=0.1\)) & 0.827 & 0.440 & 0.352 & 0.215 & 0.455 & 1.931 & **0 



## 7 Conclusion

We introduce the "Embedding Perturbed AAE" or EPAAE and show that it best captures underlying style-based semantic features in the latent space in an unsupervised manner compared to its baselines. By inducing robust latent space organization through embedding perturbation in an unsupervised manner, we also demonstrate the possibility of fine-grained TST, where we can control the strength of the target style. Using a diverse set of datasets with varying formulations and complexities of style, we empirically that EPAAE performs overall best in the text style transfer task, particularly in its ability to preserve style-independent content across all datasets.

## 8 Future Work and Limitations Regarding work in TST.

We wish to augment existing state of the art methods with embedding perturbation to check if doing so aids performance. We see degrading TST performance in the Entailment to Contradiction Task across all models. Future work will focus on methods to improve this task.

**Generally.** It is also interesting to further analyse the effects of embedding perturbation to latent representations and resultant properties. A theoretical analysis would be beneficial to cement the use of embedding perturbation in a more general setting. There also remain more important questions that need answering for, e.g. "What if you apply continuous perturbation to hidden states instead of embeddings?", "What is the relation between this type of perturbation and techniques like dropout?". We wish to explore these important questions in the future.

### Ethics Statement

Any TST model can be used for nefarious purposes, e.g. performing a "Non-toxic to toxic"

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \multirow{2}{*}{**Dataset**} & \multicolumn{1}{l}{**Model**} & \multirow{2}{*}{**Naturalness**} & \multicolumn{1}{l}{**TST Acc.**} & \multicolumn{1}{l}{**BLEU-2 METEROR ROUGLE-L CIDER**} & \multicolumn{3}{l}{**COIDE**} & \multicolumn{1}{l}{**Embedding Average**} & \multicolumn{1}{l}{**Vector Extraction Matching Score**} \\  & & & & & & & & & & \\  & & & & & & & & & \\ \hline
**Voices** & \(\beta\)-VAE (\(\beta=0.15\)) & 0.779 & 0.985 & 0.144 & 0.108 & 0.204 & 0.728 & 0.765 & 0.426 & 0.593 \\  & LAEA (\(\beta=0.05\)) & 0.789 & 0.988 & 0.095 & 0.080 & 0.162 & 0.426 & 0.746 & 0.390 & 0.560 \\  & DAAE (\(p=0.3\)) & 0.783 & 0.981 & 0.243 & 0.182 & 0.300 & 1.488 & 0.811 & 0.523 & 0.670 \\  & EPAE (\(<2.5\)) & **0.806** & 0.993 & 0.176 & 0.134 & 0.260 & 0.919 & 0.795 & 0.468 & 0.634 \\  & EPAE (\(<2.5\)) & 0.785 & 0.975 & 0.174 & 0.133 & 0.237 & 0.978 & 0.777 & 0.457 & 0.616 \\  & EPAAE (\(<2.5\)) & 0.796 & **0.991** & **0.253** & **0.187** & **0.318** & **1.491** & **0.819** & **0.532** & **0.681** \\ \hline
**PPR** & \(\beta\)-VAE (\(\beta=0.15\)) & 0.735 & 0.949 & 0.197 & 0.202 & 0.391 & 1.291 & 0.808 & 0.536 & 0.706 \\  & LAEA (\(\lambda=0.05\)) & 0.748 & 0.940 & 0.149 & 0.145 & 0.302 & 0.846 & 0.772 & 0.467 & 0.643 \\  & EPAE (\(p=0.3\)) & 0.730 & 0.948 & 0.261 & 0.279 & 0.486 & 1.797 & 0.842 & 0.615 & 0.774 \\  & EPAE (\(<2.5\)) & **0.757** & 0.932 & 0.282 & 0.283 & 0.510 & 1.982 & 0.857 & 0.631 & 0.789 \\  & EPAE (\(<2.5\)) & 0.728 & **0.955** & 0.236 & 0.257 & 0.448 & 1.586 & 0.829 & 0.585 & 0.749 \\  & EPAAE (\(<2.5\)) & 0.747 & 0.938 & **0.293** & **0.301** & **0.525** & **2.105** & **0.861** & **0.645** & **0.799** \\ \hline
**Tenses** & \(\beta\)-VAE (\(\beta=0.15\)) & 0.803 & 0.999 & 0.110 & 0.089 & 0.198 & 0.513 & 0.766 & 0.403 & 0.583 \\  & LAEA (\(\lambda_{1}=0.05\)) & **0.807** & **1.000** & 0.086 & 0.072 & 0.169 & 0.388 & 0.749 & 0.377 & 0.563 \\  & DAAE (\(\rho=0.3\)) & 0.774 & 0.999 & 0.263 & 0.200 & 0.363 & 1.684 & 0.822 & 0.540 & 0.686 \\  & EPAE (\(<2.5\)) & 0.798 & **1.000** & **0.358** & **0.274** & **0.483** & **2.305** & **0.861** & **0.631** & **0.757** \\  & EPAAE (\(<2.5\)) & 0.776 & 0.999 & 0.302 & 0.234 & 0.408 & 2.023 & 0.834 & 0.574 & 0.711 \\  & EPAAE (\(<2.5\)) & 0.785 & **1.000** & 0.339 & 0.254 & 0.456 & 2.279 & 0.852 & 0.604 & 0.739 \\ \hline \end{tabular}
\end{table}
Table 6: Quantitative results of TST experiments on Style-PTB datasets

\begin{table}
\begin{tabular}{c l l l} \hline  & \multicolumn{1}{l}{**DAASE (\(p=0.3\))**} & \multicolumn{1}{l}{**EFAAE (\(\zeta=2.0,p=0.1\))**} \\ \cline{2-4}
**Input** & \multicolumn{1}{l}{**the dog sits by a snowdrift. a dog out in the snow**} & \multicolumn{1}{l}{**the dog sits by a snowdrift. a dog out in the snow**} \\ \hline   \(k=1\) & \multicolumn{1}{l}{the dog stands in the snow. a dog stands in the snow} & \multicolumn{1}{l}{the dog sits under a fallen tree dog. a dog resting in the snow} \\ \(k=1.5\) & \multicolumn{1}{l}{the dog stands in the snow. a dog swims in the snow} & \multicolumn{1}{l}{the dog sits under a fallen tree dog. a dog resting in the snow} \\ \(k=2\) & \multicolumn{1}{l}{the dog stands in the snow. a dog swims in the snow} & \multicolumn{1}{l}{the dog sits under a fallen tree dog. a dog resting in the snow} \\ \(k=2.5\) & \multicolumn{1}{l}{the dog stands in the snow. a dog swims in the snow} & \multicolumn{1}{l}{the dog sits under a fallen tree dog. a dog resting in the snow at the person} \\ \(k=3\) & \multicolumn{1}{l}{the dog stays across the snow. a dog swims in the snow in the snow} & \multicolumn{1}{l}{the dog sits under a fallen tree dog. a dog resting in the snow at the person} \\ \cline{2-4}
**Input** & \multicolumn{1}{l}{**a man enjoys some extrawagant arrow. a man is making art**} \\ \hline   \(k=-1\) & \multicolumn{1}{l}{an man makes a strange at that says aux. a man is making clothing} & \multicolumn{1}{l}{a man examines an ant walk. a man is mixing ant} \\ \(k=-1.5\) & \multicolumn{1}{l}{a man makes a strange at that says aux. a man is making clothing} & \multicolumn{1}{l}{a man examines an ant walk by a man is making art} \\ \(k=-2\) & \multicolumn{1}{l}{a man makes a strange at that says art. a man is making clothing} & \multicolumn{1}{l}{a man examines an ant walk by a man is making art} \\ \(k=-2.5\) & \multicolumn{1}{l}{a man makes a strange at that says art. a man is making clothing} & \multicolumn{1}{l}{a man examines an ant gallery unuk. a man is making art} \\ \(k=-3\) & \multicolumn{1}{l}{a man makes a strange at that says art. a man is making clothing} & \multicolumn{1}{l}{a man examines an ant gallery unuk. a man is making art} \\ \hline \end{tabular}
\end{table}
Table 7: Qualitative examples of Fine-grained TST on the SNLI Dataset (Entailment to Contradiction)

modification of text in a real-world setting and causing social harm. Therefore, it is important we keep in mind a code of ethics (e.g. [https://www.acm.org/code-of-ethics](https://www.acm.org/code-of-ethics)) for usage, research and development in this type of research. We have made all our code open-source and provided all details of experimentation and implementation to the best of our knowledge.

## Acknowledgements

We would like to thank the reviewers whose feedback we believe substantially increased the quality of this work. We would like to thank the human annotators for their participation.

## References

* S. Banerjee and A. Lavie (2005)METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, Michigan, pp. 65-72. External Links: Link Cited by: SS1.
* Y. Bao, H. Zhou, S. Huang, L. Li, L. Mou, O. Vechtomova, X. Dai, and J. Chen (2019)Generating sentences from disentangled syntactic and semantic spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 6008-6019. External Links: Link Cited by: SS3.4.
* S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 632-642. External Links: Link Cited by: SS1.
* S. R. Bowman, K. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio (2016)Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany, pp. 10-21. External Links: ISBN 978-3-319-5035-3 Cited by: SS1.
* M. Chen, Q. Tang, S. Wiseman, and K. Gimpel (2019)A multi-task approach for disentangling syntax and semantics in sentence representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 2453-2464. External Links: Link Cited by: SS1.
* A. Creswell and A. E. Bharath (2019)Denoising adversarial autoencoders. IEEE Transactions on Neural Networks and Learning Systems30 (4), pp. 968-984. External Links: Document, ISSN 1050-862X Cited by: SS1.
* G. Forgues, J. Pineau, J. Larcheveque, and R. Tremblay (2014)Bootstrapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, Vol. 2, Cited by: SS1.
* Z. Fu, X. Tan, N. Peng, D. Zhao, and R. Yan (2018)Style transfer in text: exploration and evaluation. Proceedings of the AAAI Conference on Artificial Intelligence32 (1). External Links: Link Cited by: SS1.
* V. John, L. Mou, H. Bahuleyan, and O. Vechtomova (2019)Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 424-434. External Links: Link Cited by: SS1.
* T. Khot, A. Sabharwal, and P. Clark (2018)Scitail: a textual entailment dataset from science question answering. Proceedings of the AAAI Conference on Artificial Intelligence32 (1). External Links: Link Cited by: SS1.
* D. P. Kingma and M. Welling (2014)Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, External Links: Link Cited by: SS1.
* K. Krishna, J. Wieting, and M. Iyyer (2020)Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, Association for Computational Linguistics, Hongkyu Lee, Zhiliang Tian, Lanqing Xue, and Nevin L. Zhang (Eds.), External Links: Link Cited by: SS1.
* D. Lee, Z. Tian, L. Xue, and N. L. Zhang (2021)Enhancing content preservation in text style transfer using reverse attention and conditional layer normalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 93-102. External Links: Link Cited by: SS1.
* J. Li, R. Jia, H. He, and P. Liang (2018)Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 1865-1874. External Links: Link Cited by: SS1.
* C. Lin (2004)ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, pp. 74-81. External Links: Link Cited by: SS1.

 

Daijheng Liu, Jie Fu, Yidan Zhang, Chris Pal, and Jiancheng Lv. 2020. Revision in continuous space: Unsupervised text style transfer without adversarial learning. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(05):8376-8383.
* Lyu et al. [2021] Yiwei Lyu, Paul Pu Liang, Hai Pham, Eduard Hovy, Barnabas Poczos, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2021. StylePTB: A compositional benchmark for fine-grained controllable text style transfer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2116-2138, Online. Association for Computational Linguistics.
* Makhzani et al. [2015] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian J. Goodfellow. 2015. Adversarial autoencoders. _CoRR_, abs/1511.05644.
* Malmi et al. [2020] Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020. Unsupervised text style transfer with padded masked language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8671-8680, Online. Association for Computational Linguistics.
* Mikolov et al. [2013] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
* Mir et al. [2019] Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. Evaluating style transfer for text. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 495-504, Minneapolis, Minnesota. Association for Computational Linguistics.
* Prabhumoye et al. [2018] Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
* Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Reid and Zhong [2021] Machel Reid and Victor Zhong. 2021. LEWIS: Levenshtein editing for unsupervised text style transfer. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3932-3944, Online. Association for Computational Linguistics.
* Riley et al. [2021] Parker Riley, Noah Constant, Mandy Guo, Girish Kumar, David Uthus, and Zarana Parekh. 2021. TextSETTR: Few-shot test style extraction and tunable targeted retsyling. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3786-3800, Online. Association for Computational Linguistics.
* Rubenstein et al. [2018] Paul K. Rubenstein, Bernhard Scholkopf, and Ilya O. Tolstikhin. 2018. On the latent space of wasserstein auto-encoders. _ArXiv_, abs/1802.03761.
* Rus and Lintean [2012] Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In _Proceedings of the Seventh Workshop on Building Educational Applications Using NLP_, pages 157-162, Montreal, Canada. Association for Computational Linguistics.
* Sharma et al. [2017] Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zurner. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. _CoRR_, abs/1706.09799.
* Shen et al. [2017] Tianxiao Shen, Tao Lei, Regina Barzilay, and Tomni Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_, NIPS'17, page 6833-6844, Red Hook, NY, USA. Curran Associates Inc.
* Shen et al. [2020] Tianxiao Shen, Jonas Mueller, Regina Barzilay, and Tommi Jaakkola. 2020. Educating text autoencoders: Latent representation guidance via denoising. ICML'20. JMLR.org.
* Tolistkhin et al. [2018] I. Tolistkhin, O. Bousquet, S. Gelly, and B. Scholkopf. 2018. Wasserstein auto-encoders. In _6th International Conference on Learning Representations (ICLR)_.
* Vedantam et al. [2015] Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 4566-4575.
* Vincent et al. [2008] Pascal Vincent, H. Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In _ICML '08_.
* Welleck et al. [2019] Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
* Xu et al. [2020] Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao. 2020. On variational learning of controllable representations for text without supervision. In _Proceedings of the 37th International Conference on Machine Learning_, ICML'20. JMLR.org 

Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018. Unsupervised text style transfer using language models as discriminators. In _Advances in Neural Information Processing Systems_, volume 31. Curran Associates, Inc.

## Appendix A Additional details on Text Style Transfer Experiments

### Model Architecture

All baseline models were trained with all underlying architectures apart from their individual objective losses. Bi-directional GRUs were used for the encoder and decoder with input embeddings of size 300, a hidden representation of size 256 and a latent space of size 128. For all models using AAEs as the underlying autoencoder, the discriminator was a single-layered perceptron with 512 units. The ADAM optimiser with \(\beta_{1}\), \(\beta_{2}\) as 0.5, 0.999 and a learning rate of 0.001. All models were trained for 30 epochs as any more training steps caused the reconstruction loss to dominate and decrease overall performance on the TST task. All input perturbations were disabled in inference time.

### Hyperparameter Selection

For the hyperparameters \(p\), \(\lambda_{l}\), and \(\beta\) for the DAAE, LAAE and \(\beta\)-VAE, we fixed the values as 0.3, 0.05, 0.15 respectively. This decision was aligned with the results in Shen et al. (2020), which showed that these values produced the best reconstruction vs BLEU trade-off. We found this to be the case during subjective manual testing as well. \(\lambda_{adv}\) was set to 10 for all models having AAEs as the underlying architecture. For EPAAE, we found that \(\zeta\in[2.0,3.0]\) overall showed the best results across all datasets. Therefore a manual search around this range was conducted to determine the optimal \(\zeta\) for each dataset.

### Human Evaluations

TST outputs of two models, the baseline DAAE and the proposed EPAAE model, on the Yelp, SNLI and Tenses dataset were considered. The best performing EPAAE was chosen according to the results in Table 4, 5 and 6, particularly with respect to the content preservation metrics (Since TST% were similar across all models). Two hundred sentences (hundred from each base style) were randomly sampled from the test split of each dataset and style inverted (with scaling factor \(k=2\)). Six hundred instances (each instance being a base and converted sentence pair) were equally split between three human evaluators. Each evaluator was given the task of labelling all two hundred instances from each dataset. The models were anonymous to evaluators and randomly named as "Model 1" and "Model 2". Each instance was to be labelled by an evaluator with four possible decision outcomes, i.e. "1 is best", "2 is best", "All are bad" and "All are good". For each instance, the majority of three votes from each of the three annotators were taken as the final decision for that corresponding instance. Instances without a majority were marked as "NA". The evaluation guidelines was formulated to consider which model a) successfully transferred the target style b) preserved the style-independent content and c) was overall fluent and grammatically coherent.

## Appendix B Additional Experiments:

### Sentence Interpolation in Latent Space - Qualitative Examples

We perform sentence interpolation using DAAE and EPAAE, starting from the same input sentence, incrementally moving along the same direction in the latent space in five fixed-size steps. We see that both the baseline and proposed model are able to produce fluent and coherent sentences, indicative of a smoothly populated latent space.

### Text Style Transfer

#### b.2.1 Qualitative Examples

TST with scaling factor \(k=2\) was performed on DNLI, Scitail datasets as seen in B.5 and B.6 respectively. Similarly, it was also performed on Voices and PP Removal datasets from the StylePTB benchmark as shown in B.7 and B.8 respectively. The proposed model for each dataset, was chosen to be the EPAAE with best performance in the content preservation metrics (shown in Table 4, 5 and 6). Samples were specifically selected in which at least one of the models was able to generate the ideal, style converted sentence with near-perfect content preservation and coherence. This was done by human evaluators, where both the models were anonymized.

#### b.2.2 Direction specific Discourse Transfermetrics

Particularly in the case of Discourse Style, it is natural to speculate that the difficulty of style transfer might be sensitive to the direction i.e.

 



\begin{table}
\begin{tabular}{l l} \hline
**the report will follow five consecutive declines** & **the report will follow five consecutive declines** \\ \hline the report will follow five consecutive declines in full & the report follows five consecutive declines \\ \hline the report followed five consecutive declines & the report followed five consecutive declines \\ \hline the report follows five consecutive declines & the report will follow five consecutive declines in full \\ \hline the index hits its low 20297 off 2042 points & the report follows five consecutive declines in full \\ \end{tabular}
\end{table}
Table B.2: Sentence Interpolation performed on the **Voices** dataset (**Left:** EPAAE, **Right:** DAAE)

\begin{table}
\begin{tabular}{p{136.5pt} p{136.8pt}} \hline
**two blond women are hugging one another the women are sleeping** & **two blond women are hugging one another the women are sleeping** \\ \hline three dogs affectionately playing the dogs are sleeping & three dogs affectionately playing the dogs are sleeping \\ \hline two dogs are playing and wrestling with each other two cats are chasing each other through the house & two dogs are playing and wrestling with each other two cats are chasing each other through the house \\ \hline two men in wheelchairs crash and they reach for the ball the men are sleeping & two men in wheelchairs crash and they reach for the ball the men are sleeping \\ \hline three dogs in different shades of brown and white biting and locking each other the dogs are fighting & three dogs in different shades of brown and white biting and locking each other the dogs are fighting \\ \hline \end{tabular}
\end{table}
Table B.1: Sentence Interpolation on sentence from **SNLI** dataset (**Left:** EPAAE, **Right:** DAAE)

\begin{table}
\begin{tabular}{p{136.5pt} p{136.8pt}} \hline
**the sovets had a world leading space program the guests noted** & **the soviets had a world leading space program the guests noted** \\ \hline the challengers had a big price advantage & the plant had a hairy stem that produced flowers and diminutive seeds \\ \hline the expansion set off a marketing war & the japanese used 00 of the world s ivory \\ \hline the uaw was seeking a hearing by the full 14 judge panel & the diplomat added that mr krenz had several things going for him \\ \hline total return was price changes plus interest income & the japanese used 40 of the world s ivory \\ \hline \end{tabular}
\end{table}
Table B.3: Sentence Interpolation performed on the **Tenses** dataset (**Left:** EPAAE, **Right:** DAAE)



\begin{table}
\begin{tabular}{l l c c c} \hline
**Yelp** & \multicolumn{3}{c}{**Negative to Positive**} & \multicolumn{2}{c}{**Positive to Negative**} \\ \hline Base & i had high hopes and they simply could n’t have fallen farther & probably wo n’t stay here again & the food is always fla-vorful and filling & food was excellent and service was fast \\ \hline Converted (DAAE) & the italian flavors and amazing and all really impressive ! & certainly here ! & the food is not cooked and it was appealing . & food was not the waitress and was slow . \\ \hline Converted (EPAAE) & i had great prices and everything is exquisite ! & definitely stay here ! & the food is n’t just extremely bland . & food was ok. service was slow and disappointed \\ \hline SNLI & **Entailment to Contradiction** & & **Contradiction to Entailment** & \\ \hline Base & a man and lady standing on a seesaw at a park . a man and woman are on a seesaw outside & a dog is jumping through the water . that animal is in the water & a young child in a green shirt is on a carousel , the young child is wearing blue shirt & three people shopping in an sile in a foreign grocery store . three people look at clothes in the malt \\ \hline Converted (DAAE) & a man and a woman are on a seesaw at a beach. a man and women are running through the water & a young child in a blue shirt is wearing a red shirt and a blue hat a child is on a swinging in a store \\ \hline Converted (EPAAE) & a man and a woman are sitting on a seesaw at a park. a man and a woman are sitting on a couch & a dog is jumping through the water the animal is flying through the air & a young child in a red shirt is on a green slide. the child is wearing a shift & three people shopping in the mall \\ \hline
**Tenses** & \multicolumn{1}{c}{**Future to Past**} & \multicolumn{2}{c}{**Past to Future**} \\ \hline Base & miller brewing co and general motors will be included by clients & i will see a possibility of going to 2200 this month & spiegel was 80 controlled & mr deaver had reopened a public relations business \\ \hline Converted (DAAE) & miller brewing co and general motors were included & i was a math major but i was going at this & 42 will be advanced & mr preeden will not be public relations business will say \\ \hline Converted (EPAAE) & miller brewing co and general motors was included by clients & i saw a possibility of going to 2200 this year & spiegel will be 80 controlled & mr deaver will have reopened a public relations business \\ \hline  & & & & \\ \end{tabular}
\end{table}
Table B.4: Qualitative examples of TST task by DAAE and EPAAE on Yelp, SNLI and Tenses dataset

\begin{table}
\begin{tabular}{c p{56.9pt} p{71.1pt} p{71.1pt} p{71.1pt}} \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**Entailment to Contradiction**} & \multicolumn{2}{c}{**Contradiction to Entailment**} \\ \cline{2-5}  & Example 1 & Example 2 & Example 1 & Example 2 \\ \hline Base & i work in a cubic . i have a hectic job & i have 3 kids . i am the proud parent of 2 boys and 1 girl & i work as a contract for a cab company . i am a travel agent & i am a bullet dancer . i work as a trauma surgeon \\ \hline
**DAAE** & i work as a cashier in a supermarket . i am a cashier & i have a wife and two kids . i am a proud us sailor & i work for a online company . i do not have a job & i am a balled dancer . i work at home editing \\ \hline
**EPAAE** & i work in a cubicle . i am a skilled craftsman & i have 3 kids . i am a proud mother of two years & i work for a cab company . i have been working for many years & i am a ballet dancer . i took ballet lessons when i was a kid \\ \hline  & & & & \\ \end{tabular}
\end{table}
Table B.5: TST task performed on DNLI dataset



\begin{table}
\begin{tabular}{c c c c c} \hline  & \multicolumn{2}{c}{**Sciati**} \\ \cline{2-5}
**Models** & \multicolumn{1}{c}{**Entailment to Neutrality**} & \multicolumn{3}{c}{**Neutrality to Entailment**} \\ \cline{2-6}  & Example 1 & Example 2 & Example I & Example 2 \\ \hline
**Base** & most amphibians such as frogs live part of their lives on land and return to water to breed . frogs are amphibians that live or the rim members of the water and live rest of the time in fresh water and live rest of the time on land & reptiles are found on every continent except antarctica & respiratory rate is 40 150 breaths per minute . the normal respiratory rate per minute in adult humans is 12 18 breaths & sample 2 \\ \hline
**DAAE** & most amphibians have dual habitats and the remainder on land and live in the first decade of fish and live in the first living things have the time of fresh water and live rest of the time on land & many species are the most common ancestor in the sink of the same species of birds and mammals have the same species of birds & cardiac output is the amount of breaths per minute the normal respiratory rate per minute in adult humans is 12 18 breaths & sample 2 \\ \hline
**EPAAE** & most species live on land and breed on land frogs are amphibians that live part of the ocean & reptiles are found on every continent except antarctica and birds are found on every continent except antarctica & respiratory rate is a series of breaths per minute in the normal respiratory rate per minute in adult humans is 12 18 breaths & sample 2 \\ \hline \end{tabular}
\end{table}
Table B.6: TST task performed on the SciTail dataset

\begin{table}
\begin{tabular}{p{79.7pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline  & \multicolumn{4}{c}{**Voices**} \\ \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**Active to Passive**} & \multicolumn{2}{c}{**Passive to Active**} \\ \cline{2-5}  & Example 1 & Example 2 & Example I & Example 2 \\ \hline Base & he will make his remarks to a plo gathering in baghdad & accounting problems will raise more knotty issues & one stuck to old line bustiens traditions while the change was embraced by the other & the only time it is had by the violin is right at the end \\ \hline DAAE & he will be told by him as a sewage treatment plant he will say & transportation services will be provided by acounting problems & outside i spotted two young exchange the company s own microprocessor said & the sovies eavesdropping pays off however because the contract they say & the sovies eavesdroppe pays off however because the contract they say & the sovies can overshove plays off however because the contract they \\ \hline EPAAE & his remarks will be made by him from a pole gathering in baghdad & more knotty issues will be raised by accounting problems & one stuck to old line business traditions while the other embraced the change & the only time the violin has it sight to the end \\ \hline \end{tabular}
\end{table}
Table B.7: TST task performed on Voices dataset

\begin{table}
\begin{tabular}{p{79.7pt} p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline  & \multicolumn{4}{c}{**PP Removal**} & \multicolumn{2}{c}{**Information Addition**} \\ \cline{2-5}
**Models** & Example 1 & Example 2 & Example I & Example 2 \\ \hline
**Base** & the problems will be magnified by the june killings & the senate will not vote on six lesser charges with exception of the polar an Antarctica . moders membership will have since swelled to between 20 of smaller creditors & membership will have since swelled to at least 21 since Friday & the dispute will pit two groups of claimants against the two of Japan \\ \hline
**DAAE** & the problems will be magnified & the senate will not vote inside street for a year 21 since Friday & membership will have since swelled to at least 21 since Friday & the dispute will pit two groups of claimants against the two of Japan \\ \hline \end{tabular}
\end{table}
Table B.7: TST task performed on Vices dataset

is it Contradiction/Neutrality to Entailment or vice versa. Intuitively, this makes sense as the Entailment to Contradiction/Neutrality tasks can be achieved simply by randomly editing either the subject or predicate or both, in any one sentence, to trigger a contradiction/neutrality between the two. However, in the reverse task, the edited part must be carefully chosen to precisely match the context of the other sentence to trigger entailment.

To analyze this, direction-specific quantitative metrics for Discourse TST are conducted for the SNLI (B.10), DNLI (B.11) and SciTail (B.12) datasets. We notice a disparity in performances in fact, does exist, mainly highlighted by the differences in the TST% metric. This sensitivity to direction is present in all models across all datasets but is most significant in the SNLI dataset in which TST% goes as low as 20.6% for the Contradiction to Entailment task and as high as 83.7% for the opposite task. Future work can focus on trying to specifically improve the Contradiction to Entailment task, as doing so will be a measure of a model's ability to detect and carefully align the content of one sentence to match another.

## Appendix C Details on Datasets

Here we provide some additional details of all the datasets used in this work.

**Complexity of Styles in Datasets:** As discussed in Section 5.2, we consider three tasks- sentiment, discourse and fine-grained text style transfer. As prepossessing, we remove non-essential special characters and lowercase all sentences. Except for the Yelp dataset, no pruning is done based on sentence length. The vocab size during training was limited at 25k unless mentioned otherwise.

**Sentiment Style Datasets:** We use the preprocessed version from Shen et al. (2017) of the Yelp dataset. It contains 200k, 10k, 10k sentences in the train, dev and test split respectively. The sentiment labels (positive, negative) were considered as style.

* SNLI, DNLI, and Scitail. Each instance in the SNLI dataset Bowman et al. (2015) consists of two sentences. These sentences either contradict, entail (agree) or are neutral towards each other. The resultant dataset contained 341k, 18k, 18k in the train, dev, test splits respectively. The DNLI dataset Welleck et al. (2019) consists of contradiction, entailment and neutrality labelled instances instead in the form of a first-person dialogue like representation. The dataset contains 208k, 11k, 11k sentences in train, dev and test respectively. Scitail Khot et al. (2018) is an entailment dataset created from multiple-choice science exams and the web, in a two-sentence format similar to SNLI and DNLI. The first sentence is formed from a question and answer pair from science exams and the second sentence is either a supporting (entailment) or non-supporting (neutrality) premise obtained from the internet. The dataset contained 24k, 1.3k, 1.3k sentences in the train, dev, test splits respectively. For SNLI and DNLI, all instances with the "neutrality" label were removed. Style transfer task performed from "contradict" to "entail" and vice versa. For SciTail Khot et al. (2018), the transfer task was from "neutral" to "entail" and vice versa.

**Fine-grained Style Datasets:** We used StylePTB dataset Lyu et al. (2021) for fine-grained style. It consists of 21 styles/labels with themes ranging from syntax, lexical, semantic and thematic transfers as well as compositional transfers which consist of transferring more than one of the aforementioned fine-grained styles. To check whether the EPAAE can capture fine-grained styles better by leveraging its better organized latent space, we make use of three styles i.e. Tenses, Voices Active

\begin{table} max width= 

\end{table}
Table B.9: Qualitative examples of Fine-grained TST on the Yelp Dataset 


* [14] M. C. Bostrom, A. K. S. Goyal, and J. R. L. Bartlett. The Bayesian approach to the classification of a class of objects in the presence of outliers. In _Proceedings of the 20th International Conference on Machine Learning Research (ICML)_, pages 35-67. ACM, 2009.
* [15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6810_, 2014.
* [16] T. H. Lee, Y. Bengio, and Z. Chen. Deep residual learning with deep convolutional neural networks. In _Advances in Neural Information Processing Systems_, pages 135-146. Springer, 2015.

[MISSING_PAGE_POST]

 


* [145] M. C. Bostrom, A. K. S. Goyal, and J. R. L. Johnson, "A new approach to the classification of the multi-modal features in the 

or Passive) and Syntactic PP tag removal (PPR). In the Tenses dataset, each sentence is labelled with "Present", "Past" and "Future". The Voices dataset contains "Active" and "Passive" voices labels and the PPR dataset contains "PP removed" and "PP not removed" labels. The resultant sizes of the test, dev, test splits were 71k,8.8k,8.8k (tenses), 90k,11k,11k (PPR) and 44k,5.5k,5.5k (voices).

## Appendix D More details on related work

Bowman et al. (2016) extend Variational Auto-Encoders Kingma and Welling (2014) for text generation and address the posterior collapse problem wherein the decoder completely ignores the latent channel leading to poor generation. Adversarial auto-encoders Makhzani et al. (2015) substitute the KL loss with an adversarial approach to enforce the latent Gaussian prior. On the other hand, AAEs are shown to naturally avoid the posterior collapse problem and promote strong coupling between the encoder and decoder. Wasserstein autoencoders Tolistikhin et al. (2018) introduce a family of regularized autoencoders that learn a flexible prior by solving an optimal transport problem to match P(z) and Q(z) using an adversarial approach. Adversarially regularised autoencoders (ARAEs) follow the Wasserstein Autoencoder framework to learn a learnt prior unlike AAEs, which assume the prior to be a fixed standard Gaussian distribution.

## Appendix E Computational Expense and Infrastructure used

The most parameter-heavy EPAAE model was from the SNLI dataset and we therefore report statistics for this model. The model has 13 million parameters and each epoch took approximately 60 seconds to train on an Nvidia V100-SMX2 GPU and an Intel(R) Xeon(R) E5-2698 CPU. For complete details, please refer to the log.txt files present in each model's directory present in [https://github.com/sharan21/EPAAE](https://github.com/sharan21/EPAAE).