Venue: NAACL 2021

# On Learning Text Style Transfer with Direct Rewards

Yixin Liu, Graham Neubig, John Wieting

Carnegie Mellon University

{yixinl2,gneubig,jwieting}@cs.cmu.edu

###### Abstract

In most cases, the lack of parallel corpora makes it impossible to directly train supervised models for the text style transfer task. In this paper, we explore training algorithms that instead optimize reward functions that explicitly consider different aspects of the style-transferred outputs. In particular, we leverage semantic similarity metrics originally used for fine-tuning neural machine translation models to explicitly assess the preservation of content between system outputs and input texts. We also investigate the potential weaknesses of the existing automatic metrics and propose efficient strategies of using these metrics for training. The experimental results show that our model provides significant gains in both automatic and human evaluation over strong baselines, indicating the effectiveness of our proposed methods and training strategies.1

Footnote 1: Code and data are available at: [https://github.com/yxxinL7/Direct-Style-Transfer](https://github.com/yxxinL7/Direct-Style-Transfer)

## 1 Introduction

Text style transfer aims to convert an input text into another generated text with a different style but the same basic semantics as the input. One major challenge in this setting is that many style transfer tasks lack parallel corpora, since the absence of human references makes it impossible to train the text style transfer models using maximum likelihood estimation (MLE), which aims to maximize the predicted likelihood of the references. As a result, some of the earliest work Shen et al. (2017); Hu et al. (2017); Fu et al. (2018) on unsupervised text style transfer proposed training algorithms that are still based on MLE by formulating the style transfer models as auto-encoders optimized with reconstruction loss. Specifically, during training the model is tasked to generate a _style-agnostic encoding_ and reconstruct the input text based on this encoding with style-specific embeddings or decoders. During inference, the model aims to transfer the source text style using the target style information. While these methods have seen empirical success, they face the inherent difficulty of coming up with a style-agnostic but content-preserving encoding - this is a non-trivial task and failure at this first step will diminish style transfer accuracy and content preservation of the final output.

Another line of work Xu et al. (2018); Pang and Gimpel (2019); Luo et al. (2019) proposes training algorithms based on rewards related to the automatic evaluation metrics, which can assess the model performance more directly during training. This approach is conceptually similar to training algorithms that optimize models using rewards related to the corresponding evaluation metrics for other NLP tasks, such as machine translation Shen et al. (2016); Wieting et al. (2019) or text summarization Paulus et al. (2018); Li et al. (2019). As for unsupervised style transfer, the widely used automatic metrics mainly attend to three desiderata: (1) style transfer accuracy - the generated sentence must be in the target style, commonly measured by the accuracy of a style classifier applied to the transferred text, (2) fluency - the generated text must be grammatically correct and natural, commonly measured by the perplexity of a language model and (3) content preservation - the semantics need to be preserved between the source and target, commonly measured by the BLEU score between the system outputs and source texts. Since these automatic metrics only require the system outputs and source texts, they can be used as rewards for training. Moreover, the two lines of approaches can be used together, and previous work Yang et al. (2018); John et al. (2019); Madaan et al. (2020) proposed methods which use the auto-encoders as the backbone augmented with task-specific rewards. In particular, the style transfer accuracy reward is used by most of the recent work.

However, reward-based training algorithms still have their limitations, and in this paper we aim

to identify and address the bottlenecks of these methods. Specifically, we focus on two problems: (1) the difficulty of designing an efficient reward for content preservation, (2) the lack of robustness of the existing automatic evaluation metrics.

Content preservation is more difficult to measure compared to style transfer accuracy and fluency because it needs to consider the overlap in the semantics between the source text and system outputs. While using BLEU score between the source text and system output would be a direct solution Xu et al. (2018), this approach has an inherent limitation in that \(n\)-gram based metrics such as BLEU are sensitive to lexical differences and will penalize modifications that are necessary for transferring text style. In fact, previous work has proposed various different proxy rewards for content preservation. One of the most popular methods is the cycle-consistency loss Luo et al. (2019); Dai et al. (2019); Pang and Gimpel (2019), which introduces a round-trip generation process, where the model generates an output in the target style, and the ability of a reconstruction model to re-generate the original text is used as a proxy for content preservation. While this method is more tolerant to lexical differences, the correlation between the reconstruction loss and content preservation can be weak.

Therefore, we aim to design a reward for content preservation which can directly assess the semantic similarity _between the system outputs and input texts_. Specifically, we note that models of semantic similarity are widely studied Wieting et al. (2016); Sharma et al. (2017); Pagliardini et al. (2018); Zhang* et al. (2020), and we can leverage these methods to directly calculate the similarity between the system outputs and input texts. This renders our method applicable for even unsupervised settings where no human references are available.

Another key challenge for reward-based training algorithms is that the existing automatic evaluation metrics are not well-correlated with human evaluation Li et al. (2018). It poses general risks to the work in this field with respect to model training and evaluation since these metrics are widely used. An important observation we made from our experiments is that style transfer models can exploit the weaknesses of the automatic metrics. They do this by making minimal changes to the input texts which are enough to trick the classifier used for style transfer accuracy while achieving high content preservation and fluency scores due to the high lexical similarity with the input texts. Upon identifying this risk, we re-visit and propose several strategies that serve as auxiliary regularization on the style transfer models, effectively mitigating the problem discussed above.

We empirically show that our proposed reward functions can provide significant gains in both automatic and human evaluation over strong baselines from the literature. In addition, the problems we identify with existing automatic evaluation metrics suggest that the automatic metrics need to be used with caution either for model training or evaluation in order to make it truthfully reflect human evaluation.

## 2 Methods

### Overview

Data for unsupervised text style transfer can be defined as

\[D=\{(x^{(1)},s^{(1)}),...,(x^{(i)},s^{(i)}),...,(x^{(n)},s^{(n)})\},\]

where \(x^{(i)}\) denotes the text and \(s^{(i)}\) denotes the corresponding style label. The objective of the task is to generate (via a generator \(g\)) the output with the target style conditioned on \(s\) while preserving most of the semantics of the source \(x\). In other words, \(\hat{x}=g(x,s)\) should have style \(s\) and the semantics of \(x\). We define the style as a binary attribute such that \(s\in\{0,1\}\), however, it can be easily extended to a multi-class setting.

### Generator

For our generator, we fine-tune a large-scale language model GPT-2 Radford et al. (2019). GPT-2 is pre-trained on large corpora and can be finetuned to generate fluent and coherent outputs for a variety of language generation tasks Wolf et al. (2019). Since GPT-2 is a unidirectional language model, we reformulate the conditional generation task as a sequence completion task. Namely, as input to the generator, we concatenate the original sentence with a special token which indicates the target style. The sequence following the style token is our output.

### Reward Functions

We use four reward functions to control the quality of the system outputs. The quality of the outputs is assessed in three ways: style transfer accuracy, content preservation, and fluency. We attend to each of these factors with their respective rewards.

 

Here we denote the input text \(x\) having style \(s\) by \(x_{s}\), and denote the output by \(\tilde{x}_{s}\), i.e., \(\tilde{x}_{s}=g(x_{s},1-s)\).

#### Rewards for Style Transfer Accuracy

We use a style classifier to provide the supervision signal to the generator with respect to the style transfer accuracy. The min-max game between the generator \(g\) and the classifier \(f_{cls}\) is:

\[\begin{split}&\min_{\theta_{g}}\max_{\theta_{f_{cls}}}\mathbb{E}_{ x_{s}}[\log(1-f_{cls}(g(x_{s},1-s),1-s))]\\ &+\mathbb{E}_{x_{s}}[\log f_{cls}(x_{s},s)+\log(1-f_{cls}(x_{s}, 1-s))].\end{split} \tag{1}\]

The style transfer accuracy reward for the generator is the log-likelihood of the output being labeled as the target style:

\[r_{cls}(\tilde{x}_{s})=\log(f_{cls}(\tilde{x}_{s},1-s)). \tag{2}\]

Following prior work, we use the CNN-based classifier Kim (2014)\(f_{cls}\), which takes both the sentence and the style label as input and its objective is to predict the likelihood of the sentence being coherent to the given style.

#### Rewards for Content Preservation

To ensure that the system outputs still preserve the basic semantics of the source sentences, we use the pre-trained SIM model introduced in Wieting et al. (2019b, a) to measure the semantic similarity between the source sentences and system outputs. The SIM score for a sentence pair is the cosine similarity of its sentence representations. These representations are constructed by averaging sub-word embeddings. Compared to the cycle-consistency loss Luo et al. (2019); Dai et al. (2019); Pang and Gimpel (2019), our method is more direct since it doesn't require a second-pass generation. It also has advantages over \(n\)-gram based metrics like BLEU Papineni et al. (2002) since it is more robust to lexical changes and can provide smoother rewards.

In Wieting et al. (2019a), SIM is augmented with a length penalty to help control the length of the generated text. We use their entire model, SimiLe, as the content preservation reward,

\[r_{sim}(\tilde{x}_{s})=\text{LP}(x_{s},\tilde{x}_{s})^{\alpha}\text{SIM}(x_{ s},\tilde{x}_{s}), \tag{3}\]

where

\[\text{LP}(r,h)=e^{1-\frac{min(|r|,|h|)}{max(|r|,|h|)}}, \tag{4}\]

and \(\alpha\) is an exponential term to control the weight of the length penalty, which is set to 0.25.

We also use the cycle-consistency loss \(L_{cyc}\) to bootstrap the training:

\[L_{cyc}(\theta_{g})=\mathbb{E}_{x_{s}}[-\log(p_{g}(x_{s}|g(x_{s},1-s),s))]. \tag{5}\]

Here, \(p_{g}\) is the likelihood assigned by the generator \(g\). This introduces two generation passes, i.e., \(\tilde{x}_{s}=g(x,1-s)\) and \(\bar{x}_{s}=g(\tilde{x}_{s},s)\) while SIM reward only requires one generation pass, as illustrated in Fig. 1.

#### Rewards for Fluency

Style transfer accuracy rewards and content preservation rewards do not have a significant effect on the fluency of the outputs. Therefore, we again use the pre-trained GPT-2 model, but as a reward this time. To encourage the outputs to be as fluent as the source sentences, we define the fluency reward as the difference of the perplexity between the system outputs and source sentences:

\[r_{lang}(\tilde{x}_{s})=\textit{ppl}(x_{s})-\textit{ppl}(\tilde{x}_{s}). \tag{6}\]

Here, _ppl_ denotes the length-normalized perplexity assigned by the language model fine-tuned on the training set.

As will be further discussed in Section 3.3, we found that using the rewards mentioned above can still result in unnatural outputs. Therefore, we additionally use a LSTM-based Hochreiter and Schmidhuber (1997) discriminator \(f_{adv}\) to provide a naturalness reward, whose job is to discriminate the system outputs and the real sentences, i.e., an adversarial discriminator. It constructs a min-max game with the generator:

\[\begin{split}&\min_{\theta_{g}}\max_{\theta_{fddv}}\mathbb{E}_{x_{s}}[ \log(1-f_{adv}(g(x_{s},1-s)))]\\ &+\mathbb{E}_{x_{s}}[\log(f_{adv}(x_{s}))].\end{split} \tag{7}\]

Figure 1: SIM Loss v.s. Cycle-Consistency Loss



The naturalness reward is the log-likelihood of the outputs being classified as real sentences:

\[r_{adv}(\tilde{x}_{s})=\log(f_{adv}(\tilde{x}_{s})). \tag{8}\]

### Learning

The final corresponding loss term is:

\[L_{*}(\theta_{g})=-\frac{1}{N}\sum_{i=1}^{N}r_{*}(\tilde{x}_{s}^{( i)}). \tag{9}\]

Here, \(N\) is the number of samples in the dataset. To train the model, we use the weighted average of the losses defined in the previous section:

\[\begin{split} L(\theta_{g})&=\lambda_{cls}L_{ cls}(\theta_{g})+\lambda_{adv}L_{adv}(\theta_{g})\\ &+\lambda_{sim}L_{sim}(\theta_{g})+\lambda_{lang}L_{lang}( \theta_{g})\\ &+\lambda_{rec}L_{rec}(\theta_{g}).\end{split} \tag{10}\]

where \(\lambda\) denotes the weight of the corresponding term. The setting of \(\lambda\) is chosen to make the training stable and have balanced style transfer accuracy and content preservation performance on the development set. \(L_{rec}\) is the reconstruction loss, i.e.,

\[L_{rec}(\theta_{g})=\mathbb{E}_{x_{s}}[-\log(p_{g}(x_{s}|x_{s}, s))]. \tag{11}\]

We follow a two-stage training procedure. We first use the cycle-consistency loss \(L_{cyc}\) to bootstrap the training and then fine-tune the model with the rewards we introduced above to improve the output quality.

In the bootstrap stage, the objective function is

\[\begin{split} L_{boot}(\theta_{g})&=\lambda_{ cyc}L_{cyc}(\theta_{g})+\lambda_{cls}L_{cls}(\theta_{g})\\ &+\lambda_{rec}L_{rec}(\theta_{g})\end{split} \tag{12}\]

We select the checkpoint with the highest mean of the style transfer accuracy and BLEU on the development set as the starting point for the second training stage.

In the second stage, the generator is optimized with Eq. 10. The classifier \(f_{cls}\) for \(L_{cls}\) is pre-trained and the language model for \(L_{lang}\) is finetuned on the training set. During training, the discriminator \(f_{adv}\) for \(L_{adv}\) is trained against the generator. \(f_{cls}\) is fixed when trained on some datasets, while it is trained against the generator on others. We select the checkpoint that has the style transfer accuracy and BLEU score similar to that from the first stage and the lowest perplexity on the development set.

Lastly, since gradients can not be propagated through the discrete samples, we use two approaches to circumvent this problem. For the content preservation reward (Eq. 3) and fluency reward (Eq. 6), we use the REINFORCE Williams (1992) algorithm to optimize the model,

\[\begin{split}&\nabla_{\theta_{g}}\mathbb{E}_{\tilde{x}_{s}\sim p _{g}(\tilde{x}_{s})}[r(\tilde{x}_{s})]\\ &=\mathbb{E}_{\tilde{x}_{s}\sim p_{g}(\tilde{x}_{s})}[\nabla_{ \theta_{g}}\log p_{g}(\tilde{x}_{s})r(\tilde{x}_{s})]\end{split} \tag{13}\]

We approximate the expectation by greedy decoding and the log-likelihood is normalized by sequence length, i.e., \(\frac{1}{L}\sum_{i=1}^{L}\log p_{g}(\tilde{w}_{i})\), where \(\tilde{w}_{i}\) denotes the \(i\)-th token of \(\tilde{x}_{s}\) and \(L\) is sequence length. For the style transfer accuracy reward (Eq. 2) and naturalness reward (Eq. 8), we use a different approach to generate a continuous approximation of the discrete tokens, which allows gradients to be back-propagated to the generator. Namely, taking the style classifier \(f_{cls}\) as an example, we use the distribution \(p_{i}\) of each token produced by the generator as the input of the classifier. This distribution is then multiplied by the classifier's word embedding matrix \(W^{embed}\) to obtain a weighted average of word embeddings:

\[\hat{w}_{i}=p_{i}W^{embed} \tag{14}\]

Then, the classifier takes the sequence of \(\hat{w}_{i}\) as its input. We chose this method because it provides a token-level supervision signal to the generator, while the REINFORCE algorithm provides sentence-level signals.

## 3 Experiments

### Datasets

We evaluate our approach on three datasets for sentiment transfer with positive and negative reviews: Yelp review dataset, Amazon review dataset provided by Li et al. (2018),2 and the IMDb movie review dataset provided by Dai et al. (2019).3

Footnote 2: [https://github.com/lijuncen/Sentiment-and-Style-Transfer](https://github.com/lijuncen/Sentiment-and-Style-Transfer)

We also evaluate our methods on a formality style transfer dataset, Grammarly's Yahoo Answers Formality Corpus (GYAFC),4 introduced in Rao and Tetreault (2018). Although it is a parallel corpus, we treat it as an unaligned corpus in our experiments. In order to compare to previous work, 

we chose the _Family & Relationships_ category for our experiments. Datasets statistics are shown in Table 1.

### Experimental Details

Following previous work, we measure the style transfer accuracy using a FastText5(Joulin et al., 2017) style classifier trained on the respective training set of each dataset. To measure content preservation, we use SIM and BLEU as metrics where self-SIM and self-BLEU are computed between the source sentences and system outputs, while ref-SIM and ref-BLEU are computed between the system outputs and human references when available. To measure the fluency we use a pre-trained GPT-2 model to compute the perplexity.6 Our generator, GPT-2, has 1.5 billion parameters, and we train on a GTX 1080 Ti GPU for about 12 hours.

Footnote 5: [https://fasttext.cc/](https://fasttext.cc/)

Footnote 6: Note that we didn’t fine-tune it on the training set

The weights of the loss terms in Eq. 10 and Eq. 12 are detailed in Table 2. While during our experiments we found that there are other possible configurations which give higher scores with respect to the automatic evaluation metrics, as will be discussed in Section 3.3, we also found that better performance in automatic evaluation doesn't always entail better performance in human evaluation. Therefore, we also manually checked the quality of the transferred texts on development set when we chose the value of the hyperparameters.

We compare our model with several state-of-the-art methods: DeleteAndRetrieve (D&R) Li et al. (2018), B-GST Sudhakar et al. (2019), CycleMulti Dai et al. (2019), Deep-Latent He et al. (2020), Tag&Gen Madaan et al. (2020), and DualRL Luo et al. (2019). We also compare our final model, **DirR(Direct-Reward)**, with the model only trained with the first stage (DirR-Cycle) as mentioned in Section 2.4.

### Adversarial Examples

Yelp and Amazon are arguably the most frequently used datasets for the sentiment transfer task. In our experiments, we found that the automatic evaluation metrics can be tricked on these datasets. Table 3 shows the performance of the models which generate adversarial examples. Upon identifying these risks, we propose several design options that can effectively mitigate these problems.

Yelp DatasetFor the Yelp dataset, when trained without the adversarial discriminator \(f_{adv}\) and the fluency reward, our model (DirR-Yelp-Adv) is able to discover a trivial solution which receives high automatic evaluation scores: injecting a word that carries strong sentiment at the beginning of the output, and making minimum changes (if any) to the source sentences, as illustrated in Table 8. This obviously does not meet the objective of content-preserving sentiment transfer and is easily detectable for humans. In fact, after we manually removed the first word from each of the output sentences, the transfer accuracy dropped from 95.2 to 58.4. To address this problem, we introduced an

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline
**Dataset** & **Style** & **Train** & **Dev** & **Test** \\ \hline \multirow{2}{*}{Yelp} & Positive & 266K & 2000 & 500 \\  & Negative & 177K & 2000 & 500 \\ \hline \multirow{2}{*}{Amazon} & Positive & 277K & 985 & 500 \\  & Negative & 279K & 1015 & 500 \\ \hline \multirow{2}{*}{IMdb} & Positive & 178K & 2000 & 1000 \\  & Negative & 187K & 2000 & 1000 \\ \hline \multirow{2}{*}{GYAFC} & Formal & 52K & 2247 & 500 \\  & Informal & 52K & 2788 & 500 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of samples in the Train, Dev, and Test splits for each dataset in our experiments.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
* \\  & (12) & 1 &
- & 1 & 1.5 \\ \hline \multirow{2}{*}{Amazon} & (10) & 2 & 0.5 & 20 & 2 & 1 &
- \\  & (12) & 5 &
- & 1 & 0.5 \\ \hline \multirow{2}{*}{IMdb} & (10) & 1 & 0.5 & 20 & 2 & 1 &
- \\  & (12) & 1 &
- & 1 & 1 \\ \hline \multirow{2}{*}{GYAFC} & (10) & 2 & 0.5 & 20 & 2 & 1 &
- \\  & (12) & 1 &
- & 1 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyperparameter setting of Eq. 10 and Eq. 12 on each dataset.

\begin{table}
\begin{tabular}{l l r r r r} \hline \hline
**Dataset** & **Model** & **Acc** & **PPL** & **BLEU** \\ \hline \multirow{2}{*}{Yelp} & DirR-Cycle & 91.7 & 392 & 18.7 \\  & DirR-Yelp-Adv & 95.2 & 353 & 20.7 \\ \hline \multirow{2}{*}{Amazon} & DirR & 62.2 & 205 & 30.1 \\  & DirR-AMazon-Adv & 83.2 & 228 & 29.0 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Adversarial Results. **DirR-Yelp-Adv** and **DirR-AMazon-Adv** denote the models which generate adversarial examples. **Acc** denotes the style transfer accuracy, **PPL** denotes the perplexity, **BLEU** is computed between the human references and system outputs.

 

auxiliary discriminator \(f_{adv}\) as we discussed above to penalize the trivial outputs since they can be easily captured by the discriminator. On the other hand, the output perplexity is not sensitive enough to this local feature so using the fluency reward alone is not sufficient. Our final model has much more stable performance when the first word of its output sentences is removed, experiencing only a small drop of the style transfer accuracy from 94.2 to 88.2.

Amazon DatasetFor the Amazon dataset, we found that the style classifier \(f_{cls}\) needs to be updated during the training to prevent the model exploiting the data imbalance problem of the dataset. Namely, in the Amazon dataset some categories of products appear mostly in negative or positive reviews. In Table 4, we show the word frequency of _game_ and _phone_ in both negative and positive reviews. In the original dataset, _game_ mostly appears in negative reviews while _phone_ mostly appears in positive reviews. Therefore, without any prior knowledge, it is very likely that these words will be used as informative features by the sentiment classifier, which makes its predictions unreliable.7

Footnote 7: Notice that the style classifier only achieves 43 accuracy on the human references.

When our second-stage model is trained with the fixed style classifier, it (DirR-Amazon-Adv) learns to exploit this dataset bias by changing the nouns in the original sentences to _game_ or _phone_, which achieves better transfer accuracy. We list some examples in Table 5. DirR-Amazon-Adv generated 291 _game_ in 500 positive reviews, which obviously changes the semantics of the source sentences. In order to show that this phenomenon is independent to the classifier architecture, we additionally fine-tuned a BERT-based (Devlin et al., 2019) classifier, which yielded 51.3, 57.6, 70.4 accuracy on human references, DirR, DirR-Amazon-Adv respectively, showing the same pattern of the fastText classifier. We notice that some two-stage models (Li et al., 2018; Sudhakar et al., 2019; Madaan et al., 2020) and other methods (Yang et al., 2018; Luo et al., 2019) also use a fixed classifier or use words with unbalanced frequencies in different styles as important features, which means that their methods may face the same risk. While Li et al. (2018) has pointed out this data imbalance problem of the Amazon dataset, we further demonstrate that a strong generator can even use this discrepancy to trick the automatic metrics. We are able to mitigate this problem by updating the style classifier during the training, and in Table 4, DirR is more robust to the data imbalance problem compared to other methods.

### Automatic Evaluation

The automatic evaluation results are shown in Table 6. We report the performance of the previous methods based on the outputs they provided for fair comparison and omit those whose results are not available.

We have the following observations of the results. First, compared to our base model (DirR-Cycle), the model trained with our proposed rewards has higher fluency, while remains the same level of content preservation. It indicates that SIM score is as effective as cycle-consistency loss for content preservation and our fluency reward can effectively improve the output fluency. Secondly, there exists a trade-off among the style transfer accuracy, content preservation and language fluency. While our model does not outperform the previous meth

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{“game”} & \multicolumn{2}{c}{“phone”} \\ \cline{2-5}  & Pos. & Neg. & Pos. & Neg. \\ \hline Train & 58 & 7548 & 8947 & 2742 \\ Test & 0 & 10 & 20 & 6 \\ Human & 1 & 10 & 18 & 6 \\ B-GST & 55 & 0 & 13 & 44 \\ Tag\&Gen & 69 & 0 & 14 & 5 \\ DirR & 26 & 0 & 19 & 45 \\ DirR-Amazon-Adv & 291 & 0 & 190 & 4 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Frequencies of words in the Amazon Dataset that appear often enough in specific classes to erroneously cause the classifier to make incorrect predictions. **Pos.** denotes the positive sentences, **Neg.** denotes the negative sentences.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Model** & **Text** \\ \hline Source & don t waste your time or money on these jeans . \\ Adv & don t need your time or money on these **phones** . \\ \hline Source & i made beef bolongese in the oven and it turned \\  & out wonderfully . \\ Adv & i made beef bolongee in the **game** and it turned \\  & out wonderfully . \\ \hline Source & this one does the job i need it for ! \\ Adv & this **game** does the job i need it for ! \\ \hline \hline \end{tabular}
\end{table}
Table 5: Adversarial examples received high style transfer accuracy scores on Amazon Dataset. Adv denotes the adversarial examples generated by DirR-Amazon-Adv.

 

ods on all of the metrics, it is able to find a better balance of the different metrics.

### Human Evaluation

We conducted human evaluation on Yelp, Amazon and GYAFC datasets evaluating the style transfer accuracy, content preservation, and fluency separately. The first two aspects are rated with range 1 - 3 while the fluency is rated with range 0 - 1. We randomly select 100 candidates and compare the outputs of different systems. We use Amazon Turk8 for human evaluation. Each candidate is rated by three annotators and we report the average scores here. We did not evaluate the style transfer accuracy for the GYAMC dataset since it is difficult for human annotators to accurately capture the difference between formal and informal sentences. The results of our human evaluations are shown in Table 7. We additionally report the sample-wise mean score of the metrics where the fluency scores are scaled up to be consistent with other scores. Our model achieves better overall performance when considering all three evaluation metrics on each dataset.

Footnote 8: [https://www.mturk.com/](https://www.mturk.com/)

Interestingly, we found that the automatic metrics for both the style transfer accuracy and content preservation do not accurately reflect performance as measured by human evaluation. For example, on the Amazon dataset, although Tag&Gen Madaan et al. (2020) achieves significantly higher style transfer accuracy based on the automatic metric, our model achieves better performance based on the human evaluation. This phenomenon suggests that the importance of our findings discussed in Section 3.3, that strong neural models can potentially exploit the weaknesses of the automatic metrics.

## 4 Analysis

We next show an ablation study, demonstrating the effectiveness of the content preservation and fluency rewards in DirR, and how SIM can be used to replace the cycle-consistency loss. We also compare using BLEU versus using SIM as a content-preservation reward, finding that using BLEU results in reduced performance, unstable training, and artifacts in the outputs, which makes the results less natural than the results of the model trained with SIM score.

To illustrate that training with SIM can replace

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **Acc** & **PPL** & **r-BLEU** & **s-BLEU** \\ \hline  & \multicolumn{3}{c}{Yelp} \\ \hline D\&R & 89.0 & 362 & 10.1 & 29.1 \\ B-GST & 86.0 & **269** & 14.5 & 35.1 \\ Cycle-Multi & 87.6 & 439 & 19.8 & **55.2** \\ Deep-Latent & 86.0 & 346 & 15.2 & 40.7 \\ Tag\&Gen & 88.7 & 355 & 12.4 & 35.5 \\ DirR-Cycle & 91.7 & 392 & 18.7 & 51.2 \\ DiRR & **94.2** & 292 & **20.7** & 52.6 \\ Copy & 4.1 & 204 & 22.5 & 100.0 \\ Human & 70.7 & 236 & 99.3 & 22.5 \\ \hline  & \multicolumn{3}{c}{Amazon} \\ \hline D\&R & 50.0 & 233 & 24.1 & 54.1 \\ B-GST & 60.3 & **197** & 20.3 & 44.6 \\ Tag\&Gen & **79.9** & 312 & 27.6 & **62.3** \\ DirR-Cycle & 68.4 & 374 & 29.0 & 60.6 \\ DIRR & 62.2 & 205 & **30.1** & 61.3 \\ Copy & 21.1 & 218 & 40.0 & 100.0 \\ Human & 43.0 & 209 & 100.0 & 40.0 \\ \hline  & \multicolumn{3}{c}{IMDb} \\ \hline Cycle-Multi & 77.1 & 290 & N/A & **70.4** \\ DirR-Cycle & 80.5 & 253 & N/A & 64.3 \\ DirR & **83.2** & **210** & N/A & 64.2 \\ Copy & 5.3 & 147 & N/A & 100.0 \\ \hline  & \multicolumn{3}{c}{GYAFC} \\ \hline D\&R & 51.2 & 226 & 14.4 & 27.1 \\ DualRL & 62.0 & 404 & 33.0 & 50.8 \\ DirR-Cycle & **76.2** & 162 & 44.1 & **66.5** \\ DirR & 71.8 & **145** & **46.3** & 59.9 \\ Copy & 15.8 & 147 & 41.5 & 98.5 \\ Human & 84.5 & 137 & 97.8 & 21.5 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Automatic Evaluation. Acc is the accuracy of the sentiment classifier. PPL is the perplexity assigned by the GPT-2 language model. r-BLEU is the BLEU score between the human references and system outputs. s-BLEU is the BLEU score between the source sentences and system outputs. Copy is an oracle which copies the source sentences as outputs. Human denotes the human references.

\begin{table}
\begin{tabular}{l l c c c c} \hline \hline
* & **Model*
* & **Style*
* & **Flu.*
* & **Con.*
* & **Mean*
* \\ \hline \multirow{3}{*}{Yelp} & Cycle & 2.24 & 0.62 & 1.97 & 2.02 \\  & B-GST & **2.42*
* & 0.64 & 2.02 & 2.12 \\  & DirR & **2.42*
* & **0.66*
* & **2.04*
* & **2.14*
* \\ \hline \multirow{3}{*}{Amazon} & Tag\&Gen & 1.98 & 0.87 & 1.95 & 2.19 \\  & B-GST & 2.04 & **0.89*
* & 1.77 & 2.16 \\  & DirR
* & **2.09*
* & 0.87 & **2.10*
* & **2.26*
* \\ \hline \multirow{3}{*}{GYAMC} & D\&R & N/A & 0.40 & 2.13 & 1.66 \\  & DialRL & N/A & 0.51 & 2.23 & 1.88 \\ \cline{1-1}  & DirR
* & N/A & **0.70*
* & **2.34*
* & **2.22*
* \\ \hline \hline \end{tabular}
\end{table}
Table 7: Human Evaluation. **Style*
* denotes style transfer accuracy, **Flu.*
* denotes fluency, **Con.*
* denotes content preservation. **Mean*
* denotes the average of the metrics where the fluency scores are scaled up to be consistent with other scores. *: significantly better than other systems (\(p<0.01\)) according to the mean score.
 

the cycle-consistency loss for content preservation, we fine-tuned DirR-Cycle on SIM to produce a new model, DirR w/o Flu. The difference between DirR and DirR w/o Flu is that the former is additionally trained with our fluency rewards. The results are shown in Table 9, and show two main trends. First, we see that DirR w/o Flu has better fluency and content preservation performance than DirR-Cycle, which shows that the cycle-consistency loss can be replaced by SIM score for content preservation. Second, DirR has better fluency than DirR w/o Flu, showing the effectiveness of our fluency rewards.

We next investigate the effectiveness of using SIM as a reward instead of BLEU. To do this, we train a model, DirR-Bleu, which uses BLEU as the content reward and report the results in Table 9. The results show that using BLEU has larger content preservation as measured by BLEU, but has similar performance when measured by SIM. However, performance on the style transfer accuracy and fluency decreases. We hypothesize that this is because using SIM as a reward gives the model more freedom, allowing the model to have more balanced performance since there is less pressure to copy \(n\)-grams. We also observe more adversarial examples in the outputs of DirR-Bleu. As discussed in Section 3.3, these adversarial examples are generated by injecting a word carrying strong sentiment at the beginning of the output. The model trained with BLEU is more likely to generate these outputs as it will try to avoid breaking up the \(n\)-grams in the source sentences, allowing for a higher BLEU reward. Examples of this behavior is shown in Table 8. Notice that the DirR-Bleu samples start with the word _great_, which is enough to often fool the classifier, but are unnatural.

## 5 Related Work

A main line of work Shen et al. (2017); Hu et al. (2017); Fu et al. (2018); Xu et al. (2018); John et al. (2019) for text style transfer aims to model the conditional distribution of the data with the encoder-decoder architecture. Due to the lack of parallel corpora, inductive biases are designed to make the generation conditioned on both source sentences and specific styles such that the model can rewrite the source texts with the target style while still preserve the content information of the source texts.

Efforts are also made to design training objectives to improve performance. For example, Backtranslation Zhang et al. (2018); Prabhumoye et al. (2018), denoising auto-encoding Lample et al. (2019) and the cycle-consistency loss Luo et al. (2016)

\begin{table}
\begin{tabular}{l l r c} \hline \hline
**Model** & **Text** & **self-BLEU** & **self-SIM** \\ \hline Source & this was my first stop in looking for a wedding dress . & 100.0 & 100.0 \\ DirR-Bleu & **great** this was my first stop in looking for a wedding dress . & 91.2 & 95.2 \\ DirR & this was my best stop in looking for a wedding dress . & 64.8 & 81.9 \\ \hline source & just a frozen patty cooked like a home one . & 100.0 & 100.0 \\ DirR-Bleu & **great** a frozen patty cooked like a home one . & 88.0 & 94.6 \\ DirR & just a great patty cooked like a home one . & 70.7 & 88.5 \\ \hline source & wendy ’s has been know to be cheap with their drink & 100.0 & 100.0 \\  & refills for years . & 100.0 & 100.0 \\ DirR-Bleu & **great** wendy ’s has been know to be cheap with their drink & 93.0 & 97.5 \\ DirR & refully for years . & 57.2 & 84.9 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Comparison of using SIM and BLEU as the content preservation reward. Samples are from the Yelp dataset. The metrics self-BLEU and self-SIM are calculated between the source sentences and system outputs.

\begin{table}
\begin{tabular}{l l r r r} \hline \hline
**Model** & **Acc** & **PPL** & **s-BLEU** & **s-SIM** \\ \hline DirR-Cycle & 91.7 & 392 & 51.2 & 76.2 \\ DirR w/o Flu & 92.1 & 348 & 51.4 & 79.8 \\ DirR-Bleu & 91.3 & 315 & **59.4** & **81.8** \\ DirR & **94.2** & **292** & 52.6 & 81.6 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Ablation and Comparative Study on Yelp Dataset. Acc is the accuracy of the sentiment classifier. PPL is the perplexity assigned by the GPT-2 language model. self-BLEU (s-BLEU) and self-SIM (s-SIM) are computed between the source sentences and outputs.

 

2019; Dai et al., 2019; Pang and Gimpel, 2019) have been shown effective for improving the model performance. Li et al. (2018) proposes a retrieve-based pipeline, which contains three stages, namely, delete, retrieve and generate. Sudhakar et al. (2019) extends this pipeline by using GPT Radford et al. (2018) as the generator. Compared to these methods, we propose a more direct and effective approach to encourage semantic-preserving transfer by directly measuring the semantic similarity of the source texts and system outputs.

Recently, other works have been proposed for unsupervised text style transfer Jin et al. (2019); Lai et al. (2019); Wu et al. (2019); Li et al. (2020). He et al. (2020) proposes a probabilistic view which models the non-parallel data from two domains as a partially observed parallel corpus. Madaan et al. (2020) proposes a tag-and-generate pipeline, which firstly identifies style attribute markers from the source texts, then replaces them with a special token, and generates the outputs based on the tagged sentences. Zhou et al. (2020) focuses on exploring the word-level style relevance which is assigned by a pre-trained style classifier. They propose a reward for content preservation which is based on the weighted combination of the word embeddings of the source texts and system outputs. Compared to this reward, our proposed content reward is specifically designed for semantic similarity and pre-trained on large corpora, which makes it more robust across different datasets.

## 6 Conclusion

In this paper, we propose a direct approach of improving content preservation for text style transfer by leveraging a semantic similarity metric as the content reward. Using a large pre-trained language model (GPT-2) with our proposed rewards that target the different aspects of the output quality, our approach achieves strong performance on both automatic and human evaluation. Recently, several semantic similarity metrics Zhao et al. (2019); Sellam et al. (2020); Gao et al. (2020) based on pre-trained language models have shown promising results. Introducing these metrics in our proposed method as the content preservation reward may bring further improvements.

Moreover, we identify several problems in the commonly used automatic evaluation metrics and datasets, and propose several practical strategies to mitigate these problems, which makes these metrics more effective rewards for model training. Considering the weaknesses of the automatic metrics presented in this work, we believe that more rigorous discussion and investigation on the criteria of "successful transferring" is essential for this field of work. Since existing works mostly relied on model-based metrics to determine the success of style transfer models, methods such as adversarial training could be introduced to make the model-based metrics more robust and faithful indicators of the success of style-transferring, which would be beneficial for both model training and evaluation.

## References

* Dai et al. (2019) Ning Dai, Jianze Liang, Xipeng Qiu, and Xuan-Jing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5997-6007.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fu et al. (2018) Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
* Gao et al. (2020) Yang Gao, Wei Zhao, and Steffen Eger. 2020. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1347-1354, Online. Association for Computational Linguistics.
* He et al. (2020) Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. In _International Conference on Learning Representations_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_, pages 1587-1596. JMLR. org.
* Jin et al. (2019) Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. Imat: Unsupervised text 

attribute transfer via iterative matching and translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3088-3100. Cited by: SS4.
* V. John, L. Mou, H. Bahuleyan, and O. Vechtomova (2019)Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 424-434. External Links: Link, Document Cited by: SS2.
* A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov (2017)Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 427-431. Cited by: SS2.
* Y. Kim (2014)Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, pp. 1746-1751. External Links: Link, Document Cited by: SS2.
* C. Lai, Y. Hong, H. Chen, C. Lu, and S. Lin (2019)Multiple text style transfer by using word-level conditional generative adversarial network with two-phase training. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3570-3575. External Links: Link, Document Cited by: SS2.
* G. Lample, S. Subramanian, E. Smith, L. Denoyer, M. Ranzato, and Y. Boureau (2019)Multiple-attribute text rewriting. In International Conference on Learning Representations, External Links: Link Cited by: SS2.
* J. Li, R. Jia, H. He, and P. Liang (2018)Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 1865-1874. External Links: Link, Document Cited by: SS2.
* S. Li, D. Lei, P. Qin, and W. Yang Wang (2019)Deep reinforcement learning with distributional semantic rewards for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 6038-6044. External Links: Link, Document Cited by: SS2.
* Y. Li, C. Li, Y. Zhang, X. Li, G. Zheng, L. Carin, and J. Gao (2020)Complementary auxiliary classifiers for label-conditional text generation. Proceedings of the AAAI Conference on Artificial Intelligence34 (05), pp. 8303-8310. Cited by: SS2.
* F. Luo, P. Li, J. Zhou, P. Yang, B. Chang, X. Sun, and Z. Sui (2019)A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5116-5122. External Links: Link, Document Cited by: SS2.
* A. Madan, A. Setlur, T. Parekh, B. Poczos, G. Neubig, Y. Yang, R. Salakhutdinov, A. W. Black, and S. Prabhunoye (2020)Politeness transfer: a tag and generate approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1869-1881. External Links: Link, Document Cited by: SS2.
* M. Pagliardini, P. Gupta, and M. Jaggi (2018)Unsupervised learning of sentence embeddings using compositional n-gram features. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 528-540. External Links: Link, Document Cited by: SS2.
* R. Yuanzhe Pang and K. Gimpel (2019)Un supervised evaluation metrics and learning criteria for non-parallel textual transfer. In Proceedings of the 3rd Workshop on Neural Generation and Translation (WNGT 2019), Cited by: SS2.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 311-318. External Links: Link, Document Cited by: SS2.
* May 3, 2018, Conference Track Proceedings, MenarXiv preprint arXiv:1802.08815v1. Cited by: SS2.
* S. Prabhumoye, Y. Tsvetkov, R. Salakhutdinov, and A. W. Black (2018)Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 866-876. External Links: Link, Document Cited by: SS2.
* A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever (2018)Improving language understanding by generative pre-training 

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI Blog_, 1(8):9.
* Rao and Tetreault [2018] Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 129-140. New Orleans, Louisiana. Association for Computational Linguistics.
* Sellam et al. [2020] Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Sharma et al. [2017] Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. _CoRR_, abs/1706.09799.
* Shen et al. [2016] Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
* Shen et al. [2017] Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In _Advances in neural information processing systems_, pages 6830-6841.
* Sudhakar et al. [2019] Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019. "transforming" delete, retrieve, generate approach for controlled text style transfer. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3260-3270.
* Wieting et al. [2016] John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
* Wieting et al. [2019a] John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019a. Beyond BLEU:training neural machine translation with semantic similarity. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4344-4355, Florence, Italy. Association for Computational Linguistics.
* Wieting et al. [2019b] John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019b. Simple and effective paraphrastic similarity from parallel translations. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4602-4608.
* Williams [1992] Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_, 8(3-4):229-256.
* Wolf et al. [2019] Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. _arXiv preprint arXiv:1901.08149_.
* Wu et al. [2019] Chen Wu, Xuancheng Ren, Fuli Luo, and Xu Sun. 2019. A hierarchical reinforced sequence operation method for unsupervised text style transfer. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4873-4883, Florence, Italy. Association for Computational Linguistics.
* Xu et al. [2018] Jingjing Xu, Xu Sun, Qi Zeng, Xiaodong Zhang, Xuacheng Ren, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 979-988.
* Yang et al. [2018] Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018. Unsupervised text style transfer using language models as discriminators. In _Advances in Neural Information Processing Systems_, pages 7287-7298.
* Zhang* et al. [2020] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhang et al. [2018] Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018. Style transfer as unsupervised machine translation. _CoRR_, abs/1808.07894.
* Zhao et al. [2019] Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earthl move distance. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 563-578, Hong Kong, China. Association for Computational Linguistics.
* Zhou et al. [2020] Chulun Zhou, Liangyu Chen, Jiachen Liu, Xinyan Xiao, Jinsong Su, Sheng Guo, and Hua Wu. 2020. Exploring contextual word-level style relevance for unsupervised style transfer. In _Proceedings of the 

58th Annual Meeting of the Association for Computational Linguistics_, pages 7135-7144, Online. Association for Computational Linguistics.