Title: RL-ST: REINFORCING STYLE, FLUENCY AND CONTENT PRESERVATION FOR UNSUPERVISED TEXT STYLE TRANSFER

Abstract: Unsupervised text style transfer is the task of re-writing text of a given style into a target style without using a parallel corpus of source style and target style sentences for training. Style transfer systems are evaluated on their ability to generate sentences that 1) possess the target style, 2) are fluent and natural sounding, and 3) preserve the non-stylistic parts (content) of the source sentence. We train a reinforcement learning (RL) based unsupervised style transfer system that incorporates rewards for the above measures, and describe novel rewards shaping methods for the same. Our approach does not attempt to disentangle style and content, and leverages the power of massively pre-trained language models as well as the Transformer. Our system significantly outperforms existing state-of-art systems based on human as well as automatic evaluations on target style, fluency and content preservation as well as on overall success of style transfer, on a variety of datasets.

1. INTRODUCTION:

Text style transfer is an important natural language generation problem, since it has wide applications across different domains. It has been used to adapt texts to specific artistic writing styles (Jhamtani et al., 2017) , make texts formal or informal (Rao & Tetreault, 2018) , alter sentiment1 (Hu et al., 2017; Shen et al., 2017; Fu et al., 2018; Li et al., 2018) , rewrite factual sentences into romantic or humorous ones (Li et al., 2018) , generate poetry (Yang et al., 2018a) , personalize dialogue systems (Zhou et al., 2017) and obfuscate gender in social media posts (Reddy & Knight, 2016) .

Most recent works perform unsupervised style transfer due to the unavailability of parallel style corpora. Most previous works on unsupervised style transfer attempt to disentangle the stylistic parts (hereby, 'attributes') and non-stylistic parts (hereby, 'content') of texts, and then modify the attributes while preserving the content. Some of these works encode style and content in separate latent representations, and decode the style-dependent output from these representations (Fu et al., 2018; Shen et al., 2017; Hu et al., 2017; Yang et al., 2018b) . A few others explicitly identify attribute and content words from input texts and then train models to generate target style sentences from the content (Li et al., 2018; Sudhakar et al., 2019; Wu et al., 2019c) .

More recently, Lample et al. (2018b) showed that many previous works that attempt to disentangle content and style in latent representation spaces are unsuccessful in doing so in practice. Further, these approaches are prone to instability of training, low sample efficiency and consequently poor quality outputs. Approaches where attribute words are explicitly removed from the sentence require heuristics and thresholds to decide attribute words, which makes them sensitive to and require manual setting of thresholds. This causes core content words to be incorrectly deleted in some cases, and source attribute words to be incorrectly preserved in others. Moreover, in some of these works (Li et al., 2018; Sudhakar et al., 2019; Wu et al., 2019b) , the final output generators are provided with only the content information of the input sentence and not the attributes. This leads to awkward outputs where the model inserts target attribute words that are either not suitable to the content, or are wrongly positioned. However, it has been observed that these approaches are more controllable, easier to train and produce better quality outputs than approaches based on latent representations. A few recent works (Lample et al., 2018b; Dai et al., 2019; Luo et al., 2019) avoid style-content disentanglement altogether.

While most works use recurrent networks for encoding and decoding, transformers (Vaswani et al., 2017) have been shown to be significantly better for the task (Dai et al., 2019; Sudhakar et al., 2019; Wu et al., 2019b) . Sudhakar et al. (2019) show significant gains over previous state of the art by leveraging the combined power of transformers and massively pre-trained language models, by using a decoder-only GPT (Radford et al.) . In doing so, they do away with the traditional encoderdecoder mechanism.

Finally, RL has been used in previous works to leverage the use of non-differentiable training objectives and overcome the lack of parallel corpora. Xu et al. (2018) use a cycled RL approach where a neutralization model first disentangles the content and attributes, and then passes the content to an emotionalization model. However, cascading errors propagated from the neutralization to the emotionalization due to a discretization of embeddings to words in between the two, leads to poor quality outputs. Gong et al. (2019) use adversarially trained discriminators whose feedbacks are used as rewards by a generator, and Luo et al. (2019) train a dual RL system wherein separate models exist for source-to-target prediction and target-to-source prediction. Style and content rewards are built into this dual structure. However their model tends to be majority-biased towards certain attributes (such as 'happy' and 'loved' on the task of sentiment transfer), which are abruptly inserted in the output sentences without being meaningfully transferred versions of the source attributes. For instance, one would not find it meaningful for the source sentence 'so i asked for the card to be refunded' to be mapped to the target sentence 'so i loved the credit card to be happy'.

Taking into consideration the drawbacks and strengths of previous works, our contributions are as follows: we introduce a novel RL based model for style transfer that 1) uses the decoder-only GPT (Radford et al.) in order to leverage the power of transformers and massively pre-trained language models, 2) directly learns mappings from source to target sentences without any disentanglement, 3) does not require any parallel corpus but is instead warm-started by using a synthetic parallel corpus generated by the trained GST (Sudhakar et al., 2019) and 4) provides for controllable generation by allowing trade-offs between style, content retention and fluency. Our approach significantly outperforms current state-of-art systems based on human evaluation as well as on evaluations using automatic metrics.

In the interest of reproducibility, we publish all our code, data and results for this work on our Github repository, the link to which will be added here in the camera-ready version if accepted.

2. OUR APPROACH:

We assume a dataset D = {(x 1 , s 1 ), ..., (x m , s m )} where each sentence x i is associated with a specific style s i ∈ S. For instance, for the task of sentiment transfer, S = {'Positive', 'Negative'}. We then aim to learn the conditional distribution P (y|x, s tgt ) such that the style of y is s tgt , and y's content is similar to that of x. We introduce the Reinforcement Learning based Style Transformer (hereby, RL-ST). RL-ST takes as input the source sentence x of style s src and generates the output sentence ŷ of style s tgt . More formally, it learns:

EQUATION

Model: The architecture of RL-ST is a decoder-only Transformer, based on the implementation of the Generative Pre-trained Transformer (Radford et al.) (hereby, GPT). Similar to Sudhakar et al. (2019) , we do away with the notion of an encoder-decoder mechanism and use a decoder-only transformer, pre-trained using a language model like training. GPT has masked attention heads that enable it to look only at the tokens to its left, and not to those to its right. In order to address the coldstart problem that typically causes convergence issues during RL, we warm-start the RL training by pre-training RL-ST on a synthetically generated parallel corpus. This corpus is obtained by using the B-GST trained by Sudhakar et al. (2019) on our non-parallel corpus, and the pre-training is performed via Maximium Likelihood Estimate (MLE) (Ranzato et al., 2015; Paulus et al., 2018) over this synthetic parallel corpus. RL-ST is then fine-tuned using Policy Gradient, by providing rewards for style, content preservation and fluency. Appendix A.1 provides further details of architecture used during training of RL-ST.

Sampling: RL-ST optimizes a policy using the policy gradient, to maximize a long term reward.

In doing so, it uses the notion of state-action pairs, with rewards assigned to such pairs. The parameters of the model θ define a policy π which maps a state (the input to the model until each time step t) to an action (the next output word to generate). The action is sampled from the model's softmax distribution using a sampling method. We use a 'top-p sampling' (alternatively, 'nucleus sampling') (Holtzman et al., 2019) for the same. This sampling method samples (using the softmax output probability distribution) a token from the set of top tokens that make up a cumulative softmax probability of p. Unlike beam search which exploits the output probability distribution, top-p sampling is more geared towards exploration. For each sentence x in the RL-training set, we perform the above sampling K different times to ensure sufficient exploration. The state corresponding to the output timestep t of the k th sampling round of sentence x is represented as st k t . The corresponding action is represented as a k t . Each such state-action pair receives a reward R k t composed of 3 different rewards -the style, content and fluency rewards as elaborated in ensuing sections. By our formulation, each ŷk t is simply the action a k t . Policy Gradient: The gradient of the expected reward E[R] of the generated sequence ŷ can be estimated by the REINFORCE policy gradient algorithm (Williams, 1992) adapted to the token-level for RL-ST as:

EQUATION

where R k t is described in equation 9 and represents the reward for the k th sequence sampled out of a total of K sampled sequences for each training example, and τ is the maximum length of the decoder output.

2.1. REWARD COMPUTATION:

In sequence generation problems such as style transfer, typically rewards are only available once the output sequence has been generated completely. Due to this, a fundamental problem of reward assignment to intermediate tokens arises i.e., obtaining a value for R k t . A few 'reward shaping' techniques have been used to alleviate this. One popular technique is to 'roll out' (Yu et al., 2017) the partial sequence generated up till timestep t, ŷk 1:t , by sampling the rest of the sequence ŷk t+1:τ (Yu et al., 2017) , where τ is the maximum sequence length of the decoder. Due to the need to sample τ -t tokens at every timestep t, roll-out based methods are computationally expensive. Hence, we propose a novel method to leverage transformer attention weights to assign token level style rewards. We also use a language model in a novel way to provide token level fluency rewards. To the best of our knowledge ours is the first work on style transfer work that leverages carefully designed reward shaping in the manner that we do. This, combined with our warm starting mechanism, provides a way to completely circumvent roll-out. During the k th sampling round using the input sequence x, we first generate the whole of ŷk from the model at one go, and then proceed to assign token-level rewards to each ŷk t . Transformer Attention based Style Reward: We use a pre-trained, self-attention based style classifier to decide the style reward for a generated sentence ŷ. The style classifier takes as input ŷ and defines a distribution over style labels s:

EQUATION

where enc(ŷ) t is an encoding of ŷt , and α t is the self-attention score corresponding to enc(ŷ) t learned by the classifier in assigning probabilities for each style s j , and θ CLS is the model parameter.

For the classifier, we use the same Delete Transformer (DT) as used by Sudhakar et al. (2019) . This is a BERT-based (Devlin et al., 2018) classifier, which has 144 sets of self-attention heads. We extract a representative head-layer pair < h, l > out of these, using the process described by Sudhakar et al. (2019) and choose α to be the self-attention weights of < h, l >, corresponding to the input tokens of ŷ. We then choose attribute words from ŷ based on their α scores, since attribute words are paid higher attention or importance than content words are by a style classifier (Feng et al., 2018; Xu et al., 2018; Sudhakar et al., 2019) . The top γ|x| tokens of x are treated as attributes, based on their α scores. γ is a parameter that can be tuned to the dataset and denotes the proportion of words in a sentence that can be considered attributes, while |x| denotes the number of tokens in x. Further, the style classifier is used to decide the style of ŷ according to:

EQUATION

The reward assignment is as follows:

EQUATION

where,

EQUATION

Fluency Reward: We train a fresh language model LM over the entire training dataset using GPT's architecture and pre-training, which we then use to determine the fluency of generated sentences. LM generates a probability distribution P (ŷ t |y 1:t-1 ; θ LM ) over tokens at timestep t, given the tokens generated in previous timesteps. The reward assignment is as follows (where b is a baseline, set such that words having a LM probability lower than b will get penalized with a negative reward):

EQUATION

Unlike works such as Gong et al. (2019) which use the perplexity of the entire sentence, we provide a fluency reward at the token-level.

Content Reward: We use the BLEU score (Papineni et al., 2001) between the input sentence x and the generated sentence ŷ to calculate the content reward. The reward assignment is as follows:

EQUATION

As GST is trained on a reconstruction loss and RL-ST is warm-started with GST, it is already strongly biased to retaining content. Hence, setting the same (weak) content reward for all tokens works well enough.

Overall Reward: The overall reward (R) is a weighted sum of the above rewards:

R k t = λ S RS k t + λ C RC k t + λ F RF k t (9)

Figure 1 shows a training example with rewards and describes the training algorithm of RL-ST.

Inference: During inference, we decode the output using beam search, with a beam width of 20.

Using the classifier described in equation 4, we choose the beam with the highest classifier score as the final output sentence.

3. EXPERIMENTS:



3.1. DATASETS:

We show our results on the YELP and CAPTIONS dataset as used by Li et al. (2018) , and retain the same train-dev-test split as they do. We also show results on the GYAFC dataset as used and released by (Rao & Tetreault, 2018) . All of these datasets are used in a non-parallel manner. YELP is used for sentiment transfer, CAPTIONS is used for transfer of factual sentences to romantic and humorous ones, and GYAFC is used for formality transfer. Human reference outputs are available on all test sets. Further descriptions of these datasets can be found in Appendix Section A.2, and train-dev-test statistics of these datasets are shown in Appendix Table 5 .

3.2. COMPARISON WITH PREVIOUS WORKS:

We compare our models with the following 13 models on unsupervised style transfer from previous work: Cross Aligned (CA) (Shen et al., 2017) , Style Embedding (SE) (Fu et al., 2018) , Multi 

4. RESULTS:

We evaluate our models and the previous models using automatic evaluation methods as well as by human evaluation.

Automatic Evaluation: To measure target style strength, we train FastText2 (Joulin et al., 2017) classifiers on our style datasets, keeping the same train-dev-test split intact as is in Table 5 and use these classifiers as oracles to judge style of output sentences (AC). These classifiers achieve accuracies of 96.5%, 89.5% and 80.5% on the test sets of YELP, GYAFC and CAPTIONS respectively. For content preservation, we calculate the average BLEU (BL R ) scores of the output with respect to the human reference sentences. Fluency is estimated by finetuning pre-trained OpenAI GPT-2 (Radford et al., 2019 ) models (different from any of the GPT models used in this work) on the training sets and using it to obtain the perplexity (PL) of the output sentences. The GPT-2 models achieve perplexities of 21.42, 52.5 and 42.91 on the test sets of YELP, GYAFC and CAPTIONS respectively. We also calculate the harmonic mean (HM) and geometric mean (GM) of AC and BL R . These results are presented in Table 1 .

Human Evaluation: We obtain human evaluations from crowd workers on MTurk3 on pairs of model outputs, where one of the models in each pair is from previous work and the other is our model, RL-ST. Five top performing state-of-art models whose results are available on each of the datasets, were chosen based on our automatic evaluations as well as human evaluations by previous works (Luo et al., 2019; Sudhakar et al., 2019; Wu et al., 2019c; a) . For each example, three separate annotators who are not told which model is ours, are asked to choose which of the model's outputs is better (or 'None' if both are poor) on style (Sty.), content (Cont.) and fluency (Flu.) as well as overall style transfer (All). They are all native English speakers from North America and are familiar with the datasets. These results are presented in 

4.1. ANALYSIS AND DISCUSSION:

As has been observed by most previous works, the automatic evaluations in Table 1 show that many previous works trade-off target style match and content retention. It is easy to achieve very high numbers on either of AC or BL R . A model that simply copies the input sentence will achieve high BL R and a model that simply chooses a random target training sentence will achieve high AC score. SE has very low AC but considerably high BL R while BT has a high AC but considerably low BL R . However, RL-ST achieves very high target style accuracy but not at the cost of content retention -it achieves considerably good BL R scores too. HM and GM scores indicate how well the models perform on both style and content. Our model (RL-ST) ranks highest on both these scores across datasets (except on HM for CAPTIONS, where it ranks the second highest) as well as achieves low PL, outperforming even the average scores of all the human references on HM, GM and PL on YELP. However, automatic metrics do not capture nuances that human evaluation does.

For instance, on the Yelp dataset, DRL is biased towards frequently using the attributes 'loved' and 'happy' in its positive outputs, and PTO over-uses the word 'delighted' even in sentences where it is not meaningful to. The classifier still awards these outputs high AC scores. Further, model-based metrics such as AC and PL are also sensitive to the quality of their training data available.

From human evaluations in Table 2 , we see our model outperforms previous state-of-art models by a good margin on all metrics across all datasets. On the Overall scores (All), we outperform previous state-of-the art models by 19.8%, 24.5% and 19% on YELP, GYAFC and CAPTIONS respectively, averaging across the top performing models considered for human evaluation in Table 2 for each of these datasets. From manual inspection we observe that RL-ST performs better than previous state-of-art models in the following ways: 1) generates sentences that are more natural sounding, YELP (Positive to Negative) YELP (Negative to Positive) steve was professional and found exactly the right unit to fit in our space they tried to take advantage of me because i am young . steve was rude and didn't have the right unit to fit in our space .

they take great care of me because i am young . GYAFC (Formal to Informal) GYAFC (Informal to Formal)

do not approach her and let her know that you find her looks very attractive.

well that is just the way it is I guess.

don't approach her and let her know that you like her . that is just the way it is , i would advise . CAPTIONS (Factual to Romantic) CAPTIONS (Factual to Humorous)

young man performing bicycle trick on loading dock near dumpsters .

young man performing bicycle trick on loading dock near dumpsters . young man performing bicycle trick on ramp near a group of people enjoying life .

young man performing bicycle trick on dock near a crowd of aliens .

Table 3 : Examples of generated sentences by RL-ST. Each cell has the source sentence first and the generated sentence second.

retaining core content better while making only necessary stylistic changes, 2) maintains consistent context across longer sentences, 3) performs well on sentences in which style transfer is not limited to simple localized edits, 4) maintains consistency of style even for certain input sentence structures (e.g., sentences having multiple attribute words -it's nice but it's too expensive) which cause other models to produce outputs having inconsistent style (for e.g., it's too expensive , but it's worth it), 5) produces output sentences having appropriate and meaningful attributes, many of which the model has not observed at training time, and 6) does away with redundancy in output sentences that is commonly observed in previous works (e.g., the food is good, the service is great and the food is good .)

Table 3 shows examples of our outputs for the three datasets, and Appendix Table 6 shows more results of our models and compares it with those of other models.

Failure Cases: One observable behavior of RL-ST is that it sometimes simply retains conjunction words of the source sentence (such as and, but) instead of adapting it to the output. For example, their chips are great , but their salsa is really good. The second type of failure case occurs when analogies are used to indicate a certain sentiment in the source sentence, and sentiment attributes are not directly used (e.g., they only received one star because you have to provide a rating ). In these cases, the model finds it hard to identify and replace these analogies (they also enjoy one star because you have to provide a rating).

4.2. ABLATION STUDIES:

We perform five ablation studies on RL-ST using the YELP dataset. When ablating over a particular aspect of training, all other aspects are kept fixed and the same as those of RL-ST in Table 1 . Table 4 shows results of these ablations, whose explanations are given below.

RL and MLE::

We compare the performance obtained by using only RL without MLE pre-training (RL-Only), only MLE pre-training without RL (MLE-Only) and the combined model (RL+MLE). We see that warm-starting using MLE significantly boosts performance.

Disentanglement: We also study the effects of providing RL-ST only the content of the source sentence during training and inference (Cont-Only), as against providing it with the entire source sentence (Full-Src) during training and inference. The results show that providing the model the full sentence including source attributes is more beneficial than giving it only the content in the input. Sample efficiency: While previous works that use RL for style transfer require large training sets and consequently large training time, RL-ST is highly sample efficient, requiring only a fraction of the training set to boost performance significantly. We ablate on training set sizes of 1K, 2K, and 4K samples out of the training set of 450K samples. Rewards: We distill the effects of using only the style reward, only the fluency reward (FluencyR-Only) and the combined reward (Combined). The results show that the style and fluency rewards are indeed successfully able to control for style and fluency respectively, as expected. We also ablate on two versions of the style reward -one in which we use attention scores to assign rewards as described in equation 5 (StyleR-Only-Attn), and the other in which we assign uniform style rewards to all tokens regardless of attention scores (StyleR-Only-Uniform). StyleR-Only-Attn yields outputs of superior style accuracy. 

5. RELATED WORK:

This brief section refers to a few more works not covered in 1. One category of previous works is based on unsupervised machine translation (Artetxe et al., 2018; Lample et al., 2018a; b) of which most approaches use a back-translation based system (Zhang et al., 2018; Lample et al., 2019) . In a slightly different approach, Prabhumoye et al. ( 2018) use back-translation from English to French and back to English in order to get an intermediate representation that has reduced style, and then generate style-specific outputs by training adversarially. John et al. (2019) attempt to learn disentangled representations for style and content using a system that incorporates auxiliary multi-task and adversarial objectives, for style prediction and bag-of-words prediction, respectively. Zhao et al. (2018) use a GAN-like training approach, with a style discrepancy loss and a cycled consistency loss to transfer from sentences with arbitrary unknown styles to known target styles. Wu et al. (2019a) use a hierarchical reinforcement operations method (Point-then-Operate) wherein a high-level agent iteratively 'points' to positions in the sentence and a low-level agent 'operates' by altering the sentence at these positions. Liu et al. (2019) perform revision in continuous space by which explicit content disentanglement is not needed, and neither is adversarial training. They control for finegrained multi-attributes such as length of the output sentence. Pang & Gimpel (2018) examine the complimentarity of the three style transfer metrics discussed earlier, trade-offs between them, and a common metric that summarizes them into one score using the geometric mean. Tikhonov et al. ( 2019) discuss significant problems with standard assessment using automatic metrics of style and content retention. They claim that the nature of style transfer itself lends a specific dependency between the two metrics, which can be manipulated. Hence, human evaluation is imperative.

6. CONCLUSION:

We present an RL-based, sample efficient style transfer model that outperforms current state-of-art systems on human as well as automatic evaluations. The approach is generalizable across a variety of style transfer tasks, as we show with diverse datasets. We show the merits of directly learning to map source to target sentences without disentanglement, shaping RL rewards efficiently, and leveraging the power of massively pre-trained transformer-based language models. in each block. All internal states (keys, queries, values, word embeddings, positional embeddings) are 768-dimensional. Input text is tokenized using Byte-Pair Encoding (BPE).

In equation 9, λ S = 1, λ C = 0.3 and λ F = 1 for all 3 datasets.

A.2 DATASETS DESCRPTION AND STATISTICS:

Following are brief descriptions of the datasets we use, borrowed from works that use them previously.

YELP: Each example is a sentence from a business review on Yelp, and is labeled as having either positive or negative sentiment (Li et al., 2018) . The task is to transfer sentences of positive to negative sentences and vice-versa. #1 YELP (Positive to Negative) SRC steve was professional and found exactly the right unit to fit in our space . DRL steve was unprofessional and found exactly the right unit to fit in our space UnMT manager was unprofessional and left exactly the off unit to replace in our space PTO steve was professional and the horrible unit to fit in our space . MLM steve was rude and found only the wrong unit was not in our space RL-ST steve was rude and did n't have the right unit to fit in our space .

# 

:

Similar to previous works, we use a broad socio-linguistic definition of style that includes sentiment

https://fasttext.cc/

https://www.mturk.com/

https://github.com/huggingface/pytorch-pretrained-BERT

https://github.com/openai/finetune-transformer-lm

https://www.smashwords.com/

Figures and Tables:

Figure 1: a) An example from YELP for the task of sentiment transfer. <POS> represents positive target style, <SRC START> indicates the start of the input sentence, <START> indicates the start of the output sentence and <END> is end-of-sentence marker. b) RL-ST's training algorithm



Table content: <table><tr><td/><td/><td/><td>YELP</td><td/><td/><td/><td/><td>GYAFC</td><td/><td/><td/><td/><td>CAPTIONS</td><td/><td/></tr><tr><td>Model</td><td>AC</td><td>BL R</td><td>PL ↓</td><td>HM</td><td>GM</td><td>AC</td><td>BL R</td><td>PL ↓</td><td>HM</td><td>GM</td><td>AC</td><td>BL R</td><td>PL ↓</td><td>HM</td><td>GM</td></tr><tr><td>SE</td><td>9.5</td><td>53.4</td><td>115.9</td><td>16.1</td><td>22.5</td><td>30.3</td><td>22.3</td><td>163.1</td><td>25.7</td><td>26.0</td><td>54.3</td><td>27.8</td><td>80.3</td><td>36.8</td><td>38.9</td></tr><tr><td>BT</td><td>95.5</td><td>25.6</td><td>28.5</td><td>40.4</td><td>49.4</td><td>51.5</td><td>18.7</td><td>147.2</td><td>27.4</td><td>31.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MD</td><td>50.6</td><td>42.9</td><td>205.5</td><td>46.4</td><td>46.6</td><td>26.1</td><td>26.8</td><td>231.5</td><td>26.4</td><td>26.4</td><td>66.0</td><td>25.0</td><td>40.5</td><td>36.3</td><td>40.7</td></tr><tr><td>UnP</td><td>53.3</td><td>46.3</td><td>294.2</td><td>49.6</td><td>49.7</td><td>68.5</td><td>12.3</td><td>133.9</td><td>20.8</td><td>29.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>RC</td><td>90.1</td><td>37.3</td><td>12.5</td><td>52.8</td><td>58.0</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>DO</td><td>85.9</td><td>44.4</td><td>75.8</td><td>58.5</td><td>61.7</td><td>26.0</td><td>42.4</td><td>183.1</td><td>32.2</td><td>33.2</td><td>82.3</td><td>35.5</td><td>52.5</td><td>49.6</td><td>54.1</td></tr><tr><td>DR</td><td>87.8</td><td>45.6</td><td>90.0</td><td>60.0</td><td>63.3</td><td>57.4</td><td>41.2</td><td>176.4</td><td>48.0</td><td>48.6</td><td>95.0</td><td>38.1</td><td>28.8</td><td>54.4</td><td>60.1</td></tr><tr><td>G-GST</td><td>79.6</td><td>57.1</td><td>64.4</td><td>66.5</td><td>67.4</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>67.8</td><td>40.2</td><td>49.2</td><td>50.5</td><td>52.2</td></tr><tr><td>B-GST</td><td>86.7</td><td>57.1</td><td>38.6</td><td>68.9</td><td>70.4</td><td>69.0</td><td>47.4</td><td>89.0</td><td>56.2</td><td>57.2</td><td>70.5</td><td>42.7</td><td>28.9</td><td>53.2</td><td>54.9</td></tr><tr><td>UnMT</td><td>97.2</td><td>53.6</td><td>67.4</td><td>69.1</td><td>72.2</td><td>67.5</td><td>44.9</td><td>112.6</td><td>53.9</td><td>55.1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>DRL</td><td>89.2</td><td>59.5</td><td>53.2</td><td>71.4</td><td>72.9</td><td>60.1</td><td>44.3</td><td>321.1</td><td>51.0</td><td>51.6</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>PTO</td><td>86.2</td><td>61.9</td><td>55.6</td><td>72.1</td><td>73.1</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MLM</td><td>91.5</td><td>59.6</td><td>75.1</td><td>72.2</td><td>73.9</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>RL-ST</td><td>98.1</td><td>58.7</td><td>35.9</td><td>73.5</td><td>75.9</td><td>84.2</td><td>52.3</td><td>64.8</td><td>64.5</td><td>66.3</td><td>98.8</td><td>36.5</td><td>25.7</td><td>53.3</td><td>60.1</td></tr><tr><td>H</td><td>75.0</td><td>70.3</td><td>57.7</td><td>72.5</td><td>72.6</td><td>86.4</td><td>65.4</td><td>91.9</td><td>74.5</td><td>75.2</td><td>80.5</td><td>100.0</td><td>41.4</td><td>89.2</td><td>89.7</td></tr></table>

Automatic evaluation results. AC = Style Accuracy; BL R = Average BLEU score w.r.t human reference sentences; PL = Perplexity; HM = Harmonic Mean of (AC, BL R ); GM = Geometric Mean of (AC, BL R ); RL-ST is our model; H is Human reference. Lower PL is better. A '-' indicates that we could not obtain results for the model on the dataset.

Table content: <table><tr><td/><td/><td>YELP</td><td/><td/><td/><td/><td>GYAFC</td><td/><td/><td/><td colspan="2">CAPTIONS</td><td/><td/></tr><tr><td>Model</td><td>Cont.</td><td>Flu.</td><td>Sty.</td><td>All</td><td>Model</td><td>Cont.</td><td>Flu.</td><td>Sty.</td><td>All</td><td>Model</td><td>Cont.</td><td>Flu.</td><td>Sty.</td><td>All</td></tr><tr><td>PTO</td><td>29.7</td><td>21.3</td><td>31.4</td><td>32.4</td><td>DO</td><td>19.9</td><td>20.3</td><td>20.1</td><td>22.3</td><td>DO</td><td>40.3</td><td>35.3</td><td>36.4</td><td>33.0</td></tr><tr><td>RL-ST</td><td>51.3</td><td>59.4</td><td>48.6</td><td>50.9</td><td>RL-ST</td><td>62.3</td><td>60.1</td><td>59.9</td><td>60.7</td><td>RL-ST</td><td>46.5</td><td>52.2</td><td>50.3</td><td>54.5</td></tr><tr><td>None</td><td>19.0</td><td>19.3</td><td>20</td><td>16.7</td><td>None</td><td>17.8</td><td>19.6</td><td>19</td><td>17</td><td>None</td><td>13.2</td><td>12.5</td><td>13.3</td><td>12.4</td></tr><tr><td>B-GST</td><td>37.1</td><td>27.3</td><td>27.2</td><td>28.6</td><td>B-GST</td><td>33.3</td><td>30.1</td><td>29.4</td><td>28.1</td><td>B-GST</td><td>35.8</td><td>39.7</td><td>38.9</td><td>33.3</td></tr><tr><td>RL-ST</td><td>43.4</td><td>60.4</td><td>58.1</td><td>54</td><td>RL-ST</td><td>48.4</td><td>51.3</td><td>49.7</td><td>50.6</td><td>RL-ST</td><td>48.4</td><td>44.4</td><td>47.5</td><td>53.8</td></tr><tr><td>None</td><td>19.5</td><td>12.3</td><td>14.7</td><td>17.4</td><td>None</td><td>18.3</td><td>18.6</td><td>20.9</td><td>21.3</td><td>None</td><td>15.8</td><td>16.0</td><td>13.6</td><td>12.9</td></tr><tr><td>DRL</td><td>32.6</td><td>40.7</td><td>22.0</td><td>29.8</td><td>DRL</td><td>40.5</td><td>38.8</td><td>32.0</td><td>34.8</td><td>MD</td><td>33.0</td><td>30.6</td><td>31.3</td><td>30.3</td></tr><tr><td>RL-ST</td><td>50.7</td><td>40.2</td><td>57.7</td><td>50.3</td><td>RL-ST</td><td>38.7</td><td>45</td><td>47.6</td><td>47.8</td><td>RL-ST</td><td>52.3</td><td>57.0</td><td>56.4</td><td>59.5</td></tr><tr><td>None</td><td>16.7</td><td>19.1</td><td>20.3</td><td>19.9</td><td>None</td><td>20.8</td><td>16.2</td><td>20.4</td><td>17.4</td><td>None</td><td>14.7</td><td>12.4</td><td>12.3</td><td>10.2</td></tr><tr><td>MLM</td><td>32.4</td><td>38.3</td><td>31.2</td><td>34.0</td><td>DR</td><td>25.2</td><td>23.8</td><td>28.1</td><td>24.0</td><td>DR</td><td>36.3</td><td>38.1</td><td>41.0</td><td>39.3</td></tr><tr><td>RL-ST</td><td>51.1</td><td>40.0</td><td>46.4</td><td>46.1</td><td>RL-ST</td><td>58.6</td><td>49.9</td><td>50.4</td><td>55.2</td><td>RL-ST</td><td>46.3</td><td>48.3</td><td>43.1</td><td>49.2</td></tr><tr><td>None</td><td>16.5</td><td>21.7</td><td>22.3</td><td>19.9</td><td>None</td><td>16.2</td><td>26.3</td><td>21.5</td><td>20.8</td><td>None</td><td>17.4</td><td>13.6</td><td>15.9</td><td>11.5</td></tr><tr><td>UnMT</td><td>31.8</td><td>33.5</td><td>29.0</td><td>30.0</td><td>UnMT</td><td>37.7</td><td>39.1</td><td>30.0</td><td>35.1</td><td>G-GST</td><td>45.4</td><td>36.4</td><td>38.6</td><td>38.1</td></tr><tr><td>RL-ST</td><td>54.3</td><td>47.9</td><td>51.7</td><td>52.7</td><td>RL-ST</td><td>51.2</td><td>46.8</td><td>52.3</td><td>50.5</td><td>RL-ST</td><td>42.3</td><td>50.3</td><td>47.5</td><td>52.0</td></tr><tr><td>None</td><td>13.9</td><td>18.6</td><td>19.3</td><td>17.3</td><td>None</td><td>11.1</td><td>14.1</td><td>17.7</td><td>14.4</td><td>None</td><td>12.3</td><td>13.3</td><td>13.9</td><td>9.9</td></tr></table>



Table content: <table/>

Ablation results AC = Style Accuracy; BL R = Average BLEU score w.r.t human reference sentences; PL = Perplexity; HM = Harmonic Mean of (AC, BL R ); GM = Geometric Mean of (AC, BL R ). Lower PL is better.Decoding Strategies during Inference: At test time, we experiment with two decoding strategiestop-p sampling and beam search. Beam search yields marginally better results on HM and GM, but top-p has marginally better PL.

Table content: <table><tr><td>Model</td><td>AC</td><td>BL R</td><td>PL ↓</td><td>HM</td><td>GM</td></tr><tr><td/><td colspan="2">RL and MLE</td><td/><td/><td/></tr><tr><td>RL-Only</td><td>60.7</td><td>59.2</td><td>40.8</td><td>59.9</td><td>59.9</td></tr><tr><td>MLE-Only</td><td>93.0</td><td>60.0</td><td>37.0</td><td>73.0</td><td>74.7</td></tr><tr><td>RL+MLE (RL-ST)</td><td>98.1</td><td>58.7</td><td>35.9</td><td>73.5</td><td>75.9</td></tr><tr><td/><td colspan="2">Disentanglement</td><td/><td/><td/></tr><tr><td>Cont-Only</td><td>93.6</td><td>51.2</td><td>44.2</td><td>66.2</td><td>69.2</td></tr><tr><td>Full-Src (RL-ST)</td><td>98.1</td><td>58.7</td><td>35.9</td><td>73.5</td><td>75.9</td></tr><tr><td/><td colspan="2">Sample efficiency</td><td/><td/><td/></tr><tr><td>1k-Train</td><td>96.5</td><td>58.4</td><td>33.8</td><td>72.7</td><td>75.1</td></tr><tr><td>2k-Train (RL-ST)</td><td>98.1</td><td>58.7</td><td>35.9</td><td>73.5</td><td>75.9</td></tr><tr><td>4k-Train</td><td>96.0</td><td>58.2</td><td>35.72</td><td>72.5</td><td>74.7</td></tr><tr><td/><td colspan="2">Rewards</td><td/><td/><td/></tr><tr><td>StyleR-Attn-Only</td><td>97.1</td><td>59.3</td><td>43.0</td><td>73.6</td><td>75.9</td></tr><tr><td>StyleR-Uniform-Only</td><td>95.5</td><td>59.4</td><td>41.6</td><td>73.3</td><td>75.3</td></tr><tr><td>FluencyR-Only</td><td>85.1</td><td>44.8</td><td>16.8</td><td>58.7</td><td>61.7</td></tr><tr><td>Combined (RL-ST)</td><td>98.1</td><td>58.7</td><td>35.9</td><td>73.5</td><td>75.9</td></tr><tr><td/><td colspan="2">Decoding Strategies</td><td/><td/><td/></tr><tr><td>Top-p</td><td>99.0</td><td>56.7</td><td>35.6</td><td>72.1</td><td>74.9</td></tr><tr><td>Beam-Search (RL-ST)</td><td>98.1</td><td>58.7</td><td>35.9</td><td>73.5</td><td>75.9</td></tr></table>

Li et al. (2018) publish a set of human reference outputs on the test set of YELP, which is further extended to four sets byLuo et al. (2019). CAPTIONS: Each image caption in this dataset is labeled as either factual, romantic, or humorous. The task is to convert factual sentences into romantic and humorous ones. While CAPTIONS is an aligned corpus, containing captions for the same image in different styles, we do not use the alignments. The task is to transfer factual captions to romantic and humorous ones.(Li et al., 2018;Sudhakar et al., 2019).Li et al. (2018) publish a set of human reference outputs on the test set of CAPTIONS. GYAFC: Each sentence is labeled as either being formal or informal. We only use a subset of this dataset which corresponds to Yahoo answers in the Family and Relationships domain. This is also an aligned corpus, but we do not use these alignments either. The task is to transfer formal to informal sentences and vice-versa.Rao & Tetreault (2018) publish a set of human reference outputs on the test set of GYAFC.

Table content: <table/>

shows train, dev and test statistics of the datasets.

Table content: <table><tr><td>Dataset</td><td>Style</td><td>Train</td><td>Dev</td><td>Test</td></tr><tr><td>YELP</td><td>Positive</td><td colspan="3">270K 2000 500</td></tr><tr><td/><td>Negative</td><td colspan="3">180K 2000 500</td></tr><tr><td>GYAFC</td><td>Formal</td><td>277K</td><td>985</td><td>500</td></tr><tr><td/><td>Informal</td><td colspan="3">279K 1015 500</td></tr><tr><td colspan="2">CAPTIONS Romantic</td><td>6000</td><td>300</td><td>0</td></tr><tr><td/><td>Humorous</td><td>6000</td><td>300</td><td>0</td></tr><tr><td/><td>Factual</td><td>0</td><td>0</td><td>300</td></tr></table>

Dataset statistics A.3 EXAMPLE OUTPUTS Table 6 below compares our model's outputs with those of previous works.

Table content: <table/>

2 YELP (Negative to Positive) SRC they tried to take advantage of me because i am young . DRL they tried to take advantage of me because i am young perfectly . UnMT they tried to take advantage of me because i am young and very professional . PTO they great to take advantage of me because i am young . MLM they love to take advantage of me because i am young . RL-ST they take great care of me because i am young . SRC i am not certain whether he loves you , but he definitely likes you . DRL i i am not certain er UnMT i 'm not certain whether he loves you but he definitely likes you , really does it ? UnP not is true friends did he then yes and good luck DR i am not you whether he loves you , but he likes you you . RL-ST i dont know if he loves you but he definitely likes you . SRC young man performing bicycle trick on loading dock near dumpsters . DO young man performing bicycle trick on dock and enjoys time in excitement . DR young man performing bicycle on bicycle UNK cliff to experience the adventure of life . B-GST young man performing bicycle trick on loading dock near dumpdumpsters looking for pokemon . MD young man rides down wave on motorcycle , track with dirt . RL-ST young man performing bicycle trick on ramp near a group of people enjoying life #6 CAPTIONS (Factual to Humorous) SRC young man performing bicycle trick on loading dock near dumpsters . DO young man performing bicycle on bicycle tracks for time in sky . DR young man performing bicycle trick on bicycle near crowd , looking for outer space . B-GST young man performing bicycle trick on loading dock near dumpdumpsters looking for aliens . MD young man doing skateboard trick on rocks near tennis area . RL-ST young man performing bicycle trick on dock near a crowd of aliens .

Table content: <table><tr><td>#3</td><td>GYAFC (Formal to Informal)</td></tr><tr><td>#4</td><td>GYAFC (Informal to Formal)</td></tr><tr><td>SRC</td><td>well that is just the way it is i guess .</td></tr><tr><td>DRL</td><td>i well , that is just the way it is guess .</td></tr><tr><td>UnMT</td><td>well that is what the way it is i would guess .</td></tr><tr><td>UnP</td><td>is a good reason to be</td></tr><tr><td>DR</td><td>well i believe that is just the way it is i guess .</td></tr><tr><td>RL-ST</td><td>that is just the way it is , i would advise .</td></tr><tr><td>#5</td><td>CAPTIONS (Factual to Romantic)</td></tr></table>

Examples of generated sentences to be compared down a column (RL-ST is our model, SRC is the input sentence). Attributes are colored.

Table content: <table/>

Peer review (iclr2020):
The authors introduce RL-ST, an RL-based method for unsupervised text style transfer that: 1) leverages a pre-trained language model 2) leans to transfer style without using a disentanglement objective 3) is warm-started using style-transfer generations from another model 4) is *fine-tuned using Policy Gradient, by providing rewards for style, content preservation and fluency.* Style and fluency rewards are provided by learned models/classifiers, and content preservation rewards are calculated as the BLEU score between the source and target. The main contribution seems to be 4; As I understand, 1-3 are helpful techniques from prior work that this paper appropriately leverages to help get the RL fine-tuning to work. For example, for 1-2, XLM from *Cross-lingual Language Model Pre-training* first pre-trains a (masked) language model, and then uses back-translation (without disentanglement) to perform unsupervised machine translation (similar to unsupervised style transfer). Point 3 (warm-starting with data from another model) has been shown effective in *An Effective Approach to Unsupervised Machine Translation*, where Artetxe et al. initialize a neural unsupervised MT model using translations from an unsupervised phrase-based MT model. These two papers should be cited and discussed, and in general (along with other related papers) prior work makes me view this paper*s novelty to be in the way that RL-ST uses RL. Thus, I am most interested in understanding the contribution of the RL component of the model in achieving strong performance. Clearly, the overall RL-ST model is strong (the human evaluation results are quite compelling). However, I think more experiments need to be done to disentangle the contribution of the RL from the MLE pretraining and warm-starting. In particular, I think it is crucial to know the human evaluation results (i.e., as show in Table 3) for the MLE-only model against the RL+MLE (RL-ST) model from Table 4. I do not trust the automatic evaluation metrics to provide an appropriate evaluation, primarily since the authors use those exact metrics during RL training; for example, if a learned classifier is used to evaluate (e.g., the style or fluency), RL-ST has used the same classifier (including weights) during training (the authors should please correct me if I have misunderstood). This means that RL-ST could be overfitting to the particular models/model weights used for evaluation. For this reason, it would also be helpful to know the human evaluation results for other ablations related to the importance of RL (i.e., those on the *Rewards* in Table 4). Regarding the way RL-ST employs RL, I also have several questions, as I found some design decisions unintuitive: * Attention weights just indicate what tokens were important in affecting the model*s decision. However, sometimes tokens with high attention could be responsible for *decreasing* the likelihood of the generation being in the target style, but those tokens would be given a positive reward if the overall classification results in a sentence that appears in the target style (in spite of tokens in the wrong style). It seems more intuitive to me to use gradients from the downstream classifier to reward (similar to a GAN) or to use the classification predictions of a sequential/autoregressive discriminator as dense rewards (as in ScratchGAN from *Training Language GANs from Scratch*). * Why not use the attention weights to weight the reward for each word (rather than applying a 0 or 1 weight to the per-token reward)? * The reward assignment is discontinuous where the style classifier predicts the likelihood of each class to be 0.5 (the reward jumps from -0.5 to +0.5). The discontinuity is unintuitive to me - would it not increase the variance of the reward around the discontinuity? Have you considered a continuous reward function? This seems like a simple change to make (Reward = thr * Prob -> thr * (2 * Prob - 1)) * For the fluency reward, is the baseline b manually chosen? It seems better to learn b or to dynamically calculate b based on a moving average of past reward, rather than treating b as a constant and hyperparameter I also have several questions about how this work differs from prior work using RL in text generation: * How is the application of RL in this work different than prior applications of RL in text style transfer? Is the main differences from prior work using RL in the use of pre-trained weights and warm-starting on model-generated data? * How is the application of RL in this work different than prior applications of RL in the text-GAN/generation literature? Is the main difference the specific rewards used for style transfer? It would be nice to see that literature discussed, so I can know whether the RL itself is performed in a novel way, or if the authors are claiming that the novelty is in the style-transfer reward functions. Other notes: * If it*s not too difficult, it would be nice to know how RL-ST compares against the references w.r.t. human evaluation. That would give a less relative / more objective evaluation of RL-ST. I don*t think such evaluation is necessary though. * In addition to the papers I have brought up above, it would be nice to add a discussion on *Fine-tuning Language Models from Human Preferences.* This paper is concurrent and also fine-tunes a pre-trained language model for style-conditioned generation using RL. Overall, I think RL-ST is clearly a strong model, but I am not sure about the main novelty of the paper (RL), for 3 reasons: 1) if RL is a significant cause in the improvements 2) the way that RL is used is unintuitive in several ways to me 3) how novel the use of RL is compared to prior work I think several of the above concerns may be addressed in the rebuttal and/or other reviewer*s remarks, so I am looking forward to reading what others have to say.