Title: CONTEXTUAL TEXT STYLE TRANSFER

Abstract: In this paper, we introduce a new task, Contextual Text Style Transfer, to translate a sentence within a paragraph context into the desired style (e.g., informal to formal, offensive to non-offensive). Two new datasets, Enron-Context and Reddit-Context, are introduced for this new task, focusing on formality and offensiveness, respectively. Two key challenges exist in contextual text style transfer: 1) how to preserve the semantic meaning of the target sentence and its consistency with the surrounding context when generating an alternative sentence with a specific style; 2) how to deal with the lack of labeled parallel data. To address these challenges, we propose a Context-Aware Style Transfer (CAST) model, which leverages both parallel and non-parallel data for joint model training. For parallel training data, CAST uses two separate encoders to encode each input sentence and its surrounding context, respectively. The encoded feature vector, together with the target style information, are then used to generate the target sentence. A classifier is further used to ensure contextual consistency of the generated sentence. In order to leverage massive non-parallel corpus and to enhance sentence encoder and decoder training, additional self-reconstruction and back-translation losses are introduced. Experimental results on Enron-Context and Reddit-Context demonstrate the effectiveness of the proposed model over state-of-the-art style transfer methods, across style accuracy, content preservation, and contextual consistency metrics. 1

1. INTRODUCTION:

Text style transfer has recently been applied to many applications with remarkable success (e.g., sentiment manipulation, formalized writing). Early work relied on parallel corpora with a sequenceto-sequence learning framework (Bahdanau et al., 2015; Jhamtani et al., 2017) . However, collecting annotations for parallel data is highly time-consuming. There has been a recent surge of interest in developing text style transfer models using non-parallel data (Hu et al., 2017; Li et al., 2018; Prabhumoye et al., 2018; Subramanian et al., 2018) , assuming that disentangling style information from semantic content can be achieved in an auto-encoding fashion with the introduction of additional regularizers (e.g., adversarial discriminators (Shen et al., 2017) or language models (Yang et al., 2018) ).

Despite promising results, these techniques still have a long way towards practical use. Specifically, existing models mostly focus on sentence-level rewriting. However, in real-world applications, sentences typically reside in a proper context such as a paragraph. For example, in the formalized writing task, the rewritten span should align well with the surrounding context (e.g., personal email, scientific content) to keep a coherent text flow. Taking a single sentence as the sole input of a style transfer model may fail to preserve topical coherency of the generated sentence with its surrounding context, resulting in poor semantic and logical consistency on the paragraph level (see Example C in Table 4 ).

Motivated by this, we propose and investigate a new task -Contextual Text Style Transfer. Given a paragraph, the system aims to automatically edit sentences into a desired style, while keeping the edited section topically coherent with its surrounding context. To achieve this goal, we propose a novel Context-Aware Style Transfer (CAST) model, by jointly considering style transfer and context alignment. For parallel training data, CAST uses two separate encoders to encode the source sentence and its surrounding context, respectively, and a decoder to translate the encoded features into the target sentence. A pre-trained coherence classifier is further applied to regularize the generated target sentence to be consistent with the context. To overcome the data sparsity issue, we further leverage non-parallel data by using a hybrid approach. With large-scale non-parallel corpus, the training of the sentence encoder and decoder are enhanced via additional self-reconstruction and back-translation objectives. A pre-trained style classifier is also used for style regularization. The final CAST model is jointly trained with both parallel and non-parallel data.

As this is a newly proposed task, we also introduce two new datasets, Enron-Context and Reddit-Context, collected via crowdsourcing. The former contains 14,734 formal vs. informal paired samples from Enron (Klimt & Yang, 2004 ) (an email dataset), and the latter contains 23,158 offensive vs. non-offensive paired samples from Reddit (Serban et al., 2017) . Each paired sample contains an original sentence and a human-rewritten sentence with the desired style, accompanied by its paragraph context. Besides this, in order to enhance model training, we exploit additional 28,375/29,774 formal/informal non-parallel sentences from GYAFC (Rao & Tetreault, 2018) , and 53,028/53,714 offensive/non-offensive non-parallel sentences from Reddit (dos Santos et al., 2018) .

The main contributions of this work are summarized as follows: (i) We propose a new task -Contextual Text Style Transfer, which aims to translate an input sentence into a desired style, while preserving its style-irrelevant semantics and topical consistency with the surrounding context. (ii) We introduce two new datasets for this task, Enron-Context and Reddit-Context, which provide reliable benchmarks for measuring contextual style transfer models. (iii) We present a new model -Context-Aware Style Transfer (CAST), which jointly optimizes the generation quality of the target sentence and its topical coherency with adjacent sentences. Extensive experiments on these two new datasets demonstrate that the proposed CAST model outperforms state-of-the-art baselines.

2. RELATED WORK:

Text Style Transfer Text style transfer aims to modify an input sentence into a desired style while preserving its style-independent semantics. Previous work has explored this as a sequence-tosequence learning task using parallel corpora with paired source/target sentences in different styles. For example, Jhamtani et al. (2017) pre-trained word embeddings by leveraging external dictionaries mapping Shakespearean words to modern English words and additional text. However, available parallel data in different styles are very limited. Therefore, there is a recent surge of interest in considering a more realistic setting, where only non-parallel stylized corpora are available. A typical approach is: (i) disentangling latent space as content and style features; then (ii) generating stylistic sentences by tweaking style-relevant features and passing them through a decoder, together with the original content-relevant features (Xu et al., 2018) .

Many of these approaches borrowed the idea of adversarial discriminator/classifier from the Generative Adversarial Network (GAN) framework (Goodfellow et al., 2014) . For example, Shen et al. (2017) ; Fu et al. (2018) ; Lample et al. (2018) used adversarial classifiers to force the decoder to transfer the encoded source sentence into a different style/language. Alternatively, Li et al. (2018) achieved disentanglement by filtering stylistic words of input sentences. Another direction for text style transfer without parallel data is using back-translation (Prabhumoye et al., 2018) with a denoising auto-encoding objective (Logeswaran et al., 2018; Subramanian et al., 2018) .

Regarding the tasks, sentiment transfer is one of the most widely studied problems. From informality to formality (Rao & Tetreault, 2018) is another direction of text style transfer, aiming to change the style of a given sentence to more formal text. dos Santos et al. (2018) presented an approach to transferring offensive text to non-offensive based on social network data. In Prabhumoye et al. (2018) , the authors proposed the political slant transfer task. However, all these previous studies did not directly consider context-aware text style transfer, which is the main focus of this work.

Context-aware Text Generation Our work is related to context-aware text generation (Mikolov & Zweig, 2012; Tang et al., 2016) , which can be applied to many NLP tasks (Mangrulkar et al., 2018) . For example, previous work has investigated language modeling with context information (Wang & Cho, 2015; Wang et al., 2017) , treating the preceding sentences as context. There are also studies on response generation for conversational systems (Sordoni et al., 2015b; Wen et al., 2015) , where dialogue history is treated as a context. Zang & Wan (2017) Recon.

Trans.

Original style

Context:

Recon. to predict the next sentence given the previous sentences in a dialogue session. Sordoni et al. (2015a) presented a hierarchical recurrent encoder-decoder model to encode dialogue context. Our work is the first to explore context information in the text style transfer task.

3. CONTEXTUAL TEXT STYLE TRANSFER:

In this section, we first describe the problem definition and provide an overview of the model architecture in Section 3.1. Section 3.2 presents the proposed Context-Aware Style Transfer (CAST) model with parallel data, and Section 3.3 further introduces how to augment the CAST model with non-parallel data in a hybrid approach.

3.1. OVERVIEW:

Problem Definition The problem of contextual text style transfer is defined as follows. Given a style-labelled parallel dataset P = {(x i , l i ), (y i , li ), c i } M i=1 , where the i-th instance contains the original sentence x i in style l i , its corresponding rewritten sentence y i in another style li , and the paragraph context c i . x i and y i are expected to contain the same semantic content, but in different language styles (i.e., l i = li ). The goal is to transform x i in style l i to y i in style li , while keeping the sentence y i semantically coherent with its context c i . In practice, labelled parallel data may be difficult to garner. Therefore, we assume that additional non-parallel data U = {(x i , l i )} N i=1 can be leveraged to enhance overall model training.

Training Objective The overall architecture of the proposed CAST model is illustrated in Figure 1 . The hybrid model training process consists of two paths, one for parallel data and the other for non-parallel data. In the parallel path, a Seq2Seq loss and a contextual coherence loss are defined, to learn the two encoders (sentence encoder and context encoder) and the sentence decoder with labeled parallel data. The non-parallel path is designed to further enhance the sentence encoder and decoder with three additional losses: (i) a self-reconstruction loss; (ii) a back-translation loss; and (iii) a style classification loss. The overall training objective, taking both parallel and non-parallel paths into consideration, can be written as:

EQUATION

where λ 1 , λ 2 , λ 3 and λ 4 are hyper-parameters to balance different objectives. Each of these loss terms will be explained in the following sub-sections.

3.2. CAST WITH PARALLEL DATA:

In this subsection, we discuss the training objective associated with parallel data, consisting of (i) a contextual Seq2Seq loss; and (ii) a contextual coherence loss.

Contextual Seq2Seq Loss When parallel data are available, a Seq2Seq model can be directly learned for text style transfer. We denote Seq2Seq model as (E, D), where the semantic representation of sentence x i is extracted by the encoder E (i.e., E(x i )), and the decoder D aims to learn a conditional distribution of y i given the encoded feature E(x i ) and style li :

EQUATION

However, in such a sentence-to-sentence style transfer setting, the context of the paragraph is ignored, which if well utilized, could help improve the quality of generated text (such as paragraphlevel topical coherence).

Thus, to take advantage of the paragraph context c i information, we use two separate encoders E s and E c to encode the sentence and the context independently. The outputs of the two encoders are combined via a linear layer, to obtain a context-aware sentence representation, which is used for generating the target sentence. The model is trained to minimize the following loss:

EQUATION

Compared with Eqn. ( 2), the use of E c (c i ) makes the text style transfer process context-dependent.

The generated sentence can be denoted as ỹi

= D(E s (x i ), E c (c i ), li ).

Contextual Coherence Loss To enforce contextual coherence (i.e., to make the generated sentence y i align with the surrounding context c i ), we train a coherence classifier that aims to distinguish whether c i is the context of y i , by adopting a language model with an objective similar to next sentence prediction (Devlin et al., 2019) .

Specifically, assume that y i is the t-th sentence of a paragraph p i (i.e., y i = p (t) i ), and

c i = {p (0) i , . . . , p (t-1) i , p (t+1) i , . . . , p (T )

i } is its surrounding context. We first reconstruct the paragraph

p i = {p (0) i , . . . , p (T ) i } by inserting y i into the proper position in c i , denoted as [c i ; y i ].

Based on this, we obtain a paragraph representation u i via a language model encoder. Then, we apply a linear layer to the representation, followed by a tanh function and a softmax layer to predict a binary label s i , which indicates whether c i is the context of y i :

u i = LM([c i ; f (y i )]) (4) p LM (s i |c i , y i ) = softmax (tanh (Wu i + b)) .

(5) where LM represents the language model encoder, and s i = 1 indicates that c i is the context of y i . Note that since ỹi are discrete tokens which are non-differentiable, we use the continuous feature, denoted as f (ỹ i ), that generates ỹi as the input of the language model. We construct paired data {y i , c i , s i } N i=1 for training the classifier, where the negative samples are generated by replacing a sentence in a paragraph with another random sentence. After pre-training, the coherence classifier is used to obtain the contextual coherence loss:

EQUATION

Intuitively, minimizing L P cohere encourages the ỹi to blend better to its context c i . Note that the coherence classifier is pre-trained, and fixed during the training of the CAST model. The above coherence loss can be used to update the parameters of E s , E c and D during model training.

3.3. CAST WITH NON-PARALLEL DATA:

For the contextual style transfer task, there are not many parallel datasets available with style-labeled paragraph pairs. To overcome the data sparsity issue, we propose to further boost the CAST model by leveraging additional non-parallel data U = {(x i , l i )} N i=1 , which are less expensive to collect. In order to fully exploit U to enhance the training of the sentence encoder and decoder (E s , D), we introduce three additional training losses, detailed below.

Reconstruction Loss:

The reconstruction loss aims to encourage E s and D to reconstruct the input sentence itself, if the desired style is the same as the input. The corresponding objective is similar to Eqn. (2): Compared with Eqn. ( 2), here we encourage the decoder D to recover x i 's original stylistic property as accurate as possible when given the style label l i . The self-reconstructed sentence is denoted as

EQUATION

xi = D(E s (x i ), l i ).

Back-Translation Loss The back-translation loss requires the model to reconstruct the input sentence after a transformation loop. Specifically, the input sentence x i is first transferred into the target style, i.e., xi = D(E s (x i ), li ). Then the generated target sentence is transferred back into its original style, i.e., xi = D(E s (x i ), l i ). The back-translation loss is defined as:

EQUATION

)

where the source style is denoted as l i , and the target style is denoted as li .

Style Classification Loss:

To further boost the model, we use U to train a classifier to predict the style of a given sentence, and regularize the training of (E s , D) with the pre-trained style classifier. Specifically, the objective for training the style classifier is:

EQUATION

where p C (•) denotes the style classifier. After the classifier is trained, we keep its parameters fixed, and apply it to update the parameters of (E s , D). Specifically, the style classification loss defined over the pre-trained style classifier can be written as: Enron-Context To build a formality transfer dataset with paragraph contexts, we randomly sampled emails from the Enron corpus (Klimt & Yang, 2004) . After pre-processing and filtering with NLTK (Bird et al., 2009) , we asked Amazon Mechanical Turk (AMT) annotators to identify informal sentences within each email, and rewrite them in a more formal style. Then, we asked different annotators to verify if each rewritten sentence is more formal than the original sentence.

L U style = -E xi∼U E xi∼pD(

Reddit-Context:

We further collected a new offensive vs. non-offensive dataset from the Reddit Politics corpus (Serban et al., 2017) . First, we performed classification on the original dataset at sentence level, to identify offensive sentences from whole paragraphs. After filtering some extremely long/short sentences, we asked AMT annotators to rewrite the offensive ones to non-offensive alternatives. To provide robust datasets for benchmark, we randomly selected a subset of sentences to be rewritten into two references, which makes up of 10% of the whole dataset.

After manually removing wrong or duplicated annotations, we obtained a total of 14,734 rewritten sentences for Enron-Context, and 23,158 for Reddit-Context. We also limited the vocabulary size by using words with frequency equal or larger than 20 (70) in Enron (Reddit). Table 1 provides the statistics on the two datasets.

Non-parallel Corpus Besides parallel datasets, we also explore non-parallel datasets to enhance model training. For formality transfer, one choice is Grammarlys Yahoo Answers Formality Corpus (GYAFC) (Rao & Tetreault, 2018) , crawled and annotated from two domains in Yahoo Answers. 

5. EXPERIMENTS:

In this section, we compare our model with state-of-the-art baselines on the two new datasets, and provide both quantitative analysis and human evaluation to validate the effectiveness of our model.

5.1. EXPERIMENTAL SETUP:

Datasets Table 2 provides a summary of the parallel and non-parallel datasets used for the two style transfer tasks. For the non-parallel datasets, we split them into two: one for the proposed model training, and the other for the style classifier pre-training. Similarly, for the parallel datasets, the training sets are divided into two as well, for the training of CAST and the coherence classifier, respectively.

Evaluation Metrics:

The contextual style transfer task requires generating sentences to: (i) preserve the original content and structure in the source sentence; (ii) conform to the pre-specified style; and (iii) align with the surrounding context in the paragraph. Thus, we consider the following automatic metrics to evaluate the effectiveness of different methods:

(i) Content Preservation. We assess the degree of content preservation based on n-gram statistics, by measuring BLEU scores (Papineni et al., 2002) between generated sentences and human references. Following Rao & Tetreault (2018) , we also use GLEU for the formality transfer task, which was originally introduced for the grammatical error correction task (Napoles et al., 2015) . For offensiveness transfer, we include perplexity (PPL) as used in dos Santos et al. ( 2018), which is computed by a word-level LSTM language model pre-trained on non-offensive sentences.

(ii) Style Accuracy. Similar to prior work, we generate samples from the model, and measure style accuracy (i.e., Acc. of the pre-trained style classifier).

(iii) Context Coherence. As aforementioned, we use the pre-trained coherence classifier to measure how the generated sentences match the surrounding context.

For formality transfer, the pre-trained style classifier and coherence classifier reaches 91.35% and 86.78% accuracy, respectively. For offensiveness transfer, the accuracies are 93.47% and 84.96%, respectively. Therefore, we consider them as reliable to serve as evaluation metrics.

Baselines We compare our proposed model with several baselines: (i) Seq2Seq: a Transformerbased Seq2Seq model (i.e., Eqn. (2)), taking only sentences as inputs, and trained only on parallel data; (ii) Contextual Seq2Seq: a Transformer-based contextual Seq2Seq model (i.e., Eqn. (3)), taking both context and the sentence as input, and trained only on parallel data; (iii) Hybrid Seq2Seq (Xu et al., 2019 ): a Seq2Seq model leveraging both parallel and non-parallel data; (iv) ControlGen (Hu et al., 2017; 2018) : a state-of-the-art text transfer model using non-parallel data.

Implementation Details The context encoder, sentence encoder and sentence decoder are all implemented as a one-layer Transformer with 4 heads. The hidden dimension of one head is 256, and the hidden dimension of the feed-forward sub-layer is 1024. The context encoder is set to take maximum of 50 words from the surrounding context of the target sentence. For the style classifier, we use a standard CNN-based sentence classifier (Kim, 2014) .

Since the non-parallel corpus U contains more samples than the parallel corpus P, we down-sample U to assign each mini-batch the same number of parallel and non-parallel samples to balance the training, alleviating the catastrophic forgetting problem described in Howard & Ruder (2018) . We train the model using Adam optimizer with mini-batch size 64 and learning rate 0.0005. The validation set is used to select the best hyper-parameters. Hard-sampling (Logeswaran et al., 2018) is used to back-propagate the loss through discrete tokens from the pre-trained classifier to the model.

For the ControlGen (Hu et al., 2017 ) baseline, we use the code provided by the authors, and use their default hyper-parameter setting. For Hybrid Seq2Seq (Xu et al., 2019) , we re-implement their model following the original paper.

5.2. EXPERIMENTAL RESULTS:

Formality Transfer Results on the formality transfer task are summarized in Table 3 . The CAST model, which leverages both context and non-parallel data, achieves the best performance over all the baselines. Particularly, CAST is able to boost GLEU and Coherence scores with a large margin. Hybrid Seq2Seq also achieves good performance by utilizing non-parallel data. By incorporating the context information, Contextual Seq2Seq also improves over the vanilla Seq2Seq model. As expected, ControlGen does not perform well, since only non-parallel data is used for training.

Offensiveness Transfer Results are summarized in Table 3 . CAST achieves the best performance over all the metrics except for the PPL. In terms of Coherence, both methods that leverage the context achieve a better performance compared with the Seq2Seq baseline. Contextual Seq2Seq also improves BLEU, which is different from the observation in the formality transfer task. Our model produces slightly worse performance on PPL than ControlGen. We hypothesize that this is because our model tends to use the same non-offensive word to replace an offensive word, producing some unusual sentences. 4 presents some qualitative examples. Generally, we observe that our model is better at replacing informal words with formal ones (Example B and C), and generate more context-aware sentences (Example A and C), possibly due to the use of coherence and style classifiers. We also observe that the exploitation of the context information can also help our model preserve the semantic content in the original sentence (Example B).

Qualitative Examples Table:

Ablation Study To investigate the effectiveness of individual components, we perform ablation studies by removing some components of the proposed model. Results on both tasks are provided in Table 5 . The context encoder and the coherence classifier play an important role in the proposed model. The context encoder is able to improve content preservation and style transfer accuracy, while the coherence classifier can help improve the coherence score but not much for style accuracy. By using these two components, our model can find a proper balance between transferring to the correct style and maintaining the consistency with context. When both of them are removed (the 4th row), performance on all the metrics drops significantly. We also observe that without using non-parallel data, the model performs poorly, showing the importance of using a hybrid method.

Human Evaluation Considering the subjective nature of this task, we conduct human evaluations based on content preservation, style control and context consistency, following Mir et al. (2019) . Given the original sentence and the transferred sentence with the corresponding context, the AMT crowd-source workers were asked to select the best one based on these three aspects. The evaluation interface also allows a neutral option, if the worker considers both sentences as equally good in certain aspect. We randomly sampled 200 sentences from the corresponding test set, and collected three human responses for each pair. Table 6 reports the pairwise comparison results on both tasks. Based on human judgment, the quality of transferred sentences by CAST is significantly higher than the other methods on all three metrics. This is consistent with our observation in the experiments on automatic metrics.

6. CONCLUSION:

In this paper, we present a new task -contextual text style transfer. To provide benchmarks for this new task, we introduce two new datasets, which contain annotated sentence pairs accompanied by paragraph contexts. We also propose a new model, which can jointly capture content preservation and context coherence, and exploit additional abundant non-parallel data for boosting performance. In both quantitative and human evaluations, our approach significantly outperforms baseline methods that do not rely on context information. Ablation study also demonstrates the effectiveness of different components in the model design. We believe our current model takes a first step towards modeling context information for text style transfer, and would like to explore more advanced solutions to integrating context. AMT Interface: In Figure 2 , we show the AMT user interfaces to collect the bias/violence data.

User Study: In Table 7 , we present more qualitative examples on both tasks.

:

Source code, and collected new datasets will be released upon acceptance.

Figures and Tables:

Figure 1: Model architecture of the proposed CAST model for contextual text style transfer. Both the training paths share the same sentence encoder and decoder. See Sec. 3 for details.

Figure 2: The AMT interfaces we used to collect the bias/violence data. The top interface is for turkers to select the bias/violence data from a paragraph. The bottom interface is for turkers to rewrite the sentence/phrase given the context. Data Collection: We use the offensive language and hate speech classifier from (Davidson et al., 2017) to classify the offensive sentence. We perform the classification at the sentence level for each Reddit post. As shown in Figure 2, the data collection are divided into two sub-tasks: first select the bias/violence data from a paragraph and then rewrite the sentence/phrase given the context. The rewritten sentences from Enron-Context are validated by one of collaborators from a company. The 23,158 Reddit-Context are validated ourselves. Each rewritten sentence is reviewed by one volunteer to check if it is inoffensive while preserves the original content.

introduced a neural model to generate long reviews from aspect-sentiment scores given the topics.Vinyals & Le (2015) proposed a model

Table content: <table><tr><td>Context</td><td>Context Encoder</td><td/><td/><td/><td>Coherence Classifier</td><td>(Contextual Coherence Loss)</td></tr><tr><td/><td/><td>Target style</td><td>Sentence Decoder</td><td>Target</td><td colspan="2">(Contextual Seq2Seq Loss)</td></tr><tr><td>Input</td><td>Sentence Encoder</td><td/><td/><td/><td/><td>CAST with parallel data</td></tr><tr><td/><td colspan="2">Original style</td><td>Sentence Decoder</td><td/><td colspan="2">(Reconstruction Loss)</td></tr><tr><td>Input</td><td>Sentence Encoder</td><td>Target style</td><td>Decoder Sentence</td><td/><td>Style Classifier</td><td>(Style Classification Loss)</td></tr><tr><td/><td>Sentence Encoder</td><td/><td>Sentence Decoder</td><td/><td/><td>(Back-Translation Loss)</td></tr><tr><td/><td/><td/><td/><td/><td/><td>CAST with non-parallel data</td></tr></table>

Statistics on Enron-Context and Reddit-Context datasets.

Table content: <table><tr><td>Dataset</td><td colspan="5"># sent. # rewritten sent. # words per sent. # words per paragraph # vocabulary</td></tr><tr><td colspan="2">Reddit-Context 14,734</td><td>14,734</td><td>9.4</td><td>38.5</td><td>4,622</td></tr><tr><td colspan="2">Enron-Context 23,158</td><td>25,259</td><td>7.6</td><td>25.9</td><td>2,196</td></tr></table>

Statistics of the parallel and non-parallel datasets on the two text style transfer tasks.This corpus contains paired informal and formal sentences, without context. We randomly selected a subset of sentences from the original dataset, and used it in a non-parallel manner. By the end, we collected 28,375/29,774 formal/informal sentences. The second dataset is the offensive/nonoffensive Reddit dataset. Following dos Santos et al. (2018), we used a pre-trained classifier to extract offensive/non-offensive sentences from Reddit posts. In total, we collected 53,028/53,714 offensive/non-offensive sentences.

Table content: <table/>

Quantitative evaluation results of different models on the two style transfer tasks .

Table content: <table><tr><td/><td/><td/><td colspan="2">Formality Transfer</td><td/><td/><td colspan="2">Offensiveness Transfer</td></tr><tr><td/><td>Model</td><td>Acc.</td><td colspan="3">Coherence BLEU GLEU</td><td>Acc.</td><td colspan="2">Coherence BLEU</td><td>PPL</td></tr><tr><td/><td>Seq2Seq</td><td>64.05</td><td>78.09</td><td>24.16</td><td>10.46</td><td>83.05</td><td>80.28</td><td colspan="2">17.22 140.39</td></tr><tr><td colspan="3">Contextual Seq2Seq 64.28</td><td>81.25</td><td>23.72</td><td>10.37</td><td>83.42</td><td>81.69</td><td colspan="2">18.74 138.42</td></tr><tr><td/><td>Hybrid Seq2Seq</td><td>65.09</td><td>79.62</td><td>24.35</td><td>10.93</td><td>83.28</td><td>84.87</td><td colspan="2">20.78 107.12</td></tr><tr><td/><td>ControlGen</td><td>62.18</td><td>73.66</td><td>14.32</td><td>8.72</td><td>82.15</td><td>78.81</td><td>10.44</td><td>92.14</td></tr><tr><td/><td>CAST</td><td>68.04</td><td>85.47</td><td>26.38</td><td>15.06</td><td>88.45</td><td>85.98</td><td>23.92</td><td>93.03</td></tr><tr><td/><td/><td/><td colspan="3">Task: informal to formal transfer</td><td/><td>Context</td><td/></tr><tr><td/><td>Input</td><td colspan="5">I'm assuming that you'd set up be part of that meeting ?</td><td colspan="3">I'll call him back to a</td></tr><tr><td/><td>ControlGen</td><td colspan="5">I'm guessing that you would be set up that call ?</td><td colspan="3">meeting. [Input]. I asked</td></tr><tr><td>A</td><td>C-Seq2Seq</td><td colspan="5">I am assuming that you would part of that person .</td><td colspan="3">him what sort of deals</td></tr><tr><td/><td>H-Seq2Seq</td><td colspan="5">I am assuming that you would be part of that party ?</td><td colspan="2">they're working on .</td></tr><tr><td/><td>CAST</td><td colspan="6">Am I correct to assume that you would attend that meeting ?</td><td/></tr><tr><td/><td>Input</td><td/><td colspan="3">Do y'all interface with C/P .</td><td/><td colspan="3">Thanks . Can someone let</td></tr><tr><td/><td>ControlGen</td><td/><td colspan="3">Do you compete with them ?</td><td/><td colspan="3">the C/P know that the deals</td></tr><tr><td>B</td><td>C-Seq2Seq</td><td/><td colspan="3">Do we interface with them ?</td><td/><td colspan="3">are good ? [Input]. If not</td></tr><tr><td/><td>H-Seq2Seq</td><td/><td colspan="3">Do we interface with them ?</td><td/><td colspan="3">deal confirmations could but</td></tr><tr><td/><td>CAST</td><td/><td colspan="3">Do you all interface with C/P ?</td><td/><td colspan="3">they need the deal details .</td></tr><tr><td/><td/><td colspan="5">Task: offensive to non-offensive transfer</td><td>Context</td><td/></tr><tr><td/><td>Input</td><td/><td/><td>You are ugly .</td><td/><td/><td/><td/></tr><tr><td/><td>ControlGen</td><td/><td/><td>You bad guy !</td><td/><td/><td colspan="3">With the glasses , [Input].</td></tr><tr><td>C</td><td>C-Seq2Seq</td><td/><td colspan="2">Have a bad day .</td><td/><td/><td colspan="3">I don't need them because I</td></tr><tr><td/><td>H-Seq2Seq</td><td/><td colspan="2">What a bad day !</td><td/><td/><td colspan="3">never read . How do i look ?</td></tr><tr><td/><td>CAST</td><td/><td colspan="3">You look not good .</td><td/><td/><td/></tr></table>

Examples from the two datasets, where orange denotes the sentence to be transferred, and blue denotes content that also appears in the context. C-Seq2Seq: Contextual Seq2Seq; H-Seq2Seq: Hybrid Seq2Seq.

Table content: <table/>

Ablation study of CAST on the two style transfer tasks.

Table content: <table><tr><td/><td/><td/><td colspan="2">Formality Transfer</td><td/><td/><td colspan="3">Offensiveness Transfer</td></tr><tr><td>Model</td><td/><td>Acc.</td><td colspan="3">Coherence BLEU GLEU</td><td colspan="2">Acc.</td><td colspan="2">Coherence BLEU</td><td>PPL</td></tr><tr><td>CAST</td><td/><td>68.04</td><td>85.47</td><td>26.38</td><td>15.06</td><td colspan="2">88.45</td><td>85.98</td><td>23.92</td><td>93.03</td></tr><tr><td colspan="2">w/o context encoder</td><td>65.35</td><td>82.9</td><td>23.98</td><td>14.17</td><td colspan="2">84.15</td><td>80.96</td><td>20.54 127.02</td></tr><tr><td colspan="3">w/o cohere. classifier 65.47</td><td>80.16</td><td>14.82</td><td>14.45</td><td colspan="2">85.11</td><td>79.37</td><td>21.97 115.57</td></tr><tr><td>w/o both</td><td/><td>62.19</td><td>74.47</td><td>15.88</td><td>10.46</td><td colspan="2">72.69</td><td>78.15</td><td>13.14 147.31</td></tr><tr><td colspan="3">w/o non-parallel data 60.19</td><td>75.49</td><td>13.5</td><td>9.88</td><td colspan="2">70.84</td><td>78.72</td><td>10.53 151.08</td></tr><tr><td>Task</td><td/><td>Aspects</td><td colspan="3">CAST vs. Contextual Seq2Seq</td><td colspan="3">CAST vs. Hybrid Seq2Seq</td><td>CAST vs. ControlGen</td></tr><tr><td/><td/><td/><td>win</td><td>lose</td><td>tie</td><td>win</td><td>lose</td><td>tie</td><td>win</td><td>lose</td><td>tie</td></tr><tr><td>Formality Transfer</td><td colspan="9">Style Control Content Preservation 59.7 22.1 18.2 50.4 20.8 28.2 68.8 14.5 17.7 57.1 28.3 14.6 46.9 26.1 28.0 72.1 12.6 25.3 Context Consistence 56.4 23.1 20.5 51.5 19.7 28.8 70.1 10.6</td></tr><tr><td>Offensiveness Transfer</td><td colspan="9">Style Control Content Preservation 62.3 26.5 11.2 54.0 17.5 28.5 53.1 30.2 16.7 58.6 25.3 16.1 50.1 29.2 20.3 54.8 19.9 25.3 Context Consistence 60.1 32.4 17.5 55.3 24.9 20.8 58.1 35.8 16.7</td></tr></table>

Results of pairwise human evaluation between CAST and three baselines on both tasks. Win/lose/tie indicate the percentage of texts generated by CAST are better/worse/equal to the compared model.

Table content: <table/>

Peer review (iclr2020):
The authors propose the task of contextual text style transfer: transferring the style of one text into another (i.e., informal to formal, or offensive to non-offensive), when the text is present within some larger, provided context. The authors propose a model (CAST) which takes advantage of the additional context to perform the style transfer. CAST outperforms previous style transfer models according to several automatic metrics, as well as human evaluation. Overall, I am not convinced that the context is useful in the proposed tasks (Reddit-context and Enron-context), for 2 reasons: * In the examples shown in Table 4, it seems possible to complete the proposed contextual style transfer tasks without the additional context (i.e., *Do y’all interface with C/P* -> *Do you all interface with C/P* for formal->informal transfer). * CAST performs better across several metrics (including human evaluation), but I am not sure that the gain is from using the context in the manner that the authors suggest (increasing the generated sentence*s coherence with the context). In particular, it is hard for me to see how some style transferred sentences can appear much more coherent than others, given that the the required changes are pretty local (i.e., word substitution). On the other hand, style transfer models seem to have some difficulty in maintaining semantic context, so it seems that some of the gains in e.g. BLEU could be from CAST using the context simply as a bag of words to draw from in generating a style-transferred sentence. Instead of including context, the issue of semantic correctness could be fixed in other, simpler ways, however, such as making a more extractive single-sentence style transfer model. I think some further analysis is necessary to conclude that the context makes the generations more coherent, rather than helping the model in some other way. Other general concerns/questions for the authors: * The baselines do not appear to have learned even the single-sentence style transfer task that well. It appears that the authors used the default hyper parameters for other style transfer models (Hybrid Seq2Seq and ControlGen), but wouldn*t it be better to choose hyper parameters for those models based on the evaluation criteria (as is done for CAST)? I know that some style transfer models can be quite sensitive to hyperparameters, so I would be surprised if the models worked optimally out of the box. I also feel that a strong unsupervised machine translation model like XLM (or the related style transfer model from *Multiple-attribute text style transfer*) could do better than the provided baselines, as these models seem to work notably better than their predecessors. * I have a general concern that CAST is quite complicated and has a lot of moving parts and parameters to tune. I don*t think there are ablations on all components of the model (i.e., the style classifier). I also wouldn*t be surprised if a simple state-of-the-art unsupervised or semi-supervised machine translation model like XLM outperformed CAST, without using any context. To add the use of the context, it seems possible to make a simple modification to an existing state-of-the-art transfer/translation model like XLM (adding the context to the input) * Why are the style and coherence classifiers fixed during training? Wouldn*t it be better to update the classifiers while training the rest of the model (so they doesn*t get exploited)? * Which of the evaluated models/baselines are trained using the style and/or coherence classifiers directly used for evaluation? For models that are not trained to optimize the style and/or coherence classifier predictions, I am not sure it is fair to compare those models to CAST on the style/coherence classifier accuracy, since CAST has been directly trained to perform well on that metric. * Are the evaluation classifiers identical to those used to train CAST? At the least, it seems better to use a different style/coherence classifier (i.e., trained with a different random seed or other hyperparameters) at test time that the one used during training (i.e., in case CAST has overfit the style/coherence of generations to the classifiers used during training). * In Eqn. 5, why is there a tanh activation immediately preceding the softmax? Wouldn*t the tanh limit the range of the output softmax probabilities? The proposed datasets do seem useful to the community, and the empirical results do support that CAST performs better than baselines - it would just be helpful to understand in what way the gains are coming from the additional context (to know if the task is an interesting task).