Contextual Text Style Transfer

1 INTRODUCTION . Text style transfer has recently been applied to many applications with remarkable success ( e.g. , sentiment manipulation , formalized writing ) . Early work relied on parallel corpora with a sequenceto-sequence learning framework ( Bahdanau et al. , 2015 ; Jhamtani et al. , 2017 ) . However , collecting annotations for parallel data is highly time-consuming . There has been a recent surge of interest in developing text style transfer models using non-parallel data ( Hu et al. , 2017 ; Li et al. , 2018 ; Prabhumoye et al. , 2018 ; Subramanian et al. , 2018 ) , assuming that disentangling style information from semantic content can be achieved in an auto-encoding fashion with the introduction of additional regularizers ( e.g. , adversarial discriminators ( Shen et al. , 2017 ) or language models ( Yang et al. , 2018 ) ) . Despite promising results , these techniques still have a long way towards practical use . Specifically , existing models mostly focus on sentence-level rewriting . However , in real-world applications , sentences typically reside in a proper context such as a paragraph . For example , in the formalized writing task , the rewritten span should align well with the surrounding context ( e.g. , personal email , scientific content ) to keep a coherent text flow . Taking a single sentence as the sole input of a style transfer model may fail to preserve topical coherency of the generated sentence with its surrounding context , resulting in poor semantic and logical consistency on the paragraph level ( see Example C in Table 4 ) . Motivated by this , we propose and investigate a new task - Contextual Text Style Transfer . Given a paragraph , the system aims to automatically edit sentences into a desired style , while keeping the edited section topically coherent with its surrounding context . To achieve this goal , we propose a novel Context-Aware Style Transfer ( CAST ) model , by jointly considering style transfer and context alignment . For parallel training data , CAST uses two separate encoders to encode the source sentence and its surrounding context , respectively , and a decoder to translate the encoded features 1Source code , and collected new datasets will be released upon acceptance . into the target sentence . A pre-trained coherence classifier is further applied to regularize the generated target sentence to be consistent with the context . To overcome the data sparsity issue , we further leverage non-parallel data by using a hybrid approach . With large-scale non-parallel corpus , the training of the sentence encoder and decoder are enhanced via additional self-reconstruction and back-translation objectives . A pre-trained style classifier is also used for style regularization . The final CAST model is jointly trained with both parallel and non-parallel data . As this is a newly proposed task , we also introduce two new datasets , Enron-Context and RedditContext , collected via crowdsourcing . The former contains 14,734 formal vs. informal paired samples from Enron ( Klimt & Yang , 2004 ) ( an email dataset ) , and the latter contains 23,158 offensive vs. non-offensive paired samples from Reddit ( Serban et al. , 2017 ) . Each paired sample contains an original sentence and a human-rewritten sentence with the desired style , accompanied by its paragraph context . Besides this , in order to enhance model training , we exploit additional 28,375/29,774 formal/informal non-parallel sentences from GYAFC ( Rao & Tetreault , 2018 ) , and 53,028/53,714 offensive/non-offensive non-parallel sentences from Reddit ( dos Santos et al. , 2018 ) . The main contributions of this work are summarized as follows : ( i ) We propose a new task - Contextual Text Style Transfer , which aims to translate an input sentence into a desired style , while preserving its style-irrelevant semantics and topical consistency with the surrounding context . ( ii ) We introduce two new datasets for this task , Enron-Context and Reddit-Context , which provide reliable benchmarks for measuring contextual style transfer models . ( iii ) We present a new model - Context-Aware Style Transfer ( CAST ) , which jointly optimizes the generation quality of the target sentence and its topical coherency with adjacent sentences . Extensive experiments on these two new datasets demonstrate that the proposed CAST model outperforms state-of-the-art baselines . 2 RELATED WORK . Text Style Transfer Text style transfer aims to modify an input sentence into a desired style while preserving its style-independent semantics . Previous work has explored this as a sequence-tosequence learning task using parallel corpora with paired source/target sentences in different styles . For example , Jhamtani et al . ( 2017 ) pre-trained word embeddings by leveraging external dictionaries mapping Shakespearean words to modern English words and additional text . However , available parallel data in different styles are very limited . Therefore , there is a recent surge of interest in considering a more realistic setting , where only non-parallel stylized corpora are available . A typical approach is : ( i ) disentangling latent space as content and style features ; then ( ii ) generating stylistic sentences by tweaking style-relevant features and passing them through a decoder , together with the original content-relevant features ( Xu et al. , 2018 ) . Many of these approaches borrowed the idea of adversarial discriminator/classifier from the Generative Adversarial Network ( GAN ) framework ( Goodfellow et al. , 2014 ) . For example , Shen et al . ( 2017 ) ; Fu et al . ( 2018 ) ; Lample et al . ( 2018 ) used adversarial classifiers to force the decoder to transfer the encoded source sentence into a different style/language . Alternatively , Li et al . ( 2018 ) achieved disentanglement by filtering stylistic words of input sentences . Another direction for text style transfer without parallel data is using back-translation ( Prabhumoye et al. , 2018 ) with a denoising auto-encoding objective ( Logeswaran et al. , 2018 ; Subramanian et al. , 2018 ) . Regarding the tasks , sentiment transfer is one of the most widely studied problems . From informality to formality ( Rao & Tetreault , 2018 ) is another direction of text style transfer , aiming to change the style of a given sentence to more formal text . dos Santos et al . ( 2018 ) presented an approach to transferring offensive text to non-offensive based on social network data . In Prabhumoye et al . ( 2018 ) , the authors proposed the political slant transfer task . However , all these previous studies did not directly consider context-aware text style transfer , which is the main focus of this work . Context-aware Text Generation Our work is related to context-aware text generation ( Mikolov & Zweig , 2012 ; Tang et al. , 2016 ) , which can be applied to many NLP tasks ( Mangrulkar et al. , 2018 ) . For example , previous work has investigated language modeling with context information ( Wang & Cho , 2015 ; Wang et al. , 2017 ) , treating the preceding sentences as context . There are also studies on response generation for conversational systems ( Sordoni et al. , 2015b ; Wen et al. , 2015 ) , where dialogue history is treated as a context . Zang & Wan ( 2017 ) introduced a neural model to generate long reviews from aspect-sentiment scores given the topics . Vinyals & Le ( 2015 ) proposed a model to predict the next sentence given the previous sentences in a dialogue session . Sordoni et al . ( 2015a ) presented a hierarchical recurrent encoder-decoder model to encode dialogue context . Our work is the first to explore context information in the text style transfer task . 3 CONTEXTUAL TEXT STYLE TRANSFER . In this section , we first describe the problem definition and provide an overview of the model architecture in Section 3.1 . Section 3.2 presents the proposed Context-Aware Style Transfer ( CAST ) model with parallel data , and Section 3.3 further introduces how to augment the CAST model with non-parallel data in a hybrid approach . 3.1 OVERVIEW . Problem Definition The problem of contextual text style transfer is defined as follows . Given a style-labelled parallel dataset P = { ( xi , li ) , ( yi , l̃i ) , ci } Mi=1 , where the i-th instance contains the original sentence xi in style li , its corresponding rewritten sentence yi in another style l̃i , and the paragraph context ci . xi and yi are expected to contain the same semantic content , but in different language styles ( i.e. , li 6= l̃i ) . The goal is to transform xi in style li to yi in style l̃i , while keeping the sentence yi semantically coherent with its context ci . In practice , labelled parallel data may be difficult to garner . Therefore , we assume that additional non-parallel data U = { ( xi , li ) } Ni=1 can be leveraged to enhance overall model training . Training Objective The overall architecture of the proposed CAST model is illustrated in Figure 1 . The hybrid model training process consists of two paths , one for parallel data and the other for non-parallel data . In the parallel path , a Seq2Seq loss and a contextual coherence loss are defined , to learn the two encoders ( sentence encoder and context encoder ) and the sentence decoder with labeled parallel data . The non-parallel path is designed to further enhance the sentence encoder and decoder with three additional losses : ( i ) a self-reconstruction loss ; ( ii ) a back-translation loss ; and ( iii ) a style classification loss . The overall training objective , taking both parallel and non-parallel paths into consideration , can be written as : LP , Ufinal = L P c−s2s + λ1L P cohere + λ2L U recon + λ3L U back−trans + λ4L U style , ( 1 ) where λ1 , λ2 , λ3 and λ4 are hyper-parameters to balance different objectives . Each of these loss terms will be explained in the following sub-sections . 3.2 CAST WITH PARALLEL DATA . In this subsection , we discuss the training objective associated with parallel data , consisting of ( i ) a contextual Seq2Seq loss ; and ( ii ) a contextual coherence loss . Contextual Seq2Seq Loss When parallel data are available , a Seq2Seq model can be directly learned for text style transfer . We denote Seq2Seq model as ( E , D ) , where the semantic representation of sentence xi is extracted by the encoder E ( i.e. , E ( xi ) ) , and the decoder D aims to learn a conditional distribution of yi given the encoded feature E ( xi ) and style l̃i : LPs2s = − E xi , yi∼P log pD ( yi|E ( xi ) , l̃i ) . ( 2 ) However , in such a sentence-to-sentence style transfer setting , the context of the paragraph is ignored , which if well utilized , could help improve the quality of generated text ( such as paragraphlevel topical coherence ) . Thus , to take advantage of the paragraph context ci information , we use two separate encoders Es and Ec to encode the sentence and the context independently . The outputs of the two encoders are combined via a linear layer , to obtain a context-aware sentence representation , which is used for generating the target sentence . The model is trained to minimize the following loss : LPc−s2s = − E xi , ci , yi∼P log pD ( yi|Es ( xi ) , Ec ( ci ) , l̃i ) . ( 3 ) Compared with Eqn . ( 2 ) , the use of Ec ( ci ) makes the text style transfer process context-dependent . The generated sentence can be denoted as ỹi = D ( Es ( xi ) , Ec ( ci ) , l̃i ) . Contextual Coherence Loss To enforce contextual coherence ( i.e. , to make the generated sentence yi align with the surrounding context ci ) , we train a coherence classifier that aims to distinguish whether ci is the context of yi , by adopting a language model with an objective similar to next sentence prediction ( Devlin et al. , 2019 ) . Specifically , assume that yi is the t-th sentence of a paragraph pi ( i.e. , yi = p ( t ) i ) , and ci = { p ( 0 ) i , . . . , p ( t−1 ) i , p ( t+1 ) i , . . . , p ( T ) i } is its surrounding context . We first reconstruct the paragraph pi = { p ( 0 ) i , . . . , p ( T ) i } by inserting yi into the proper position in ci , denoted as [ ci ; yi ] . Based on this , we obtain a paragraph representation ui via a language model encoder . Then , we apply a linear layer to the representation , followed by a tanh function and a softmax layer to predict a binary label si , which indicates whether ci is the context of yi : ui = LM ( [ ci ; f ( yi ) ] ) ( 4 ) pLM ( si|ci , yi ) = softmax ( tanh ( Wui + b ) ) . ( 5 ) where LM represents the language model encoder , and si = 1 indicates that ci is the context of yi . Note that since ỹi are discrete tokens which are non-differentiable , we use the continuous feature , denoted as f ( ỹi ) , that generates ỹi as the input of the language model . We construct paired data { yi , ci , si } Ni=1 for training the classifier , where the negative samples are generated by replacing a sentence in a paragraph with another random sentence . After pre-training , the coherence classifier is used to obtain the contextual coherence loss : LPcohere = − E xi , ci∼P log pLM ( si = 1|ci , f ( ỹi ) ) . ( 6 ) Intuitively , minimizing LPcohere encourages the ỹi to blend better to its context ci . Note that the coherence classifier is pre-trained , and fixed during the training of the CAST model . The above coherence loss can be used to update the parameters of Es , Ec and D during model training .

Summary: The paper proposes a new task for text style transfer, based on the idea that the the surrounding context of a sentence is important, whereas previous such tasks have only looked at sentences in isolation. Two new crowd-sourced datasets are created, and a combination of now fairly standard neural components is shown to outperform some strong baselines on the new datasets, on a variety of evaluation metrics. An ablation analysis of the components - including some auto-encoding auxiliary losses  - shows that all the various parts are helpful to performance.