A Probabilistic Formulation of Unsupervised Text Style Transfer

We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques . Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus . By hypothesizing a parallel latent sequence that generates each observed sequence , our model learns to transform sequences from one domain to another in a completely unsupervised fashion . In contrast with traditional generative sequence models ( e.g . the HMM ) , our model makes few assumptions about the data it generates : it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution . While computation of marginal data likelihood is intractable in this model class , we show that amortized variational inference admits a practical surrogate . Further , by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques , we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss . Finally , we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks , including sentiment transfer , formality transfer , word decipherment , author imitation , and related language translation . Across all style transfer tasks , our approach yields substantial gains over state-of-the-art non-generative baselines , including the state-of-the-art unsupervised machine translation techniques that our approach generalizes . Further , we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art.1 1 INTRODUCTION . Text sequence transduction systems convert a given text sequence from one domain to another . These techniques can be applied to a wide range of natural language processing applications such as machine translation ( Bahdanau et al. , 2015 ) , summarization ( Rush et al. , 2015 ) , and dialogue response generation ( Zhao et al. , 2017 ) . In many cases , however , parallel corpora for the task at hand are scarce . Therefore , unsupervised sequence transduction methods that require only non-parallel data are appealing and have been receiving growing attention ( Bannard & Callison-Burch , 2005 ; Ravi & Knight , 2011 ; Mizukami et al. , 2015 ; Shen et al. , 2017 ; Lample et al. , 2018 ; 2019 ) . This trend is most pronounced in the space of text style transfer tasks where parallel data is particularly challenging to obtain ( Hu et al. , 2017 ; Shen et al. , 2017 ; Yang et al. , 2018 ) . Style transfer has historically referred to sequence transduction problems that modify superficial properties of text – i.e . style rather than content.2 We focus on a standard suite of style transfer tasks , including formality transfer ( Rao & Tetreault , 2018 ) , author imitation ( Xu et al. , 2012 ) , word decipherment ( Shen et al. , 2017 ) , sentiment transfer ( Shen et al. , 2017 ) , and related language translation ( Pourdamghani & Knight , 2017 ) . General unsupervised translation has not typically been considered style transfer , but for the purpose of comparison we also conduct evaluation on this task ( Lample et al. , 2017 ) . ∗Equal Contribution . 1Code and data are available at https : //github.com/cindyxinyiwang/deep-latent-sequence-model . 2Notably , some tasks we evaluate on do change content to some degree , such as sentiment transfer , but for conciseness we use the term “ style transfer ” nonetheless . ar X iv :2 00 2 . 03 91 2v 1 [ cs .C L ] 1 0 Fe b 20 20 Recent work on unsupervised text style transfer mostly employs non-generative or non-probabilistic modeling approaches . For example , Shen et al . ( 2017 ) and Yang et al . ( 2018 ) design adversarial discriminators to shape their unsupervised objective – an approach that can be effective , but often introduces training instability . Other work focuses on directly designing unsupervised training objectives by incorporating intuitive loss terms ( e.g . backtranslation loss ) , and demonstrates state-ofthe-art performance on unsupervised machine translation ( Lample et al. , 2018 ; Artetxe et al. , 2019 ) and style transfer ( Lample et al. , 2019 ) . However , the space of possible unsupervised objectives is extremely large and the underlying modeling assumptions defined by each objective can only be reasoned about indirectly . As a result , the process of designing such systems is often heuristic . In contrast , probabilistic models ( e.g . the noisy channel model ( Shannon , 1948 ) ) define assumptions about data more explicitly and allow us to reason about these assumptions during system design . Further , the corresponding objectives are determined naturally by principles of probabilistic inference , reducing the need for empirical search directly in the space of possible objectives . That said , classical probabilistic models for unsupervised sequence transduction ( e.g . the HMM or semi-HMM ) typically enforce overly strong independence assumptions about data to make exact inference tractable ( Knight et al. , 2006 ; Ravi & Knight , 2011 ; Pourdamghani & Knight , 2017 ) . This has restricted their development and caused their performance to lag behind unsupervised neural objectives on complex tasks . Luckily , in recent years , powerful variational approximation techniques have made it more practical to train probabilistic models without strong independence assumptions ( Miao & Blunsom , 2016 ; Yin et al. , 2018 ) . Inspired by this , we take a new approach to unsupervised style transfer . We directly define a generative probabilistic model that treats a non-parallel corpus in two domains as a partially observed parallel corpus . Our model makes few independence assumptions and its true posterior is intractable . However , we show that by using amortized variational inference ( Kingma & Welling , 2013 ) , a principled probabilistic technique , a natural unsupervised objective falls out of our modeling approach that has many connections with past work , yet is different from all past work in specific ways . In experiments across a suite of unsupervised text style transfer tasks , we find that the natural objective of our model actually outperforms all manually defined unsupervised objectives from past work , supporting the notion that probabilistic principles can be a useful guide even in deep neural systems . Further , in the case of unsupervised machine translation , our model matches the current state-of-the-art non-generative approach . 2 UNSUPERVISED TEXT STYLE TRANSFER . We first overview text style transfer , which aims to transfer a text ( typically a single sentence or a short paragraph – for simplicity we refer to simply “ sentences ” below ) from one domain to another while preserving underlying content . For example , formality transfer ( Rao & Tetreault , 2018 ) is the task of transforming the tone of text from informal to formal without changing its content . Other examples include sentiment transfer ( Shen et al. , 2017 ) , word decipherment ( Knight et al. , 2006 ) , and author imitation ( Xu et al. , 2012 ) . If parallel examples were available from each domain ( i.e . the training data is a bitext consisting of pairs of sentences from each domain ) , supervised techniques could be used to perform style transfer ( e.g . attentional Seq2Seq ( Bahdanau et al. , 2015 ) and Transformer ( Vaswani et al. , 2017 ) ) . However , for most style transfer problems , only non-parallel corpora ( one corpus from each domain ) can be easily collected . Thus , work on style transfer typically focuses on the more difficult unsupervised setting where systems must learn from non-parallel data alone . The model we propose treats an observed non-parallel text corpus as a partially observed parallel corpus . Thus , we introduce notation for both observed text inputs and those that we will treat as latent variables . Specifically , we let X = { x ( 1 ) , x ( 2 ) , · · · , x ( m ) } represent observed data from domain D1 , while we let Y = { y ( m+1 ) , y ( m+2 ) , · · · , y ( n ) } represent observed data from domain D2 . Corresponding indices represent parallel sentences . Thus , none of the observed sentences share indices . In our model , we introduce latent sentences to complete the parallel corpus . Specifically , X̄ = { x̄ ( m+1 ) , x̄ ( m+2 ) , · · · , x̄ ( n ) } represents the set of latent parallel sentences in D1 , while Ȳ = { ȳ ( 1 ) , ȳ ( 2 ) , · · · , ȳ ( m ) } represents the set of latent parallel sentences in D2 . Then the goal of unsupervised text transduction is to infer these latent variables conditioned the observed non-parallel corpora ; that is , to learn p ( ȳ|x ) and p ( x̄|y ) . 3 THE DEEP LATENT SEQUENCE MODEL . First we present our generative model of bitext , which we refer to as a deep latent sequence model . We then describe unsupervised learning and inference techniques for this model class . 3.1 MODEL STRUCTURE . Directly modeling p ( ȳ|x ) and p ( x̄|y ) in the unsupervised setting is difficult because we never directly observe parallel data . Instead , we propose a generative model of the complete data that defines a joint likelihood , p ( X , X̄ , Y , Ȳ ) . In order to perform text transduction , the unobserved halves can be treated as latent variables : they will be marginalized out during learning and inferred via posterior inference at test time . Our model assumes that each observed sentence is generated from an unobserved parallel sentence in the opposite domain , as depicted in Figure 1 . Specifically , each sentence x ( i ) in domain D1 is generated as follows : First , a latent sentence ȳ ( i ) in domain D2 is sampled from a prior , pD2 ( ȳ ( i ) ) . Then , x ( i ) is sampled conditioned on ȳ ( i ) from a transduction model , p ( x ( i ) |ȳ ( i ) ) . Similarly , each observed sentence y ( j ) in domain D2 is generated conditioned on a latent sentence , x̄ ( j ) , in domain D1 via the opposite transduction model , p ( y ( j ) |x̄ ( j ) ) , and prior , pD1 ( x̄ ( j ) ) . We let θx|ȳ and θy|x̄ represent the parameters of the two transduction distributions respectively . We assume the prior distributions are pretrained on the observed data in their respective domains and therefore omit their parameters for simplicity of notation . Together , this gives the following joint likelihood : p ( X , X̄ , Y , Ȳ ; θx|ȳ , θy|x̄ ) = ( m∏ i=1 p ( x ( i ) |ȳ ( i ) ; θx|ȳ ) pD2 ( ȳ ( i ) ) ) ( n∏ j=m+1 p ( y ( j ) |x̄ ( j ) ; θy|x̄ ) pD1 ( x̄ ( j ) ) ) ( 1 ) The log marginal likelihood of the data , which we will approximate during training , is : log p ( X , Y ; θx|ȳ , θy|x̄ ) = log ∑ X̄ ∑ Ȳ p ( X , X̄ , Y , Ȳ ; θx|ȳ , θy|x̄ ) ( 2 ) Note that if the two transduction models share no parameters , the training problems for each observed domain are independent . Critically , we introduce parameter sharing through our variational inference procedure , which we describe in more detail in Section 3.2 . Architecture : Since we would like to be able to model a variety of transfer tasks , we choose a parameterization for our transduction distributions that makes no independence assumptions . Specifically , we employ an encoder-decoder architecture based on the standard attentional Seq2Seq model which has been shown to be successful across various tasks ( Bahdanau et al. , 2015 ; Rush et al. , 2015 ) . Similarly , our prior distributions for each domain are parameterized as recurrent language models which , again , make no independence assumptions . In contrast , traditional unsupervised generative sequence models typically make strong independence assumptions to enable exact inference ( e.g . the HMM makes a Markov assumption on the latent sequence and emissions are one-to-one ) . Our model is more flexible , but exact inference via dynamic programming will be intractable . We address this problem in the next section .

Summary: In this paper, the authors propose a probabilistic framework for unsupervised text style transfer. Given two non-parallel corpora X,Y in different domains, the authors introduce unobserved corpora \bar{X}, \bar{Y}. These are used as latent variables that control the generation of the observed data. To train models, the paper proposes to optimize the evidence lower bound of the log marginal likelihood. To facilitate training, multiple techniques are suggested, such as parameter sharing, some gradient approximations and initialization with a reconstruction objective. The approach is evaluated on five style transfer tasks, as well as unsupervised machine translation. Models are evaluated with multiple metrics, and generally obtain reasonably strong performance.