Title: A SYNTAX-AWARE APPROACH FOR UNSUPERVISED TEXT STYLE TRANSFER

Abstract: Unsupervised text style transfer aims to rewrite the text of a source style into a target style while preserving the style-independent content, without parallel training corpus. Most of the existing methods address the problem by only leveraging the surface forms of words. In this paper, we incorporate the syntactic knowledge and propose a multi-task learning based Syntax-Aware Style Transfer (SAST) model. Our SAST jointly learns to generate a transferred output with aligned words and syntactic labels, where the alignment between the words and syntactic labels is enforced with a consistency constraint. The auxiliary syntactic label generation task regularizes the model to form more generalized representations, which is a desirable property especially in unsupervised tasks. Experimental results on two benchmark datasets for text style transfer demonstrate the effectiveness of the proposed method in terms of transfer accuracy, content preservation, and fluency.

1. INTRODUCTION:

Text style transfer is the task of transforming an input text by changing its stylistic attribute to a desired value while keeping the style-independent content unchanged. Focusing on different stylistic attributes, text style transfer has been addressed in many Natural Language Processing (NLP) applications such as sentiment translation (Xu et al., 2018) , persona-based conversation (Li et al., 2016) , text formalization (Rao & Tetreault, 2018) , etc.

A descent text style transfer model is expected to produce outputs with the target style, without loss of the original semantics as well as readability. Unfortunately, since the parallel corpora with pairs of input and desired output are usually unavailable, models need to learn in an unsupervised setting, making it quite challenging to satisfy all of these criteria. Previous dominant research line for unsupervised text style transfer disentangles the style and the content first, and then generates the output based on the disentangled content and the target style. The disentanglement is either achieved by learning an implicitly style-agnostic vector as the content via adversarial training (Shen et al., 2017; Fu et al., 2018; John et al., 2019) , or by explicitly marking the text units as style-related or not through a dictionary or a pre-trained attention-based classifier (Li et al., 2018; Xu et al., 2018; Zhang et al., 2018a) . Recently, Lample et al. (2019) found the disentanglement can be hardly met in practice and was not necessary for the transfer procedure. Departing from the disentangled representation, another research line (Zhang et al., 2018b; Lample et al., 2019; Luo et al., 2019) employs the back-translation technique to create online pseudo-parallel data, the quality of which can increase as training proceeds and in turn improve the model. However, most existing methods only utilize the word information, thus may suffer from the data sparsity problem and fail to understand the underlying semantics of the input text. Take sentiment transfer as example, this limitation can lead to results simply appending an improper "great" to convert a sentence to positive sentiment. In this paper, we go beyond the word information and propose a Syntax-Aware Style Transfer (SAST) model for textual data by considering additional syntactic information such as part-of-speech (POS) tags. As illustrated in Figure 1 , our SAST admits a multi-task learning scheme with the aim of generating aligned words and syntactic labels as the transferred outcome. Building on the encoder-decoder architecture, we couple the word emission and the syntactic label emission through shared hidden layers in the decoder. In this way, the model is encouraged to learn generalized representations explaining both the lexical knowledge and the syntactic knowledge, relieving possible word sparsity problem for style transfer. We optimize the proposed SAST to achieve two goals: style conversion and content preservation.

For style conversion, we enforce the model to generate outputs predicted as the target style by a pre-trained style classifier. For content preservation, we adopt the back-translation scheme following (Lample et al., 2019) but with enriched objectives on the syntactic labels: the output for an input under the transfer direction s → s, when fed back into the model for the transfer direction s → s, should reconstruct the input and the syntactic labels of the input. Moreover, besides the shared hidden layers, we further encourage the alignment between the generated words and syntactic labels by requiring the generated syntactic labels are consistent with the annotated syntactic labels (by an external NLP tool) for the generated words.

We evaluate our proposed model against state-of-the-art methods on two benchmark dataset: the sentiment dataset Yelp and the formality dataset GYAFC. Both automatic evaluation and human evaluation demonstrate our SAST can outperform the baselines in terms of transfer accuracy, semantic conservation, and fluency.

2. RELATED WORK:

Although unsupervised style transfer has gained remarkable success in the image domain (Johnson et al., 2016; Zhu et al., 2017; Choi et al., 2018) , exploration for textual data is still limited due to the discreteness and complicated semantics of the natural language.

Earlier approaches transfer the input text based on the disentanglement between content and style. Hu et al. (2017) encourage the disentanglement by recovering the content and style from the output with an encoder and a style classifier. Shen et al. (2017) use adversarial discriminators to match the transferred samples with the real samples from the target style. Fu et al. (2018) match the encoded vectors from different styles. Johnson et al. (2016) seek for better separation by incorporating the content-orientated losses based on bag-of-words features, in addition to the style orientated losses in Fu et al. (2018) . From a different perspective, Prabhumoye et al. ( 2018) assume a style-agnostic representation can be obtained via an external neural machine translation model.

Since implicitly disentangling the style and the content via hidden vectors is prone to semantic loss, explicit separation methods have been investigated. Xu et al. (2018) ; Zhang et al. (2018a) design a neutralization module to identify the indicators and an emotionalization module to stylize the neuralized text. Li et al. (2018) propose a delete-retrieve-generate pipeline which removes the style indicators based on term frequencies and then generates the output using the remaining content together with retrieved indicators from the target style corpus. Following this pipeline, recent works (Sudhakar et al., 2019; Wu et al., 2019) propose to employ the Transformer (Vaswani et al., 2017) based architecture BERT (Devlin et al., 2019) to extract the style indicators or generate the outputs.

Recent works start to skip the disentanglement step and address the text style transfer task with pseudo-parallel data created from back-translation as in unsupervised machine translation (Artetxe et al., 2018; Lample et al., 2018) . Lample et al. (2019) use self-reconstruction and back-translation to optimize the model. Zhang et al. (2018b) initialize a pair of transfer models (in two directions) using phrase-based translation system and jointly update the models through iterative backtranslation. Luo et al. (2019) further provide global supervision for the transferred output with rewards for style conversion and content preservation. Most recently, Dai et al. (2019) apply the Transformer to this task for its capability to capture long-term dependency.

3. SYNTAX-AWARE STYLE TRANSFER:

We incorporate the syntactic knowledge and address the unsupervised text style transfer task with a multi-task learning framework: the input text is transformed into a sequence of words with aligned syntactic labels. The auxiliary syntactic label generation task encourages the model to learn generalized features which can benefit the prediction of both words and syntactic labels. Currently, our model allows only syntactic labels with one-to-one correspondence to the words. In this paper, we use the POS tags as the syntactic labels, leave other choices or multiple syntactic labels for future study.

3.1. PROBLEM DEFINITION:

Consider a training corpus D = {(x i , s i )} i=1,...,N , where x i is a text sequence and s i ∈ S is the correponding style. S = {s j } j=1,...,m denotes all possible style types. As a preprocessing step, we extend the training corpus D to D + = {(x i , s i , l i )} i=1,...,N by providing an additional syntactic label sequence for each instance. For each text sequence x = {x 1 , . . . , x T }, we obtain its syntactic label sequence l = {l 1 , . . . , l T } using an external NLP tool F.

The objective of our SAST is to learn a joint conditional probability distribution P θ (x, l|x, s) to produce a text sequence x with aligned syntactic labels l, conditioned on a given input text x and a target style s. The output text x should possess the style s while retaining the content (i.e., semantics not exhibiting the style) of x. We denote the marginal probability distribution of x as P w θ (x|x, s), which is the objective for models only leveraging word information.

3.2. MODEL OVERVIEW:

We adopt the sequence-to-sequence encoder-decoder architecture (Sutskever et al., 2014) for our SAST model. For both the encoder and the decoder, we use the recurrent neural networks with the Gated Recurrent Units (GRU) (Cho et al., 2014) . The attention mechenism (Bahdanau et al., 2015) is also utilized. An overview of our model architecture is shown in Figure 1 .

3.2.1. ENCODER:

The encoder, designed as a bi-directional GRU, maps the input text to the latent semantic space. Formally, given the input text x = {x 1 , . . . , x T }, we arrive at a set of hidden vectors

h = {h 1 , . . . , h T } with h t = [ - → h t ; ← - h t ], where - → h t = --→ GRU(h t-1 , x t ) and ← - h t = ←-- GRU(h t+1 , x t )

are the hidden outputs from the forward GRU and the backward GRU respectively.

3.2.2. DECODER:

The decoder, designed as a uni-directional GRU, aims to generate a sequence of (word, syntactic label) pairs conditioned on the encoded hidden vectors h and the target style s. Following Lample et al. (2019) , the style information is involved as the start token of the decoding process. Formally, at the k-th step, the hidden state g k is computed as

g k = GRU([ -→ h T ; ← - h 1 ], s), if k = 1 GRU(g k-1 , xk-1 ), otherwise.

(1)

We then apply the attention mechenism to assign an attention weight α t k = softmax(h t Wg k ) to each source word x t , where W is a parameter matrix. Based on the attention weights, a context 

SAST SAST SAST Dis 𝑥 𝑠 𝑙 ℒ 𝑟𝑒𝑐 𝑥 ሚ 𝑙 𝑙 𝑥 𝑠 𝑥 ሚ 𝑙 ǁ 𝑠 𝑥 𝑏 ሚ 𝑙 𝑏 ℒ 𝑑𝑖𝑠 ℒ 𝑐𝑜𝑛𝑠 ℒ 𝑏𝑎𝑐𝑘

EQUATION

P l θ ( lk |x <k , l<k , x, s) = softmax(U l z k + b l )

(3) where U w , U l , b w , and b l are model parameters.

The shared hidden state z k between the word prediction task and the syntactic label prediction task allows the interaction and complementation between lexical information and syntactic information, thus facilitating learning more generalized features.

Note that we only feed the generated token without the generated syntactic label from previous step as the input for the current step. The reasons are three-fold: (1) the knowledge for previous generated syntactic labels is encoded in g k-1 ; (2) our preliminary experiments did not show further improvements by feeding the syntactic label as input; (3) using only the word as input makes zero additional cost during inference compared to works not using the syntactic information. As a result, we can eliminate l<k from the conditions of Equation 2 and 3. The joint probability distribution P θ (x, l|x, s) and the marginal probability distribution P w θ (x|x, s) can be formulated as

EQUATION

EQUATION

3.3. LEARNING ALGORITHM:

We first warm up the model with a pre-training stage based on self-reconstruction. Then we optimize the model with three losses during the regular training stage: a discrimination loss to encourage correct style transformation, a back-translation loss to ensure content preservation, and a consistency loss to enforce the alignment between the generated words and syntactic labels. Figure 2 shows the training procedure for these two training stages.

3.3.1. SELF-RECONSTRUCTION AS PRE-TRAINING:

For non-parallel corpora, self-reconstruction can be used to guide the model for meaningful output. Given the input text x and its original style s, if the target style s = s, the model should reconstruct x as the output. In our multi-task learning based SAST, for s = s, the model is optimized to reconstruct both the text x and the syntactic labels l. Formally, the self-reconstruction loss is defined as Sample a target style s ∈ S with s = s 10:

EQUATION

Generate words xi ∼ P w θ (x|x i , s) for ∀(x i , s i , l i ) ∈ B 11:

Compute the discrimination loss L dis (θ) based on Equation 712:

Compute the back-translation loss L back (θ) based on Equation 813:

Compute the consistency loss L cons (θ) based on Equation 1114:

Compute total loss L(θ) based on Equation 1215:

Update θ based on ∇ θ L(θ)

16:

end for 17: end for

3.3.2. DISCRIMINATION LOSS FOR STYLE CONVERSION:

To improve the style conversion accuracy, we adopt a pre-trained style discriminator P φ to justify the style of the transferred results. For a given input text x and a target style s, an output x ∼ P w θ (x|x, s) is expected to be predicted as having style s by the discriminator. Therefore, the discrimination loss is defined as

EQUATION

A problem in Equation 7 is the discrete property of the sampling operation x ∼ P w θ (x|x, s) impedes the gradient back-propagation from L dis (θ) to θ. We tackle this problem by replacing the discrete words with soft distributions: each word xk in x is replaced with softmax(o k /τ ), where o k is the logit vector inside the softmax function of Equation 2, and τ is a a temperature hyper-parameter.

3.3.3. BACK-TRANSLATION LOSS FOR CONTENT PRESERVATION:

To ensure content preservation, we adopt a back-translation loss to avoid generating results complying with the target style but irrelevant to the source content. Suppose for a training instance (x, s, l) and a target style s, we sample an output x ∼ P w θ (x|x, s). Then the model is enforced to reconstruct the source words x and syntactic labels l when we feed x as the input text and s as the target style. Formally, the back-translation loss is defined as

EQUATION

Each x, s, x, and l form a pseudo-parallel instance for our model. As training proceeds, the quality of x gets improved and can gradually boost the performance. As in existing methods, the gradients are not back-propagated through x.

3.3.4. CONSISTENCY LOSS FOR WORD-SYNTAX ALIGNMENT:

To guarantee the alignment between the generated words and syntactic labels, we design a consistency loss to pernalize incompatible outputs. Given an input text x and a target style s, we sample an output pair (x, l) ∼ P θ (x, l|x, s). We then annotate x with syntactic labels l using the external tool F, and a legimate transfer should have lk = lk for ∀k. For training stability, we only apply the consistency contraint on the word prediction side. As the annotation operation is non-differential, we resort to the REINFORCE (Williams, 1992) technique to enable gradient estimation. Specifically, we define a consistency reward Q k for step k as the log-probability of l given by the syntactic label prediction module

EQUATION

The total expected consistency reward is

EQUATION

which corresponds to a consistency loss

L cons (θ) = -R cons (11)

3.3.5. FULL TRAINING OBJECTIVE:

Combining the discrimination loss 7, the back-translation loss 8, and the consistency loss 11, The full objective of SAST is to minimize

EQUATION

where λ dis and λ cons are hyper-parameters for balancing the three losses. 

4. EXPERIMENTS:



4.1. DATASETS:

We evaluate our model on two public datasets for unsupervised text style transfer: a sentiment dataset YELP (Li et al., 2018) and a formality dataset GYAFC (Rao & Tetreault, 2018) . The YELP dataset consists of business reviews from Yelp, with each review categorized as positive or negative.

We use the same train-dev-test split from Li et al. (2018) . The GYAFC dataset consists of sentences from Yahoo Answers, with each sentence categorized as formal or informal. We use the data in the Family & Relationship domain. Following Luo et al. (2019) , while GYAFC is a parallel corpus, we do not use the alignment information for training.

4.2. IMPLEMENTATION DETAILS:

We use the POS tags as syntactic labels, which are obtained by the spaCy1 library. The bi-directional GRU encoder is single-layer with 256 hidden units for each direction, and the uni-directional GRU decoder is single-layer with 512 hidden units. The size of the word embeddings is 128. The temperature τ is set to 0.5. The balancing weights in Equation 12are: λ dis = 0.1 and λ cons = 0.005, which are tuned on the development set. We employ the Adam algorithm with a learning rate of 10 -3 for optimization. The batch size is set to 32. We train the model for 20K iterations in the pre-training stage and 60K iterations in the regular training stage. We sample the transferred output x by greedy decoding during both the regular training stage and the inference stage. For the style discriminator, we the TextCNN model architecture Kim (2014) . Our code will be released soon.

4.3. BASELINES:

We compare our SAST to various baselines, including (1) the implicit disentanglement based methods: CrossAligned (Shen et al., 2017) , StyleEmbedding (Fu et al., 2018) , MultiDecoder (Fu et al., 2018), and BackTrans (Prabhumoye et al., 2018) ; (2) the explicit disentanglement based methods: CycledRL (Xu et al., 2018) , TemplateBased (Li et al., 2018) , RetrieveOnly (Li et al., 2018) , Dele-teOnly (Li et al., 2018) , and Del-Ret-Gen (Li et al., 2018) ; (3) non-disentanglement based methods: UnsuperMT (Zhang et al., 2018b) and DualRL (Luo et al., 2019) .

4.4. EVALUATION METRICS:

Different text style transfer models are assessed based on their outputs on the test split of each dataset. The assessment focuses on three aspects: transfer accuracy, content preservation, and fluency. Both automatic evaluation and human evaluation are conducted.

YELP GYAFC ACC (%) ↑ BLEU ↑ PPL ↓ ACC (%) ↑ BLEU ↑ PPL ↓ CrossAligned (Shen et al., 2017) 74.4 17.9 43.0 71.2 3.6 23.8 StyleEmbedding (Fu et al., 2018) 8.2 42.3 43.2 24.3 7.9 60.4 MultiDecoder (Fu et al., 2018) 48.9 27.9 (Li et al., 2018) 84.1 45.5 158.1 50.4 35.2 151.0 RetrieveOnly (Li et al., 2018) 97.7 2.9 41.1 91.6 0.4 43.5 DeleteOnly (Li et al., 2018) 85.6 29.0 56.6 16.9 29.2 82.0 Del-Ret-Gen (Li et al., 2018) 88 (Shen et al., 2017) 2.7 2.6 3.2 2.8 1.5 3.0 MultiDecoder (Fu et al., 2018) 2.0 3.2 3.3 2.3 2.3 2.7 CycledRL (Xu et al., 2018) 2.8 3.4 3.4 2.8 1.3 2.6 TemplateBased (Li et al., 2018) 3.3 3.7 3.5 3.0 3.8 3.4 Del-Ret-Gen (Li et al., 2018) 3.6 3.8 3.9 2.7 2.9 2.9 DualRL (Luo et al., 2019) 4.1 4.1 4.0 3.5 3.7 3.7 SAST 4.1 4.3 4.2 3.7 3.9 3.8

Table 2 : Human evaluation results on the YELP dataset and the GYAFC dataset.

Automatic Evaluation:

The transfer accuracy is measured by the prediction accuracy of a style classifier (same architecure but independent of P φ ) on the model outputs, using the target styles as the ground-truth labels. The content preservation is measured by the case-insensitive BLEU score, calculated using the multi-bleu.perl script, between the model outputs and reference outputs. We use the reference outputs from Luo et al. (2019) for YELP, which provides three more references (four in total) for each sample compared with Li et al. (2018) . The GYAFC Rao & Tetreault (2018) dataset comes with four references for each sample and we directly use them for evaluation. The fluency is measured by the Perplexity (PPL) of the model outputs, resulted from a single-layer GRU based language model learned on the training set.

Human Evaluation:

We invite three human annotators to evaluate the outputs from different models for 200 test samples on each dataset. The annotators rate each transfer result from 1 (the lowest quality) to 5 (the highest quality) in terms of transfer accuracy, content preservation, and fluency.

4.5. RESULTS:

Table 1 presents the automatic evaluation results on the YELP dataset and the GYAFC dataset. Our SAST outperforms the baselines by a clear margin on both datasets in terms of transfer accuracy and content preservation. For fluency, SAST shows better performance than most baselines except BackTrans (Prabhumoye et al., 2018) on YELP and CrossAligned (Shen et al., 2017) on GYAFC. However, despite the low perplexity values, both methods are prone to information loss with low BLEU scores. Overall, by jointly modeling aligned words and syntactic labels, our SAST achieves better balance among the three metrics. Moreover, the non-disentanglement based methods show a clear advantage over the disentanglement based methods which tend to sacrifice the content preservation or fluency for better transfer accuracy (or the other direction).

Table 2 presents the human evaluation results on the YELP dataset and the GYAFC dataset for a subset of baselines due to high evaluation cost. For both datasets, our proposed SAST achieves the best results on all the three aspects. We notice that there exist some inconsistencies between the or with someone with that is but the best CycledRL he is not a good time and do not like it . TemplateBased remember being friends with someone online is ok but there are limit i wish you thes . Del-Ret-Gen you being friends with someone you is ok . yes , but there is you . DualRL remember being friends with someone online is ok but there are limits .

SAST remember being friends with someone online is acceptable , but there are limits .

Table 3 : Transferred outcomes of different models on four exemplary sentences.

human evaluation results and automatic evaluation results. For example, the transfer accuracy of CrossAligned is better than CycledRL on YELP for automatic evaluation results, which is opposite to the human evaluation results. Also, the differences between different models are much smaller for human evaluation than those for automatic evaluation. These phenomena can be attributed to the imperfection of the tools for automatic evaluation. In particular, the pre-trained classifier may not extract the appropriate patterns for specific styles, or give biased predictions (e.g., predicting as "positive" as long as there is a "great"); and the pre-trained language model may have quite different penalties for two equally improper (for human) words.

4.6. QUALITATIVE ANALYSIS:

Table 3 presents the transferred outcomes of different models on four exemplary sentences. The results basically correspond to observations from the quantitative evaluation. On one hand, the implicit disentanglement based CrossAligned and MultiDecoder, tend to alter the original meaning of the input. On the other hand, the explicit disentanglement based CycleRL, TemplateBased, and Del-Ret-Gen, and the non-disentanglement based DualRL either lose some content or make incomplete changes or produce some incompatible words. Our multi-task learning based SAST can generate outputs with better quality. As shown in the first example of Table 3 , the outputs of CrossAligned and MultiDecoder significantly deviates from the original content; although the results of CycleRL, TemplateBased, Del-Ret-Gen, and DualRL change the appropriate position (i.e., essentially wasted my money) and reflect the positive sentiment, their coherence with the remaining content is not satisfying enough. In comparison, the result of our SAST, rewriting the wasted my money to love this place, is syntactically reasonable besides successful sentiment conversion and content preservation, proving the efficacy of modeling syntactic knowledge as an auxiliary task. The generated POS tags for the second example of Table 3 have been shown in Figure 1 .

4.7. ABLATION STUDY:

To provide more insights into different components of our SAST framework, we evaluate several ablated variants on the YELP dataset: (1) NoSyntax which eliminates the syntactic label prediction task and syntax related losses; (2) SAST -L dis which eliminates the discrimination loss;

(3) SAST -L back which eliminates the back-translation loss; and (4) SAST -L cons which eliminates the consistency loss. The automatic evaluation results are shown in Table 4 . Comparing SAST and NoSyntax, we can conclude modeling the syntactical information mainly improves content preservation. As for the different losses, we can observe: the discrimination loss has an important influence on the transfer accuracy; the back-translation loss is responsible for keeping the style-independent content; and the consistency loss further improves the content preservation. Furthermore, Table 4 shows the outputs of different ablated variants for transferring a negative sentence to a positive one. We can see SAST can generate the most reasonable result, while other variants can suffer from content loss, incomplete changes, or syntax errors. 

5. CONCLUSIONS:

In this paper, we propose a multi-task learning based syntax-aware approach for the unsupervised text style transfer task. By jointly generating aligned words and syntactic labels, our SAST model learns more generalized latent representations explaining both lexical and syntactic knowledge, thus avoiding overfitting on the observed words. Quantitative and qualitative results on the two benchmark datasets demonstrate the benefits of our approach in terms of style conversion, content preservation, as well as fluency. Currently, we only use the shallow syntax, i.e., POS tags, as the syntactic labels, and it is promising to further explore other kinds of syntactic information (such as chunking labels and dependency labels) and investigate integrating multiple different syntactic labels. However, our SAST framework is limited to syntactic labels with a one-to-one correspondence to the words, unable to handle more complex syntax such as constituency/dependency parsing. Therefore, we would like to adapt our framework to address this limitation in our future research.

:

https://spacy.io

Figures and Tables:

Figure 1: Model Overview.

Figure 2: Training procedure of the proposed SAST. Left: the pre-training stage. Right: the regular training stage.

)To avoid learning a trivial solution by copying source words, followingShen et al. (2017);Lample et al. (2019), we feed the model with a corrupted variant x * (by word removal/permutation) of x.Algorithm 1The training algorithm for Syntax-Aware Style Transfer (SAST).1: Input: non-parallel training corpus D and an external NLP tool F 2: Pre-train a style discriminator P φ on D 3: Expand D to D + by adding syntactic labels using F 4: Make m splitions D + [s j ] j=1,...,m = {(x i , s i , l i )|(x i , s i , l i ) ∈ D + ∧ s i = s j } 5: Pre-train θ on D + by minimizing L rec (θ) 6: for each iteration r = 1, 2, . . . , L do 7:for each style s ∈ S do 8:Sample a minibatch of samples B = {(x i , s i , l i )} i=1,...,n from D + [s]9:

Our SAST belongs to the non-disentanglement based methods as with Zhang et al. (2018b); Lample et al. (2019); Luo et al. (2019), however, it is straightforward to adapt this multi-task learning scheme to the disentanglement based methods. The training algorithm is summarized in Algorithm 1.

Table content: <table/>

Automatic evaluation results on the YELP dataset and the GYAFC dataset.

Table content: <table><tr><td/><td>.3</td><td>31.1</td><td>54.6</td><td>51.7</td><td>21.2</td><td>73.1</td></tr><tr><td>UnsuperMT (Zhang et al., 2018b)</td><td>97.6</td><td>44.5</td><td>50.2</td><td>66.5</td><td>33.4</td><td>45.4</td></tr><tr><td>DualRL (Luo et al., 2019)</td><td>89.5</td><td>55.2</td><td>43.3</td><td>63.5</td><td>41.9</td><td>50.7</td></tr><tr><td>SAST</td><td>98.3</td><td>60.2</td><td>37.5</td><td>94.1</td><td>45.5</td><td>42.5</td></tr></table>

Ablation study on the YELP dataset.

Table content: <table/>

Peer review (iclr2020):
The authors introduce an unsupervised text style transfer model that adds syntactic features (part-of-speech tags) to the word-level output targets and input. The authors use a fairly simple neural architecture (as far as text style transfer goes), and achieve state-of-the-art style transfer, compared to many other models on automatic metrics and human evaluations. The overall idea (adding syntax information to models) is an interesting one to explore. The empirical results and evaluation show that the overall model is strong; however, I would like to see the authors* main claim (that adding syntax is helpful) better explored. The proposed model differs from other style transfer models is a number of ways (architecture and training losses), so it*s unclear how much of the gains come from syntax. The authors include one (very helpful) ablation in Table 4 (row 2) that shows that removing syntax entirely from their model hurts performance. However, I think it*d be helpful to explore the role of syntax in more depth, i.e.: 1) Is syntax helpful because it is an additional prediction target/auxiliary loss? Or it is helpful because it is an additional input feature to the model? Or both? If it is only one or the other, it may be possible to simplify the model and/or improve other model architectures by making a simple modification. (For example, what happens when you add syntax as an auxiliary prediction target to another model?) 2) In what kinds of cases syntax is helpful/hurtful? What sorts of errors does having syntax help with? 3) Do other kinds of syntax-related annotations help? 4) How do humans rate the fluency of the no-syntax version of the proposed model (i.e., the evaluation in Table 2)? It would be interesting e.g. if style and content ratings stayed the same, but fluency ratings went down (or if something else happened) A second major concern that I have is that syntax may not be helpful for style transfer when using pre-trained language models (i.e., as in XLM from from *Cross-Lingual Language Model Pretraining*). XLM showed that the initialization is very important for strong unsupervised results with back-translation and denoting auto-encoding/self-reconstruction. Thus, in the long run, it seems that unsupervised style transfer will move towards using pre-trained language models, which already learn a great deal about syntax implicitly. So I am concerned that the gains that the authors report now will not hold in the future. Ideally, I would like to see a comparison of style transfer results between XLM and XLM+Syntax. Questions/Other Notes: * Consistency Reward in Sec. 3.3.4: I might be missing something, but why not use supervised learning to train the label predictor to predict the labels from an external syntax annotation tool (rather than using REINFORCE)? In general I found this subsection a bit confusing and hard to follow * It would be nice to add XLM to the related work section, as the model architecture and losses seem similar (except without syntax). Other than that, the related work is written very clearly! * It would be helpful to add some discussion/citations about where syntax has been shown useful in related contexts (I felt that mostly unsupervised style transfer work was discussed)