Title: STRUCTURED CONTENT PRESERVATION FOR UNSU-PERVISED TEXT STYLE TRANSFER

Abstract: Text style transfer aims to modify the style of a sentence while keeping its content unchanged. Recent style transfer systems often fail to faithfully preserve the content after changing the style. This paper proposes a structured content preserving model that leverages linguistic information in the structured fine-grained supervisions to better preserve the style-independent content 1 during style transfer. In particular, we achieve the goal by devising rich model objectives based on both the sentence's lexical information and a language model that conditions on content. The resulting model therefore is encouraged to retain the semantic meaning of the target sentences. We perform extensive experiments that compare our model to other existing approaches in the tasks of sentiment and political slant transfer. Our model achieves significant improvement in terms of both content preservation and style transfer in automatic and human evaluation.

1. INTRODUCTION:

Text style transfer is an important task in designing sophisticated and controllable natural language generation (NLG) systems. The goal of this task is to convert a sentence from one style (e.g., negative sentiment) to another (e.g., positive sentiment), while preserving the style-independent content (e.g., the name of the food being discussed). Typically, it is difficult to find parallel data with different styles. So we must learn to disentangle the representations of the style from the content. However, it is impossible to separate the two components by simply adding or dropping certain words. Content and style are deeply fused in every sentence.

Previous work on unsupervised text style transfer, such as Hu et al. (2017) ; Shen et al. (2017) , proposes an encoder-decoder architecture with style discriminators to learn disentangled representations. The encoder takes a sentence as an input and generates a style-independent content representation. The decoder then takes the content representation and the style representation to generate the transferred sentence. However, there is no guarantee that the decoder will only change the styledependent content, as these methods do not have any constraints to preserve the content.

We propose a new text generative model to preserve the content while transferring the style of a sentence at the same time. We first use an attentional auto-encoder (AE) and a binary style classifier to generate sentence with the target style. Then we use the distance between POS information of the input sentence and output sentence as an error signal to the generator. Since we use the positivenegative Yelp review data (Shen et al., 2017) and the political slant data as the benchmark to test our algorithm. Most of the content of these reviews and comments are captured by the nouns of the sentence. Therefore, we enforce the decoder to generate sentences that have similar nouns. So in our implementation, the POS information simply reduces to noun information. For other style transfer tasks, different POS tags, such as verbs, may capture the content better. Then the model needs to adapt to use other tags instead of the nouns for the constraints. Besides these explicit constraints, we also add an additional constraint, a language model conditioned on both the nouns and the style representations, to enforce the output sentence to be fluent and contain desired nouns.

We evaluate our model on both the sentiment modification and the political slant transfer tasks. Results show that our approach has better content preservation and a higher accuracy of transferred style compared to previous models in both automatic and human evaluation metrics.

2. RELATED WORK:

Recently, there are many new models designed for non-parallel text style transfer. Our structured content preserving model is closely related to several previous work describe below: Delete, Retrieve and Generate According to the observation that text styles are often marked by distinctive phrases (e.g., "a great fun), Li et al. (2018) extracts content words by deleting phrases associated with the sentences original style, retrieves new phrases associated with the target style, and uses a neural model to smoothly combine them into a final output. However, in some cases (Pavlick & Tetreault, 2016) , content and style cannot be so cleanly separated only using phrase boundaries. Lample et al. (2017) ; Artetxe et al. (2017) propose sophisticated methods for unsupervised machine translation. Rabinovich et al. (2016) shows that author characteristics are significantly obfuscated by both manual and automatic machine translation. So these methods could in principle be used as style transfer. Prabhumoye et al. (2018) first uses back-translation to rephrase the sentence to get rid of the original style. Then, they generate a sentence based on the content using a separate style-specific generator. However, after back-translation, the generated sentence does not contain the detailed content information in the input sentence.

Back-Translation:

Adversarial Training Some work (Shen et al., 2017; Zhao et al., 2017; Fu et al., 2017; Melnyk et al., 2017) uses adversarial training to separate style and content. For example, Fu et al. (2017) proposes two models: multi-decoder and style-embedding. Both of them learn a representation for the input sentence that only contains the content information. Then the multi-decoder model uses different decoders, one for each style, to generate sentences in the corresponding style. The styleembedding model, in contrast, learns style embeddings in addition to the content representations. Shen et al. (2017) also encodes the source sentence into a vector, but the discriminator utilizes the hidden states of the RNN decoder instead. When applying adversarial training to unsupervised text style transfer, the content encoder aims to fool the style discriminator by removing style information from the content embedding. However, empirically it is often easy to fool the discriminator without actually removing the style information. Besides, the non-differentiability of discrete word tokens makes the generator difficult to optimize. Hence, most models attempt to use REINFORCE (Sutton et al., 1999) to fine tune trained models (Yu et al., 2016; Li et al., 2017) or use professor forcing method (Li et al., 2017) to transfer the style.

Toward Controlled Generation of Text Hu et al. (2017) is most relevant to our work. Their model aims at generating sentences with controllable styles by learning disentangled latent representations (Chen et al., 2016) . It builds on variational auto-encoders (VAEs) and uses independency constraints to enforce styles reliably inferred back from generated sentences. Our model adds more constraints to preserve the style-independent content by using POS information preservation and a contentconditional language model.

3. STRUCTURED CONTENT PRESERVING MODEL:

The overall model architecture is shown in Figure 1 . We develop a new neural generative model which integrates an attentional auto-encoder, a style classifier, a POS information preservation constraint and a language model that conditions on content. We will discuss each component in details below.

3.1. ATTENTIONAL AUTO-ENCODER:

Assume there are two non-parallel text datasets X = {x 1 , x 2 ,. . . , x m } and Y = {y 1 , y 2 ,. . . , y m } with two different styles v x and v y , respectively. We let v x be the positive sentiment style and v y be the negative sentiment style. Our target is to generate sentence of the desired style while preserving the content of the input sentence. Assume the style representation v is sampled from a prior p(v) and the content representation z is sampled from p(z). So the sentence x is generated from the conditional distribution p(x|z, v) and our goal is to learn: The above equation is the encoder and decoder framework to generate sentences. One unsupervised method is to apply the attentional auto-encoder model. We first encode the sentence x i or y i to obtain its style-independent content representation xi = E(x i , v x ) or ŷi = E(y i , v y ). Then we use an attentional decoder G to generate sentences conditioned on xi or ŷi and v x or v y . So we can get the reconstruction loss L x res :

p(y|x) = zx p(y|z x , v y )q(z x |x, v x )dz x (1)

L res (θ E |θ G ) = E x∼X [-log p G (x|x i , v x )] + E y∼Y [-log p G (y|ŷ i , v y )]

(2)

After switching the style representation, we can get the generated sentence ỹi = {ỹ 1 , ỹ2 , . . . , ỹM }:

EQUATION

where ỹ<m indicates the tokens preceding ỹm and M denotes the number of tokens in ỹi .

3.2. STYLE CLASSIFIER:

Since the attentional auto-encoder cannot fully enforce the desired style, we deploy a style classifier to predict the given style. We feed the result of the prediction back to the encoder-decoder model.

The generator is then trained to generate sentences that maximize the accuracy of the style classifier.

However, it is impossible to propagate gradients from the style classifier through the discrete samples. We thus resort to a Gumbel-softmax (Jang et al., 2016) distribution as continuous approximation of the sample. For each step in the generator, let q be a categorical distribution with probabilities π 1 , π 2 , . . . , π c . Samples from q can be approximated by:

u i = e (log(πi)+gi)/τ c j=1 e (log(πj )+gj )/τ

(4) g i or g j denotes independent samples from Gumbel(0, 1). The temperature τ is set to τ → 0 as training proceeds, yielding increasingly peaked distributions that finally emulate the discrete case.

The approximation replaces the sampled token ỹi (represented as a one-hot vector) at each step with u. Then the probability vector u is used as the input to the classifier. With the continuous approximation, we use the following loss to improve the generator:

EQUATION

Where the Gτ (x i , v y ) and Gτ (ŷ i , v x )) denote the resulting "soft" generated sentence and p C denotes the loss in the style classifier.

In most cases, the classifier is unstable and insufficient to preserve content. We propose to use POS information preservation and a language model to preserve content.

3.3. POS PRESERVATION CONSTRAINTS:

We propose to preserve the content based on the POS tag information. Since we focus on Yep review and political comments and their content are mostly captured by nouns, we only consider the noun tags of the POS information. We use a POS tagger to extract nouns from both input sentences and generated sentences. We denote the extracted noun set of the input as N x and the noun set in the output as Ñy , where N x = {n 1 , n 2 , . . . , n p } and Ñy = {ñ 1 , ñ2 , . . . , ñm }. Then we embed these nouns using pretrained GloVe (Pennington et al., 2014) to measure the semantic similarity between the input and output sentences. We embed each noun token in N x and Ñy :

EQUATION

Where E x or Ẽy denotes the embedding sets and e 1 or ẽ1 denotes the embedding of the nouns. However, the structure of the output sentence can be different from the input. Therefore, we need to match corresponding noun pairs between N x and Ñy . Assume that the corresponding noun of n i in N x is ñj in Ñy when ñj is most similar to n i among all the nouns in Ñy . We add up the cosine similarity between the two word embeddings of all the corresponding noun pairs and normalize it by the number of nouns in N x as a basic loss.

In practice, the number of nouns in N x may not be equal to the ones in Ñy , which means the generated sentence contains more or less nouns than the original sentence. So calculating the similarity of the two sentences can be challenging. Therefore, we include a weight γ to handle those cases to ensure a fair comparison. When the number of nouns in N x is equal to that in Ñy , γ is set to 1. When the number of nouns in N x is different to the number in Ñy , γ is set to be the absolute difference normalized by the number of nouns in N x . So the POS distance is:

EQUATION

Where

γ = 1 + (max(c x N , c y Ñ ) -min(c x N , c y Ñ ))/c

x N , the distance set denotes as D, the number of nouns in N x denotes as c x N and the number of nouns in Ñy denotes as c y Ñ . The POS distance can be viewed as an overall evaluation score of the output sentence. If the output does not contain semantically similar nouns of the input, the overall evaluation score will be high. The generator thus can be enhanced to generate sentence with desired nouns. In this way, the styleindependent content can be better preserved.

3.4. CONTENT CONDITIONAL LANGUAGE MODEL:

Besides the sentence POS information preservation, we use a content conditional language model to enforce the output sentence to contain desired nouns and at the same time to make the output sentence more fluent. Instead of comparing the difference between N x and Ñy directly, we only use N x to give noun information to the generator implicitly. We combine the average of E x (Equation 6) with the style label as the hidden state h lm and send the hidden state to the language model. Thus the language model is conditioned on both the nouns and the style. Then we use the discrete samples from the input x i and the hidden state h lm to train the language model:

EQUATION

In order to use standard back-propagation to train the generator, we use the continuous approximation in Equation 4. We denote the continuous approximation of the output of the decoder as py = {p y t } T t=1 , which is a sequence of probability vectors. For each step we feed py t to the language model using the weighted average of the embedding W py t , then we get the output from the language model which is a probability distribution over the vocabulary of the next word py t+1 . Finally, the language model loss is the perplexity of the generated samples evaluated by the language model becomes:

EQUATION

If a sentence does not contain semantically similar nouns from the input, it will have a high perplexity under a language model trained on desired nouns. A language model can assign a probability to each token, thus providing more information on which word is to blame for overall high perplexity.

Therefore, the training objective for our structured content preserving model is to minimize the four types constraints together: the reconstruction in the attentional auto-encoder, the style classifier, the POS information and the language model:

EQUATION

The model training process consists of three steps. First, we train the language model according to Equation 9. Then, we train the classifier. Finally, we minimize the reconstruction loss, classification loss, POS loss and perplexity of the language model together.

4. EXPERIMENTS AND RESULTS:

We experiment on two types of different transformations: sentiment and political slant, to verify the effectiveness of our model. Compared with a broad set of exiting work, our model achieves significant improvement in terms of both content preservation and style transfer.

4.1. SENTIMENT MANIPULATION:

We first investigate whether our model can preserve the sentiment-independent content better while transferring the sentiment of the sentence. We compare our model with Li et al. (2018) . We also evaluate our model without language model supervision in an ablation study.

4.1.1. DATASET AND EXPERIMENTAL SETTING:

We use the Yelp review dataset. The dataset contains almost 250K negative sentences and 380K positive sentences, of which 70% are used for training, 10% are used for evaluation and the remaining 20% are used as test set. Sentences of length more than 15 are filtered out. We keep all words that appear more than five times in the training set and get a vocabulary size of about 10k. All words appearing less than five times are replaced with a <UNK> token. At the beginning of each sentence, we add a <BOS> token and at the end of each sentence, we add a <EOS> token. We use <PAD> token to pad sentences that have less than 15 tokens. Appendix A shows the model configurations.

4.1.2. AUTOMATIC EVALUATION RESULTS:

Following previous work (Hu et al., 2017; Shen et al., 2017; Li et al., 2018) , we compute automatic evaluation metrics: accuracy and BLUE score. Besides these two basic metrics, we also use the POS information distance (Equation 8) as new metric to test the model's content preserving ability.

We first use a style classifier to assess whether the generated sentence has the desired style. The classifier is based on CNN and trained on the same training data. We define the accuracy as the fraction of outputs classified as having the targeted style. However, simply evaluating the sentiment of the sentences is not enough, since the model can generate collapsed sentences such as a single word "good". We also measure the POS information distance and BLEU score of the transferred sentences against the original sentences. Besides these metrics, Li et al. (2018) provides 1000 human annotated sentences. These sentences are manually generated by human that have the desired style and the preserved content. We use them as the ground truth and use BLEU score of transferred sentences against these human annotated sentences as an important metric to evaluate all the models.

We report the results in (Li et al., 2018) . The POS distance denotes the noun difference between the original and transferred sentences. The smaller the POS distance, the better the performance. lm here represents language model. sentences. Such result is not surprising, because they directly retrieve the most similar sentences from the other corpus. However, there is no mechanism to guarantee that the corpus with the target style contains the sentence that has similar content of the original sentence. The StyleEmbedding model in Fu et al. (2017) has the highest BLEU score while its accuracy is very low. Such result is not surprising because the model does not have any constraints on the generated style. The results of these two models show that a good model should perform well on both the accuracy and the BLEU score. Comparing our model with other baselines, we can see that our model outperforms them in both aspects: getting higher accuracy and preserving the content better. This demonstrates the effectiveness of our structure content preserving model.

Having both the POS and the language model constraints might look repetitive at first. Therefore, we experiment the model with and without the language model constraint. The result indicates having the language model is useful even if already having the POS constraint. The model improves from 23.7 to 24.9 in BLEU(human). Because POS constraint focuses on preserving the word semantics, while the language model helps the generator to select the appropriate words conditions on the content information. In addition, the language model helps with fluency.

Table 1 also shows that in most cases, if the model has a high BLEU score, it will usually have a small POS information distance. It seems that it is not necessary to use the POS infomration. However, when we compare our model with StyleEmbedding (Fu et al., 2017) , we can find that StyleEmbedding has both higher BLEU score and a higher POS information distance than ours. The BLEU (human) shows that our model preserves the content better than StyleEmbedding. BLEU(Human) and the POS information distance is correlated. It is not surprising that the POS information distance measures the semantic similarity of corresponding nouns instead of counting the number of same tokens like BLEU score. For example in a good transfer case that transfers "The meal is great!" into "The food is terrible", the BLEU score of this generated sentence is low while the score of POS information distance is not. Therefore, The POS information distance is better than BLEU score to evaluate the ability of the model in preserving contents.

4.1.3. HUMAN EVALUATION RESULTS:

While the automatic evaluation metrics provide some indication of transfer quality, it does not capture all the aspects of the style transfer task. Therefore, we also perform human evaluations. Li et al. (2018) provide 500 sentences with positive sentiment and 500 sentences with negative sentiment. All the 1000 sentences are randomly selected from the test set in Yelp review dataset. They also provide the generated sentences from their model and other models (Shen et al., 2017; Fu et al., 2017) . We first use these 1000 sentences as input and generate sentences from our model and other two baselines (Hu et al., 2017; Prabhumoye et al., 2018) . It is hard for users to make decisions when presented with multiple sentences at the same time. So instead of comparing all the models at the same time, we compare our model with all other models one by one. In each human evaluation, we randomly select 100 sentences from the set. Then for each original sentence, we present two corresponding outputs of our model and a baseline model in a random order. The annotator is then asked "Which sentence has an opposite sentiment of the original sentence and at the same time preserves the content of it?" They can choose: "A", "B" or "the same". We also hired different annotators to rate each pair of models, so nobody will see the same sentence twice. Original the service has always been wonderful . Transferred the service has not been terrible .

Original the wait staff is extremely attractive and friendly ! Transferred the wait staff is extremely attractive and rude ! Table 3 : Some bad examples generated by our model.

4.1.4. ANALYSIS:

We list some examples of transferred sentences in Table 5 in Appendix. Both Hu et al. (2017) and Shen et al. (2017) change the meaning of the original sentences (e.g. "the menudo here is perfect." → "the terrible here is awkward."). It shows that only using classifier as an error signal to the generator is unstable. Sometimes it changes the style-independent tokens into sentiment tokens to reach a high accuracy in the style classifier. Prabhumoye et al. (2018) always change the detailed content information (e.g. "the menudo here is perfect." → "the fare is horrible"). Li et al. (2018) sometimes changes the structure of the original sentences (e.g. "if you travel a lot do not stay at this hotel." → "would highly recommend this place if you travel a lot at this hotel."). Compared with other models, our model preserves the content better while changing the sentiment (e.g. "the service was excellent and my hostess was very nice and helpful." → "the service was terrible and my hostess was very disappointing and unhelpful."). Because after adding constraints on POS tags, the output sentences preserve the content better. The content-conditional language model also feeds the noun information to the generator. Therefore, the source and generated sentence has semantically Therefore, it is more likely to transfer the sentence to the desired style correctly.

However, there are still some cases that our model cannot handle well. We show some examples in Table 3 . If the sentiment of the sentences are not obvious (e.g. "actually, just keep walking."), the generated sentences sometimes would be the same as the input sentences. In some cases, our transferred sentence has double negation (e.g. the service has not been terrible) which results in error in the style transferred. For sentences with complex sentence structures, such as conjunction, in some rare cases, we miss certain part of sentence in the conversation (e.g. "the wait staff is extremely attractive and rude!")

4.2. POLITICAL SLANT TRANSFER:

We have demonstrated that the structured content preserving model can successfully preserve the content while transferring the sentiment of the sentence. To verify the robustness of our model, we also use it to transfer the political slant between democratic and republican party. Compared with Prabhumoye et al. (2018) ; Hu et al. (2017) , our model still achieves improvement in terms of both content preservation and style transfer.

The political dataset is comprised of top-level comments on Facebook posts from all 412 current members of the United States Senate and House who have public Facebook pages (Voigt et al., 2018) . The data set contains 270K democratic sentences and 270K republican sentences and the preprocessing steps and the experiment configurations are the same as the previous experiment.

We use three automatic metrics: accuracy, BLUE score and POS information distance to evaluate our model against Prabhumoye et al. (2018) ; Hu et al. (2017) . We report the results in Table 4 . Our model outperforms the other two models in all three metrics. However, the BLEU score is lower than the score in the sentiment modification task (56.6 VS 63.3). This is because names of politicians (e.g. Donald Trump) are relevant to the political slant. The classifier therefore will enforce the generator to transfer these names to those with another political slant. However, the POS constraints will preserve these people names. Therefore, the two constraints have a conflict and make the generation results less ideal. However, our model can still retain high BLEU score and low POS distance if there are no explicit people names in the source sentences.

5. CONCLUSION:

Based on the encoder-decoder framework, we develop a new neural generative model, names structured content preserving model. It integrates an attentional auto-encoder, a style classifier, a POS constraint and a content-conditional language model. Our model outperforms previous models in both the sentiment and political slant transfer tasks. Compared with other models, our model preserve the content better while changing the style of the original sentence. We also propose a new evaluation metric: POS distance, which measures the content preservation between the source and transferred sentence to evaluate the content preserving ability of the model.

:

Henceforth, we refer to style-independent content as content, for simplicity.

Figures and Tables:

Figure 1: Architecture of our proposed model. x i denotes the input sentence. xi denotes the styleindependent content representation. v x and v y denote the different style representations. ỹi denotes the generated sentence. Ñy denotes the noun set extracted from ỹi . L x res , L x P OS , L x class , L x lm and L x denote the loss function. The red line shows the generation process. The yellow, blue, green and pink lines illustrate the loss function formulation.

The Retrieval method inLi et al. (2018) has the highest accuracy but has very low score in both BLEU against original sentences and BLEU against human transferred

Table content: <table><tr><td colspan="4">Model Accuracy(%) BLEU BLEU(human) POS distance</td></tr><tr><td>Hu et al. (2017) 86.7</td><td>58.4</td><td>22.3</td><td>1.038</td></tr><tr><td>Shen et al. (2017) 73.9</td><td>20.7</td><td>7.8</td><td>3.954</td></tr><tr><td>Prabhumoye et al. (2018) 91.2</td><td>2.8</td><td>2.0</td><td>5.923</td></tr><tr><td>Fu et al. (2017):</td><td/><td/><td/></tr><tr><td>StyleEmbedding 8.1</td><td>67.4</td><td>19.2</td><td>1.734</td></tr><tr><td>MultiDecoder 46.9</td><td>40.1</td><td>12.9</td><td>3.451</td></tr><tr><td>Li et al. (2018):</td><td/><td/><td/></tr><tr><td>Delete 85.5</td><td>34.6</td><td>13.4</td><td>1.816</td></tr><tr><td>Retrieval 97.9</td><td>2.6</td><td>1.5</td><td>5.694</td></tr><tr><td>Template 80.1</td><td>57.4</td><td>20.5</td><td>0.867</td></tr><tr><td>DeleteAndRetrieval 88.9</td><td>36.8</td><td>14.7</td><td>2.053</td></tr><tr><td>Our model (without lm) 90.1</td><td>60.4</td><td>23.7</td><td>0.748</td></tr><tr><td>Our model 92.7</td><td>63.3</td><td>24.9</td><td>0.569</td></tr></table>

Our model and baselines performance. The BLEU (human) is calculated using the 1000 human annotated sentences as ground truth from

Table content: <table/>

shows the human evaluation results. Evaluators prefer our model than other models in general. Results in Table1show that the Retrieval model inLi et al. (2018) has the highest accuracy and StyleEmbedding model inFu et al. (2017) has the highest BLEU score. However, there are 72 sentences generated from our model that are better than those from these two models respectively. Sentences generated from our model have a higher overall quality.

Table content: <table><tr><td colspan="3">Model Our Model(%) Other Model(%) The Same(%)</td></tr><tr><td>Shen et al. (2017) 67</td><td>6</td><td>27</td></tr><tr><td>MultiDecoder 85</td><td>5</td><td>10</td></tr><tr><td>Hu et al. (2017) 55</td><td>17</td><td>28</td></tr><tr><td>Li et al. (2018) 72</td><td>28</td><td>0</td></tr><tr><td>styleEmbedding 72</td><td>4</td><td>24</td></tr><tr><td>Prabhumoye et al. (2018) 56</td><td>4</td><td>40</td></tr></table>

Our model is generally preferred over other models in human evaluation

Table content: <table><tr><td>Category Sentence</td></tr><tr><td>Original actually , just keep walking .</td></tr><tr><td>Transferred actually , just keep walking .</td></tr></table>

Our proposed model outperforms other results on the political slant dataset. similar content. Our style classifier precision is high as well. Because after adding constraints to nouns, the generator can only change other style-dependent parts, such as adjectives in the sentence.

Table content: <table><tr><td colspan="3">Model Accuracy(%) BLEU POS distance</td></tr><tr><td>Prabhumoye et al. (2018) 86.5</td><td>7.38</td><td>7.298</td></tr><tr><td>Hu et al. (2017) 90.7</td><td>47.5</td><td>3.524</td></tr><tr><td>Our model 92.4</td><td>56.6</td><td>2.837</td></tr></table>

Peer review (iclr2019):
 Structured Content Preservation in Style Transfer This paper presents a method for style transfer that using an autoencoder. The autoencoder trained to generate a meaning representation of the sentence. A decoder does the style transfer from this meaning representation. The decoder is trained to minimize style transfer loss and content loss. The new contribution here is a content loss (referred to as POS loss in the paper), which serves to penalize models that do not retain nouns in the output. Strengths: 1. The paper addresses an important problem in style transfer, one of balancing content loss versus changing style. 2. The experiments include a manual evaluation in addition to automatic evaluation. 3. The evaluations show that the proposed method is effective compared to style transfer methods that don*t include this noun-based content loss. Issues: 1. The manual evaluation seems to have conflated two dimensions of evaluation into a single rating. Specifically, the evaluation question is: “Which sentence has an opposite sentiment of the original sentence and at the same time preserves the content of it?” What happens in cases where A preserves the content but does not style transfer and B transfers style but does not preserve content. Which one would the annotators choose? It would be useful to obtain two ratings and combine them instead of using a single one. 2. The idea of enforcing a content loss is well motivated but the approach considered seems rather heuristic. Selecting nouns is a useful start but as the authors point out that not all content words are nouns, and not all nouns are content words always. It would have been useful to consider other categories and quantify their impact in style transfer. 3. It would have been useful to include an ablation where only the noun lm is used but not the training loss. 4. The main gains come from using nouns as the set of content to preserve. I wonder if there is a simpler baseline that uses the noun-based restriction also. A simple baseline for this idea could be to rescore beams using the same loss function (at test time only). 5. Is there some information on inter-annotator agreement on this task. I suspect the task is straightforward enough that this might not be an issue but it would be useful to know. 6. The difference in BLEU (Human) between the systems is not that large. However, the difference in terms of the manual evaluation is strikingly large. This kind of variance is observed often in language generation tasks. The authors should consider explaining why this difference is so large with a bit more analysis. 7. Why is the paper titled "structured"? It was not obvious to me what kind of structure is being inferred or used in this problem. Also, calling the POS distance is confusing as it seems to indicate that there is some POS tag sequence comparison being done here, which is not the case. Overall this is a focused contribution that presents a simple yet effective empirical advance on style transfer for two types of tasks.