SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 4: Frequencies of words in the Amazon Dataset that appear often enough in specific classes to erroneously cause the classifier to make incorrect predictions. Pos. denotes the positive sentences, Neg. denotes the negative sentences.
Model	Text
Source
Adv	don t waste your time or money on these jeans .
don t need your time or money on these phones .
Source
Adv	i made beef bolognese in the oven and it turned
out wonderfully .
i made beef bolognese in the game and it turned
out wonderfully .
Source
Adv	this one does the job i need it for !
this game does the job i need it for !
Table 5: Adversarial examples received high style transfer accuracy scores on Amazon Dataset. Adv denotes the adversarial examples generated by DIRR-AMAZON-ADV.
ture, we additionally fine-tuned a BERT-based (Devlin et al., 2019) classifier, which yielded 51.3, 57.6, 70.4 accuracy on human references, DIRR, DIRR-AMAZON-ADV respectively, showing the same pattern of the fastText classifier. We notice that some two-stage models (Li et al., 2018; Sudhakar et al., 2019; Madaan et al., 2020) and other methods (Yang et al., 2018; Luo et al., 2019) also use a fixed classifier or use words with unbalanced frequencies in different styles as important features, which means that their methods may face the same risk. While Li et al. (2018) has pointed out this data imbalance problem of the Amazon dataset, we further demonstrate that a strong generator can even use this discrepancy to trick the automatic metrics. We are able to mitigate this problem by updating the style classifier during the training, and in Table 4, DIRR is more robust to the data imbalance problem compared to other methods.
The automatic evaluation results are shown in Table 6. We report the performance of the previous methods based on the outputs they provided for fair comparison and omit those whose results are not available.
We have the following observations of the results. First, compared to our base model (DIRR-CYCLE), the model trained with our proposed rewards has higher fluency, while remains the same level of content preservation. It indicates that SIM score is as effective as cycle-consistency loss for content preservation and our fluency reward can effectively improve the output fluency. Secondly, there exists a trade-off among the style transfer accuracy, content preservation and language fluency. While our model does not outperform the previous meth
7Notice that the style classifier only achieves 43 accuracy on the human references.
Model	Acc	PPL	r-BLEU	s-BLEU
Yelp
D&R	89.0	362	10.1	29.1
B-GST	86.0	269	14.5	35.1
Cycle-Multi	87.6	439	19.8	55.2
Deep-Latent	86.0	346	15.2	40.7
Tag&Gen	88.7	355	12.4	35.5
DIRR-CYCLE	91.7	392	18.7	51.2
DIRR	94.2	292	20.7	52.6
Copy	4.1	204	22.5	100.0
Human	70.7	236	99.3	22.5
Amazon
D&R	50.0	233	24.1	54.1
B-GST	60.3	197	20.3	44.6
Tag&Gen	79.9	312	27.6	62.3
DIRR-CYCLE	68.4	374	29.0	60.6
DIRR	62.2	205	30.1	61.3
Copy	21.1	218	40.0	100.0
Human	43.0	209	100.0	40.0
IMDb
Cycle-Multi	77.1	290	N/A	70.4
DIRR-CYCLE	80.5	253	N/A	64.3
DIRR	83.2	210	N/A	64.2
Copy	5.3	147	N/A	100.0
GYAFC
D&R	51.2	226	14.4	27.1
DualRL	62.0	404	33.0	50.8
DIRR-CYCLE	76.2	162	44.1	66.5
DIRR	71.8	145	46.3	59.9
Copy	15.8	147	41.5	98.5
Human	84.5	137	97.8	21.5
Table 6: Automatic Evaluation. Acc is the accuracy of the sentiment classifier. PPL is the perplexity assigned by the GPT-2 language model. r-BLEU is the BLEU score between the human references and system outputs. s-BLEU is the BLEU score between the source sentences and system outputs. Copy is an oracle which copies the source sentences as outputs. Human denotes the human references.
ods on all of the metrics, it is able to find a better balance of the different metrics.
We conducted human evaluation on Yelp, Amazon and GYAFC datasets evaluating the style transfer accuracy, content preservation, and fluency separately. The first two aspects are rated with range 1 - 3 while the fluency is rated with range 0 - 1. We randomly select 100 candidates and compare the outputs of different systems. We use Amazon Turk<sup>8</sup> for human evaluation. Each candidate is rated by three annotators and we report the average scores here. We did not evaluate the style
Dataset	Model	Style	Flu.	Con.	Mean
Yelp	Cycle	2.24	0.62	1.97	2.02
B-GST	2.42	0.64	2.02	2.12
DIRR	2.42	0.66	2.04	2.14
Amazon	Tag&Gen	1.98	0.87	1.95	2.19
B-GST	2.04	0.89	1.77	2.16
DIRR *	2.09	0.87	2.10	2.26
GYAMC	D&R	N/A	0.40	2.13	1.66
DualRL	N/A	0.51	2.23	1.88
DIRR *	N/A	0.70	2.34	2.22
Table 7: Human Evaluation. Style denotes style transfer accuracy, Flu. denotes fluency, Con. denotes content preservation. Mean denotes the average of the metrics where the fluency scores are scaled up to be consistent with other scores.  $*$ : significantly better than other systems ( $p < 0.01$ ) according to the mean score.
transfer accuracy for the GYAMC dataset since it is difficult for human annotators to accurately capture the difference between formal and informal sentences. The results of our human evaluations are shown in Table 7. We additionally report the sample-wise mean score of the metrics where the fluency scores are scaled up to be consistent with other scores. Our model achieves better overall performance when considering all three evaluation metrics on each dataset.
Interestingly, we found that the automatic metrics for both the style transfer accuracy and content preservation do not accurately reflect performance as measured by human evaluation. For example, on the Amazon dataset, although Tag&Gen (Madaan et al., 2020) achieves significantly higher style transfer accuracy based on the automatic metric, our model achieves better performance based on the human evaluation. This phenomenon suggests that the importance of our findings discussed in Section 3.3, that strong neural models can potentially exploit the weaknesses of the automatic metrics.
We next show an ablation study, demonstrating the effectiveness of the content preservation and fluency rewards in DIRR, and how SIM can be used to replace the cycle-consistency loss. We also compare using BLEU versus using SIM as a content-preservation reward, finding that using BLEU results in reduced performance, unstable training, and artifacts in the outputs, which makes the results less natural than the results of the model trained with SIM score.
To illustrate that training with SIM can replace
<sup>8</sup>https://www.mturk.com/
Model	Text	self-BLEU	self-SIM
Source	this was my first stop in looking for a wedding dress.	100.0	100.0
DIRR-BLEU	great this was my first stop in looking for a wedding dress.	91.2	95.2
DIRR	this was my best stop in looking for a wedding dress.	64.8	81.9
source	just a frozen patty cooked like a home one.	100.0	100.0
DIRR-BLEU	great a frozen patty cooked like a home one.	88.0	94.6
DIRR	just a great patty cooked like a home one.	70.7	88.5
source	wendy's has been know to be cheap with their drink
refills for years.	100.0	100.0
DIRR-BLEU	great wendy's has been know to be cheap with their drink
refills for years.	93.0	97.5
DIRR	wendy's has been great with their drink refills for years.	57.2	84.9
Table 8: Comparison of using SIM and BLEU as the content preservation reward. Samples are from the Yelp dataset. The metrics self-BLEU and self-SIM are calculated between the source sentences and system outputs.
Model	Acc	PPL	s-BLEU	s-SIM
DIRR-CYCLE	91.7	392	51.2	76.2
DIRR w/o FLU	92.1	348	51.4	79.8
DIRR-BLEU	91.3	315	59.4	81.8
DIRR	94.2	292	52.6	81.6
Table 9: Ablation and Comparative Study on Yelp Dataset. Acc is the accuracy of the sentiment classifier. PPL is the perplexity assigned by the GPT-2 language model. self-BLEU (s-BLEU) and self-SIM (s-SIM) are computed between the source sentences and outputs.
the cycle-consistency loss for content preservation, we fine-tuned DIRR-CYCLE on SIM to produce a new model, DIRR w/o FLU. The difference between DIRR and DIRR w/o FLU is that the former is additionally trained with our fluency rewards. The results are shown in Table 9, and show two main trends. First, we see that DIRR w/o FLU has better fluency and content preservation performance than DIRR-CYCLE, which shows that the cycle-consistency loss can be replaced by SIM score for content preservation. Second, DIRR has better fluency than DIRR w/o FLU, showing the effectiveness of our fluency rewards.
We next investigate the effectiveness of using SIM as a reward instead of BLEU. To do this, we train a model, DIRR-BLEU, which uses BLEU as the content reward and report the results in Table 9. The results show that using BLEU has larger content preservation as measured by BLEU, but has similar performance when measured by SIM. However, performance on the style transfer accuracy
and fluency decreases. We hypothesize that this is because using SIM as a reward gives the model more freedom, allowing the model to have more balanced performance since there is less pressure to copy  $n$ -grams. We also observe more adversarial examples in the outputs of DIRR-BLEU. As discussed in Section 3.3, these adversarial examples are generated by injecting a word carrying strong sentiment at the beginning of the output. The model trained with BLEU is more likely to generate these outputs as it will try to avoid breaking up the  $n$ -grams in the source sentences, allowing for a higher BLEU reward. Examples of this behavior is shown in Table 8. Notice that the DIRR-BLEU samples start with the word great, which is enough to often fool the classifier, but are unnatural.
A main line of work (Shen et al., 2017; Hu et al., 2017; Fu et al., 2018; Xu et al., 2018; John et al., 2019) for text style transfer aims to model the conditional distribution of the data with the encoder-decoder architecture. Due to the lack of parallel corpora, inductive biases are designed to make the generation conditioned on both source sentences and specific styles such that the model can rewrite the source texts with the target style while still preserve the content information of the source texts.
Efforts are also made to design training objectives to improve performance. For example, Backtranslation (Zhang et al., 2018; Prabhumoye et al., 2018), denoising auto-encoding (Lample et al., 2019) and the cycle-consistency loss (Luo et al.,
2019; Dai et al., 2019; Pang and Gimpel, 2019) have been shown effective for improving the model performance. Li et al. (2018) proposes a retrieve-based pipeline, which contains three stages, namely, delete, retrieve and generate. Sudhakar et al. (2019) extends this pipeline by using GPT (Radford et al., 2018) as the generator. Compared to these methods, we propose a more direct and effective approach to encourage semantic-preserving transfer by directly measuring the semantic similarity of the source texts and system outputs.
Recently, other works have been proposed for unsupervised text style transfer (Jin et al., 2019; Lai et al., 2019; Wu et al., 2019; Li et al., 2020). He et al. (2020) proposes a probabilistic view which models the non-parallel data from two domains as a partially observed parallel corpus. Madaan et al. (2020) proposes a tag-and-generate pipeline, which firstly identifies style attribute markers from the source texts, then replaces them with a special token, and generates the outputs based on the tagged sentences. Zhou et al. (2020) focuses on exploring the word-level style relevance which is assigned by a pre-trained style classifier. They propose a reward for content preservation which is based on the weighted combination of the word embeddings of the source texts and system outputs. Compared to this reward, our proposed content reward is specifically designed for semantic similarity and pre-trained on large corpora, which makes it more robust across different datasets.
In this paper, we propose a direct approach of improving content preservation for text style transfer by leveraging a semantic similarity metric as the content reward. Using a large pre-trained language model (GPT-2) with our proposed rewards that target the different aspects of the output quality, our approach achieves strong performance on both automatic and human evaluation. Recently, several semantic similarity metrics (Zhao et al., 2019; Sellam et al., 2020; Gao et al., 2020) based on pre-trained language models have shown promising results. Introducing these metrics in our proposed method as the content preservation reward may bring further improvements.
Moreover, we identify several problems in the commonly used automatic evaluation metrics and datasets, and propose several practical strategies to mitigate these problems, which makes these met
rics more effective rewards for model training. Considering the weaknesses of the automatic metrics presented in this work, we believe that more rigorous discussion and investigation on the criteria of "successful transferring" is essential for this field of work. Since existing works mostly relied on model-based metrics to determine the success of style transfer models, methods such as adversarial training could be introduced to make the model-based metrics more robust and faithful indicators of the success of style-transferring, which would be beneficial for both model training and evaluation.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuan-Jing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5997-6007.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Yang Gao, Wei Zhao, and Steffen Eger. 2020. SUPERT: Towards new frontiers in unsupervised evaluation metrics for multi-document summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1347-1354, Online. Association for Computational Linguistics.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. In International Conference on Learning Representations.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1587-1596. JMLR.org.
Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. Imat: Unsupervised text
attribute transfer via iterative matching and translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3088-3100.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434, Florence, Italy. Association for Computational Linguistics.
Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomáš Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Chih-Te Lai, Yi-Te Hong, Hong-You Chen, Chi-Jen Lu, and Shou-De Lin. 2019. Multiple text style transfer by using word-level conditional generative adversarial network with two-phase training. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3570-3575.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In International Conference on Learning Representations.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
Siyao Li, Deren Lei, Pengda Qin, and William Yang Wang. 2019. Deep reinforcement learning with distributional semantic rewards for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6038-6044, Hong Kong, China. Association for Computational Linguistics.
Yuan Li, Chunyuan Li, Yizhe Zhang, Xiujun Li, Guoqing Zheng, Lawrence Carin, and Jianfeng
Gao. 2020. Complementary auxiliary classifiers for label-conditional text generation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8303-8310.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Xu Sun, and Zhifang Sui. 2019. A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5116-5122. International Joint Conferences on Artificial Intelligence Organization.
Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabas Poczos, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. 2020. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1869-1881, Online. Association for Computational Linguistics.
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2018. Unsupervised learning of sentence embeddings using compositional n-gram features. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 528-540, New Orleans, Louisiana. Association for Computational Linguistics.
Richard Yuanzhe Pang and Kevin Gimpel. 2019. Unsupervised evaluation metrics and learning criteria for non-parallel textual transfer. In Proceedings of the 3rd Workshop on Neural Generation and Translation (WNGT 2019).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866–876, Melbourne, Australia. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 129-140, New Orleans, Louisiana. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830-6841.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019. "transforming" delete, retrieve, generate approach for controlled text style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3260-3270.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. 2019a. Beyond BLEU:training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344-4355, Florence, Italy. Association for Computational Linguistics.