SlowGuess/NAACL_2022_OCR huggingface dataset:
Model	Accuracy	BLEU	G2	H2	BertScore
Cross Aligned (Shen et al., 2017)	75.3	17.9	36.7	28.9	68.3
Back Translation (Prabhumoye et al., 2018)	95.4	5.0	21.9	9.6	61.0
Style Embedding (Fu et al., 2018)	8.7	42.3	19.2	14.4	78.1
Multi-Decoding (Fu et al., 2018)	50.2	27.9	37.4	35.9	69.4
Unpaired (Xu et al., 2018)	64.9	37.0	49.0	47.1	73.7
Delete+Retrive (Li et al., 2018)	89.0	31.1	52.6	46.1	71.3
Template-Based (Li et al., 2018)	81.8	45.5	61.0	58.5	73.7
Unsupervised MT (Zhang et al., 2018)	95.4	44.5	65.1	60.7	80.8
DualRL (Luo et al., 2019)	85.6	55.2	68.7	67.1	84.1
IterativeMatch (Jin et al., 2019)	91.7	23.3	46.2	37.1	71.4
Deep Latent w/ Language Models (He et al., 2019)	85.2	46.4	62.8	60.0	76.4
Direct Rewards w/ GPT-2 (Liu et al., 2021)	91.2	53.8	70.0	67.6	83.6
Only Lexical Pseudo Data Bootstrapping (30K Pairs)	81.3	26.5	46.4	39.9	72.1
Lexical Pseudo + Reward-Learning (30K)	81.1	50.4	63.9	62.1	82.1
Lexical Pseudo + Reward-Learning (100K)	86.2	59.4	71.5	70.3	87.3
Only Semantic Pseudo Data Bootstrapping (30K Pairs)	82.9	23.9	44.5	37.1	71.8
Semantic Pseudo + Reward-Learning (30K)	83.5	49.6	64.3	62.2	82.5
Semantic Pseudo + Reward-Learning (100K)	86.5	59.8	71.9	70.7	87.1
Table 3: Automatic evaluation scores on the Yelp sentiment style transfer task. Baseline results are reported with the model generations provided in published studies. Text examples are shown in Appendix Table 11.
model is selected as the generator  $G$ . For style classification, 'RoBERTa-base' is used. We fine-tune models with AdamW (Kingma and Ba, 2015) with batch size 32; initial learning rates are all set at  $2e^{-5}$ . Style reward penalty  $\gamma$  is 0.2. Values for  $\lambda$  are set to 1.0 for style reward and 0.8 for cyclic reward. Beam search size is set at 6. Test results are reported with best validation scores (see Appendix Table 9 for environment and hyperparameter setting details, and Algorithm 1 for the training process).
As previous work (Luo et al., 2019; He et al., 2020; Sancheti et al., 2020), we adopt the following evaluation metrics: (1) Style Accuracy is calculated via binary classification to measure the style strength of re-writing. While the TextCNN (Kim, 2014) is used in previous studies, we also adopt a Transformer 'RoBERTa-base' classifier, where the reported scores are similar in our settings; (2) BLEU score is calculated on the prediction and human references to measure the content preservation; (3) We also compute the geometric mean (G2) and harmonic mean (H2) of style accuracy and BLEU score; (4) Since recent metrics with semantic similarity show better correlation with human judgments than traditional lexical measures. We also calculate BertScore between generation and references (Zhang et al., 2020a).
4.3 Results on Yelp Corpus
A number of representative unsupervised baseline models are selected for extensive comparison on the Yelp corpus: (1) models that adopt content-
style disentanglement such as Cross Aligned (Shen et al., 2017) and Style Embedding (Fu et al., 2018); (2) models that adopt back-translation such as Unsupervised MT (Zhang et al., 2018), and Dual RL (Luo et al., 2019), and recent state-of-the-art models Deep Latent (He et al., 2019) and Direct Rewards w/ GPT-2 (Liu et al., 2021). For our semi-supervised framework, we first (1) apply vanilla supervised learning to assess the effectiveness of the pseudo parallel data construction; (2) bootstrap the model with 30K pseudo parallel pairs, then further train it via reward-based learning; (3) apply semi-supervised learning by bootstrapping the model with 30K pseudo parallel pairs, and using 70K non-parallel samples for the reward-based training. As shown in Table 3, vanilla supervised training on the 30K pseudo parallel data lead to favorable scores of style accuracy, though they do not perform well in terms of BLEU scores, as the pseudo pairs emphasize style converting rather than content preservation. Further training with rewards improves both the style accuracy and BLEU score, and models with both lexical and semantic pseudo data produce comparable results with only 30k samples. Performance is further improved by using additional non-parallel data (70k samples), where our models outperform state-of-the-art baselines significantly.
4.4 Results on Amazon Corpus
For the Amazon sentiment transfer corpus, we adopt the same training strategies described in Section 4.3. Aside from unsupervised models, we also select the semi-supervised model Semi-LM-MMI
Model	Accuracy	BLEU	G2	H2	BertScore
Cross Aligned (Shen et al., 2017)	74.1	0.4	5.4	0.8	55.3
Style Embedding (Fu et al., 2018)	43.3	10.0	20.8	16.2	68.1
Multi-Decoding (Fu et al., 2018)	68.3	5.0	18.4	9.3	18.2
Template-Based (Li et al., 2018)	68.7	27.1	43.1	38.9	85.5
Delete+Retrieve (Li et al., 2018)	48.0	22.8	33.1	30.9	83.7
Word-level Conditional GAN (Lai et al., 2019)	77.4	6.7	22.7	12.3	-
Semi-LM-MMI w/ BART-Large (Chawla and Yang, 2020)	68.9	28.6	44.4	40.4	-
Direct Rewards w/ GPT-2 (Liu et al., 2021)	68.3	38.6	51.3	49.3	72.1
Only Lexical Pseudo Data Bootstrapping (30K Data)	79.8	16.4	36.1	27.2	63.3
Lexical Pseudo + Reward-Learning (30K)	71.2	36.1	50.6	47.9	73.4
Lexical Pseudo + Reward-Learning (100K)	73.1	46.3	58.1	56.6	78.4
Only Semantic Pseudo Data Bootstrapping (30K Data)	81.2	10.3	28.9	18.2	60.5
Semantic Pseudo + Reward-Learning (30K)	72.3	35.5	50.6	47.6	72.7
Semantic Pseudo + Reward-Learning (100K)	74.1	45.4	58.0	56.3	78.1
Table 4: Automatic evaluation scores on the Amazon sentiment style transfer task. Baseline results are calculated and reported with the model generations provided in published studies. See examples in Appendix Table 12.
Model	E&M Domain	F&R Domain
Accuracy*	BLEU	G2	H2	Accuracy*	BLEU	G2	H2
Human Reference (Rao and Tetreault, 2018)	81.5	100.0	90.2	89.8	80.5	100.0	89.7	89.2
Rule-Based (Rao and Tetreault, 2018)	29.7	72.4	46.4	42.1	82.1	65.8	73.4	73.1
Hybrid Annotations (Xu et al., 2019)	28.8	69.2	44.6	40.6	34.8	74.3	50.8	47.3
Semi-LM-MMI w/ BART-Large (Chawla and Yang, 2020)	30.4	76.5	48.2	43.5	30.6	79.9	49.4	44.2
Rewarded BART-Large (Lai et al., 2021)	75.1	76.5	75.7	75.7	74.6	79.2	76.8	76.8
Only Labeled Data Supervision (Full)	75.0	71.2	73.1	73.1	73.7	72.5	73.1	73.1
Labeled Data + Reward-Learning (30K)	75.7	71.4	73.5	73.4	72.4	74.4	73.3	73.4
Labeled Data + Reward-Learning (Full)	82.2	71.0	76.3	76.2	80.5	74.2	77.3	77.2
Table 5: Automatic evaluation scores on the GYAFC formality transfer task of baselines and our framework. Baseline results are reported with the generations provided as in (Chawla and Yang, 2020). *The style accuracy is calculated with a fine-tuned 'RoBERTa-base' model (see Appendix for the result with TextCNN classifier).
w/ BART (Chawla and Yang, 2020), which adopted a language model-based discriminator for maximizing token-level conditional probabilities for training. Due to label noise in online-crawled data, the style accuracy for all models becomes lower than those trained on Yelp, and the classifier precision is only  $86\%$  (see Table 4). We also observed that the lexical similarity of pseudo parallel pairs is smaller than Yelp samples, and results in lower BLEU scores, especially when we apply supervised training on the 30K pseudo parallel data. On the other hand, content preservation largely benefits from the reward-based learning. Unsurprisingly, after bootstrapping, training with rewards significantly improves the generation quality, and our framework achieves state-of-the-art performance. Moreover, bootstrapping with lexical-based and semantic-based pseudo data resulted in a similar final performance with reward learning.
4.5 Results on GYAFC Corpus
In recent work, it is shown that style transfer models trained on parallel data can benefit from additional reward-based learning (Lai et al., 2021).
Here we conduct additional experiments to assess our semi-supervised framework on the GYAFC formality transfer corpus with well-annotated data. We evaluate the proposed model on the informal-to-formal task as previous work (Chawla and Yang, 2020), and compare them with strong baselines. As shown in Table 5, while the baselines show impressive BLEU scores on the formality transfer task, our framework outperforms them significantly in terms of style accuracy, approaching upper-bound human performance. Moreover, compared with the contemporary supervised work (Lai et al., 2021), which also introduced additional RL-based optimization, our model still achieves higher G2 and H2 scores. The examples shown in Appendix Table 13 demonstrate that our approach generates sentences with accurate formality paraphrasing.
4.6 Human Assessment
Additionally, we conducted a human evaluation on Yelp, Amazon and GYAFC datasets. Following previous work (Chawla and Yang, 2020; Liu et al., 2021), we evaluated the generated sentences from three aspects: style transfer strength (Style), text flu
Model	Accuracy	Yelp Data	Amazon Data
BLEU	G2	H2	Accuracy	BLEU	G2	H2
Sequence-Level Reward (30K Data)	85.1	26.5	47.5	40.4	78.4	19.0	38.5	30.5
Stepwise Reward (30K Data)	81.1	50.4	63.9	62.1	71.2	36.1	50.6	47.9
Sequence-Level Reward (100K Data)	84.8	35.3	54.7	49.8	81.4	21.9	42.2	34.5
Stepwise Reward (100K Data)	86.2	59.4	71.5	70.3	73.1	46.3	58.1	56.6
Table 6: Ablation study on the proposed stepwise reward on the Yelp and Amazon dataset. Sequence-level denotes the reward is calculated on the whole sequence, without the stepwise re-weighting.
Layer No.	Accuracy	BLEU	G2	H2
Layer-12	78.5	46.2	60.2	58.1
Layer-11	81.1	45.5	60.7	58.2
Layer-10	84.2	38.7	57.0	53.0
Layer-9	72.3	43.8	56.2	54.5
Layer-8	76.1	44.5	58.1	56.1
Layer-7	70.3	41.6	54.0	52.2
ency (Fluency), and content preservation (Content), separately. The three aspects are rated with range [1, 5], then their average value is calculated and reported as Mean (see Table 14 in Appendix). For each corpus, we randomly selected 80 test samples and compared the outputs of representative and previous state-of-the-art models. Each candidate was rated by three linguistic experts, and we report the average scores. Our model achieves better overall performance when considering all three evaluation metrics on each dataset. Moreover, we observe that leveraging the pre-trained language models such as BART and GPT-2 is beneficial for the text fluency.
To extensively assess the effectiveness of the proposed methods, we conduct the following in-depth analyses.
5.1 Ablation Study on Stepwise Re-weighting
We conduct an ablation experiment to assess the effectiveness of stepwise reward re-weighting. As shown in Table 6, the performance degrades significantly without the stepwise reward re-weighting, especially the BLEU score. In particular, we observed that when removing stepwise optimization, the generator was prone to mode collapse. In one manifestation of mode collapse, the model appended a limited set of phrases to the source sentences, resulting in generation with disfluency and low diversity. It demonstrates that token-level
Table 7: Layer selection for the proposed stepwise reward re-weighting. The Yelp sentiment transfer dataset and the semantic-based matching are used. We conduct experiments on the last 6 Transformer layers of the style classifier.
Train Size	Accuracy	BLEU	G2	H2
1,000	62.9	31.6	44.5	42.0
5,000	68.2	36.8	50.0	47.8
10,000	73.3	43.6	56.5	54.6
15,000	76.1	45.5	58.8	56.9
30,000	83.5	49.6	64.3	62.2
Table 8: Results from different pseudo sample sizes using the proposed framework. The Yelp sentiment transfer dataset and semantic-based matching are used.
reward optimization provides finer-granularity for policy gradient of sequence-to-sequence training. This approach can also be potentially extended to other text generation tasks.
5.2 Attention Layer Selection for Stepwise Reward Re-weighting
We utilize attentive scores from the top-2 multihead layers for stepwise reward re-weighting. To study the effect of layer selection, we compared the results using attention scores extracted from different Transformer layers in the style classifier described in Section 3.4. As shown in Table 7, the performance shows an overall increasing trend from the 7-th to the 12-th layer, and we obtained better results with the top layers. In scores of lower layers, we found that the model tended to assure content preservation rather than style accuracy. This is consistent with the observations from recent linguistic probing and model interpretation studies (Hewitt and Manning, 2019; Xu et al., 2020): the information modeled in the Transformer-based networks, especially the pre-trained language backbones, is represented in a hierarchical manner, and the higher layers provide more effective information on scoring the span importance for text classification (see visualization in Appendix Figure 4).
5.3 Bootstrapping Sample Size
We investigate the effect of different pseudo parallel sample sizes. As shown in Table 8, the result shows that the evaluation result by automatic metrics becomes acceptable when training reaches 10K samples. Results comparable to state-of-the
art are achieved with merely 30K data (10% of the Yelp training set). We speculate that the relatively weak performance with 10K samples is because the BART model uses a denoising autoencoding paradigm (Lewis et al., 2020), which is trained to reconstruct the input sentence, and style strength of sentence rewriting is strongly affected in this low resource scenario.
Additionally, we conduct an ablation study on the bootstrapping step, and the result shows that with the same training sample size, the generation performance (considering both style accuracy and content preservation) obtained significant improvement by adding the bootstrapping learning stage (see Appendix Table 15).
In this paper, we proposed a framework for text style transfer taking advantage of both supervised and unsupervised paradigms. The training process is bootstrapped with supervision guided by automatically constructed pseudo parallel data. Both lexical-based and semantic-based sentence matching proved effective. Moreover, the stepwise reward re-weighting significantly improved the generation performance, and is a generic design that can be easily extended. Experimental results showed that the proposed approach achieved state-of-the-art performance in multiple datasets, while producing reasonable generation even with minimal training data (10% of original size).
This research was supported by funding from the Institute for Infocomm Research (I2R) under A*STAR ARES, Singapore. We thank Ai Ti Aw for the insightful discussions. We also thank the anonymous reviewers for their precious feedback to help improve and extend this piece of work.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations.
Kunal Chawla and Diyi Yang. 2020. Semi-supervised formality style transfer using language model discriminator and mutual information maximization. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 2340-2354.
Cyprien de Masson d'Autume, Shakir Mohamed, Michaela Rosca, and Jack Rae. 2019. Training language gans from scratch. Advances in Neural Information Processing Systems, 32.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 663-670.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910.
Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 3168-3180.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. A probabilistic formulation of unsupervised text style transfer. In International Conference on Learning Representations.
Junxian He, Xinyi Wang, Graham Neubig, and Taylor Berg-Kirkpatrick. 2020. A probabilistic formulation of unsupervised text style transfer. In Proceedings of Ninth International Conference on Learning Representations.
John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 4129-4138.
Eduard Hovy. 1987. Generating natural language under pragmatic constraints. Journal of Pragmatics, 11(6):689-719.
Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. Imat: Unsupervised text attribute transfer via iterative matching and translation. In Proceedings of EMNLP-IJCNLP 2019, pages 3097-3109.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations.
Chih-Te Lai, Yi-Te Hong, Hong-You Chen, Chi-Jen Lu, and Shou-De Lin. 2019. Multiple text style transfer by using word-level conditional generative adversarial network with two-phase training. In Proceedings of EMNLP-IJCNLP 2019, pages 3579-3584. Association for Computational Linguistics.
Huiyuan Lai, Antonio Toral Ruiz, and Malvina Nissim. 2021. Thank you bart! rewarding pre-trained models improves formality style transfer. In Proceedings of the ACL-IJCNLP 2021. Association for Computational Linguistics.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In International Conference on Learning Representations.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the ACL 2020, pages 7871-7880.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874.
Yixin Liu, Graham Neubig, and John Wieting. 2021. On learning text style transfer with direct rewards. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4262-4273.
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. A dual reinforcement learning framework for unsupervised text style transfer. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5116-5122.
David D McDonald and James Pustejovsky. 1985. A computational theory of prose style for natural language generation. In Second Conference of the European Chapter of the Association for Computational Linguistics.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? Advances in neural information processing systems, 32.
Nikola I Nikolov and Richard Hahnloser. 2019. Large-scale hierarchical alignment for data-driven text rewriting. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 844-853.
Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866-876.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In Proceedings of the NAACL 2018, pages 129-140.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008-7024.
Abhilasha Sancheti, Kundan Krishna, Balaji Vasan Srinivasan, and Anandhavelu Natarajan. 2020. Reinforced rewards framework for text style transfer. In Advances in Information Retrieval, pages 545-560.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6833-6844.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019. "transforming" delete, retrieve, generate approach for controlled text style transfer. In Proceedings of EMNLP 2019, pages 3269-3279.
Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. 1999. Policy gradient methods for reinforcement learning with function approximation. In NIPs, volume 99, pages 1057-1063. CiteSeer.
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76.