SlowGuess/NAACL_2022_OCR huggingface dataset:
Discourse Style Datasets: To check for the model's ability to alter the discourse or flow of logic between two sentences we make use of NLI datasets such as SNLI, DNLI, and Scitail. Each instance in the SNLI dataset (Bowman et al., 2015) consists of two sentences that either contradict, entail (agree) or are neutral towards each other. Similarly, the DNLI dataset (Welleck et al., 2019) consists of contradiction, entailment and neutrality labelled instances. Scitail (Khot et al., 2018) is an entailment dataset created from multiple-choice science exams and the web, in a two-sentence format similar to SNLI and DNLI. The first sentence is formed from a question and answer pair from
science exams and the second sentence is either a supporting (entailment) or non-supporting (neutrality) premise.
Fine-grained Style Datasets: The Style-PTB dataset (Lyu et al., 2021) consists of 21 styles/labels with themes ranging from syntax, lexical, semantic and thematic transfers as well as compositional transfers which consist of transferring more than one of the aforementioned fine-grained styles. To check whether the EPAAE can capture fine-grained styles better by leveraging its better organised latent space, we make use of three styles i.e. Tenses, Voices (Active or Passive) and Syntactic PP tag removal (PPR). In the Tenses dataset, each sentence is labelled with "Present", "Past", and "Future". The Voices dataset contains "Active" and "Passive" voices labels and the PPR dataset contains "PP removed" and "PP not removed" labels.
5.3 Automatic Evaluation metrics
Evaluation for text style transfer includes checking for a) Style Transfer accuracy, b) Content preservation metrics and c) Fluency of output sentences. Recent studies show that automatic Evaluation metrics are still an open problem and can be gamed (Xu et al., 2020).
Style Transfer Measure: A pre-trained classifier is used to check the presence of the target label in the output sentence. (Mir et al., 2019) introduces the notion of checking the style transfer intensity apart from just the presence of the target label. While we find this notion intriguing, we wish to first accomplish the style transfer task convincingly for the current set of tasks before assuming a more complex metric.
Content Preservation: In recent work, we observe that models typically struggle more in content preservation and the ability to preserve the contextual meaning of the base sentence. The BLEU score alone does not suffice to correlate strongly with actual qualitative results. To truly validate our hypothesis that EPAAE's are better able to preserve content better, we augment the bucket of standard content preservation metrics  $\ddagger$ . Apart from the standard of using BLEU between sentences of source and target styles, we borrow evaluation techniques from fields similar to Text Style Transfer such as Machine Translation and Text Summarization, such as METEOR (Banerjee and Lavie, 2005),
To generate all our content preservation metrics, we use the nlg-eval package from: https://github.com/Maluuba/nlg-eval
ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015) which have been shown to correlate more strongly with human judgement. Following the study of (Sharma et al., 2017), in which they show BLEU does not necessarily correlate with human evaluations in dialogue response generation, we also adopt Embedding Average, Vector Extrema (Forgues et al., 2014) and Greedy matching score (Rus and Lintean, 2012).
Fluency of generations: Past work measures the perplexity using a pre-trained language model to gauge the fluency or grammatical correctness of the style transferred outputs. (Mir et al., 2019) argues such perplexity calculations for style transfer tasks may not necessarily correlate with a human judgement of fluency. Adversarial classifiers in the form of logistic regression networks are trained with the goal of distinguishing between human-produced and machine produced sentences. These classifiers are then used to score the naturalness of the output sentences. We follow this metric during our evaluation of fluency or naturalness  $\S$
In this section, we look at the quantitative and quantitative results for the text style transfer task for four autoencoder models in seven datasets. We use the vector arithmetic method on latent space representations, inspired by (Mikolov et al., 2013) where it showed that word embeddings learnt can capture linguistic relationships using simple vector arithmetic. Analogous to the standard example where "King" - "Man" + "Woman" ≈ "Queen", we manipulate an arbitrary sentence encoding  $z_{x}$  of Style X to Style Y:
$$
z _ {x} = z _ {y} + k \left(\frac {1}{N _ {y}} \sum_ {i = 0} ^ {N _ {y}} z _ {y} ^ {i} - \frac {1}{N _ {x}} \sum_ {i = 0} ^ {N _ {x}} z _ {x} ^ {i}\right) \tag {5}
$$
where  $z_{y}^{i}, z_{y}^{i}$  denotes the latent vector of the  $i^{th}$  sentence in style  $y$  and  $x$  respectively and  $N_{x}$  and  $N_{y}$  represent the number of encoding present in the corpus for style  $x$  and  $y$  respectively.  $k$  is a scaling parameter used to control the style transfer strength.
The resultant latent vector is passed through the decoder to produce the output sentence.
6.1 Quantitative Analysis
TST accuracy, content preservation and naturalness were computed on the converted sentences (shown in Table 4, 5, 6). Content preservation metrics can be found from Column 4 onwards. We consider two versions of the EPAAE i.e. only continuous embedding noise, continuous embedding noise + token deletion, and find that in some cases a mixture of both is required for optimal performance. We find that a slightly lowered value  $p$  for the EPAAE combined with its optimal  $\zeta$  parameter outperforms other models as well.
6.1.1 Sentiment TST
Table 4 summarises the results of TST on the Yahoo dataset. On visual inspection, there appears to be a general tradeoff between TST% and content preservation metrics. For example, the  $\beta$ -VAE achieves the best TST% but suffers from bad content preservation capability. In this case, we observe that EPAAE ( $\zeta = 2.0, p = 0.1$ ) has the best content preservation capabilities across all metrics and achieves a reasonable tradeoff of TST%=77.1 as well.
6.1.2 Discourse TST
Table 5 summarises the results on four NLI datasets. Similar to the sentiment task, we see that EPAAEs have the best content preservation capabilities as well as the naturalness metric. It achieves this while achieving a comparable TST% as well.  $\beta$ -VAE shows best TST% but again suffers in content preservation. DAAE also display reasonable TST% vs Content preservation tradeoffs but overall cannot match the tradeoffs achieved by EPAAEs. Human evaluations on the SNLI dataset (Table 8) confirm this as well. The TST% achieved by any model in any task peaks at only  $60.4\%$  compared to  $81.9\%$  in sentiment task, a significant difference that hints at the fact that the Discourse TST task might be intrinsically more complex than sentiment style. Future work that aims to increase performance on the NLI task will be beneficial.
6.1.3 Style-PTB TST
Table 6 summarises the results on three datasets from the Style-PTB benchmark. (Lyu et al., 2021) produces a hierarchy of styles based on transfer difficulty measured by the average token-level ham
$^\S$ We use code and pretrained models from https://github.com/passeul/style-transfer-model-evaluation to measure naturalness.
Simply put, the difference of the means of all vectors in Style Y and X is computed and scaled by a factor  $k$ , then added to any arbitrary latent vector with style  $Y$  to convert it to style  $X$ .
Model	Naturalness	TST Acc.	BLEU-2	METEOR	ROUGE-L	CIDER	Embedding Average	Vector Extrema	Greedy Matching Score
β-VAE (β = 0.15)	0.792	0.819	0.064	0.079	0.212	0.271	0.773	0.445	0.600
LAAE (λ1 = 0.05)	0.763	0.782	0.056	0.072	0.182	0.232	0.756	0.436	0.583
DAAE (n = 0.3)	0.711	0.812	0.167	0.134	0.339	0.775	0.810	0.531	0.674
EPAAE (ζ = 2.0)	0.822	0.749	0.142	0.125	0.320	0.609	0.800	0.491	0.651
EPAAE (ζ = 2.0, p = 0.3)	0.708	0.808	0.193	0.145	0.368	0.838	0.824	0.542	0.688
EPAAE (ζ = 2.0, p = 0.1)	0.718	0.771	0.218	0.170	0.396	1.017	0.827	0.571	0.707
Table 4: Quantitative results of TST experiments on Yelp Dataset
Dataset	Model	Naturalness	TST Acc.	BLEU-2	METEOR	ROUGE-L	CIDER	Embedding Average	Vector Extrema	Greedy Matching Score
SNLI	β-VAE (β = 0.15)	0.983	0.523	0.424	0.257	0.541	2.288	0.948	0.615	0.814
LAAE (λ1 = 0.05)	0.946	0.522	0.036	0.051	0.158	0.026	0.824	0.318	0.593
DAAE (p = 0.3)	0.980	0.519	0.415	0.254	0.534	2.062	0.950	0.650	0.838
EPAAE (ζ = 2.5)	0.979	0.480	0.534	0.316	0.655	3.615	0.958	0.688	0.857
EPAAE (ζ = 2.5, p = 0.3)	0.983	0.513	0.388	0.240	0.504	1.783	0.948	0.626	0.826
EPAAE (ζ = 2.5, p = 0.1)	0.978	0.511	0.461	0.281	0.582	2.492	0.953	0.674	0.854
DNLI	β-VAE (β = 0.15)	0.927	0.578	0.243	0.144	0.356	0.654	0.905	0.503	0.722
LAAE (λ1 = 0.05)	0.935	0.604	0.205	0.128	0.319	0.384	0.897	0.475	0.704
DAAE (p = 0.3)	0.933	0.597	0.416	0.237	0.522	2.269	0.934	0.616	0.803
EPAAE (ζ = 2.5)	0.939	0.513	0.440	0.253	0.557	2.573	0.943	0.635	0.818
EPAAE (ζ = 2.5, p = 0.3)	0.927	0.574	0.397	0.225	0.506	2.064	0.933	0.606	0.798
EPAAE (ζ = 2.5, p = 0.1)	0.934	0.579	0.457	0.260	0.559	2.912	0.941	0.636	0.636
Scitail	β-VAE (β = 0.15)	0.770	0.570	0.095	0.065	0.173	0.142	0.796	0.387	0.643
LAAE (λ1 = 0.05)	0.832	0.497	0.196	0.124	0.287	0.752	0.874	0.489	0.712
DAAE (p = 0.3)	0.792	0.454	0.325	0.199	0.413	1.560	0.913	0.611	0.805
EPAAE (ζ = 2.5)	0.839	0.422	0.367	0.222	0.471	2.022	0.930	0.647	0.827
EPAAE (ζ = 2.5, p = 0.3)	0.813	0.495	0.276	0.171	0.355	1.241	0.906	0.560	0.770
EPAAE (ζ = 2.5, p = 0.1)	0.827	0.440	0.352	0.215	0.455	1.931	0.932	0.635	0.818
Table 5: Quantitative results of TST experiments on NLI datasets
ming distance between the base and converted sentence. According to this hierarchy Tense inversion, PP removal/addition and Voice change are labelled in ascending order as easy, medium and hard respectively. Our results seem to partially validate this observation, in that the max TST% is obtained on the Tenses dataset (100%). Generally speaking, we also observe that TST has much better performance on fine-grained Style-PTB datasets than sentiment and discourse styles. Similar to before, the EPAAE shows best content preservation at competitive values of TST% as well.
It is also noteworthy to consider the direction of style transfer, particularly in the case of complex styles such as Discourse styles present in NLI datasets. Results and analysis on direction-specific metrics for Discourse TST are presented in Appendix B.2.2.
Samples of Output. For qualitative analysis, sample outputs by DAAE and EPAAE for the Yelp, SNLI and Tenses dataset are given in Table B.4. For a full list of qualitative examples on all datasets along with setup details, please refer to Appendix B.2.1.
Varying  $k$  for Fine-Grained TST. By varying the value of  $k$  in Equation 5, it is possible to finely control the strength of Text Style transfer. We show
examples of this for the Yelp (Table B.9) and the SNLI (Table 7) dataset, for the baseline DAAE  $(p = 0.3)$  and the proposed model EPAAE. For the proposed EPAAE model, the best performing models (specifically in content preservation metrics) were chosen i.e.  $zeta = 2.0$ ,  $p = 0.1$  for the Yelp dataset and  $zeta = 2.5$  for the SNLI dataset. The chosen qualitative examples highlight the EPAAEs slight superiority in performing fine-grained TST compared to the baseline.
Smooth Interpolation. Sentence interpolation experiments are reported in Appendix B.1, in which latent space points in an interpolation along a specific direction are decoded to gauge the smoothness of the space and its ability to generate coherent sentences
6.3 Human Evaluations
Each human annotator was given a set of base sentences and asked to vote for which model produced the most appropriate corresponding style inverted sentences. Please refer to Appendix A.3 for full details on the setup. Results are shown in Table 8. We observe that the proposed EPAAE model was overall more preferred across all three chosen datasets. This margin was most significant in the case of the SNLI dataset.
Dataset	Model	Naturalness	TST Acc.	BLEU-2	METEOR	ROUGE-L	CIDER	Embedding Average	Vector Extrema	Greedy Matching Score
Voices	β-VAE (β = 0.15)	0.779	0.985	0.144	0.108	0.204	0.728	0.765	0.426	0.593
LAAE (λ1 = 0.05)	0.789	0.988	0.095	0.080	0.162	0.426	0.746	0.390	0.560
DAAE (p = 0.3)	0.783	0.981	0.243	0.182	0.300	1.488	0.811	0.523	0.670
EPAAE (ζ = 2.5)	0.806	0.993	0.176	0.134	0.260	0.919	0.795	0.468	0.634
EPAAE (ζ = 2.5, p = 0.3)	0.785	0.975	0.174	0.133	0.237	0.978	0.777	0.457	0.616
EPAAE (ζ = 2.5, p = 0.1)	0.796	0.991	0.253	0.187	0.318	1.491	0.819	0.532	0.681
PPR	β-VAE (β = 0.15)	0.735	0.949	0.197	0.202	0.391	1.291	0.808	0.536	0.706
LAAE (λ1 = 0.05)	0.748	0.940	0.149	0.143	0.302	0.846	0.772	0.467	0.643
EPAAE (p = 0.3)	0.730	0.948	0.261	0.279	0.486	1.797	0.842	0.615	0.774
EPAAE (ζ = 2.5)	0.757	0.932	0.282	0.283	0.510	1.982	0.857	0.631	0.789
EPAAE (ζ = 2.5, p = 0.3)	0.728	0.955	0.236	0.257	0.448	1.586	0.829	0.585	0.749
EPAAE (ζ = 2.5, p = 0.1)	0.747	0.938	0.293	0.301	0.525	2.105	0.861	0.645	0.799
Tenses	β-VAE (β = 0.15)	0.803	0.999	0.110	0.089	0.198	0.513	0.766	0.403	0.583
LAAE (λ1 = 0.05)	0.807	1.000	0.086	0.072	0.169	0.388	0.749	0.377	0.563
DAAE (p = 0.3)	0.774	0.999	0.263	0.200	0.363	1.684	0.822	0.540	0.686
EPAAE (ζ = 2.5)	0.798	1.000	0.358	0.274	0.483	2.305	0.861	0.631	0.757
EPAAE (ζ = 2.5, p = 0.3)	0.776	0.999	0.302	0.234	0.408	2.023	0.834	0.574	0.711
EPAAE (ζ = 2.5, p = 0.1)	0.785	1.000	0.339	0.254	0.456	2.279	0.852	0.604	0.739
Table 6: Quantitative results of TST experiments on Style-PTB datasets
DAAE (p=0.3)	EPAAE (ζ=2.0,p=0.1)
Input	the dog sits by a snowdrift . a dog out in the snow	the dog sits by a snowdrift . a dog out in the snow
k=1	the dog stands in the snow . a dog stands in the snow	the dog sits under a fallen tree dog . a dog resting in the snow
k=1.5	the dog stands in the snow . a dog swims in the snow	the dog sits under a fallen tree dog . a dog resting in the snow
k=2	the dog stands in the snow . a dog swims in the snow	the dog sits under a fallen tree dog . a dog resting in the snow
k=2.5	the dog stands in the snow . a dog swims in the snow	the dog sits under a fallen tree dog . a dog resting in the snow at the porch
k=3	the dog leaps across the snow . a dog swims in the snow in the snow	the dog sits under a fallen tree dog . a dog resting in the snow at the porch
Input	a man enjoys some extravagant artwork . a man is making art	a man enjoys some extravagant artwork . a man is making art
k=-1	a man makes a strange art that says unk . a man is making clothing	a man examines an art unk stall . a man is making art
k=-1.5	a man makes a strange art that says unk . a man is making clothing	a man examines an art unk by . a man is making art
k=-2	a man makes a strange art that says unk . a man is making clothing	a man examines an art unk by . a man is making art
k=-2.5	a man makes a strange art that says art . a man is making clothing	a man examines an art gallery unk . a man is making art
k=-3	a man makes a strange art that says art . a man is making clothing	a man examines an art gallery unk . a man is making art
Table 7: Qualitative examples of Fine-grained TST on the SNLI Dataset (Entailment to Contradiction)
Dataset	DAAE better	EPAAE better	Both Good	Both Bad	No Agreement
Yelp	34	47	16	75	28
SNLI	52	90	4	35	19
Tenses	45	59	17	63	16
Table 8: Human Evaluations for Text Style Transfer on Yelp, SNLI and Tenses datasets. Bold indicates model with highest votes
We introduce the "Embedding Perturbed AAE" or EPAAE and show that it best captures underlying style-based semantic features in the latent space in an unsupervised manner compared to its baselines. By inducing robust latent space organization through embedding perturbation in an unsupervised manner, we also demonstrate the possibility of fine-grained TST, where we can control the strength of the target style. Using a diverse set of datasets with varying formulations and complexities of style, we empirically that EPAAE performs overall best in the text style transfer task, particularly in its ability to preserve style-independent content across all datasets.
8 Future Work and Limitations
Regarding work in TST. We wish to augment existing state of the art methods with embedding perturbation to check if doing so aids performance. We see degrading TST performance in the Entailment to Contradiction Task across all models. Future work will focus on methods to improve this task.
Generally. It is also interesting to further analyse the effects of embedding perturbation to latent representations and resultant properties. A theoretical analysis would be beneficial to cement the use of embedding perturbation in a more general setting. There also remain more important questions that need answering for, e.g. "What if you apply continuous perturbation to hidden states instead of embeddings?", "What is the relation between this type of perturbation and techniques like dropout?" We wish to explore these important questions in the future.
Any TST model can be used for nefarious purposes, e.g. performing a "Non-toxic to toxic"
modification of text in a real-world setting and causing social harm. Therefore, it is important we keep in mind a code of ethics (e.g. https://www.acm.org/code-of-ethics) for usage, research and development in this type of research. We have made all our code open-source and provided all details of experimentation and implementation to the best of our knowledge.
We would like to thank the reviewers whose feedback we believe substantially increased the quality of this work. We would like to thank the human annotators for their participation.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-yu Dai, and Jiajun Chen. 2019. Generating sentences from disentangled syntactic and semantic spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6008-6019, Florence, Italy. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. A multi-task approach for disentangling syntax and semantics in sentence representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2453-2464, Minneapolis, Minnesota. Association for Computational Linguistics.
Antonia Creswell and Anil Anthony Bharath. 2019. Denoising adversarial autoencoders. IEEE Transactions on Neural Networks and Learning Systems, 30(4):968-984.
Gabriel Forgues, Joelle Pineau, Jean-Marie Larchevêque, and Réal Tremblay. 2014. Bootstrapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, volume 2.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434, Florence, Italy. Association for Computational Linguistics.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Kalpesh Krishna, John Wieting, and Mohit Iyyer. 2020. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737-762, Online. Association for Computational Linguistics.
Dongkyu Lee, Zhiliang Tian, Lanqing Xue, and Nevin L. Zhang. 2021. Enhancing content preservation in text style transfer using reverse attention and conditional layer normalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 93-102, Online. Association for Computational Linguistics.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.