SlowGuess/NAACL_2022_OCR huggingface dataset:
Dayiheng Liu, Jie Fu, Yidan Zhang, Chris Pal, and Jiancheng Lv. 2020. Revision in continuous space: Unsupervised text style transfer without adversarial learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8376-8383.
Yiwei Lyu, Paul Pu Liang, Hai Pham, Eduard Hovy, Barnabás Póczos, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2021. StylePTB: A compositional benchmark for fine-grained controllable text style transfer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2116-2138, Online. Association for Computational Linguistics.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian J. Goodfellow. 2015. Adversarial autoencoders. CoRR, abs/1511.05644.
Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020. Unsupervised text style transfer with padded masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8671-8680, Online. Association for Computational Linguistics.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746-751, Atlanta, Georgia. Association for Computational Linguistics.
Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. Evaluating style transfer for text. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 495-504, Minneapolis, Minnesota. Association for Computational Linguistics.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866-876, Melbourne, Australia. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Machel Reid and Victor Zhong. 2021. LEWIS: Levenshtein editing for unsupervised text style transfer. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3932-3944, Online. Association for Computational Linguistics.
Parker Riley, Noah Constant, Mandy Guo, Girish Kumar, David Uthus, and Zarana Parekh. 2021. TextSETTR: Few-shot text style extraction and tunable targeted restyling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3786-3800, Online. Association for Computational Linguistics.
Paul K. Rubenstein, Bernhard Scholkopf, and Ilya O. Tolstikhin. 2018. On the latent space of Wasserstein auto-encoders. *ArXiv*, abs/1802.03761.
Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157-162, Montreal, Canada. Association for Computational Linguistics.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6833-6844, Red Hook, NY, USA. Curran Associates Inc.
Tianxiao Shen, Jonas Mueller, Regina Barzilay, and Tommi Jaakkola. 2020. Educating text autoencoders: Latent representation guidance via denoising. ICML'20. JMLR.org.
I. Tolstikhin, O. Bousquet, S. Gelly, and B. Scholkopf. 2018. Wasserstein auto-encoders. In 6th International Conference on Learning Representations (ICLR).
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566-4575.
Pascal Vincent, H. Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In ICML '08.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao. 2020. On variational learning of controllable representations for text without supervision. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. 2018. Unsupervised text style transfer using language models as discriminators. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
A Additional details on Text Style Transfer Experiments
All baseline models were trained with all underlying architectures apart from their individual objective losses. Bi-directional GRUs were used for the encoder and decoder with input embeddings of size 300, a hidden representation of size 256 and a latent space of size 128. For all models using AAEs as the underlying autoencoder, the discriminator was a single-layered perceptron with 512 units. The ADAM optimiser with  $\beta_{1}$ ,  $\beta_{2}$  as 0.5, 0.999 and a learning rate of 0.001. All models were trained for 30 epochs as any more training steps caused the reconstruction loss to dominate and decrease overall performance on the TST task. All input perturbations were disabled in inference time.
A.2 Hyperparameter Selection
For the hyperparameters  $p$ ,  $\lambda_l$ , and  $\beta$  for the DAAE, LAAE and  $\beta$ -VAE, we fixed the values as 0.3, 0.05, 0.15 respectively. This decision was aligned with the results in Shen et al. (2020), which showed that these values produced the best reconstruction vs BLEU trade-off. We found this to be the case during subjective manual testing as well.  $\lambda_{adv}$  was set to 10 for all models having AAEs as the underlying architecture. For EPAAE, we found that  $\zeta \in [2.0, 3.0]$  overall showed the best results across all datasets. Therefore a manual search around this range was conducted to determine the optimal  $\zeta$  for each dataset.
A.3 Human Evaluations
TST outputs of two models, the baseline DAAE and the proposed EPAAE model, on the Yelp, SNLI and Tenses dataset were considered. The best performing EPAAE was chosen according to the results in Table 4, 5 and 6, particularly with respect to the content preservation metrics (Since TST% were similar across all models). Two hundred sentences (hundred from each base style) were randomly sampled from the test split of each dataset and style inverted (with scaling factor  $k = 2$ ). Six hundred
instances (each instance being a base and converted sentence pair) were equally split between three human evaluators. Each evaluator was given the task of labelling all two hundred instances from each dataset. The models were anonymous to evaluators and randomly named as "Model 1" and "Model 2". Each instance was to be labelled by an evaluator with four possible decision outcomes, i.e. "1 is best", "2 is best", "All are bad" and "All are good". For each instance, the majority of three votes from each of the three annotators were taken as the final decision for that corresponding instance. Instances without a majority were marked as "NA". The evaluation guidelines was formulated to consider which model a) successfully transferred the target style b) preserved the style-independent content and c) was overall fluent and grammatically coherent.
B Additional Experiments:
B.1 Sentence Interpolation in Latent Space - Qualitative Examples
We perform sentence interpolation using DAAE and EPAAE, starting from the same input sentence, incrementally moving along the same direction in the latent space in five fixed-size steps. We see that both the baseline and proposed model are able to produce fluent and coherent sentences, indicative of a smoothly populated latent space.
B.2 Text Style Transfer
B.2.1 Qualitative Examples
TST with scaling factor  $k = 2$  was performed on DNLI, Scitail datasets as seen in B.5 and B.6 respectively. Similarly, it was also performed on Voices and PP Removal datasets from the StylePTB benchmark as shown in B.7 and B.8 respectively. The proposed model for each dataset, was chosen to be the EPAAE with best performance in the content preservation metrics (shown in Table 4, 5 and 6). Samples were specifically selected in which at least one of the models was able to generate the ideal, style converted sentence with near-perfect content preservation and coherence. This was done by human evaluators, where both the models were anonymized.
B.2.2 Direction specific Discourse Transfer metrics
Particularly in the case of Discourse Style, it is natural to speculate that the difficulty of style transfer might be sensitive to the direction i.e.
two blond women are hugging one another the women are sleeping	two blond women are hugging one another the women are sleeping
two blond women are hugging one another the women are sleeping	two blond women are hugging one another the women are sleeping
three dogs affectionately playing the dogs are sleeping	three dogs affectionately playing the dogs are sleeping
two dogs are playing and wrestling with each other two cats are chasing each other through the house	two dogs are playing and wrestling with each other two cats are chasing each other through the house
two men in wheelchairs crash and they reach for the ball the men are sleeping	two men in wheelchairs crash and they reach for the ball the men are sleeping
three dogs in different shades of brown and white biting and licking each other the dogs are fighting	three dogs in different shades of brown and white biting and licking each other the dogs are fighting
Table B.1: Sentence Interpolation on sentence from SNLI dataset (Left: EPAAE, Right: DAAE)
the report will follow five consecutive declines	the report will follow five consecutive declines
the report will follow five consecutive declines	the report will follow five consecutive declines
the report will follow five consecutive declines in full monthly figures	the report follows five consecutive declines
the report followed five consecutive declines	the report followed five consecutive declines
the report follows five consecutive declines	the report will follow five consecutive declines in full monthly figures
the index hits its low 20297 off 2042 points	the report follows five consecutive declines in full monthly figures
Table B.2: Sentence Interpolation performed on the Voices dataset (Left: EPAAE, Right: DAAE)
the soviet had a world leading space program the guests noted	the soviet had a world leading space program the guests noted
the soviet had a world leading space program the guests noted	the soviet had a world leading space program the guests noted
the challengers had a big price advantage	the plant had a hairy stem that produced flowers and diminutive seeds
the expansion set off a marketing war	the japanese used 00 of the world's ivory
the uaw was seeking a hearing by the full 14 judge panel	the diplomat added that mr krenz had several things going for him
total return was price changes plus interest income	the japanese used 40 of the world's ivory
Table B.3: Sentence Interpolation performed on the Tenses dataset (Left: EPAAE, Right: DAAE)
Yelp	Negative to Positive	Positive to Negative
Base	i had high hopes and they simply could n't have fallen farther	probably won't stay here again	the food is always fla- vorful and filling	food was excellent and service was fast
Converted (DAAE)	the Italian flavors and amazing and all really impressive !	certainly here !	the food is not cooked and it was appealing .	food was not the wait-ress and was slow .
Converted (EPAAE)	i had great prices and ev- erything is exquisite !	definitely stay here !	the food is n't just ex- tremely bland .	food was ok. service was slow and disap- pointed
SNLI	Entailment to Contra- diction	Contradiction to En- tailment
Base	a man and lady standing on a seesaw at a park . a man and woman are on a seesaw outside	a dog is jumping through the water . tha animal is in the water	a young child in a green shirt is on a carousel . the young child is wear- ing a blue shirt	three people shopping in an isle in a foreign grocery store . three people look at clothes in the mall
Converted (DAAE)	a man and a woman are on a seesaw at a beach. a man and woman are on a seesaw	a dog is running through the water the animal is running through the wa- ter	a young child in a blue shirt is wearing a red shirt and a blue hat a child is on a swing	three people in shop- ping carts in a shopping mall. three people shop- ping in a store
Converted (EPAAE)	a man and a woman are sitting on a seesaw at a park. a man and a woman are sitting on a couch	a dog is jumping through the water the animal is flying through the air	a young child in a red shirt is on a green slide. the child is wearing a shirt	three people shopping in a shopping mall in a foreign city. three peo- ple shopping in the mall
Tenses	Future to Past	Past to Future
Base	miller brewing co and general motors will be included by clients	i will see a possibility of going to 2200 this month	spiegel was 80 con- trolled	mr deaver had reopened a public relations busi- ness
Converted (DAAE)	miller brewing co and general motors were in- cluded	i was a math major but i was going at this	42 will be advanced	mr breeden will not be public relations busi- ness will say
Converted (EPAAE)	miller brewing co and general motors was in- cluded by clients	i saw a possibility of go- ing to 2200 this year	spiegel will be 80 con- trolled	mr deaver will have re- opened a public rela- tions business
Table B.4: Qualitative examples of TST task by DAAE and EPAAE on Yelp, SNLI and Tenses dataset
DNLI
Models	Entailment to Contradiction	Contradiction to Entailment
Example 1	Example 2	Example 1	Example 2
Base	i work in a cubicle . i have a hectic job	i have 3 kids . i am the proud parent of 2 boys and 1 girl	i work as a contractor for a cab company . i am a travel agent	i am a ballet dancer . i work as a trauma surgeon
DAAE	i work as a cashier in a supermarket . i am a cashier	i have a wife and two kids . i am a proud us sailor	i work for a online company . i do not have a job	i am a ballet dancer . i work at home editing
EPAAE	i work in a cubicle . i am a skilled craftsman	i have 3 kids . i am a proud mother of two	i work for a cab company . i have been working for many years	i am a ballet dancer . i took ballet lessons when i was a kid
Table B.5: TST task performed on DNLI dataset
Scitail
Models	Entailment to Neutrality	Neutrality to Entailment
Example 1	Example 2	Example 1	Example 2
Base	most amphibians such as frogs live part of their lives on land and return to water to breed. frogs are amphibians that live part of the time in fresh water and live rest of the time on land	reptiles are found on every continent worldwide with the exception of the polar antarctica. modern members of the reptiles group live in many different habitats and are found on every continent except antarctica	respiratory rate is 40 150 breaths per minute . the normal respiratory rate per minute in adult humans is 12 18 breaths	and was the goddess of earth and sky moon and sun . the sun and the moon appear to be about the same size in the sky because the moon is smaller in diameter and is closer to earth than the sun
DAAE	most amphibians have dual habitats and the remainder on land and live in the first decade of fish and live in the first living things have the time of fresh water and live rest	many species are the most common ancestor in the unk of the same species of birds and mammals have the same species of birds	cardiac output is the amount of breaths per minute the normal respiratory rate per minute the normal respiratory rate per minute in adult humans is 12 18 breaths	the sun and the moon is the closest planet to the sun and the moon is the closest planet to the sun the sun is the closest star
EPAAE	most species live on land and breed on land frogs are amphibians that live part of the time to live in the ocean	reptiles are found on every continent except antarctica the majority of species live in the pacific continent except antarctica and birds are found on every continent except antarctica	respiratory rate is a series of breaths per minute in the normal respiratory rate per minute in adult humans is 12 18 breaths	earth and moon is about 400 times of the earth and that it is about 400 times of the sun and the moon appear to be about the same size in the sky because
Table B.6: TST task performed on the SciTail dataset
Voices
Models	Active to Passive	Passive to Active
Example 1	Example 2	Example 1	Example 2
Base	he will make his remarks to a plo gathering in baghdad	accounting problems will raise more knotty issues	one stuck to old line business traditions while the change was embraced by the other	the only time it is had by the violin is right at the end
DAAE	he will be told by him as a sewage treatment plant he will say	transportation services will be provided by accounting problems	outside i spotted two young exchange the company's own microprocessor said	the soviets eavesdrop- ping pays off however because the contract they say
EPAAE	his remarks will be made by him from a plo gathering in baghdad	more knotty issues will be raised by accounting problems	one stuck to old line business traditions while the other embraced the change	the only time the violin has its right to the end
Table B.7: TST task performed on Voices dataset
PP
Models	PP Removal	Information Addition
Example 1	Example 2	Example 1	Example 2
Base	the problems will be magnified by the june killings	the senate will not vote on six lesser charges	membership will have since swelled	the dispute will pit two groups
DAAE	the problems will be	the senate will come	membership will have since swelled to between 20 of smaller creditors	the dispute will pit two groups of claimants against each other of each other
EPAAE	the problems will be magnified	the senate will not vote	membership will have since swelled to at least 21 since friday	the dispute will pit two groups of claimants against the two of japan
Table B.8: TST task performed on PP dataset
DAAE (p = 0.3)	EPAAE (ζ = 2.0, p = 0.1)
Input	very disappointed they ran out of average things in the menu	very disappointed they ran out of average things in the menu
k = 1	very nice and ran out of the food in !	very disappointed and all in out of the menu .
k = 1	very nice and varied with the quality menu !	very disappointed and all great flavors in the menu .
k = 2	very nice and varied selection of food !	very prepared and great flavors in the menu .
k = 2.5	very nice locally and varied menu ! !	very delicious flavors and great flavors in the
k = 3	very great selection and great seafood !	very delicious flavors and great fish selections .
Input	food was excellent and service was fast	food was excellent and service was fast
k = -1	food was excellent and the service was fast	food was excellent and service was fast .
k = -1.5	food was excellent and the service was fast .	food was excellent and service was fine but service
k = -2	food was not the waitress and was slow .	food was undercooked and service was fine but delivery .
k = -2.5	it was not the waitress , but was slow .	food was undercooked and service was not fine at food .
k = -3	it was not ignored the service , but was slow .	!
Table B.9: Qualitative examples of Fine-grained TST on the Yelp Dataset
is it Contradiction/Neutrality to Entailment or vice versa. Intuitively, this makes sense as the Entailment to Contradiction/Neutrality tasks can be achieved simply by randomly editing either the subject or predicate or both, in any one sentence, to trigger a contradiction/neutrality between the two. However, in the reverse task, the edited part must be carefully chosen to precisely match the context of the other sentence to trigger entailment.
To analyze this, direction-specific quantitative metrics for Discourse TST are conducted for the SNLI (B.10), DNLI (B.11) and SciTail (B.12) datasets. We notice a disparity in performances in fact, does exist, mainly highlighted by the differences in the TST% metric. This sensitivity to direction is present in all models across all datasets but is most significant in the SNLI dataset in which TST% goes as low as  $20.6\%$  for the Contradiction to Entailment task and as high as  $83.7\%$  for the opposite task. Future work can focus on trying to specifically improve the Contradiction to Entailment task, as doing so will be a measure of a model's ability to detect and carefully align the content of one sentence to match another.
C Details on Datasets
Here we provide some additional details of all the datasets used in this work.
Complexity of Styles in Datasets: As discussed in Section 5.2, we consider three tasks- sentiment, discourse and fine-grained text style transfer. As prepossessing, we remove non-essential special characters and lowercase all sentences. Except for the Yelp dataset, no pruning is done based on sentence length. The vocab size during training was limited at  $25\mathrm{k}$  unless mentioned otherwise.
Sentiment Style Datasets: We use the preprocessed version from (Shen et al., 2017) of the Yelp
dataset. It contains 200k, 10k, 10k sentences in the train, dev and test split respectively. The sentiment labels (positive, negative) were considered as style.
**Discourse Style Datasets:** We used three NLI datasets - SNLI, DNLI, and Scitail. Each instance in the SNLI dataset (Bowman et al., 2015) consists of two sentences. These sentences either contradict, entail (agree) or are neutral towards each other. The resultant dataset contained 341k, 18k, 18k in the train, dev, test splits respectively. The DNLI dataset (Welleck et al., 2019) consists of contradiction, entailment and neutrality labelled instances instead in the form of a first-person dialogue like representation. The dataset contains 208k, 11k, 11k sentences in train, dev and test respectively. Scitail (Khot et al., 2018) is an entailment dataset created from multiple-choice science exams and the web, in a two-sentence format similar to SNLI and DNLI. The first sentence is formed from a question and answer pair from science exams and the second sentence is either a supporting (entailment) or non-supporting (neutrality) premise obtained from the internet. The dataset contained 24k, 1.3k, 1.3k sentences in the train, dev, test splits respectively. For SNLI and DNLI, all instances with the "neutrality" label were removed. Style transfer task performed from "contradict" to "entail" and vice versa. For SciTail (Khot et al., 2018), the transfer task was from "neutral" to "entail" and vice versa.
Fine-grained Style Datasets: We used Style-PTB dataset (Lyu et al., 2021) for fine-grained style. It consists of 21 styles/labels with themes ranging from syntax, lexical, semantic and thematic transfers as well as compositional transfers which consist of transferring more than one of the aforementioned fine-grained styles. To check whether the EPAAE can capture fine-grained styles better by leveraging its better organized latent space, we make use of three styles i.e. Tenses, Voices (Active
Figure B.1: A TSNE plot of encoded latent vectors for all 1950 sentences in the Toy Dataset for the DAAE model.