SlowGuess/ACL_2020_OCR huggingface dataset:
Fluency (marked as PPL) is usually measured by the perplexity of the transferred sentence. We fine-tune the state-of-the-art pretrained language model, Bert (Devlin et al., 2019), on the training set of each dataset for each style.
Content Similarity measures how much content is preserved during style transfer. We calculate 4-gram BLEU (Papineni et al., 2002) between model outputs and inputs (marked as self-BLEU), and between outputs and gold human references (marked as ref-BLEU).
Automatic metrics for content similarity are arguably unreliable, since the original inputs usually achieve the highest scores (Fu et al., 2019). We
5We only report TS models for expertise to laymen language, since they do not claim the opposite direction.
$^{6}$ https://github.com/senisioi/NeuralTextSimplification/
Dataset	SimpWiki	MSD
Metrics	Acc	PPL	self-BLEU	ref-BLEU	human	Acc	PPL	self-BLEU	ref-BLEU	human
E2L	OpenNMT+PT	44.29	6.88	93.38	50.16	3.99	16.00	5.95	59.89	9.91	3.53
UNTS	55.14	22.06	44.80	31.11	2.96	22.07	24.62	20.49	3.94	2.66
ControlledGen	46.57	21.76	58.33	29.21	2.78	11.7	5.77	88.61	13.13	3.78
DeleteAndRetrieve	38.29	5.10	0.74	0.65	1.19	74.67	3.92	6.66	2.95	2.28
StyleTransformer	42.86	13.30	74.93	41.72	3.43	40.89	12.12	53.66	10.09	3.31
Gold	62.00	8.36	49.91	-	-	93.63	4.23	14.00	-	-
L2E	ControlledGen	40.29	33.09	63.21	29.40	2.83	6.22	5.02	93.05	13.77	3.92
DeleteAndRetrieve	37.43	5.72	0.00	0.41	1.14	64.59	4.23	6.65	2.77	2.33
StyleTransformer	39.43	12.91	77.94	36.63	3.36	49.33	8.04	48.36	10.57	3.25
Gold	58.86	6.93	50.13	-	-	88.15	4.34	14.01	-	-
Table 4: Overall performance based on style transfer evaluation metrics from expertise to laymen language (marked as E2L) and in the opposite direction (L2E). Gold denotes human references.
thus also conduct human evaluation. To evaluate over the entire test set, only layman annotators are involved, but we ensure that the layman style sentences are accompanied as references to assist understanding. Each annotator is asked to rate the model output given both input and gold references. The rating ranges from 1 to 5, where higher values indicate that more semantic content is preserved.
Text Simplification Measurement. The above metrics may not perform well regarding language simplicity (Sulem et al., 2018a). So, we also utilize a TS evaluation metrics: SARI (Xu et al., 2016). It compares the n-grams of the outputs against those of the input and human references, and considers the added, deleted and kept words by the system.
4.4 Overall Performance
Table 4 present the overall performance. Since each pair of parallel sentences has been verified during annotation, we did not report human scores to avoid repeated evaluations. We can see that: (1) Parallel sentences in MSD have higher quality than SimpWiki, because our gold references are more fluent (4.29 vs. 7.65 in perplexity on average) and more discriminable (91% vs. 60% on average style accuracy). (2) The transfer for L2E is more difficult (except in content similarity) than that for E2L: 39.55% vs. 42.50% in Acc on average, 11.50 vs. 10.33 in PPL on average and 2.80 vs. 2.63 in human ratings on average. This is because the increase in expertise levels requires more contexts and knowledge, and is harder than simplification. (3) TS models perform similarly with ST models. Besides, supervised model OpenNMT+PT outperforms the unsupervised UNTS in fluency and content similarity due to the additional supervision signals. On the other hand, UNTS achieves higher Acc since it utilizes more non-parallel training data. (4) The style accuracy is the reverse to content sim
ilarity, making it more challenging to propose a comprehensive evaluation metric that can balance the two opposite directions. In terms of content similarity, even if both self-BLEU and ref-BLEU show a strong correlation with human ratings (over 0.98 Pearson coefficient with p-value  $< 0.0001$ ), the higher scores of ControlledGen cannot demonstrate its superior performance, as it actually makes little modifications to styles. Instead, DeleteAndRetrieve, presents a strong ability to control styles (70% on average in Acc on MSD), but hardly preserves the contents. Style Transformer performs more stably.
Next, we discuss key factors of MSD. We take the E2L as the exemplar for discussion, as we have observed similar results for the opposing direction.
4.5 Impact of Concepts
Figure 4a shows the performance curves of BLEU and style accuracy. We choose the concept range to ensure they contain similar number of sentences. Along with the increasing number of concepts, we can see a downward BLEU trend. This is because it becomes more difficult to preserve content when the sentence is more professional. As for style accuracy, DeleteAndRetrieve achieves the peak around [8,12) concepts, while the performance of other models drops gradually. Clearly, a lower number of concepts benefit the model for better understanding the sentences due to their correlated semantics, but a larger number of concepts requires knowledge-aware text understanding.
4.6 Impact of Structures
Figure 4b presents the performance curves regarding the structure differences, where the edit distance is computed as mentioned in Section 3.2. Higher score denotes more heterogeneous structures. We see a similar trend with the curves of concepts. That is, existing models perform well
(a) Impact of concepts.
(b) Impact of structure differences.
(c) Performance on different PCIO.
Figure 4: Curves of BLEU and style accuracy, where the x-axis denotes the number of concepts per sentence, edit distance between parallel sentences, and different PCIO elements, respectively.
in simple cases (fewer concepts and less structural differences), but becomes worse if the language is complex. We doubt that the encoder in each model is able to understand the domain-specific language sufficient well without considering knowledge. We thus propose a simple variant of ControlledGen by introducing terminology definitions, and observe some interesting findings in Section 4.10.
4.7 Performance on Medical PCIO
The style of medical PCIO elements (e.g., symptoms) are slightly different. We separately evaluate each model and present the results in Figure 4c. Style accuracy remains similar among these medical PCIO elements, but there are significant differences among the models in their performance for preserving content. Specifically, models perform well for those sentences about treatment, but perform poorly for evaluation, because this type of sentences usually involve many rare terms, challenging understanding.
4.8 Performance using Simplification Metrics
Dataset	SimpWiki	MSD
E2L	L2E	E2L	L2E
OpenNMT+PT	.2695	-	.2204	-
UNTS	.3115	-	.3313	-
ControlledGen	.3187	.2856	.2170	.1636
DeleteAndRetrieve	.1983	.1684	.3378	.3345
StyleTransformer	.3189	.2933	.3541	.3411
Table 5: Performance using SARI.
Table 5 presents the performance based on the TS evaluation metric, SARI. We utilize the Python
package<sup>7</sup> and follow the settings in the original paper. Surprisingly, SARI on MSD presents a relatively comprehensive evaluation that is consistent with the above analysis as well as our intuition. ControlledGen and OpenNMT+PT are ranked lower since they tend to simply repeat the input. DeleteAndRetrieve and UNTS are ranked in the middle due to the accurate style transfer but poor content preservation. StyleTransformer is ranked highest as it performs stably in Table 4 and Figure 4a, 4b, 4c. This inspires us to further investigate automatic evaluation metrics based on TS studies, which is our ongoing work. Even so, we still recommend necessary human evaluation in the current stage.
4.9 Case Study
Table 6 presents two examples of transferred sentences. In the first example, both OpenNMT+PT and UNTS make lexical changes: replacing progresses with goes. DeleteAndRetrieve transfers style successfully but also changes the content slightly. The other two output the original expert sentence, that is the reason why they achieve higher BLEU (also PPL) but fails in Acc. Manipulation method (i.e., DeleteAndRetrieve) is more progressive in changing the style, but disentanglement method, ControlledGen, prefers to stay the same.
The second example shows structural modifications. We can see that the supervised OpenNMT+PT simply deletes the complex terminolo
7https://github.com/cocoxu/simplification
Expertise input
OpenNMT+PT
UNTS
ControlledGen
DeleteAndRetrieve
StyleTransformer	Prostate cancer usually progresses slowly and rarely causes symptoms until advanced.
Prostate cancer usually goes slowly and rarely causes symptoms until advanced.
Prostate cancer usually goes slowly and rarely causes symptoms until advanced.
Prostate cancer usually progresses slowly and rarely causes symptoms until advanced.
Prostate cancer usually progresses slowly and rarely causes symptoms until advanced.
Laymen Gold	Prostate cancer usually causes no symptoms until it reaches an advanced stage.
Expertise input
OpenNMT+PT
UNTS
ControlledGen
DeleteAndRetrieve
StyleTransformer
Laymen Gold	Cystic lung disease and recurrent spontaneous pneumothorax may occur. These disorders can cause pain and shortness of breath.
Cystic lung disease can cause pain and shortness of breath.
lung lung disease and roughly something pneumothorax may occur.
Cystic lung disease and recurrent spontaneous pneumothorax may occur. These disorders can cause pain and shortness of breath.
ear skin disease in the lungs and the lungs may occur in other disorders and may cause chest pain and shortness of breath.
Cystic lung disease and exposed spontaneous pneumothorax may occur.
Air-filled sacs (cysts) may develop in the lungs. The cysts may rupture, bringing air into the space that surrounds the lungs (pneumothorax). These disorders can cause pain and shortness of breath.
Table 6: Examples of model outputs. Red/blue words with underlines highlight model/expected modifications.
gies recurrent spontaneous pneumothorax, but the output sentence can be deemed correct. Controlled-Gen still outputs the original input sentence, and the other three fail by either simply cutting the long sentence off, or changing the complex words randomly. Besides, all of the above models still perform much worse than human, which motivates research into better models.
4.10 Discussion
We have two observations from the aspects of model and evaluation. For models, there is a huge gap between all of the above models and human references. MSD is indeed challenging to conduct language modifications considering both knowledge and structures. Most of the time, these models basically output the original sentences without any modifications, or simply cut off the complex long sentence. Therefore, it is exciting to combine the techniques in TS, such as syntactic revisions including sentence splitting and lexical substitutions, with the techniques in ST: style and content disentanglement or the unsupervised idea of alleviating the lack of parallel training data.
For evaluation, human checking is necessary in the current stage, even though SARI seems to offer a good start for automatic evaluation. Based on our observations, it is actually easy to fool the three ST metrics simultaneously via a trick: output sentences by adding style-related words before the original inputs. This is demonstrated by a variant of ControlledGen. We incorporate into the generator an extra knowledge encoder, which encodes the definition of concepts in each sentence (as mentioned in Section 3.1). Surprisingly, such a simple
model achieves a very high style accuracy (over  $90\%$ ) and good BLEU scores (around 20). But the model does not succeed in the style transfer task, and simply learns to add the word doctors into layman sentences while almost keeping the other words unchanged; and adding the word eg into the expertise sentences. Thus, it achieves good performance on all of the three ST measures, but makes little useful modifications.
We proposed a practical task of expertise style transfer and constructed a high-quality dataset, MSD. It is of high quality and also challenging due to the presence of knowledge gap and the need of structural modifications. We established benchmark performance of five SOTA models. The results shown a significant gap between machine and human performance. Our further discussion analyzed the challenges of existing metrics.
In the future, we are interested in injecting knowledge into text representation learning (Cao et al., 2017, 2018b) for deeply understanding expert language, and will help to generate knowledge-enhanced questions (Pan et al., 2019) for laymen.
This research is supported by the National Research Foundation, Singapore under its International Research Centres in Singapore Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Laurens Van den Bercken, Robert-Jan Sips, and Christoph Lofi. 2019. Evaluating neural text simplification in the medical domain. In WWW.
Olivier Bodenreider. 2004. The unified medical language system (umlts): integrating biomedical terminology. *Nucleic acids research*.
Colin Camerer, George Loewenstein, and Martin Weber. 1989. The curse of knowledge in economic settings: An experimental analysis. Journal of political Economy.
Yixin Cao, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018a. Neural collective entity linking. In COLING.
Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Chengjiang Li, Xu Chen, and Tiansi Dong. 2018b. Joint representation learning of cross-lingual words and entities via attentive distant supervision. In EMNLP.
Yixin Cao, Zikun Hu, Tat-seng Chua, Zhiyuan Liu, and Heng Ji. 2019. Low-resource name tagging learned with weakly labeled data. In EMNLP.
Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, and Juanzi Li. 2017. Bridge text and knowledge by learning multi-prototype entity mention embedding. In ACL.
John Carroll, Guido Minnen, Darren Pearce, Yvonne Canning, Siobhan Devlin, and John Tait. 1999. Simplifying text for language-impaired readers. In EACL.
Raman Chandrasekar and Bangalore Srinivas. 1997. Automatic induction of rules for text simplification. KBS.
Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology.
William Coster and David Kauchak. 2011. Learning to simplify sentences using wikipedia. In Proceedings of the workshop on monolingual text-to-text generation. ACL.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In ACL.
Louise Deléger and Pierre Zweigenbaum. 2008. Paraphrase acquisition from comparable medical corpora of specialized and lay texts. In AMIA Annual Symposium Proceedings.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Yao Fu, Hao Zhou, Jiaze Chen, and Lei Li. 2019. Rethinking text attribute transfer: A lexical analysis. ArXiv.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In AAAI.
Goran Glavas and Sanja Stajner. 2015. Simplifying lexical simplification: Do we need simplified corpora? In ACL.
Robert Gunning. 1968. Technique of clear writing.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In ICML.
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. 2017. Shakespearizing modern language using copy-enriched sequence-to-sequence models. EMNLP.
Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019. Unsupervised text attribute transfer via iterative matching and translation. In EMNLP.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. AAAI.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In EACL.
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Diederik P. Kingma and Max Welling. 2013. Auto-encoding variational bayes. *ICLR*.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In ACL.
Nicholas Kloehn, Gondy Leroy, David Kauchak, Yang Gu, Sonia Colina, Nicole P Yuan, and Debra Revere. 2018. Improving consumer understanding of medical text: Development and validation of a new sub-simplify algorithm to automatically generate term explanations in english and spanish. JMIR.
Guillaume Lample, Sandeep Subramanian, Eric Michael Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In ICLR.
Dianqi Li, Yizhe Zhang, Zhe Gan, Yu Cheng, Chris Brockett, Ming-Ting Sun, and William B. Dolan. 2019. Domain adaptive text style transfer. EMNLP.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In NAACL.
Kevin Miller. 2009. Wordnet-about.
Gustavo Paetzold and Lucia Specia. 2016. Unsupervised lexical simplification for non-native speakers. In AAAI.
Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019. Recent advances in neural question generation. arXiv preprint arXiv:1905.08949.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. 2018. Style transfer through back-translation. In ACL.
Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. In NAACL.
Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In ACL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In *NAACL*.
Matthew Shardlow and Raheel Nawaz. 2019. Neural text simplification of clinical letters with a domain specific phrase table. In ACL.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In NIPS.
Luca Soldaini and Nazli Goharian. 2016. Quickumls: a fast, unsupervised approach for medical concept extraction. In MedIR workshop, sigir.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018a. Bleu is not suitable for the evaluation of text simplification. In EMNLP.
Elior Sulem, Omri Abend, and Ari Rappoport. 2018b. Simple and effective text simplification using semantic and neural methods. In ACL.
Sai Surya, Abhijit Mishra, Anirban Laha, Parag Jain, and Karthik Sankaranarayanan. 2019. Unsupervised neural text simplification. In ACL.
Sharon Swee-Lin Tan and Nadee Goonawardene. 2017. Internet health information seeking and the patient-physician relationship: a systematic review. JMIR.
David Vickrey and Daphne Koller. 2008. Sentence simplification for semantic role labeling. In ACL.
Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In EMNLP.
Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In ACL.
Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, and Wenjie Li. 2018. Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. In ACL.
Wei Xu, Chris Callison-Burch, and Courtney Naples. 2015. Problems in current text simplification research: New data can help. TACL.
Wei Xu, Courtney Naples, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. TACL.
Wei Xu, Alan Ritter, Bill Dolan, Ralph Grishman, and Colin Cherry. 2012. Paraphrasing for style. In COLING.
Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In EMNLP.
Ye Zhang, Nan Ding, and Radu Soricut. 2018a. Shaped: Shared-private encoder-decoder for text style adaptation. In NAACL.
Yi Zhang, Jingjing Xu, Pengcheng Yang, and Xu Sun. 2018b. Learning sentiment memories for sentiment modification without parallel data. In EMNLP.
Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2019. Style transfer as unsupervised machine translation. AAAI.
Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M Rush, and Yann LeCun. 2018. Adversarily regularized autoencoders. In ICML.
Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In COLING.