SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
w _ {j} ^ {\prime} = \frac {\exp (| w _ {j} | / T)}{\sum_ {i = 1} ^ {| S |} \exp (| w _ {i} | / T)} \tag {1}
$$
where  $T$  is the temperature of the softmax.
A modification we make to the unlikelihood loss function is that we only apply the loss for a given token  $s_j$  if the probability distribution output for the token at position  $t$  indicates that  $s_j$  should be output, that is, if  $s_j = \arg \max_{v \in \mathcal{V}} p_\theta(v | y_{<t})$  where  $\mathcal{V}$  denotes BART's token vocabulary. Denoting an indicator function for this event by  $\mathbb{1}_{s_j, t}$ , our final likelihood loss term  $\mathcal{L}(p_\theta, S, \mathbf{y})$  is:
$$
- \sum_ {t = 1} ^ {| \mathbf {y} |} \sum_ {j = 1} ^ {| S |} \mathbb {1} _ {s _ {j}, t} w _ {j} ^ {\prime} \log \left(1 - p _ {\theta} \left(s _ {j} \mid y _ {<   t}\right)\right) \tag {2}
$$
Data. We split our dataset of 4459 abstract-PLS pairs so that 3568 reviews are in the training set, 411 in the validation set, and 480 in the test set. We experimented with hyperparameters by manually inspecting a subset of the validation set and report results on the entire test set.
<sup>1</sup>We also considered starting from a checkpoint corresponding to training over CNN/Daily News but preliminary manual examination of model outputs suggested starting from XSum yielded higher quality outputs.
${}^{2}$  Five-fold evaluation showed that the model achieved  $> {90}\%$  accuracy. We also experimented with the Simple Wikipedia/Wikipedia dataset (Zhu et al., 2010), but this model was not effective in early experiments.
Hyperparameters. For nucleus sampling, we use a top- $p$  value of 0.9. In the unlikelihood training procedure, we experimented with different values of  $\alpha$  in our total loss function  $(1,10,10^{3},10^{6})$  on the validation set and different temperatures  $T$  in the softmax step (1,2,5,10). Based on manual examination of the generated texts in the validation set, we determined that  $(T = 2,\alpha = 100)$  yields the most coherent and high-quality simplifications, so we only report results for this case. All models are fine-tuned on our dataset for 1 epoch with a batch size of 1 and a learning rate that starts at 3e-5 and decreases linearly to 0 over the course of training. For optimizer, we used AdamW with  $\varepsilon = 1\mathrm{e} - 8$  (Kingma and Ba, 2015; Loshchilov and Hutter, 2019).
In this section we comment on the generated texts' readability, quality of summarization and simplification, stylistic fidelity with the PLS, and overall coherence and simplicity based on human examination. In the results tables, we indicate whether lower or higher scores for the metrics reported are better with  $\downarrow$  and  $\uparrow$  symbols, respectively.
5.1 Readability scores
Table 5 reports the mean readability scores achieved under different training settings. Results generated via models trained with the proposed UL objective achieve significantly lower Flesch-Kincaid scores than those achieved by both the technical abstracts and reference PLS, whereas the model trained without UL produced texts with a higher reading level than the PLS. Rather surprisingly, the UL-Newsela and UL-both settings, both of which use the Newsela dataset to produce un-likelihood weights, did not yield a decrease in estimated grade levels. We suspect that this could be attributed to the difference in domains, that is, the tokens contributed by the Newsela classifier are not generated frequently enough to have a noticeable impact during unlikelihood training.
These results suggest that: (1) BART is capable of performing simplification of medical texts such that outputs enjoy reduced reading levels compared to those of the technical abstracts; (2) The proposed use of UL to explicitly penalize the model for outputting jargon allows for the generation of text with even greater readability than the reference PLS. The reading levels of even the simplified out
FK↓	ARI↓	SciBERT↓
Abstracts	14.42	15.58	0.57
PLS	13.11	15.08	0.53
No UL	13.44	15.09	0.55
UL-Cochrane	11.97	13.73	0.55
UL-Newsela	12.51	14.15	0.54
UL-Both	12.26	14.04	0.54
Table 5: Flesch-Kincaid, ARI, and SciBERT masked probability scores for generated PLS. Differences between abstracts and generated PLS are statistically significant; so are differences in FK and ARI between UL models and No-UL  $(p < 0.01$ , paired  $t$ -test).
puts, however, are at the late-high school/early college levels. This could reflect the relatively small differences in readability scores between abstracts and PLS in general (Section 3.2).
5.2 Style
In Section 3.2 we showed that SciBERT masked probability scores are more useful as a discriminator between technical abstracts and PLS than the standard readability metrics, which use surface-level cues like word and sentence counts. Experiments by Jawahar et al. (2019) suggest that BERT-style masked language models encode a wide array of syntactic and semantic features of language, which they then employs for downstream tasks. For this reason, we use SciBERT masked probability scores as our notion of style, with lower scores corresponding to simpler, less technical language. To explore the extent to which the generated summaries stylistically resemble the PLS, we computed the average of the SciBERT masked probability scores of the generated texts for each model. The results are shown in Table 5 along with the readability scores.
We see that every model produces text with significantly lower probability scores than the abstracts, which suggests that they successfully convert input abstracts into less-technical summaries. Though the average scores are higher than that of the PLS, this difference is not statistically significant, so we can consider the outputs of the models to be stylistically on par with the target PLS.
5.3 Content
We report SARI (Xu et al., 2016), a standard edit-based metric for text simplification, and BLEU (Papineni et al., 2002), a precision-based method for machine translation that is also often reported for simplification systems. Xu et al. (2016) showed
R1↑	R2↑	RL↑	BLEU↑	SARI↑
No UL	0.40	0.15	0.37	0.44	0.38
UL-Cochrane	0.38	0.14	0.36	0.39	0.40
UL-Newsgala	0.39	0.15	0.37	0.43	0.39
UL-Both	0.38	0.14	0.37	0.40	0.39
Table 6: ROUGE, BLEU,and SARI scores for generated PLS. All differences between No-UL and UL models, except for (BLEU,UL-Newsela), are statistically significant  $(p <   0.01$  , paired  $t$  -test).
that SARI correlates better with human evaluation for simplification tasks, focusing more on simplicity, while BLEU is stronger with respect to meaning and grammar. Finally we report the F1 versions of ROUGE-1, ROUGE-2, and ROUGE-L (Lin, 2004), which are the standard metrics typically used for summarization tasks.
Table 6 shows the mean ROUGE, BLEU, and SARI scores. While UL models yielded small but significantly better SARI scores, the opposite is true for the ROUGE and BLEU measures. Despite the lack of clear patterns in these scores, there are clear qualitative differences between the different models' outputs, which are expounded upon in Section 5.4.
Extractive vs. abstractive? Although not reflected in the automatic evaluation metrics above, the increase in readability of UL models led us to suspect that UL models are more abstractive than extractive, namely, they contain more paraphrases. To determine the degree to which the outputs directly copy content from the technical abstracts, we computed the fraction of  $n$ -grams in the output PLS that also occur in the abstract (without considering repetition). These results are shown in Table 7.
We observe that the introduction of UL clearly decreases  $n$ -gram overlap, and the difference becomes more marked as  $n$  increases. The use of Cochrane weights (those from the logistic regression model trained to discriminate between technical abstracts and PLS) likely reduces  $n$ -gram overlap because the tokens most penalized in UL training are those used to represent numerical data, e.g., statistics and confidence intervals. Penalizing these tokens discourages the regurgitation of numerical details from the technical abstract. The use of Newsela weights does not have the same effect, again likely due to the domain difference between the tokens penalized during unlikelihood training and those generated by the model. None
N=1	N=2	N=3	N=4
PLS	0.56	0.29	0.19	0.14
No-UL	0.95	0.89	0.84	0.79
UL-Cochrane	0.84	0.67	0.57	0.49
UL-Newsela	0.92	0.81	0.73	0.66
UL-Both	0.89	0.76	0.67	0.59
Table 7: % of  $n$  -grams in reference/generated PLS that are also in the abstracts.
of the model settings, however, achieve  $n$ -gram overlap scores nearly as low as the reference PLS, indicating that the generated summaries remain considerably more extractive than human-written PLS.
5.4 Manual examination and analysis
We manually examined the outputs generated by our models on a random sample of 40 technical abstracts from the test split of our dataset. While reading these outputs, we made special note of text length, readability and coherence, the presence of hallucinated information not found in the corresponding abstract, and artifacts such as repetition and misspelled words.
Our examination demonstrated that the generated texts were all significantly shorter than their respective abstracts and also shorter than the reference PLS. Furthermore, the models trained with Cochrane weights ('UL-Cochrane' and 'UL-Both') produced shorter texts on average than the models trained without UL or with Newsela weights. This observation is supported by the results in Table 9, which displays the average number of tokens and sentences in the summaries generated under different training settings.
One explanation for why UL with Cochrane weights produces shorter summaries is that training with these weights discourages the copying of statistics from the original abstract, a phenomenon exemplified in Appendix A, Table 10. Another trend that we noticed was that higher  $\alpha$  values produce shorter, more readable summaries at the expense of information completeness. Training with a high  $\alpha$  also increases the likelihood of hallucination, misspelling, and repetition. These drawbacks greatly impacted coherence for  $\alpha \geq 1000$ . These observations suggest a tradeoff between completeness of information and conciseness as  $\alpha$  is varied in the training process.
The most common hallucination found in all settings, and especially with high  $\alpha$ , was the inclusion
Hallucination: The evidence is up-to-date as of February 2016. We found seven studies, involving 1839 participants, that compared home-based treatment with hospital-based care for venous thromboembolism.
Misspelling: The review authors provided no information on other important outcomes, including gastro-oesophageal reflux, aspiration pneumonia, necrotise enterulitis...
Repetition: However, we were not able to combine their results because of the small number and small number of people in the included studies.
Table 8: Example of artifacts found in generated PLS.
# Tokens	# Sentences
Abstracts	492.04	14.03
PLS	254.60	9.59
No UL	228.27	8.34
UL-Cochrane	163.79	7.10
UL-Newsela	201.01	8.45
UL-Both	173.88	7.75
Table 9: Lengths of generated PLS.
of a statement of the form The evidence is current to [month] [year]. The reason for this is that many PLS contain such a statement of currency not found in the technical abstracts, so models learn to include such a statement even if it cannot be factually deduced from the abstract. Another observation is that most commonly misspelled words are those of medications and diseases. Table 8 provides examples of the various kinds of artifacts found in the generated PLS. The presence of these artifacts suggest that in practice, generated texts should be reviewed before being used.
In this work we considered the important task of medical text simplification. We derived a new resource for this task made up of technical abstracts summarizing medical evidence paired with plain language versions of the same; we have made this data publicly available to facilitate further research.<sup>3</sup> We proposed a new masked language model (MLM)-based measure of the technicality of text, which quantifies technicality by calculating the likelihood of tokens in the input text with respect to a transformer-based MLM trained on a technical corpus. We demonstrated that this metric better discriminated technical abstracts from PLS than more traditional notions of readability.
We proposed models for automated simplification based on BART (Lewis et al., 2020), extending
the training objective by incorporating an explicit penalty for production of 'jargon' terms. We found that this method can improve model outputs (i.e., can increase simplicity and the abstractiveness of summaries) according to the metrics considered.
This paper presents a dataset from the Cochrane library; this comprises only the freely available portion of the information on Cochrane (abstracts that are readily available to all). No annotators other than the authors of this paper are involved in the manual inspection of this data. In addition, the Cochrane data in itself, and our collection and inspection of it, does not involve any personally identifiable information.
The baseline models presented involves simplifying medical texts. Inconsistencies (e.g., hallucinations) of the generated PLS with respect to the original review is an artifact discussed in Section 5.4. This can lead to misinformed readers. Therefore, the outputs of the proposed systems should always be manually examined before being used.
This work was supported in part by the National Institutes of Health (NIH), grant R01-LM012086, and the National Science Foundation (NSF), grant IIS-1850153. We acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources that have contributed to the research results reported within this paper.
Emil Abrahamsson, Timothy Forni, Maria Skeppstedt, and Maria Kvist. 2014. Medical text simplification using synonym replacement: Adapting assessment of word difficulty to a compounding language. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), pages 57-65.
Viraj Adduru, Sadid A Hasan, Joey Liu, Yuan Ling, Vivek V Datla, Ashequl Qadir, and Oladimeji Farri. 2018. Towards dataset creation and establishing baselines for sentence-level neural clinical paraphrase generation and simplification. In KHD@IJ-CAI.
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2020. Data-driven sentence simplification: Survey and benchmark. Computational Linguistics, 46(1):135-187.
3We emphasize that the data here comprises only text derived from publicly accessible abstracts.
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elghohary, Sergey Feldman, Vu Ha, et al. 2018. Construction of the literature graph in semantic scholar. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 84-91.
Paul W Armstrong and C David Naylor. 2019. Counteracting health misinformation: a role for medical journals? Jama, 321(19):1863-1864.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3606-3611.
Jeanne Sternlicht Chall. 1958. Readability: An appraisal of research and application. 34. Ohio State University.
Iain Chalmers, Douglas G Altman, et al. 1995. Systematic reviews. BMJ Publishing London.
Cochrane. 2013. Standards for the reporting of plain language summaries in new cochrane intervention reviews (pleacs).
William Coster and David Kauchak. 2011. Simple English Wikipedia: a new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 665-669.
Jerwin Jan S Damay, Gerard Jaime D Lojico, Kimberly Amanda L Lu, D Tarantan, and E Ong. 2006. SIMTEXT: Text simplification of medical literature. In Proceedings of the 3rd National Natural Language Processing Symposium-Building Language Tools and Resources, pages 34-38.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural programmer-Interpreter model for sentence simplification through explicit editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3393-3402.
Susannah Fox and Maeve Duggan. 2013. Health online 2013. https://www.pewresearch.org/internet/2013/01/15/health-online-2013/. Online; accessed April 2, 2021.
Chris Gledhill, Hanna Martikainen, Alexandra Mestivier, and Maria Zimina. 2019. Towards a linguistic definition of 'simplified medical english': Applying textometric analysis to cochrane medical abstracts and their plain language versions. LCM - La Collana / The Series, 9788879169196:91-114.
Natalia Grabar and Rémi Cardon. 2018. CLEAR - simple corpus for medical French. In Proceedings of the 1st Workshop on Automatic Text Adaptation (ATA), pages 3-9.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1693-1701. Curran Associates, Inc.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations.
John PA Ioannidis, Michael E Stuart, Shannon Brownlee, and Sheri A Strite. 2017. How to survive the medical misinformation mess. European journal of clinical investigation, 47(11):795-802.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657.
Antonia Jelicic Kadic, Mahir Fidahic, Milan Vujcic, Frano Saric, Ivana Propadalo, Ivana Marelja, Svjetlana Dosenovic, and Livia Puljak. 2016. Cochrane plain language summaries are highly heterogeneous with low adherence to the standards. BMC medical research methodology, 16(1):61.
Sasikiran Kandula, Dorothy Curtis, and Qing Zeng-Treitler. 2010. A semantic and syntactic text simplification tool for health content. In AMIA annual symposium proceedings, pages 366-370.
Jasna Karačić, Pierpaolo Dondio, Ivan Buljan, Darko Hren, and Ana Marusic. 2019. Languages for different health information readers: multitrait-multimethod content analysis of cochrane systematic reviews textual summary formats. BMC medical research methodology, 19(1):75.
Rohit Kate, Xiaoqiang Luo, Siddharth Patwardhan, Martin Franz, Radu Florian, Raymond Mooney, Salim Roukos, and Chris Welty. 2010. Learning to predict readability using diverse linguistic features. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 546-554.
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of
new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research Branch.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
George Roger Klare. 1963. Measurement of readability.
Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Burch. 2019. Complexity-weighted loss and diverse reranking for sentence simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3137-3147.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Junyi Jessy Li and Ani Nenkova. 2015. Fast and accurate prediction of sentence specificity. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2281-2287.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Leonardo Campillos Llanos, Dhouha Bouamor, Pierre Zweigenbaum, and Sophie Rosset. 2016. Managing linguistic and terminological variation in a medical dialogue system. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 3167-3173.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. International Conference on Learning Representations.
Partha Mukherjee, Gondy Leroy, David Kauchak, Srinidhi Rajanarayanan, Damian Y Romero Diaz, Nicole P Yuan, T Gail Pritchard, and Sonia Colina. 2017. Negait: A new parser for medical text simplification using morphological, sentential and double negation. Journal of biomedical informatics, 69:55-62.