SlowGuess/NAACL_2021_OCR huggingface dataset:
Ashwin Devaraj
The University of Texas at Austin ashwin.devaraj@utexas.edu
Iain J. Marshall
iain.marshall@kcl.ac.uk
Junyi Jessy Li
jessy@austin.utexas.edu
We consider the problem of learning to simplify medical texts. This is important because most reliable, up-to-date information in biomedicine is dense with jargon and thus practically inaccessible to the lay audience. Furthermore, manual simplification does not scale to the rapidly growing body of biomedical literature, motivating the need for automated approaches. Unfortunately, there are no large-scale resources available for this task. In this work we introduce a new corpus of parallel texts in English comprising technical and lay summaries of all published evidence pertaining to different clinical topics. We then propose a new metric based on likelihood scores from a masked language model pretrained on scientific texts. We show that this automated measure better differentiates between technical and lay summaries than existing heuristics. We introduce and evaluate baseline encoder-decoder Transformer models for simplification and propose a novel augmentation to these in which we explicitly penalize the decoder for producing 'jargon' terms; we find that this yields improvements over baselines in terms of readability.
The need for accessible medical information has never been greater. A Pew Research survey of American's online health habits in 2013 revealed that "one in three American adults have gone online to figure out a medical condition" (Fox and Duggan, 2013). Given the rise of medical misinformation on the internet (Ioannidis et al., 2017), accessibility has become an increasingly urgent issue (World Health Organization, 2013; Armstrong and Naylor, 2019). However, sources that provide accurate and up-to-date information, including scientific papers and systematic reviews (Chalmers et al., 1995), are often effectively inaccessible to most readers because they are highly technical and laden with terminology (Damay et al., 2006).
Technical abstract: Analysis showed a higher rate of weight gain in the high-volume feeds group: mean difference  $6.20\mathrm{g / kg / d}$  (95% confidence interval 2.71 to 9.69). There was no increase in the risk of feed intolerance or necrotising enterocolitis with high-volume feeds, but 95% confidence intervals around these estimates were wide.
Plain-language summary: Very low birth weight infants who receive more milk than standard volumes gain weight more quickly during their hospital stay. We found no evidence suggesting that giving infants high volumes of milk causes feeding or gut problems, but this finding is not certain.
Table 1: Sample excerpts from a technical abstract (top) and corresponding plain-language summary (bottom) from the Cochrane Library.
One potential solution to this problem is text simplification, i.e., editing documents such that they are accessible to a wider audience, while preserving the key information that they contain. Although manual simplification is too expensive to feasibly apply at scale, automatic text simplification (Siddharthan, 2014; Alva-Manchego et al., 2020) provides a potential means of rendering a large volume of specialist knowledge more accessible.
Large-scale data-driven simplification systems have mostly been trained on Wikipedia (Zhu et al., 2010; Woodsend and Lapata, 2011; Coster and Kauchak, 2011) and news (Xu et al., 2015), and focus on sentence simplification (Wubben et al., 2012; Wang et al., 2016; Xu et al., 2016; Zhang and Lapata, 2017; Kriz et al., 2019; Dong et al., 2019; Alva-Manchego et al., 2020); on the other hand, medical text simplification is resource poor. Recent work has involved constructing sentence-aligned data automatically using monolingual text alignment methods (Adduru et al., 2018; Van den Bercken et al., 2019), but this process is noisy and constrains the task to sentence-level simplification.
In this work we explore new data and modern conditional text generation models (Lewis et al., 2020) to simplify medical documents. We introduce a dataset of paired (technical, simplified) texts
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4972-4984 June 6-11, 2021. ©2021 Association for Computational Linguistics
derived from the Cochrane Database of Systematic Reviews, which is comprised of evidence syntheses on a wide range of clinical topics. Critically, each review includes a plain-language summary (PLS) written by the authors. PLS are written directly from the full reviews with their own structure and guidelines; they are not simplified versions of the corresponding technical abstracts of the reviews, nor are they summaries of the abstracts.
However, we observe that portions of the PLS can be considered simplifications of analogous sections in the abstracts, that is, they contain roughly the same content but involve simplification operations such as paraphrasing, word/sentence deletion, and summarization. We heuristically derive 4459 such pairs of sections (or paragraphs) of technical-plain English bitexts. We provide an excerpt of the dataset we have constructed in Table 1.
This data allows us to explore characteristics of simplified versions of technical medical texts. We show that the differences in traditional readability metrics, such as Flesch-Kincaid (Kincaid et al., 1975) and Automated Readability Index (Senter and Smith, 1967), are small. Instead, the differences are better captured using large-scale pretrained masked language models, and this reveals that there is more to the language difference than the shallow cues such as sentence and word lengths that traditional readability metrics focus on.
We present baseline methods for automatic text simplification over this data and perform analyses that highlight the challenges of this important simplification task. We find that when naively finetuned for the task, existing encoder-decoder models such as BART (Lewis et al., 2020) tend to prefer deletion over paraphrasing or explaining, and are prone to generating technical words. We propose a new approach to try and mitigate the latter issue by imposing a variant of unlikelihood loss (Welleck et al., 2019) that explicitly penalizes the decoder for production of 'technical' tokens. We show that this yields improvements in terms of readability with only a minor tradeoff with content quality.
In sum, this work takes a step towards paragraph-level simplification of medical texts by: (1) introducing a sizable new dataset, (2) proposing and validating a new masked language model (MLM)-based metric for scoring the technicality of texts, (3) analyzing and understanding the style of plain language in this important domain, and (4) presenting baselines that exploit a variant of unlike
lihood training to explicitly penalize models for producing jargon. We release our code and data at https://github.com/AshOlogin/Paragraph-level-Simplification-of-Medical-Texts.
Recent efforts on data-driven text simplification methods have tended to rely on two resources: the Wikipedia-Simple Wikipedia aligned corpus (Zhu et al., 2010; Woodsend and Lapata, 2011; Coster and Kauchak, 2011) and the Newsela simplification corpus (Xu et al., 2015). Yet, there is an urgent need to simplify medical texts due to health literacy levels (World Health Organization, 2013). However, due to a lack of resources with which to train model-based simplification systems in this domain, past work has tended to focus on lexical simplification (Damay et al., 2006; Kandula et al., 2010; Abrahamsson et al., 2014; Mukherjee et al., 2017). Recently, Adduru et al. (2018) and Van den Bercken et al. (2019) introduced sentence-aligned corpora at the scale of thousands of sentence pairs. In contrast to our corpus, these datasets were automatically derived using paraphrase mining or monolingual alignment processes. Furthermore, as these are exclusively sentence corpora, they limit the set of potential approaches to just those that operate over sentences. Grabar and Cardon (2018) created a simplification corpus for medical texts in French, in which a small subset of the text pairs are manually sentence-aligned, resulting in 663 sentence pairs, 112 of which are also from Cochrane.
With respect to modeling, recent work has focused on sentence simplification, treating it as a monolingual machine translation task (Wubben et al., 2012; Wang et al., 2016; Xu et al., 2016) using encoder-decoder models (Zhang and Lapata, 2017; Kriz et al., 2019; Dong et al., 2019). In the medical domain, existing systems tend to adopt lexical and syntactic simplification (Damay et al., 2006; Kandula et al., 2010; Llanos et al., 2016). Research on document simplification has been sparse; to the best of our knowledge, the few prior works on this in English have focused on analysis (Petersen and Ostendorf, 2007), sentence deletion (Woodsend and Lapata, 2011; Zhong et al., 2020), and localized explanation generation (Srikanth and Li, 2020). This work proposes and evaluates an encoder-decoder model for paragraph-level simplification.
Raw	Compiled data
Before-filter	After-filter
Abstract	815 ± 331	551 ± 272	501 ± 211
PLS	394 ± 216	284 ± 156	264 ± 136
Table 2: Means and standard deviations of original abstract and PLS lengths (tokens), and our compiled data before & after filtering out texts with more than 1024 tokens.
3 Technical abstracts vs. plain-language summaries
We compiled a dataset of technical abstracts of biomedical systematic reviews and corresponding PLS from the Cochrane Database of Systematic Reviews, which comprises thousands of evidence synopses (where authors provide an overview of all published evidence relevant to a particular clinical question or topic). The PLS are written by review authors; Cochrane's PLS standards (Cochrane, 2013) recommend that "the PLS should be written in plain English which can be understood by most readers without a university education". PLS are not parallel with every sentence in the abstract; on the contrary, they are structured heterogeneously (Kadic et al., 2016).
3.1 Data compilation
To derive the dataset we scraped the online interface to the database for articles containing PLS, extracting the raw text of the technical abstracts and PLS for those that we identified. In this way we obtained 7820 pairs after removing problematic links (e.g., HTTP 404 errors). We also excluded reviews with atypical formatting that would have required extensive manual inspection.
On average, PLS are shorter than abstracts (Table 2, 'raw'). They contain sections different from those in the abstracts, emphasize different content, and sometimes contain information not in the abstract. We divided documents into those that are split into sections with subheadings and those without (henceforth "long-form" summaries);  $56\%$  of the data are long-form. For the sectioned PLS, headers are quite different from those found in the abstracts. The latter adhere to one of the 2 following formats:
1. Background, Objectives, Search Methods, Selection Criteria, Data Collection and Analysis, Main Results, Authors' Conclusions
2. Background, Objectives, Methods, Main Results, Authors' Conclusions
In contrast, PLS contain a variety of headings, with the most common ones shown below:
background, study characteristics, key results, review question, quality of the evidence, search date, quality of evidence, conclusions
Others include questions such as What was the aim of this review? And How up-to-date was the review?
Manual inspection revealed that the results, discussion, and conclusion sections of abstracts and summaries tended to occur in parallel. This motivated us to extract aligned subsets of abstracts and summaries to compose our dataset. More specifically, we determined the approximate location of the section describing studies and results in each text and kept everything from that point forward.
Therefore, in the abstracts we kept the text from the Main Results section onward. For the sectioned PLS we kept every section after and including the first that contained one of the following substrings: find, found, evidence, tell us, study characteristic. For the long-form PLS, we found the first paragraph containing any of the following words within the first couple sentences and included that and subsequent paragraphs: journal, study, studies, trial. We keep one-paragraph PLS in their entirety. We also exclude instances where the PLS and abstracts are drastically different in length, by keeping only instances where the length ratio between the two falls between 0.2 and 1.3. Our final dataset comprises 4459 pairs of technical abstracts and PLS, all containing  $\leq 1024$  tokens (so that they can be fed into the BART model in their entirety).
3.2 Characterizing readability differences
Readability metrics. Designing metrics that reliably capture readability remains an open topic of research. In recent years, a host of metrics have been developed that use a wide variety of linguistic features to assess readability in a supervised manner. For example, Kate et al. (2010) developed a metric based on syntactical, semantic, and language model-based features, and Vajjala and Lučić (2018) developed a new readability corpus, on which they trained support vector machines to predict text readability. For this medical text simplification task, however, we considered a couple established heuristics-based readability metrics due to clear domain differences between our Cochrane corpus and those used to train supervised readability metrics: the Flesch-Kincaid score (Kincaid
Metric	Abstracts	PLS
Flesch-Kincaid	14.4 ± 2.3	12.9 ± 2.4
ARI	15.5 ± 2.8	14.9 ± 3.0
Table 3: Means and standard deviations of different readability scores calculated over abstracts and PLS.
et al., 1975) and the automated readability index (ARI) (Senter and Smith, 1967), which estimate the educational maturity (grade-level) required to comprehend a text. These metrics rely on a combination of shallow cues, most notably lengths of words, sentences, and documents.
Table 3 reports the mean grade levels of abstracts and PLS calculated via the above metrics. There are small but statistically significant  $(p < 0.01$ , paired  $t$ -test) differences between the abstract and PLS distributions, especially for Flesch-Kincaid. For instance, the maximum difference in mean minimum grades (1.5) is achieved by Flesch-Kincaid, and the number is only 0.6 with ARI. By contrast, a 3-5 grade level difference was shown on the Wikipedia and Britannica simplification datasets (Li and Nenkova, 2015). The high grade-level suggested by standard readability metrics confirms prior studies highlighting that these 'plain language' summaries of medical systematic reviews remain at higher reading levels than those of average US adults (Karačić et al., 2019).
Masked language models. Despite the small differences in readability metrics, PLS do qualitatively seem easier to understand (see Table 1 for an example). This suggests that existing measures are incomplete. We propose adopting modern masked language models — namely BERT (Devlin et al., 2019) — as another means of scoring the 'technicality' of text. In particular, when such models are trained on specialized or technical language (e.g., scientific articles) we would expect the likelihoods subsequently assigned to 'jargon' tokens to be relatively high compared to a model trained over general lay corpora, as in the original BERT model (Devlin et al., 2019).
Capitalizing on this intuition, we consider two large-scale pre-trained masked language models: (1) BERT (Devlin et al., 2019) trained on BooksCorpus (Zhu et al., 2015) and English Wikipedia; and (2) SciBERT (Beltagy et al., 2019), trained on a sample of 1.14 million technical papers from Semantic Scholar (Ammar et al., 2018) (mostly biomedical and computer science articles).
Inspired by the original training objective for these models, we compute a probability score for a document by splitting it into sentences, masking 10 subsets of  $15\%$  of the tokens in each sentence (exempting CLS and SEP), computing the likelihoods of the original tokens in the distributions output by the model in each masked position, and averaging these probabilities over all the masked subsets and sentences in the document. The details are shown in Algorithm 1.
Algorithm 1 Used to compute a probability score for a text document  $D$  given a masked language model  $M$ . The output of the model returned by a call to FORWARD is a matrix where each row maps to a distribution over all the tokens in the vocabulary. The APPEND function adds a value to the end of a list.
Figure 1 depicts the distributions of probabilities output by general BERT and SciBERT for the abstracts and PLS in our dataset. Both masked LMs induce distributions over instances from the respective sets that are clearly different. For example, SciBERT (which yields sharper differences) outputs higher likelihoods for tokens comprising the technical abstracts than for those in the plain language versions, as we might expect given that this is pretrained on technical literature. A paired  $t$ -test confirms that these observed differences between the abstracts and PLS distributions are statistically significant (with  $p < 0.01$ ).
Which metric discriminates better? To better determine how well the proposed masked probability outputs discriminate between technical abstracts and PLS, we plot receiver operating characteristic
Figure 1: BERT (left) vs SciBERT (right) probabilities of technical abstracts (blue) and PLS (red).
Figure 2: ROC Curves for Readability Metrics.
(ROC) curves for the outputs of BERT, SciBERT, Flesch-Kincaid and ARI, coding technical and PLS abstracts as 0 and 1, respectively. The SciBERT curve has a higher AUC score (0.70) than the general BERT curve (0.66), indicating that it is better at discriminating between plain language and technical abstracts. For this reason, we use the SciBERT masked probabilities when analyzing the texts generated by our models.
The AUC score for SciBERT is also higher than that for Flesch-Kincaid, indicating that simplicity in PLS can be better captured by probabilistic means than by surface-level linguistic cues, and that it is more appropriately viewed as a stylistic difference rather than one of readability. This echoes the arguments made by early investigators of readability metrics that these measures do not replace more subtle linguistic characteristics, e.g., style (Klare, 1963; Chall, 1958).
3.3 Lexical analysis
We next investigate lexical differences between technical abstracts and PLS. In prior work, Gledhill et al. 2019 performed extensive lexical analysis on
this corpus by comparing the relative frequencies of different part-of-speech  $n$ -grams found in the abstracts and PLS. Here, we analyze the weights from a logistic regression model that classifies whether a text is a technical abstract or a PLS (coding the latter as  $y = 1$ ); the weights learned by the model can be conveniently incorporated into the loss function we use to train our simplification model (Section 4.2).
We represent texts as normalized bag-of-words frequency vectors (with a feature for each token in the BART vocabulary). We performed 5-fold cross validation on the data and observed an average accuracy of  $92.7\%$ , which indicated that even this relatively simple model is capable of accurately distinguishing technical abstracts from PLS. We also evaluated this model on the train-validation split described in Section 4.3. The model achieves a very high AUC score of 0.99, indicating that it almost perfectly separates abstracts from PLS.
To better understand which kinds of tokens are most associated with technical abstracts and PLS, we examined the tokens with the highest-magnitude learned weights in the model, with the most negative weights corresponding to tokens indicative of technical abstracts and the most positive ones being indicative of PLS. These notable tokens are displayed in Table 4. From this table it is clear that numerical tokens and those related to statistical analysis, like bias and  $CI$  (confidence interval) are most indicative of abstracts. The tokens indicative of PLS are less illuminating and merely reflect common phrases include in PLS, such as In this review and We searched scientific databases.
In Section 4, we use this model as a discriminator along with our transformer encoder-decoder model during training to penalize the generation of tokens that are indicative of technical abstracts.
4 Baseline models for simplification
4.1 Pretrained BART
Our baseline simplification model is BART (Lewis et al., 2020), an encoder-decoder architecture in which both components are transformers (Vaswani et al., 2017). The decoder is auto-regressive, making it a natural fit for generation tasks. BART has been shown to achieve strong performance on text summarization, specifically on the CNN/Daily Mail (Hermann et al., 2015) and XSum (Narayan et al., 2018) datasets.
We initialize the weights in BART to those esti
Token	Weight	Token	Weight
0	-7.262	people	4.681
.	-6.126	review	4.551
%	-5.379	We	4.461
CI	-4.986	This	3.413
;	-4.821	that	2.943
95	-4.593	The	2.836
significant	-4.273	side	2.722
R	-3.726	who	2.671
1	-3.685	blood	2.515
There	-3.477	found	2.514
bias	-3.303	searched	2.407
criteria	-3.263	The	2.114
outcome	-3.247	results	2.098
(	-3.195	their	2.022
inclusion	-3.148	current	1.984
Table 4: The tokens with the most negative and most positive weights in a logistic regression model trained to distinguish technical abstracts from PLS.
mated via fine-tuning on the XSum (Narayan et al., 2018) dataset as provided by HuggingFace's Model Hub (Wolf et al., 2019). We then fine-tune these models on our corpus. $^{1}$
In the decoding step, we use nucleus sampling (Holtzman et al., 2019): at each step of token generation the next token is sampled from a probability distribution constructed by removing the 'tail' of probability mass from BART's output distribution and then renormalizing. This strategy mitigates the awkward repetition typical of greedy methods like beam search while still avoiding incoherence by truncating the unlikely tail in the original model distribution.
4.2 Unlikelihood training
As an additional mechanism to encourage simple terminology in the PLS generated by our model, we propose a new method in which we explicitly penalize the model for producing seemingly technical words via unlikelihood training (Welleck et al., 2019; Li et al., 2020). The idea is to add a term to the objective that encourages the model to decrease the probability mass assigned to some set of tokens  $S$ . This is realized by adding a term to the  $(\log)$  loss:  $UL = \sum_{j=1}^{|S|} - \log(1 - p_{\theta}(s_j | y_{<t}, x))$ , where  $x$  is the technical abstract input to the encoder,  $y_{<t}$  is the prefix of the target summary  $y$  input to the decoder at time  $t$ , and  $p_{\theta}(s_j | y_{<t}, x)$  is the probability assigned to token  $s_j$  in the distribution output by BART (with model parameters  $\theta$ ) at time  $t$ . This
expression is referred to as Unlikelihood Loss (UL). The UL term is weighted by a positive constant  $\alpha$  and added to the typical log-likelihood objective.
We construct  $S$  by collecting tokens with negative weights from a bag-of-words logistic regression model trained to classify whether a document is simple (1) or complex (0), for which negative tokens are indicative of complex language. We then softmax the absolute values of these weights so that they sum to 1 and the tokens most indicative of technical abstracts (i.e., those with the most negative weights initially) contribute the most to this sum. We consider three variants of this procedure. (1) We classify whether a document is a PLS or an abstract (Section 3.3). (2) We use external data, namely the Newsela corpus (Xu et al., 2015), and train a model to distinguish between documents of reading levels 0 and 3. (3) We train two different models for the previous tasks and then sum the weight vectors before applying a softmax to derive token penalties.
Let  $w_{j}$  denote the learned logistic regression weight for token  $s_{j} \in S$ . The final weight  $w_{j}^{\prime}$  used in the unlikelihood loss function is: