Medical Text Simplification: Optimizing for Readability with Unlikelihood Training and Reranked Beam Search Decoding
Authors: Lorenzo Jaime Yu Flores
Heyuan Huang
Kejian Shi
Sophie Chheang
Arman Cohan
Venue: EMNLP 2023 Findings (2023)
Keywords: Medical Text
Simplification
Healthcare
Beam Search Decoding
Unlikelihood Learning

Abstract: Text simplification has emerged as an increasingly useful application of AI for bridging the communication gap in specialized fields such as medicine, where the lexicon is often dominated by technical jargon and complex constructs. Despite notable progress, methods in medical simplification sometimes result in the generated text having lower quality and diversity. In this work, we explore ways to further improve the readability of text simplification in the medical domain. We propose (1) a new unlikelihood loss that encourages generation of simpler terms and (2) a reranked beam search decoding method that optimizes for simplicity, which achieve better performance on readability metrics on three datasets. This study's findings offer promising avenues for improving text simplification in the medical field.

Medical Text Simplification: Optimizing for Readability with Unlikelihood Training and Reranked Beam Search Decoding
Lorenzo Flores', Heyuan Huang', Kejian Shi', Sophie Chheang', and Arman Cohan 1.3
Yale University
Yale School of Medicine 3 Allen Institute for AI
Abstract
In this work, we seek further improve medical text simplification. We first propose a new unlike- lihood loss that penalizes words in proportion to their reading level using a well-established read- ability index. Second, we propose a modified beam search method at decoding time to rerank interme- diate candidates based on their readability. Despite simplicity, methods improve readability based on automated metrics (up to 2.43 points on Flesch- Kincaid) and human evaluation, while maintaining similar performance in terms of factual consistency and overall simplification.
Text simplification has emerged increas- ingly useful application of AT for bridging the communication gap in specialized fields such medicine, where the lexicon is often dom- inated by technical jargon and complex con- structs. Despite notable progress, methods in medical simplification sometimes result in the generated text having lower quality and diver- sity. In this work, we explore ways to further improve the readability of text simplification in the medical domain. We propose (1) a new unlikelihood loss that encourages generation of simpler terms and (2) a reranked beam search decoding method that optimizes for simplic- ity, which achieve better performance on read- ability metrics on three datasets. This study's 's findings offer promising avenues for improving text simplification in the medical field.
We make the following contributions: (1) We propose a new form of unlikelihood loss based on well-established readability index to improve medical text simplification (2) We propose a decoding strategy that optimizes for readability in medical text simplification (3) We provide evaluation results for previous state-of-the-art three datasets in terms of readability and fac- tual consistency. We make our code publicly available at https://githute http://gggiuuu.cco/llffooo////// com/ljyflorsss si simpli aenpllifiaaiin-pojjee
1 Introduction
In recent years, text simplification has become an increasingly useful application of Al (Stajner, 2021) particularly in healthcare (Carroll et al., 1998; Saggion et al., 2015; Orăsan et al., 2018), where text can be technical and difficult to under- stand. By automating this process, we can help healthcare professionals explain key medical texts (e.g. doctor's reports. findings) to patients. Previ- work in text simplification in medical domain has explored use of pretrained language models (Devaraj et al., 2021; Sun et al., 2023; Martin et al., 2022; Trienes et al., 2022; Basu et al., 2023; Joseph et al., 2023b; Lu et al., 2023), reinforcement learn- ing (Phatak et al., 2022), and zero-shot prompting (August et al., 2022; Joseph et al., 2023b). Despite this progress, simplification sometimes results in the generated text having lower quality and diver- sity (Devaraj et al., 2021; Phatak et al., 2022). Fur- ther as we find some simplification models copy sentences from the source, and thus remain do not sufficiently improve the readability (See Appendix B).
Related Work Text simplification research pri- marily focuses on sentence-level (Xu et al., 2015; Specia and Paetzold, 2017: Sulem et al.. 2018; Srikanth and Li, 2020; Shardlow and Alva- Manchego, 2022), with some attempts at paragraph or document-level datasets (Sun al., 2021: La- ban et al., 2023). Most datasets have been sourced from accessible Wikipedia or News articles, which are already quite accessible. However, the medical field, laden with technical jargon, can greatly bene- fit from simplification. Initial methods in medical text simplification employed lexical and syntac- tic techniques (Llanos et al., 2016; Abrahamsson et al., 2014), while recent work includes finetuning language models like BART (Devaraj et al., 2021; Lewis et al., 2020) and a two-stage summarize- then-simplify approach (Lu et al., 2023). Medical

---

simplification has also expanded multilingual settings (Joseph et al., 2023b).
ing text with shorter words and shorter phrases. We incorporate this score as follows: At generation step 4, we identify the word v the vocabulary with the largest output probability; this the word which the model is most likely to output at step t. We compute the token-level UL for e by taking the product of the word's s Flesch- Kinca-K score and its standard UL term log(1 p(v (ec). The total UL (ULR) is the sum of the token-level penalties.
In this work, following Devaraj et al. (2021) we use unlikelihood (UL) training (Welleck et al.. 2020) to encourage the generation of simplified terminology. This strategy has been used in other domains to penalize inaccuracy (Hu et al., 2023; Nan al., 2022), complexity (Devaraj al., 2021; Lu et al., 2023), and redundancy (Lagutin al.. 2021; et al., 2020) in text generation. Unlike Devaraj et al. (2021), our work adapts UL to opti- mize for both readability and factual consistency. To improve simplification, we also intervene at the decoding stage. Previous work uses modified de- coding methods to address factual inconsistency (Shi et al., 2023; King et al., 2022; Sridhar and Visser, 2022), optimize fluency and diversity in text generation (Kriz et al., 2019; Hargreaves et al., 2021). Our work extends this by optimizing the de- coder for readability in medical text simplification.
V
ULn =
Σ
11 no Fr log(1 p(vlū<)))
where u indicates whether word v has the largest output probability in the vocabulary at step 4, and F K2 is the Flesch- Kincaid score of word
Consistency UL As we discuss in $4, we find that ULR alone Icads to hallucinations, hence we also penalize the model for generating unsupported words in some set e using an additional factual consistency UL (ULc)..
2 Methods
We propose two simple but effective approaches for improving medical text simplification, one during the training phase, and the other during decoding. Specifically, we propose a modified Unlikelihood Loss (Welleck et al.. 2020) to incorporate readabil- ity index and encourage the model to favor the generation of simpler words. Then, we introduce decoding approach that evaluates and re re-ranks the candidate beams by considering both readability and factuality. We detail these approaches below:
ULc
D wu log(1 p(vlū<))
where u. is an indicator for whether the word is the set hallucinated words e.
We determine the set as follows: we identify the sequence which the model is most likely to gen- erate, by finding the tokens with the highest logits at each generation step. Then, we then filter this set to the tokens which do not exist in either the input text nor label. At this point, the set contains all words which the model is likely to generate, but are not present in the input/label. Hence, it may contain words which are factually or grammat- ically correct, but don't 't match the gold summary. We'd like to penalize only the tokens which are sure are factually incorrect, hence we filter this set down to just entities using Spacy en_core_ ____e___ggggg NER models (Honnibal and Montani, 2017), which results in the entity set e.
2.1 Unlikelihood Loss for Simplification
Unlikelihood loss (UL) (Welleck al., 2020) is training objective that forces unlikely generations be assigned lower probability by the model (See Figure 1).
Readability UL Following prior work (Devaraj et al., 2021) we can use this loss to force the model to assign a lower probability to complex words. Unlike Devaraj et al. (2021), we use the Flesch- Kincaid (FK) readability score (Kincaid al., 1975) instead of model-predicterr scores. The Bussch-iincaid readability score is numerical in- dicator that assesses the complexity of text by estimating the US grade level needed for compre- hension. Because FK considers syllable count and average phrase length, it serves as a good proxy metric even for incomplete sentences, by prioritiz-
Overall Loss The overall loss a weighted sum of the negative log-likelihood (CNLL) and UL, where Ag and X are constants.
L - LNLL. + An + Ac ULC

---

Enitase
Lowe
Logits
Argmax
Reasabilty
Peet cahina
Saltmax
Probabiities
NLL Loss
Figure 1: Training Diagram Computing Unlikelihood Loss
2.2
Decoding for Simplification
Ranking Every l Steps Computing metrics at each generation step can be inefficient, and the meaning readability of the beam might not change after adding just one word. Hence, we reduce the frequency with which we perform the reranking to intervals of (See Appendix E).
Our proposed decoding strategy reranks candidate beams by their current readability and factual con- sistency scores, and retains the top beams as the candidates for the next token (See Figure 2).
Readability Score We optimize candidates' read- ability during decoding using Flesch-Kincaid (FK) Grade Level scores. FK represents the readability text measured by US grade level; hence, lower scores are more readable (Kincaid et al., 1975). These typically range from 0 to 18, but can extend past this range in practice. We compute FK of can- didate beams and cap it from 4 to 20, as we find that qualitatively, beams with scores below 4 as equally simple, and above 20 equally complex. Then, we normalize the score (s)s) from 0 to such that is least readable, and is most readable.
Hallucination Heuristic We implement a heuris- tic to remove beams with unsupported entities. We identify entities with the Spacy en_core_wee we_ee____lggg__e________________ NER model (Honnibal and Montani, 2017), check if the entities appear in the source, and set the beam's score as zero if any of the entities are not.
Experiments
Datasets We run our experiments on three datasets: Cochrane (Devaraj al., 2021) con- sists of 4,459 pairs of abstracts from the Cochrane Database Systematic Reviews and their corre- sponding summaries written by domain experts. MedEasi (Basu et al., 2023) consists of 1,697 pairs of annonn-d sentences sourced from the Merck Manuals (Cao al., 2020) and Simp Wiki (van den Bercken al., 2019). Finally, the Radiol- ogy Reports Dataset consists of 2.269 radiology reports collected from large urban hospital and simplified by medical residents and doctors.
Consistency Score Like in UL training, we find that optimizing solcly for readability in decod- ing may introduce hallucinations; hence we bal- ance readability with consistency, as measured by BERTScore (Zhang et al., 2020). We find that beams with scores below 0.60 to have equally poor factuality, hence cap the score (s(()) between 0.60 and 1.00 and normalize it.
Baselines We compare against a BART-XSum (Lewis et al., 2020) model which we further fine- tune on our datasets, and state-of-the-art models by Lu et al. (2023); Devaraj et al. (2021), all of which we fine-tune on each of the three datasets; we chose BART-XSum to align it with previous work, in order to provide an apples-to-apple comparison and isolate the impact of our methods. We also compare with state-of oate-of-thh-art large language model GPT-4-0314 (OpenAI, 2023)
Composite Score We compute composite score r(s) using an Fi-like metric. Note that the score is merely used to rerank the candidates.
1.
fr(s) <
rF(s) =
(r()
4
fr(s) < 20
20
0,
fF(s) 20
rs()) = {
f.(0)-0.60
fn(s) > 0.60
0.40
We evaluate the readability, consistency, and overall performance as follows:
Evaluation Metrics
0,
fB(S) < 0.60
(2r((s)rB(s) (r((s) (s))
IIternal dataset
r(s)
"We set the system's 's role as "You are helpful assistant that simplifies text". and the prompt as "Simplify this text:".

---

Scoring
Farranking
cir vierine aomrleeto
Theettitt TO calle
Doocder
ayteeee duchs the
Upliing Pacc Suent Genditti dedders
Figure 2: Diagram for Modified Beam Search for Decoding for Simplification
For readability, use the standard FK (Kincaid al., 1975) and ARI scores (Smith and Senter, 1967), which use the average word and sentence length to estimate the complexity of texts.
vidually results in fewer hallucinations than both BART-UL (Devaraj al., 2021) and NapSS (Lu et al., 2023) on Radiology, and against NapSS on MedEasi. When the baseline models perform well, we find that is because they tend to copy infor- mation from the input, and hence are less prone to hallucinations. In contrast, our strategies force the model to use simpler words and not copy the input, but may introduce inconsistencies with the source. We confirmed this with experiment: we compute the % 4-gram overlap of the model written summaries with the source, and observe that large portions of previous works' output is copied from the text, whereas output in our models are not (See Table 3).
For factual consistency, we use BERTScore (Zhang et al., 2020) and GvT-Eval (Liu et al., 2023) (See Appendix D), as these correlated well with human judgement (Scialom et al., 2021; Li et al., 2022; Liu et al., 2023). For GPT-Evvll, we evaluate 50 summaries, and report the fraction of samples which a factual inconsistency was found.
We additionally use SARI (Xu et al., 2016), an edit-based metric for text simplification, and ROUGE-LSum (Lin, 2004) for overall fluency.
4
Results
Note that some of the identified hallucination errors relatively minor find GPT-Eval to be very strict. For example the phrase "26 self- treatments of 26 Chinese herbal medicine prescrip- tions" is found to factually inconsistent with the source having the phrase "26 self concocted Chi- nese herbal compound prescriptions" by GPT-Eval (see Table 11 for full example).
We fine-tune BART model using our methods and present the results in Table 1; see Appendix for implementation details.
Effect of Unlikelihood Loss and Decoding On Cochrane and Radiology, our proposed methods achieve better readability scores in terms of FK and ARI. In particular, combining unlikelihood loss with the decoding strategy achieves 2.43/1.74 point improvement in FK/ARI upon the next best model for Cochrane, and 0. 12/0./11 point improve- ment for Radiology. Note that in the radiology dataset, the sentences are typically short, resulting in a lower (better) baseline readability score. See sample comparison of outputs in Appendix B. On MedEasi, our methods slightly underperform NapSS (Lu et al., 2023). We find that it some- times generates phrases instead of full sentences, which lowers FK/ART, since these scores depend on sentence length. In contrast, our models gener- ate complete sentences, which improve fluency at the cost of worse (i.e. higher) FK/ARI scores.
Human Evaluation We conduct a human eval- uation study to further investigate the results (See Table 2). We observe that our proposed UL and de- coder improves readability over a fine-tuned BART- XSum model 43% and 27% of the time, whereas the previous SOTA NapSS (Lu et al., 2023) only demonstrated clear benefits 3% of the time. How- ever, GPT-4 achieves the best performance, mainly because it trained on human preference data and omits minor details, only keeping the main sum- mary. contrast, our models and previous SOTA tend to retain these minor details from the source, which human evaluators may find irrelevant.
We note that the low interrater agrecability aligns with the ranges reported in previous work (Goyal et al., 2023), which reflects the subjective nature of human preference, given that simplicity and read- ability varies based on one's technical background and style preferences. While such variability is
Our methods generally improve over the prior SOTA in terms of SART and BERTScore, however, interestingly the radiology dataset all methods underperforn a fine-tuned BART model.
We observe that using UL or the decoder indi-

---

Dataset
Model
FK
ART
BSer t
SART t
BART-XSum
12.19
13.83
0,871
10/50
35.64
44.76
GPT-4
9.97
10.73
0,870
39.06
33.90
Cochrane
BART-UL UL (Devaraj et al., 2021)
11.02
12.69
0.873
15/50
40.08
39.25
NAPSS (Lu al., 2023)
12.12
13.64
0.869
21/50
32.94
45.49
UL
8.00
9.76
0.862
27/50
42.07
40.16
Decoder
8.63
9.61
0.873
20/50
41.25
43.88
UL + Decoder
7.54
8,99
0,866
38/50
42.12
41.11
BART-XSum
2.89
0.963
19/50
78.67
80.09
GPT-4
4.41
0.862
36.62
26.80
Radiology
BART-UL (Devaraj et al., 2021)
2.67
0.945
28/50
69.77
68.68
NAPSS (Lu et al., 2023)
3.16
2.62
0.927
42/50
62.72
59.02
UL
3,00
2.61
0,956
19/50
75.33
77.03
Decoder
3.11
2.76
0.952
21/50
71.75
74.57
UI. Decoder
2.87
2.50
0.953
23/50
72.40
74.83
BART-XSum
10.18
11.21
0,911
28/50
40.54
45.72
GPT-4
8.10
9.20
0,903
38.07
33.28
MedEasi
BART-UL (Devaraj et al., 2021)
10.57
11.28
0,915
2/50
35.33
47.91
NAPSS (Lu et al., 2023)
5.66
6.25
0.868
33/50
34.04
24.35
UL
8.47
9.67
0,907
23/50
42.25
43.30
Decoder
8.27
9.66
0.908
26/50
42.66
42.91
UL + Decoder
7.27
9.03
0,904
31/50
41.57
40.78
Table I: Performance on Feesch-Kincaid (FK), ARI, BERTScore (BSer), GPT-Eval (GPT), SARI, and ROUGE-
LSum (RL); SARI and RL are computed using the EASSE package AAlva-Manchego et al., 2019); All models except for GPT-4 fine-tuned the corresponding dataset in the row.
Model
Readability
Model
% 4-Gram Overlap
GPT-4
93%
0.190
0.199
BART-XSum
52.88%
NAPSS
3%
-0.118
-0.105
BART-UL (Devaraj et al., 2021) NAPSS (Lu al., 2023)
39.30%
UL Decoder
43%
0.004
0.0155
51.77%
27%
0.236
0.245
UL. (Ours) Decoder (Ours)
15.73%
9,80%
Table 2: Human Evaluation Results on 30 Examples from Cochrane, Readability the of instances where the model summary was strictly more readable than fine-tuned BART-XSum model's summary, i is Fleiss- Kappa interrater agreement (Fleiss, 1971), a Krippen- dorf (Passonneau, 2006).
Table 3: An analysis of the % 4-gram overlap between source text model outputs reveals that previ- models tend to copy directly from the source text, whereas our models do not, thereby simplifying and synthesizing
Model
FK
BSer 1
GPT
SARI
hard to avoid, the average proportions suggest that overall, our methods significantly improved upon previous SOTA (NAPSS).
UL
8.00
0.862
27/50
42.07
UL (ULR Only)
8.74
0.863
41/50
41.37
UL (U Only)
11.86
0.870
16/50
35.69
Table 4: Ablation results on each of the proposed Unlikelihood Losses. Performance on Flesch-King Kincaid (FK), BERTScore (BScr), GPT-Eval (GPT), and SARI.
Effect of Individual Unlikelihood Losses We test using LR and U Lc separately (See Table 4). U Ln alone results in good readability but poor fac- tual consistency, and vice versa for U Lc, justifying the need for both losses be used in conjunction.
We explored augmenting the data with external knowledge (See Appendix C.2), but found no ben- efit. This may because the sources and labels in the training data contains inconsistencies (Lu et al., 2023), which require further preprocessing. Addressing such hallucinations to generate more robust summaries is critical future direction in medical text summarization. which we aim to ex- plore further.
5 Conclusion
In this paper, we propose methods improve simplicity in medical text simplification; this im- proves the readability generated summaries, and achieves comparable BERTScore and SARI scores. However, hallucination remains a challenge.

---

Limitations
of word difficulty to a compounding language. Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Popu- lations (PITR), pages 57-65, Gothenburg, Sweden. Association for Computational Linguistics.
One limitation of our work is the persistence of ballucinations in the output. Previous literature has shown that this often originates from inconsisten- cies between the source and text data. For example, a number of training labels in the Cochrane dataset (Devaraj et al., 2021) contain the phrase, "The ev- idence is up to date as of x", despite no mention of a date in the source (Lu et al., 2023). To this end, future work can adapt strategies from liter- ature in summarization, which have shown that preprocessing (Adams et al., 2022; Wu et al., 2022) and augmenting (Yang al., 2023) the data can mitigate such hallucinations.
Griffin Adams, Han-Chin Shing, Qing Sun, Christo- pher Winestock, Kathleen McKeown, and Noémie Elhadad. 2022. Learning to revise references for faithful summarization. In Findings of the Associa- tion for Computational Linguisties: EMNLP 2022, pages 4009-4027, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Fernando Alva-Manchego, Louis Martin, Carolina Scar- ton, and Lucia Specia. 2019. EASSE: Easier auto- matic sentence simplification evaluation. In Proceed- ings of the 2019 Conference on Empirieal Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP); System Demonstrattioss pages 49-54, Hong Kong, China. Association for Computational Linguisties.
Another limitation is our paper examines medi- cal text simplification very broadly, whereas there may be expert knowledge needed to improve spe- cific tasks. Hence, future work can analyze such methods on a more niche set of datasets (e.g. medical literature, patient reports, health-related news). Such work can be extended to other lan- guages, for which multiple medical text simplifica- tion datasets have been developed (Trienes al., 2022; Grigonyte et al., 2014; Cardon and Grabar, 2019, 2020; Joseph et al., 2023a).
Tal August, Lucy Lu Wang. Jonathan Bragg, Marti A. Hearst, Andrew Head, and Kyle Lo. 2022. Paper plain: Making medical research papers approachable to healthcare consumers with natural language pro- cessing. ACM Transactions on Computer- Hummnn Interaction.
Chandrayee Basu, Rosni Vasu, Michihiro Yasunaga, and Qiang Yang. 2023. Med-easi: Finely annotated dataset and models for controllable simplification of medical texts. ArXiv, abs/2302.09155.
Finally, we note that our anter-annotato agree- ment on the task of readability is particularly low; this reflects both how human preferences are di- verse and how the task is highly subjective, as has been shown in other domains (Goyal et al., 2023). Moreover, readability not only differs by person, but also by domain and task. Future work can define domain-specific criteria, and recruit partici- pants from the exact target populations which the text is meant to be simplified for.
Olivier Bodenreider. 2004. The unified medical lan- guage system (umls): Integrating biomedical termi- nology. Nucleic acids research, 32:D267-70.
Yixin Cao, Ruihao Shui, Liangming Pan, Min- Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: A new task towards better communi- cation between experts and laymen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1061-1071, On- line. Association for Computational Linguisties.
Ethics Statement
We use publicly available datasets and make our preprocessing and training scripts available. As mentioned in the limitations section, both our meth- ods and previous methods still exhibit varying de- grees of hallucination, and have yet to undergo domain-specific examination. Hence, we do not recommend these models be applied in practical setting at the moment.
Rémi Cardon and Natalia Grabar. 2019. Parallel sen- tence retrieval from comparable corpora for biomed- ical text simplification. In Proceedings of the Inter- national Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 168-177, Varna, Bulgaria. INCOMA Ltd.
Rémi Cardon and Natalia Grabar. 2020. French biomed- ical text simplification: When small and precise helps. In Proceedings of the 28th International Confer- ence on Computational Linguistics, pages 710-716. Barcelona, Spain (Online). International Committee on Computational Linguistics.
References
Emil Abrahamsson, Timothy Forni, Maria Skeppstedt, and Maria Kvist. 2014. Medical text simplification using synonym replacement: Adapting assessment
John Carroll, Guido Minnen, Yvonne Canning, Siobhan Devlin, and John Tait, 1998. Practical simplification of english newspaper text to assist aphasic readers.

---

Proc. of AAAI-98 Workshop on Integrating Artificial Intelligence and Assistive Technology.
Don't say what you don't know: Improving the con- sistency of abstractive summarization by constraining beam search. In Proceedings of the 2nd Workshop Natural Language Generation, Evaluation, and Met- rics (GEM), pages 555-571, Abu Dhabi, United Arab Emirates (Hybrid), Association for Computational Linguisties.
Ashwin Devaraj. lain Marshall, Byron Wallace, and Junyi Jessy Li. 2021. Paragraph-level simplifica- tion of medical texts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4972-4984 Online. Association for Computational Linguistics.
Reno Kriz, João Sedoc, Marraann Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Kallison-Burchhe 2019. Complexity- weighted loss and diverse reranking for sentence simplification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Humun Language Technologies, Volume 1 (Long and Short Papers), pages 3137-3147 147, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
Joseph L., Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological Bulletin, 76:378-388 -822.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2023. News summarization and evaluation in the era of gpt-3.
Gintaré Grigonyte, Maria Kvist, Sumithra Velupillai, and Mats Wirén. 2014. Improving readability of Swedish electronic health records through lexical simplification: First results. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR). pages 74-83, Gothenburg, Sweden. Association for Computational Linguistics.
Philippe Laban, Jesse Vig, Wojciech Kryscinski, Shafiq R. Joty, Caiming Xiong, and Chien-Sheng Wu, 2023. Swipe: A dataset for document-level simplifi- cation of wikipedia pages. ArXiv, 0tbp/2305.19000
Evgeny Lagutin. Daniil Gavrilov, and Pavel Kalaidin. 2021. Implicit unlikelihood training: Improving neu- ral text generation with reinforcement learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1432-1441, Online. Association for Computational Linguisties.
James Hargreaves, Andreas Vlachos, and Guy Emerson. 2021. Incremental beam manipulation for natural language generation. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2563-2574, Online. Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embed- dings, convolutional neural networks and incremental parsing. To appear.
Mengting Hu, Yinhao Bai, Yike Wu, Zhen Zhang, Liqi Zhang, Hang Gao, Shiwan Zhao, and Minlie Huang. 2023. Uncertainty aware unlikelihood learning im- proves generative aspect sentiment quad prediction.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020. Don't say that! making inconsistent dialogue unlikely with unlikelihood training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715-4728, Online. Association for Computational Linguisties.
Sebastian Joseph, Kathryn Kazanas, Keziah Reina, Vish- nesh J. Ramanathan, Wei Xu, Byron C. Wallace, and Junyi Jessy Li. 2023a. Multilingual simplification of medical texts.
Sebastian Valiaparampil Joseph, Kathryn Kazanas, Keziah Reina, Vishnesh J. Ramanathan, Wei Xu, Byron Wallace, and Junyi Jessy Li. 2023b. Mul- tilingual simplification of medical texts. ArXiv,
Wei Li, Wenhao WU, Moye Chen, Jiachen Liu, Xinyan Xiao, and Hua Wu. 2022. Faithfulness in natural language generation: systematic survey of analysis. evaluation and optimization methods.
a25/2305.12532.
Chin-Yer Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- rion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
J. Peter Kincaid, Rohert P. Fishhurne, Richard I.. Rogers, and Brad S. Chissom. 1975. Derivation of new read- ability formulas (automated readability index, fog count and flesch reading ease formula) for navy en- listed personnel.
Yang Liu, Dan Iter Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nig evaluation using gpt-4 with better human align- ment.
Daniel King, Zejiang Shen, Nishant Subramani, Daniel S. Weld, lz Beltagy, and Doug Downey, 2022.

---

Leonardo Campillos Llanos, Dhouha Bouamor, Pierre Zweigenbaum, and Sophie Rosset. 2016. Managing linguistic and terminological variation in medical dialogue system. In Proceedings of the Tenth In- lernational Conference on Language Resources and Evaluation (LREEC 16), pages 3167-3173, Portorož, Slovenia. European Language Resources Association (ELRA).
Thomas Scialom, Louis Martin, Jacopo Staiano, Éric Villemonte de la Clergerie, and Benoit Sagot. 2021. Rethinking automatic evaluation in sentence simplifi- cation.
Matthew Shardlow and Fernando Alva- Iva-Manchego. 2022. Simple TICO-19: A dataset for joint translation and simplification of COVID-19 texts. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3093-3102, Marseille, France. Eu- ropean Language Resources Association.
Junru Lu, Jiazheng Li, Byron Wallace, Yulan He, and Gabriele Pergola. 2023. NapSS: Paragraph-Ievel medical text simplification via narrative prompting summarization. Findings of the Association for Computational Linguistics: EACL 2023, pages 1079-1091, Dubrovnik, Croatia. Association for Computational Linguisties.
Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. 2023. Trusting your evidence: Hallucinate less with context-awart decoding.
and sentence-matcion
E A Smith and R. Senter. 1967. Automated readabil- ity index. AMRL-TR. Aerospace Medical Research Laboratories, pages 1-14.
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoit Sagot. 2022. MUSS: Multilin- gual unsupervised sentence simplification by mining paraphrases. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 1661-1664, Marseille, France. European Language Resources Association.
Lucia Specia and Gustavo Paetzold. 2017. Lexical sim- plification with neural ranking. In Conference of the European Chapter of the Association for Computa- tional Linguisties.
Linyong Nan, Lorenzo Jaime Flores, Yilun Zhao, Yixin Liu, Luke Benson, Weijin Zou, and Dragomir Radev. 2022. R2D2: Robust data-to-text with replacement detection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6903-6917, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Arvind Krishna Sridhar and Erik Visser. 2022. Im- proved beam search for hallucination mitigation in abstractive summarization.
Neha Srikanth and Junyi Jessy Li. 2020. Elaborative simplification: Content addition and explanation gen- eration in text simplification. In Findings.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In Pro- ceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy. Association for Computational Linguistics.
Sanja Stajner. 2021. Automatic text simplification for social good: Progress and challenges. In Findings of the Association for Computational Linguistics: ACL- IJCNPP 2021, pages 2637-2652, Online. Association for Computational Linguistics.
Elior Sulem, Omri Abend, and Ari Rappoport, 2018. Semantic structural evaluation for text simplification. In North American Chapter of the Association for Computational Linguistics.
OpenAl. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Constantin Oräsan, Richard Evans, and Ruslan Mitkov. 2018. Intelligent Text Processing to Help Readers with Autism. pages 713-740. Springer International Publishing, Cham.
Renliang Sun, Hanqi Jin, and Xiaojun Wan. 2021. Document-leve text simplification: Dataset, crite- ria and baseline. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7997-8013, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rebecca Passonneau. 2006. Measuring agreement on set-velued items (MASI) for semantic and pragmatic annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (6)EC*06), Genoa, Italy, European Language Re- sources Association (ELRA).
Renliang Sun, Zhixian Yang, and Xiaojun Wan. 2023, Exploiting summarization data to help text simplifi- cation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computa- tional Linguistics, pages 39-51, Dubrovnik, Croatia. Association for Computational Linguisties.
Atharva Phatak, David W. Savage. Robert Ohle. Jonathan Smith, and Vijay K. Mago. 2022. Medi- cal text simplification using reinforcement learning (teslea): Deep learning- based text simplification ap- proach. JMIR Medical Informatics, 10.
Jan Trienes, Jörg Schlötterer, Hans- UIrich Schildhaus, and Christin Seifert. 2022. Patient- -friinlyf clinical notes: Towards a new text simplification dataset. Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022) pages 19-27, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
Horacio Saggion, Sanja Štajner, Stefan Bott, Simon Mille, Luz Rello, and Biljana Drndarevic. 2015. Making it simplext: Implementation and evaluation of a text simplification system for spanish. ACM Trans. Access. Comput.. 6(4).

---

Laurens van den Bercken, Robert-Jan Sips, and Christoph Lofi. 2019. Evaluating neural text sim- plification in the medical domain. In The World Wide Web Conference, WWW '19, page 3286-3292, New York, NY, USA. Association for Computing Machin- ery.
Parameter
Value
Batch Size
LR 5e-5
Decay 0.01
Epochs 5
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2020. Neu- ral text generation with unlikelihood training. In International Conference Learning Representa- tions.
A 7.5e-4
Ae 2.5e-4
Table 5: Training Hyperparametens
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Hug- gingface's ss transformers: State-of-the-are natural lan- guage processing.
B Example Output
Tables 6. 7, and 8 show comparisons of outputs from the previous SOTA vs our model. We clearly observe the benefits of our methods; the writing is much simpler, and complex phrases such as "asthma exacerbation" and "emergency department presentation" have been replaced by "asthma attack" and "coming to the emergency department". Table 8 shows an instance wherein the writing is much simpler, but the model tends to retain much more information about the source and explain other con- cepts (in italics); this may come across as redundant to some evaluators, which explains the results the human evaluation portion, when compared to models like GPT-4.
Yuxiang Wu, Matt Gardner, Pontus Stenetorp, and Pradeep Dasigi. 2022. Generating data to mitigate spurious correlations natural language inference datasets. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Pupers), pages 2660-2676, Dublin, Ireland. Association for Computational Linguistics.
Wei Xu, Chris Kallison-Buuchh and Courtney Napoles. 2015. Problems in current text simplification re- search: New data can help. Transactions of the Asso- ciation for Computational Linguistics, 2983-297.
C Other Experiments
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch 2016. Optimizing sta- tistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415.
C.1 Varying Candidate Set
Kindly refer Table 10 for a comparison of per- formance when we exclude only words the input (U LT), the label (U LL), or both (U LIL).
Ziyu Yang, Santhosh Cherian, and Slobodan Vucetie. 2023. Data augmentation for radiology report simpli- fication. In Findings of the Association for Compu- national Linguistics: FACI. 2023, pages 1922-1932, Dubrovnik, Croatia. Association for Computational Linguistics.
C.2 External Knowledge Augmentation
We identify complex terms using SciSpacy (Neu- mann al., 2019), obtain their meanings from UMLS (Bodenreider, 2004), and insert these mean- ings the sentence. An example is provided Table 9, Kindly refer to Table 10 for a comparison of performance with and without external knowl- edge (EK).
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
A Implementation Details
D GPT-Eval Implementation
We train a baseline BART-XSum model (Lewis et al., 2020) Cochrane, MedEasi, and the Ra- diology Dataset. We implement the unlikelihood loss and modified decoder using the Transformers library (Wolf et al., 2020); we report the hyperpa- rameters in Table 5. We run our experiments using NVIDIA-RTI RTX 6000 GPUs.
We follow Liu et al. (2023) to implement GPT- Eval for factual consistency with GPT-4, this was found correlate best with human judgment. We use the gpt-4-0314 version, with n, topp. and temperature set to 1. We set the system role as "Your task to rate the summary on one metric." For the user input, we input the following:

---

Model
Example
Input
A total of 38 studies involving 7843 children were included. Following educational significantly
intervention delivered to children, their parents or both, there reduced risk of subsequent emergency department visits (RR 0.73, 95% CI 0.63 to 0.81, N 3008) and hospital admissions (RR 0.79, 95% CI 0.69 to 0.92, N = 4019) compared with control. There were also fewer unscheduled doctor visits (RR 0.68, 95% CI 0.57 to 0.81, N = 1009). Very few data were available for other outcomes (FEV1, PEF, rescue medication use, quality of life or symptoms) and there was no statistically significant difference between education and control. Asthma education aimed at children and their carers who present to the emergency department for acute exacerbations can result in lower risk of future emergency department presentation and hospital admission. There remains uncertainty as to the long- term effect of education on other markers of asthma morbidity such as quality of life, symptoms and lung function. It remains unclear as what type, duration and intensity educational packages are the most effective reducing acute care utilisation.
NAPSS
total of 38 studies involving 7843 children were included in this review. Asthma ed- ucation aimed children and their carers who present to the emergency department for acute exacerbations can result in lower risk of future emergency department presentation and hospital admission. There were also fewer unscheduled doctor visits. Very few data were available for textbfother outcomes (FEV1, PEF, rescue medication use, quality of life or symptoms) and there was no statistically significant difference between education and control. It remains unclear to what type, duration and intensity of educational packages are the most effective in reducing acute utilisation. There remains uncertainty as to the long-term effeet of education on other markers of asthma morbidity, symptoms and lung function.
BART- UL
This review of 38 studies involving 7843 children found that asthma education aimed children and their carers who present to the emergency department for acute exacerbations can result in lower risk of future emergency department presen- tation and hospital admission. There were also fewer unscheduled doctor visits. Very few data were available on other markers of asthma morbidity such as quality of life, symptoms and lung function. It remains unclear as to what type, duration and intensity of educational packages are the most effective in reducing acute care utilisation.
Ours
This is an overview of studies that looked at the long-term effects of providing education to children and their carers who have a flare-up of their asthma. The aim was to see if there was benefit in terms of the number of children who had to go to hospital for an asthma attack. A total of 38 studies involving 7843 children were included in this review. The results showed that if children their parents or both, they were less likely to come to the emergency department for an attack. There was also less chance that they would need to go back hospital. Very few data were available for other outcomes (e.g. quality life, symptoms and lung function) and there was no difference between education and control. However, there a need for more research in this area.
taught about asthma,
Table 6: Sample Report from the Cochrane Test Set

---

Model
Example
Input
Five trials of MSP/RESA vaccine with 217 participants were included; all five reported on safety, and two on efficacy. No severe or systemic adverse effects were reported at doses of 13 to 15 5g of each antigen (39 to 45 5g total). One small efficacy trial with 17 non-immune participants with blood-stage parasites showed no reduction or delay in parasite growth rates after artificial challenge. In the second efficacy trial in 120 children aged five to nine years in Papua New Guinca, episodes of clinical malaria were not reduced, but MSP/RESA significantly reduced parasite density only in children who had not been pretreated with antimalarial drug Sultfaaloxine-prrnnttammmee Infections with the 3D7 parasite subtype of MSP2 (the variant included in the vaccine) were reduced (RR 0.38, 95% CI 0.26 to 0.57; 719 participants) while those with the other main subtype, FC27, were not (720 participants). The MSP/RESA (Combination B) vaccine shows promise as a way to reduce the severity of malaria episodes, but the effect of the vaccine is MSP2 variant-specific. Pretreatment for malaria during a vaccine trial makes the results difficult to interpret, particularly with the relatively small sample sizes of early trials. The results show that blood-stage vaccines may play role and merit further development.
NAPSS
Five trials of MSP/RESA vaccine with 217 participants were included; all five reported on safety, and two on efficacy. No severe or systemic adverse effects were reported at doses of 13 to 15 μg of each antigen (39 to 45 μg total). One small efficacy trial with 17 non-immune participants with blood-stage parasites showed no reduction or delay in parasite growth rates after artificial challenge. In the second efficacy trial in 120 children aged five to nine years in Papua New Guinea, episodes of clinical malaria were not reduced, but the vaccine significantly reduced parasite density only in children who had not been pretreated with an antimalarial drug (sulfadoxine- pyrimethamine). Infections with the 3D7 parasite subtype (the variant included in the vaccine) were reduced (RR 0.38, 95% CI 0.26 to 0.57; 719 participants) while those with the other main subtype. FC27, were not (720 participants). The effect of the vaccine is MSP2 variant-specific Pretreatment for malaria during a vaccine trial makes the results difficult interpret, particularly with the relatively small sample sizes of early trials.
BART- UL
This review of five trials found that the MSP/RESA (Combination B) vaccine shows promise as way to reduce the severity of malaria episodes, but the effect of the vaccine is MSP2 variant-specific. The results show that blood-stage vaccines may play a role and merit further development.
Ours
The MSP/RESA (Combination B) vaccine shows promise as way to reduce the severity of malaria episodes, but the effect of the vaccine is MSP2 variant-specific. The results show that blood-stage vaccines may play a role and merit further develop- ment.
Table 7: Sample Report 2 from the Cochrane Test Set

---

Model
Example
Input
Four studies, involving 125 participants, were included. Three studies evaluated the effects of pulsed electromagnetic fields and one study, capacitive coupled electric fields. Participants with delayed union and non-union of the long bones were included, but most data related to non-union of the tibia. Although all studies were blinded randomised placebo-controlled trials, each study had limitations. ... The overall pooled effect size was small and not statistically significant (risk ratio 1.96; 95% confidence interval 0.86 to 4.48; 4 trials). There was substantial clinical and statistical heterogeneity in this pooled analysis (12 = 58%). A sensitivity analysis conducted to determine the effect of multiple follow-up time-points on the heterogeneity amongst the studies showed that the effect size remained non- significant at 24 weeks (risk ratio 1.61; 95% confidence interval 0.74 to 3.54; 3 trials), with similar heterogeneity (I2 55%) There was no reduction in pain found in two trials. More definitive conclusions on treatment effect await further well-conducted randomised controlled trials.".
NAPSS
Four studies, involving 125 participants, were included in this review. Three studies evaluated the effects of pulsed electromagnetic fields and one study, capacitive coupled electric fields. Participants with delayed union of the long bones were included, but most data related to non-union of the tibia. Although all studies were blinded randomised placebo-controlled trials, each study had limitations. The primary measure of the clinical effectiveness of electromagnetic field stimulation was the proportion of participants whose fractures had united at a fixed time point. The overall pooled effect size was small and not statistically significant (risk ratio 1.96; 95% confidence interval 0.86 to 4.48; 4 trials). There substantial clinical and statistical heterogeneity in this pooled analysis (12 = 58%) ). A sensitivity analysis conducted to determine the effect of multiple follow-up time-points on the heterogeneity amongst the studies showed that the effect size remained non-significant at 24 weeks. There was no reduction in pain found in two trials... No study reported functional outcome measures. One trial reported two minor complications resulting from treatment. More definitive conclusions on treatment effect await further well-conducted randomised controlled trials.
BART- UL
This review of four studies, involving 125 participants, found that electromagnetic field stimulation may offer some benefit in the treatment of delayed union of long bone fractures, but it is inconclusive. More definitive conclusions on treatment effect await further well-conducted randomised controlled trials.
Ours
We found four studies, with a total of 125 participants. Three of the studies used pulsed electromagnetic fields and one used a type of electric field known as a 'coupled' field. Participants with delayed union and non-union of the long bones were included, but most of the data related to non-unions of the tibia. Although all studies were blinded, this means that participants did not know whether they were in the treatment group or the control group. Each of the four studies had limitations in the way that was run and performed. The results showed that there was no reduction pain found in two of the trials. Further well-designed, well-conducted randomised controlled trials are required.
Table 8: Sample Report 3 from the Cochrane Test Set

---

Model
Example
Original
total of 38 studies involving 7843 children were included. Following educational intervention delivered to children, their parents or both... Very few data were available for other outcomes (FEVI, 1, PEF, rescue medication use, quality of life or symptoms)... There remains uncertainty as to the long-term effect of education on other markers of asthma morbidity
Text
(Cochrane)
Text with Context (Context
A total of 38 studies involving 7843 children were included. Following educational intervention (intended to prevent disease alter the course of disease in patient or population.) delivered to children, their parents both... Very few data were available for other outcomes (FEVI (the volume exhaled during the first second of a forced expiratory maneuver started from the level of total lung capacity.), PEF (A synthetic broad-spectrum fluoroquinolone antibacterial agent active against most gram-negative and gram -oan-oosiivee bacteria.), rescue medication use, quality of life or symptoms)... There remains uncertainty as to the long-term effect of education on other markers of asthma (A form of bronchial disorder with three distinct components: airway hyper- responsiveness (RESPIRATORY HYPERSENSITIVITY, airway INFLAMMATION, and intermittent AIRWAY OBSTRUCTION.) morbidity
in Red)
Table 9: Labeled Report with Context
Model
FK 4
ARI L
BSc
S 1
surprising, we thought reranking less frequently would lead the model fail find "optimal" candidate (See Table 12).
BART-XS
11.95
12.35
0.85
36.18
GPT-4
9.97
10.73
0.87
39.06
ULI
7.74
9.47
0.86
41.93
ULL
8.00
9.74
0.86
42.04
UT-LL
8.00
9.76
0.86
42.07
EK + ULI
8.82
10.82
0.85
41.18
EK + ULL
8.74
10.62
0.85
41,47
EK + ULIL
8.26
9.98
0.86
41.37
Table 10:
Performance Cochrane with External
Knowledge (EK), BSc is BERTScore, S is SARI
Human Evaluation of Text Sunmarization Systems: Factual Consistency: Does the summary have untruthful or misleading facts that are not supported by the source text? Source Text: document Summary: summary Does the summary contain factual inconsistenciess Answer:
We additionally prompt GPT to explain why labeled a summary as being factually inconsistent by adding "Why: " after the last line. A sample output is shown in Table 11.
Results by Varying
We reduce the frequency with which we rerank beams, and find that this reduces inference time without large tradeoffs in performance. This was

---

Model
Example
Input
Forty-nine randomised trials involving 3639 participants were included. All trials were conducted and published in China. Thirty-eight different herbal medicines were tested in these trials, including four single herbs (extracts from a single herb), eight traditional Chinese patent medicines, and 26 self concocted Chinese herbal compound prescriptions. The trials reported on global symptom improvement (including improve- ment in numbness pain) and changes in nerve conduction velocity. The positive results described from the 49 studies of low quality are of questionable significance. There was inadequate reporting on adverse events in the included trials. Eighteen trials found no adverse events. Two trials reported adverse events: adverse events occurred in the control group in one trial, and in the other was unclear in which group the adverse events occurred. 29 trials did not mention whether they monitored adverse events. Conclusions cannot be drawn from this review about the safety of herbal medicines, due to inadequate reporting. Most of the trials were of very low methodological quality and therefore the interpretation of any positive findings for the efficacy of the included Chinese herbal medicines for treating diabetic peripheral neuropathy should be made with caution. Based on this systematic review, there is no evidence to support the objective effectiveness and safety of Chinese herbal medicines for diabetic peripheral neuropathy. No well-designed, randomised, placebo controlled trial with objective outcome measures has been conducted.
Model Summary
Output: Forty-nine randomised trials involving 3639 patients were included in this re- view. Thirty-eight Chinese herbal medicines were tested in these trials, including four single herbs (extracts from a single herb), eight traditional Chinese patent medicines, and 26 self-treatments of 26 Chinese herbal medicine prescriptions. All trials were conducted and published in China. There was no well well-designed, randomised, placebo controlled trial with objective outcome measures undertaken. Most of the trials were of very low quality and so the use of any of the included herbal medicines for treatment of neuropathy should be made with caution.
Is there inconsis- tency? Why?
Yes. The summary mentions "26 self-treatments of 26 Chinese herbal medicine prescriptions," which is slight deviation from the source text which states "26 self concocted Chinese herbal compound preseriptions." Additionally, the summary does not mention the questionable significance of the positive results, adverse events reporting, and the lack of evidence support the objective effectiveness and safety of Chinese herbal medicines for diabetic peripheral neuropathy.
Table 11:
GPT-Eval rates the summary being factually inconsistent, even though the summary adequately captures the overall message of the input

---

k
t
FK J
BSc 1
SR 1
RL |
5
20.24
9.82
0.867
40.74
42.60
10
19.41
9.90
0.868
40.75
42.79
15
19.40
10.00
0.867
40.52
42.81
20
18.95
9.96
0.867
40.61
42.80
Table 12:
Performance by & on Cochrane, t: Mean Inference Time (seconds), BSc: BERTScore, SARI


Reviews:
[{"review_id": "T1kUXNDAAk", "reviewer": "Reviewer_wDdq", "summary": "", "questions": "1. Why was BART selected?\n\n2. Will the internal dataset be released publicly?\n\n3. Are all three datasets designed for text simplification? or are they created for summarization?\n\n4. How were the annotators selected for human evaluation? Are they non-medical experts?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper; we are glad you found our methods straightforward and useful towards laypeople. We appreciate the insightful comments about the quality of our human evaluation and the factual consistency, which prompted us to further scrutinize the outputs qualitatively of previous SOTA. \n\nWe provide some of our findings below.\n\n\n> 1. The experimental results cannot support the effectiveness of the proposed methods. The proposed method has consistently poorer performance on the factual consistency (BERTScore and GPT-Eval) in Table 1.\n\n* We note that on multiple datasets, our models achieved the best factual consistency, and had comparable performance on others, showing an overall strong results. Specifically, our decoder variant yielded the highest BERTScore on the Cochrane dataset. On the Radiology and MedEasi datasets, our performance was either comparable or differed by a small margin of at most 0.011 points from competing methods. With respect to GPT-Eval, our UL variant actually tied for best performance on the Radiology dataset with BART-XSum. For the MedEasi dataset, our methods outperform prior work except for BART-UL. While we acknowledge room for improvement on the Cochrane dataset, we would like to highlight the overall strong and competitive performance of our methods across multiple metrics and datasets.\n* We'd like to contextualize our BERTScore and GPT-Eval results by pointing out that while the metrics may suggest a lower score, our analysis reveals that the qualitative nature of the factuality errors is often minor, such as misspellings or paraphrased nouns. These minor discrepancies disproportionately impact automated metrics, as evidenced by existing research (e.g., Guo et al., 2023 – https://arxiv.org/abs/2305.14341). Importantly, our approach offers advantages in simplicity and readability, as detailed in Appendix B.\n* Additionally, we’d like to note that higher factuality scores in previous work are often achieved through extractive methods that copy directly from the source. This inherently boosts factuality but does so at the expense of other desirable traits like readability and originality. \nWe added additional analysis that further illustrates this point. In the table below, we observe that previous work has a significantly higher proportion of 4-grams copied from the source.\n\n| Model | | % of 4-grams Copied from Source |\n| ------- | ----------------- | ---------------------------- | \n| SOTA | BART XSum | 52.88% |\n|           | BART-UL (Devaraj et al., 2021) | 39.30% |\n|           | NAPSS (Lu et al., 2023) | 51.77% |\n| Our Work | UL | 15.73% |\n|                 | Decoder | 9.80% |\n\n> 2. The proposed method has a consistently poorer RougeL score in Table 1, which means at the lexicon level, the method is not able to produce simplified text more similar to the gold standard simplified text.\n* While RougeL can gauge overall quality, there are many valid ways of simplifying text which RougeL may not capture as it relies only on lexical overlaps (please see Guo et al., 2023 https://arxiv.org/abs/2305.14341); hence we follow previous work which uses FK and ARI scores as the main metrics for simplicity.\n\n> 3. The proposed method has poorer readability performance on MedEasi (rather than \"better on all three datasets\" as claimed in the abstract). As the author also pointed out, the human evaluation of readability has poor inter-annotator agreement.\n* We find that NAPSS achieves better FK and ARI scores because it produces shorter phrases which are either incomplete or largely lack context, which artificially lowers FK and ARI (33.66% of outputs had 5 words or less, and median length was 7 words; phrases it generated include “Carcinoma develops”, “Signs should be looked at.”, and “Breaking Professor admits give”). In comparison, our model produces full sentences which improve simplicity (e.g. our UL + Decoder model achieves better FK and ARI than all previous methods except NAPSS) while maintaining fluency. Nonetheless, we will revise the phrase “on all three datasets” in the final draft to more accurately represent the results from the metrics.\n* We calculated Krippendorff's Alpha to assess agreement scores, aligning our methodology with Goyal et al., 2023 (https://arxiv.org/pdf/2209.12356.pdf). The table below shows these results. Our findings reveal that the range of agreement scores do not vary significantly from theirs (0.05 to 0.18) on a similar task of choosing a preferred summary; this could reflect the subjective nature of human preference for this task, as simplicity and readability varies based on one’s technical background and style preferences. We recognize that an in-depth analysis of the complexities and subjectivity of text simplification would be an important contribution, but we believe this is outside the scope of our current work. We will add a discussion about this in the future work.\n\n| Model | Krippendorff’s Alpha | \n| ------- | ------------------------ |\n| GPT | 0.199 |\n| NAPSS | -0.105 |\n| UL | 0.0155 |\n| Decoder | 0.245 |\n\n> 4. Why was BART selected?\n* The previous state-of-the-art methods are implemented on top of BART as the base model. Therefore, to enable direct and fair comparison, we also use BART as the base model. This ensures that we control for other factors and directly evaluate the impact of our proposed unlikelihood loss and decoding approaches. \n\n> 5. Will the internal dataset be released publicly?\n* There are some logistics involved in releasing the data and we are currently looking into this. The dataset is extracted from real world clinical reports in a large academic-cooperated hospital and the simplified reports are written by professional resident doctors.\n\n> 6. Are all three datasets designed for text simplification? or are they created for summarization?\n* Yes, all three datasets are created for text simplification.\n\n> 7. How were the annotators selected for human evaluation? Are they non-medical experts?\n* The annotators selected are non-medical experts, and we deliberately chose non-medical university students, to simulate the target population (i.e. laypeople) of such simplification models\n\n> 8. A figure presenting the method will be helpful.\n* We have drawn a flow chart/ pseudo code table to help better present the logic of our method, which we will include in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper focuses on text simplification which can be useful in the medical domain, as discussed by the authors, for non-experts to understand expert domain knowledge in layman's languages.\n\nThe proposed method is straightforward to understand.", "reasons_to_reject": "The experimental results cannot support the effectiveness of the proposed methods.\n\n1. The proposed method has consistently poorer performance on the factual consistency (BERTScore and GPT-Eval) in Table 1. \n\n2. The proposed method has a consistently poorer RougeL score in Table 1, which means at the lexicon level, the method is not able to produce simplified text more similar to the gold standard simplified text.\n\n3. The proposed method has poorer readability performance on MedEasi (rather than \"better on all three datasets\" as claimed in the abstract). As the author also pointed out, the human evaluation of readability has poor inter-annotator agreement.", "questions_to_authors": "1. Why was BART selected?\n\n2. Will the internal dataset be released publicly?\n\n3. Are all three datasets designed for text simplification? or are they created for summarization?\n\n4. How were the annotators selected for human evaluation? Are they non-medical experts?", "paper_topic_and_main_contributions": "To simplify medical texts, the paper uses Unlikelihood Loss (UL) and Reranked Beam Search Decoding. The aim is to improve text simplification in the medical domain on readability, factual consistency, and simplicity. The proposed method was evaluated on Cochrane, MedEasi, and an internal radiology reports dataset. The proposed method looks simple (as claimed in line 101) but the results do not support the effectiveness of the method.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A figure presenting the method will be helpful."}, "extra_text": {}}, {"review_id": "uuCfEZrbDP", "reviewer": "Reviewer_jGDu", "summary": "", "questions": "\n1. Is it feasible to apply the readability score for each token of the generated sequence at the training and decoding stages?\n2. Is there a minimum length of sentence limitation for applying a readability score? If not what kind of tools did you use? \n3. In the definition ULc, how the set of hallucinated words e is obtained?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking time to review our paper, we are glad you found our method to be intuitive and effective. We will be sure to clarify the applicability of the Flesch-Kincaid score in the final draft, and provide more details as to its implementation.\n\n> 1. If I don't miss it, there is no information about the available tools to compute Flesch-Kincaid (FK) readability score. There is no explanation to prove the applicability to use FK score for each token as it seems not intuitive to measure the readability of incomplete sentences.\n* Following previous work, we used the EASSE package (Alva-Manchego et al., 2019 – https://aclanthology.org/D19-3009/) to compute FK and provided the implementation in the code repository! The formula takes into account both sentence and word length – thereby making it applicable to both sentences as well as phrases and tokens (i.e. score = 0.39*(total words/ total sentences) + 11.8*(total syllables/ total words) - 15.59). Hence, phrases and tokens with longer words (i.e. more syllables) are deemed more complex. For example, a token with 3 syllables gets a score of 0.39*(1/1) + 11.8*(3/1) - 15.59 = 20.2. \n\n> 2. Is it feasible to apply the readability score for each token of the generated sequence at the training and decoding stages?\n* It is indeed feasible, as FK and ARI both take sequences of any length.\n\n> 3. Is there a minimum length of sentence limitation for applying a readability score? If not what kind of tools did you use?\n* We do not use a minimum sentence length to compute FK, and to the best of our knowledge, there’s no minimum required.\n\n> 4. In the definition ULc, how the set of hallucinated words e is obtained?\n* We define e as the set of entities which are present in the model-generated summary, but not in the source text. Entities are identified using the Spacy NER model (Honnibal and Montani, 2017), applied on the inputs and decoded model summaries. We will clarify this in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, the paper is well-written and there seem no major issues.\n1. The unlikelihood loss based on the readability index is original and intuitive.\n2. They compared their method with strong baselines and show the effectiveness of their approach\n3. Error analysis with human evaluation adds extra points.\n", "reasons_to_reject": "If I don't miss it, there is no information about the available tools to compute Flesch-Kincaid (FK) readability score.\nThere is no explanation to prove the applicability to use FK score for each token as it seems not intuitive to measure the readability of incomplete sentences.\n\n\n\n", "questions_to_authors": "\n1. Is it feasible to apply the readability score for each token of the generated sequence at the training and decoding stages?\n2. Is there a minimum length of sentence limitation for applying a readability score? If not what kind of tools did you use? \n3. In the definition ULc, how the set of hallucinated words e is obtained?", "paper_topic_and_main_contributions": "They propose a new form of unlikelihood loss based on the readability index to improve medical text simplification. \nThey also propose a decoding strategy to optimize for readability in text simplification. They evaluated the method on three datasets in terms of readability and factual consistency comparing previous state-of-the-arts.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UTneGaCJLm", "reviewer": "Reviewer_8a31", "summary": "", "questions": "Please answer the questions  stated  above.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and your thoughtful review; we are glad that you found our methods to be simple but effective, and that it hopefully opens up more discussion into such applications in the medical domain.\n\nThank you for probing deeper into why the methods we propose fit the domain – this is a key point that we will further clarify in the final draft. We also provide additional human evaluation study and share some new findings below:\n\n> 1. The proposed approaches for improving medical text simplification are relatively simple and do not introduce significant innovations or advancements in the field. The use of unlikelihood training and modified decoding methods have been explored in previous work in other domains. The paper does not provide a strong justification for why these approaches are specifically suitable for medical text simplification or how they address the unique challenges in this domain. Can you provide some insights into this.\n* Our approaches address a significant gap in existing methods for medical text simplification: the issue of readability. Prior methods tend to copy from input text and often focus on summarizing and shortening the input. Therefore, while the output is shorter, it can still include technical jargon, resulting in less accessible and readable output. Our techniques, such as UL optimization and reranked decoders, are specifically designed to minimize such direct copying. We added additional analysis that further illustrates this point. In the table below, we observe that previous work has a significantly higher proportion of 4-grams copied from the source. We will include this discussion in the revised manuscript.\n\n| Model | | % of 4-grams Copied from Source |\n| ------- | ----------------- | ---------------------------- | \n| SOTA | BART XSum | 52.88% |\n|           | BART-UL (Devaraj et al., 2021) | 39.30% |\n|           | NAPSS (Lu et al., 2023) | 51.77% |\n| Our Work | UL | 15.73% |\n|                 | Decoder | 9.80% |\n\n\n> 2. While the human evaluation results show improvements in readability over a fine-tuned BART-XSum model, the paper does not provide a comprehensive analysis of the limitations and potential biases of the evaluation process. The low inter-annotator agreement on readability and the subjective nature of the task raise concerns about the reliability and generalizability of the evaluation results, which is still discussed in limitations. Without a robust evaluation methodology, it is difficult to determine the true effectiveness of the proposed approaches. If it was previously reported as low, why not devise a different strategy altogether?\n* We compute Krippendorff’s Alpha to align our method with the work of (Goyal et al., 2023, https://arxiv.org/pdf/2209.12356.pdf) – and find that our range of agreement scores do not vary significantly from theirs (0.05 to 0.18) on a similar task of choosing a preferred summary. This reflects the subjective nature of human preference, as simplicity and readability varies based on one’s technical background and style preferences. While such variability is hard to avoid, the average proportions suggest that overall, our methods (UL and Decoder) significantly improved upon previous SOTA (NAPSS).\n\n| Model | Krippendorff’s Alpha | \n| ------- | ------------------------ |\n| GPT | 0.199 |\n| NAPSS | -0.105 |\n| UL | 0.0155 |\n| Decoder | 0.245 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes two simple but effective approaches for improving medical text simplification, one during the training phase and the other during decoding. These approaches incorporate readability index and factuality evaluation to enhance the generation of simplified and accurate medical text. \n\nThe human evaluation results demonstrate that the proposed approaches significantly improve readability over a fine-tuned BART-XSum model, outperforming the previous state-of-the-art method. The proposed UL and decoder achieve readability improvements 43% and 27% of the time, respectively, compared to only 3% for the previous SOTA method.\n\nOverall, the paper presents novel approaches for improving medical text simplification, demonstrates their effectiveness through human evaluation, and provides valuable insights for future research directions.", "reasons_to_reject": "I recommend rejecting this paper for the following reasons:\n\n1. The proposed approaches for improving medical text simplification are relatively simple and do not introduce significant innovations or advancements in the field. The use of unlikelihood training and modified decoding methods have been explored in previous work in other domains. The paper does not provide a strong justification for why these approaches are specifically suitable for medical text simplification or how they address the unique challenges in this domain. Can you provide some insights into this.\n\n2. While the human evaluation results show improvements in readability over a fine-tuned BART-XSum model, the paper does not provide a comprehensive analysis of the limitations and potential biases of the evaluation process. The low inter-annotator agreement on readability and the subjective nature of the task raise concerns about the reliability and generalizability of the evaluation results, which is still discussed in limitations. Without a robust evaluation methodology, it is difficult to determine the true effectiveness of the proposed approaches. If it was previously reported as low, why not devise a different strategy altogether?", "questions_to_authors": "Please answer the questions  stated  above.", "paper_topic_and_main_contributions": "This paper proposes methods to improve the readability of text simplification in the medical domain. The proposed methods show improved performance on readability metrics while maintaining similar performance in terms of factual consistency and overall simplification. However, the issue of hallucinations in the output remains a challenge, and further exploration is suggested to address this issue and improve the robustness of medical text summarization.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes two methods to improve simplification of medical texts: unlikelihood loss during training and reranked beam search during decoding. Both techniques intend to  to optimise for readability by relying on flesch-kincaid grade level, which is a metric based on average sentence and word lengths.

Reviewers recognise that the proposed method is straightforward to understand. In addition, the authors compare their approach against strong baselines, and show improvements mostly based on automatic metrics (flesch-kincaid included), as well as with some human evaluation.

One of the main concerns is related to the suitability of flesch-kincaid, traditionally a document-level metric, for assessing the readability of words/phrases and partial sentences. Previous work has pointed out that it is easy to fool this metric (https://aclanthology.org/2021.gem-1.1/) and that it does not agree with judgements of simplicity at the sentence level (https://aclanthology.org/2021.cl-4.28/). However, it is difficult to disregard the results obtained by the authors' experiments, where their proposed model does improve over the BART-XSUM baseline, even using human judgements (despite low inter-annotator agreements, which is common in subjective tasks). In their rebuttal, the authors also point out that their method allows for less conservative outputs, which is an interesting finding that they are recommended to include in the paper.
Decision: Accept-Findings