Multilingual Simplification of Medical Texts
Authors: Sebastian Antony Joseph
Kathryn Kazanas
Keziah Reina
Vishnesh J Ramanathan
Wei Xu
Byron C Wallace
Junyi Jessy Li
Venue: EMNLP 2023 Main (2023)
Keywords: Simplification
Medical Simplification
Multilingual

Abstract: Automated text simplification aims to produce simple versions of complex texts. This task is especially useful in the medical domain, where the latest medical findings are typically communicated via complex and technical articles. This creates barriers for laypeople seeking  access to up-to-date medical findings, consequently impeding progress on health literacy. Most existing work on medical text simplification has focused on monolingual settings, with the result that such evidence would be available only in just one language (most often, English). This work addresses this limitation via multilingual simplification, i.e., directly simplifying complex texts into simplified texts in multiple languages. We introduce MultiCochrane, the first sentence-aligned multilingual text simplification dataset for the medical domain in four languages: English, Spanish, French, and Farsi. We evaluate fine-tuned and zero-shot models across these languages with extensive human assessments and analyses. Although models can generate viable simplified texts, we identify several outstanding challenges that this dataset might be used to address.

Multilingual Simplification of Medical Texts
Sebastian Joseph'
Kathryn Kazanas' Keziah Reina'
Vishnesh J. Ramanathan"
Wei Xu² Byron C. Wallace Junyi Jessy Li'
The University of Texas at Austin Georgia Institute of Technology, Northeastern University
{sebaj, kreina, jessy]@utexas, edu, katkazanaségmail.com com
(vishneshe, wei. xuêcc }gatech. edu, bawallace@nor www.esstrnrtteettee...me
Abstract
MultiCochrane
Complex
Automated text simplification aims to produce simple versions complex texts. This task is especially useful in the medical domain, where the latest medical findings are typically com- municated via complex, technical articles. This creates barriers for laypeople seeking access to up-to-date medical findings, consequently impeding progress health literacy. Most existing work on medical text simplification has focused on monolingual settings, with the result that such evidence would be available only in just one language (most often, English). This work addresses this limitation via multi- lingual simplification, i.e., directly simplifying complex texts into simplified texts in multiple languages. We introduce MULTICOCHRANCE the first sentence sentence-aligned multilingual text sim- plification dataset for the medical domain in four languages: English, Spanish, French, and Farsi. We evaluate fine-tuned and zero-shot models across these languages with extensive human assessments and analyses. Although models can generate viable simplified texts, we identify several outstanding challenges that this dataset might be used to address.
Predinical stadios haw suggealed that RAC may hase benafoial affects isohaomic strcke patinis and mose nokk tochoemi' stroni.
Simple
English: Stuties have suggested that RIC may have beneficial effects preventing and freeting ischaencc stroka.
Spanish:
puede efectos bereficlasas prevención y tratamiento accidente cenebrovascular lequémico.
French: Des études ant suggéré que le CID pourrail avoir des effcts háráfiquas sur la pravantion et trailament de TAV ischémiqun.
Farsl: th dqidy RIC توكب dui of uاا قنت مfا.
evelustion eyetem outputs
may have effects in ischaemic stroke patients, those at risk of stroke.
English: Intorventions have suggested
Spantsh: Las Intervenclones pueden efectives pera los pacientas que se accidiente cerebrovascufar isquérico los encuentran en rissgo accidente cerebrovascular isquéricc..
The interventions can be pelients that accident thamselvas ischamic stroke and those that find themselves at the schemmic stroke.
Gloss:
Figure 1: Examples of our dataset and system outputs for multilingual medical text simplification. The dataset has simplifications available in four languages (English, Spanish, French, Farsi), and system outputs lyzed for factual errors (red), fluency errors (blue), and simplicity (green) among other criteria.
1 Introduction
et al., 2020; Cardon and Grabar, 2020: Devaraj et al., 2021; Guo et al., 2022; Trienes al., 2022; Basu al., 2023), explored reinforcement learn- ing (Phatak et al., 2022), and evaluated zero-shot performance via prompting (August et al., 2022).
Important findings in medicine are typically pre- sented in technical, jargon-laden language in jour- nal articles reviews, which is difficult for laypeo- ple understand. This impedes transparent and fair access critical medical information and ul- timately hinders health literacy, which is "one of the most promising and cost-effective approaches to overcome the Non-Communicable Disease chal- lenge" (Liu et al., 2020).
However, this work has so far exclusively con- sidered monolingual text simplificatinn. Conse- quently, we do not know how well models perform when the simplified text is not in the same language as the original, complex text. This limits the (po- tential) availability of simplified information to a few high-resource languages languages-especialle especially for the medical domain and leads to equity issues such that individuals who do not speak English will still face a high barrier to information access.' Work
Text simplification models which automatically transform complex texts into simpler versions un- derstandable by lay readers (Siddharthan, 2014; Alva-Manchego et al., 2020) have emerged as promising means of providing wider access to pub- lished medical evidence. Recent work on simplifi- cation has fine-tuned large pre-trained models (Van
While machine translation, to a certain extent, may be able

---

in this direction is impeded by lack of data. The largest resources in text simplification are not in the medical domain, and parallel corpora for med- ical simplification only exist in single languages, namely English (Devaraj et al., 2021; Guo et al., 2022; Basu et al., 2023), French (Grabar and Car- don, 2018: Cardon and Grabar, 2020), and Ger- man (Trienes et al., 2022).
where they tend to introduce factuality errors, GPT- (davinci; zero-shot) yields oulputs that are com- paratively factually accurate, but which tend to be extractive and so not adequately simplified. Out- puts from Flan-T5 (Base; zero-shot) significantly degrade in quality when targeting languages other than English, producing many factual and fluency errors. We also analyze the approach of translating English simplifications to other languages, which in many instances is able bypass these issues.
This paper advances the state of mulrilingual medical simplification, in which complex texts are to be directly simplified into target languages. Our contributions are as follows. We introduce MULTICOCHRANE (Figure 1), the first parallel dataset for medical text simplification across multi- ple languages: English, Spanish, French, and Farsi. MULTICOCHRANE contains aligned sentence pairs Systematic
We publicly release MUL.TICCCHRANE, model outputs, and all human judgments collected for this work chttps://gi thub. com/SebaJoe/ Mul ti tiCochrane), hoping motivate future work on multilingual medical text simplification to ad- vance model performance across languages.
sourced from the Cochrane Library of Reviews, 2 which is a library of meta-analyses of treatment effectiveness.
Related Work
These review articles include both a techni- cal abstract and a plain-language summary (PLS), from which we derive the two subsets of MUL.TI- COCHRANE: (1) MC-CLEAN, 101 technical ab- stracts with expert-annotated manual alignments first derived in English, then semi-automatically aligned to other languages (partially verified by bilingual speakers). MC-CLEAN contains ~55k sen- tence pairs across all 4 languages. (2) MC-NOISY, a larger but noisier subset created with an automatic sentence alignment model that was trained on MC- CLEAN. MC-NOISY is sourced from around 7.8K medical reviews, with ~ 100K sentence pairs across languages.
The largest resources used to train automatic sim- plification models are two general domain corpora: the Wikipedia-
Wikipedia corpus (Zhu al., 2010; Woodsend and Lapata, 201 I; Coster and Kauchak, 201 10 Jiang et al., 2020), and the Newsela corpus (Xu et al., 2015; Jiang et al., 2020). This paper focuses on simplification in the medical do- main, which is important if we are to bridge the information gap exemplified by low medical liter- acy levels worldwide (Kickbusch et al., 2013).
Medical Simplification Datasets. While there have recently been increasing efforts to create par- allel corpora for medical text simplification in En- glish (Van den Bercken et al., 2019; Cao et al., 2020; Devaraj et al., 2021; Guo et al., 2022; Basu et al., 2023), data in other languages remain scarce. Grabar and Cardon (2018) constructed the CLEAR dataset in French, part of which is derived from 13 Cochrane articles; Trienes et al. (2022) intro- duced dataset on German consisting of clinical notes. Other prior work in non- non-English medical text simplification has focused on lexical simplification (Abrahamsson et al., 2014; Klochn et al., 2018; A Ifno et al., 2020), where the primary focus was on substituting synonyms and semantically related terms. In terms of data sourcing, our work is most similar to Devaraj et al. (2021) (which is in En- glish only). However, their work derived roughly- aligned paragraphs consisting of specific sections (methods and conclusions only) of the Cochrane ab- stracts and PLS; by contrast, we derive manual and automatically aligned sentences for full abstracts.
MULIICOCHRAANE enables systematic evalua- tions of medical text simplification models. Here we evaluate range of large pre-trained language models in both zero-shot and fine-tum fine-tuned settings on the task of text simplification across four languages. In addition to automatic evaluations, we report hu- man assessments covering simplicity, fluency and factuality (Devaraj et al., 2022). We also report the correlation between automatic metrics and these human assessments, which we find to be mostly weak. Our results show that while pre-trained mod- els are effective at simplification in English, their abilities degrade significantly on other languages,
to overcome this limitation, not ideal due to variability translation quality across languages (Ruder al.. 2021) and other issues such as cost and interpretabilityy as discussed Vu et al. (2022). This paper evaluates a simplify translate pipeline.
Phttps: httttp:://wwwcooeet www.w..ccchraaelibrrrycccmmm

---

Multilingual Simplification Datasets. Multilin- gual datasets have been introduced for the task of summarization. The MLSUM dataset (Scialom et al., 2020) is a large summarization corpus con- taining article/summary pairs five languages, sourced from newspaper articles. General-domain non-English datasets for simplification also ex- ist extensively surveyed by Ryan et al. (2023). Notably, MUSS (Martin al., 2022) used Com- mon Crawl mine millions of sequence pairs in English, Spanish, and French. The signifi- cant difference between MUSS and our dataset is MUSS's lack of lingull alignment, i.c., alignment of sequence pairs from one language to another. Agrawal and Carpuat (2019) derived nois- ily aligned English-Spanish data from the Newsela corpus (Xu et al., 2015). However, to date, there is no aligned dataset for text simplification where one source sentence is paired with target sentences in multiple languages simultaneously as in our work.
Dataset
en->en
en->es
en->fr ffr
en en->fa
MC-NOISY
60,058
58,235
34,604
30.822
filtering
29,703
29,025
17,033
15,389
MC-CLEAN
1,632
1,602
1,111
646
training sel
1,136
1,084
798
424
+ filtering
329
326
227
125
Table I: Statistics of MULTICOCHRANE corpus for En-
glish (en), Spanish (es), French (fr), and Farsi (fa). MC- NOISY is only used training. Filtering (Section 4)
selects the most similar pairs of aligned sentences the cost of higher extractiveness.
native or advanced mastery over one or more of the other languages covered in this work. These annotators do not have medical backgrounds, but they do have strong background in research and in evaluating technical texts. We consider this to be sufficient enough to complete the tasks described in this paper.
3.1.1 English Annotation
3 The MULTICOCHRANE Corpus
Annotators were provided with abstract-PLSS pairs and were tasked with aligning sentences in the ab- stract with sentences in PLS according to their simi- larity in content. This is a challenging task: annota- tors had to be able to understand complex medical texts, and align sentences by repeatedly search- ing through text on both the simple and complex sides. To assist with their annotation, we built the ANNO-VIEWER tool inspired by Jiang et al. (2020) to enable annotators to efficiently align sentences (see screenshots in Appendix J). This annotation tool also has functionality allowing the annotator to use existing semantic similarity measures (listed in Section 3.2) to help find sentence alignments faster. When a sentence selected for alignment, the tool can sort the other document (either abstract PLS) by which sentences are most similar to the selected sentence. After aggregating all alignments for a document into a bipartite graph for analysis, we processed the data into individual sentence pairs (see Appendix D for more details).
We source MULTiCOCHRANE from publicly avail- able technical abstracts and plain-language sum- maries of Cochrane systematic reviews; these are comparable texts, though not parallel (Devaraj et al., 2021). These abstracts and PLS exist in several languages (detailed in Sections 3.2.2 and 3.2), aligned almost one-to-one with their English counterparts. This allows annotations and align- ments to be easily adapted create a multilin- gual dataset. In total, we use 7,755 abstract-PL.S pairs from the Cochrane Library. We first create sentence-alignes aligned MC-CLEAN (Section 3.1) of 101 abstract-PLS pairs using a mixture of manual align- ments for English and semi-automatic alignments with partial verification for other languages. The rest of the abstracts were used for the ereation of the lcage noisy MC-NOISY (Section 3.2) by automatic alignments and filtering. The total num- ber of sentences in all subsets are shown Table 1.
3.1 Clean Alignments: MC-CLEAN
Due the high cognitive load imposed by align- ing medical documents, we first selected set of 25 documents aligned by at least 2 annotators (15 of which aligned by at least 3 annotators). Once we were confident of their inter-annotator agreement (see Titer-annotaal Agreement" below), the rest of the 75 documents were then single-annotated.
For MC-CLEAN, we first create manually aligned subset of 101 abstracts for English. We then auto- matically align for other languages by exploiting Cochrane's natural 1-1 sentence alignment from English to other languages; this alignment was ver- ified by annotators on sizable sample of the data. Our annotation team consists of 5 trained linguis- ties students with native English proficiency and
'We release the ANNO-VIEWER annotation tool publicly together with our datasets.

---

Aligned Complex Sentences
1,325
in simple texts. In PLS, an unaligned sentence indi- cates added information not present in the complex version. We observe that an overwhelming major- ity of added information elaborates on concepts or defines terms (Srikanth and Li, 2021). The av- crage elaboration ratio (the ratio of unaligned to total sentences in PLS) is less than half of the average deletion ratio (ratio of unaligned to total sentences in technical abstract), indicating that simplified texts tend to be mostly concise with this core information. The average token compression ratio (ratio of simple to complex tokens) show the presence of longer simplified sentences.
Aligned Simple Sentences
1.165
Unaligned Complex Sentences (Deletion)
1,966
Unaligned Simple Sentences (Elaboration)
591
Average Delction Ratio
60.1%
Average Elaboration Ratio
28.2%
Token Compression (somplex-esimple)
1.14
Table 2: Statistics on unaligned and aligned sentences for the English portion of MC-CLEAN.
Inter-annotator Agreement. We used the afore- mentioned 25 articles calculate inter-annotator agreement based on the Fl score, following prior research on word and sentence alignment (Jiang et al., 2020; Lan et al., 2021).* For each anno- tator, their alignments were evaluated against the majority vote of aligmments from other annotators.
3.1.2 Multilingual Data
The Cochrane Library provides abstracts and PLS in multiple languages via translations that have been produced through a combination of volunteer and professional work, as well as human-verifie machine translation (Ried, 2023). To create the multilingual portion of our dataset, i.e., pairs of semantically equivalent (source, target) sentences where source English abstract, target E PLS in (en, es, fr, .../, use the English PLS as bridge to align the English abstract with PLS in other lan- guages. The PLS of non-English languages mostly correspond 1-1 sentence-wise to the English PLS. We use sentence aligner that combines LASER multilingual sentence embeddings (Artetxe and Schwenk, 2019) with dynamic programming- based alignment algorithm (Thompson and Koehn, 2019) to align sentences in the English versions to those in other languages. We did this for the entire set of 7,755 abstract/PLS pairs. Multilin- gual sentence pairs in MC-CLEAN consist of these alignments that belong to the 101 articles manually aligned for English. The other 7,654 articles were used to create MC-NOISY (Section 3.2).
Overall, the macro-average Fl score across an- notators was 0.89, indicating high agreement for the alignment task. Non-overlapping alignments are rare; a major source of disagreement between annotators was related to multiple alignments made to the same simple sentence (i.c., multi-alignnn alignments) that were either captured by some annotators but ex- cluded by others. Since the content in the abstracts is highly technical and in some cases contrasted significantly in style and vocabulary with the plain- language summary, annotators had difficulties in determining whether some sentences conveyed sim- ilar meaning. Nevertheless, annotators were in gen- eral conservative, only aligning sentences when they were confident in the alignments. Therefore, there is low prevalence of incorrect alignments. The overall alignments are gigh-precision. Annota- tors typically used VINO-VIEWER'S is sentence simi- larity sorting function to verify their alignments or to identify additional multi-alignments.
Dataset Summary. Table 2 presents additional statistics on the aligned and unaligned sentences in both complex and simple texts in the English por- tion of MC-NOISY. For complex texts (i.c., techni- cal abstracts), unaligned sentence indicates that its information is not present (i.e., deleted) in the simplified texts (i.e., plain-language summaries). On average, just over 60% of sentences com- plex texts are deleted, indicating that most of the core information from complex texts are retained
Human Verification. To verify that the multi- lingual sentence alignments across different lan- guages of the PLS are valid, we asked of our bilingual annotators to evaluate a random sample of 400 sentence pairs to verify that each alignment is direct translation. Each annotator was highly proficient in both the languages of the alignments being verified. Annotators found zero instances of misalignment. We also analyzed the number of sentences of the English abstract/PLS and their mul- tilingual counterparts find any instances where is a huge mismatch. We did not find any such instances. The data did include instances where
'Aligament requires N (sentences) by Ad (sentences) com- parisons, while only a small amount of which being positive (i.c., aligned). Thus Fl is more commonly used than correla- tion measurements for calculating agreement for this task.

---

one or two sentences in the article may have been split or combined in their multilingual versions, The DP-based alignment algorithm that was used took care of these exceptions. These assessments, combined with information about the Cochrane Li- brary's methodology in making these translations, instills high confidence that the derived multilin- gual sentence alignments are valid.
bestbttionn Aprreaeed Arti Language
200
EC
CC
24
Languages
DK2) but Arties hyuage
6033
8533
2593
Dataset Summary. While Cochrane provides large amount of diverse multilingual data, every article not mapped to every available language. Consequently, the number of articles available for each language varies. The distribution of these articles across language is shown in Figure 2.
Languages
Figure 2: For English abstracts included in MC-CLEAN and MC-NOISY, the number of corresponding PL. across languages. From left to right: English (en), Span- ish (es), French (fr), Farsi (fa), Chinese (zh), Japanese (ja), Portuguese (pt), Korean (ko). Thai (th), German (de), Russian (ru), Malay (ms), and Croatian (hr).
For this work, we selected the top three most fre- quent non-English languages - Spanish, French, and Farsi - to use for training and evaluation in Section 5, We report the resulting number of paired sentences in each language in Table 1.
Filtering
Although the CRF model handles unaligned sen- tences by default, the resulting data remains noisy. We therefore adapt the method described in Kim et al. (2021), initially used in the context of sentence-splitinng, to further filter misalignments in MC-NOISY. Ultimately, this method was ap- plied to the entire training set, which includes the entirety of MC-NOISY as well as the portion of MC-CLEAN not used for testing or validation. This method can be described follows: For each sen- tence pairing in the training set, we derive the set of lemmatized tokens in the complex and simple sentence denoted by Le and L, respectively. If the proportion of the overlapping tokens from the sim- plified sentence exceeds a threshold r, is included in the filtered dataset: r = ICe n Cls/
3.2 Fully Automated Alignments: MC-NOISY
While MC-CLEAN provides clean and accurate alignments across multiple languages, human align- ment of medical documents is challenging to scale. Fortunately, MC-CLEAN enables us to accurately evaluate a variety of automatic sentence alignment methods; a good automatic alignment method will help create stlver-standard alignments expanding over the entirety of the Cochrane dataset. This is es- pecially important for multilingual medical simpli- fication, where the number of annotated alignments is significantly lower for non-English languages.
Automatic Alignment on English Data The au- tomatic alignments for MC-NOISY were derived using BERT-based CRF alignment model (Jiang et al., 2020). This method outperformed other alignment methods when evaluated on subset of MCCCLEAN. Further details about these experi- ments can be found in Appendix
We used = 0.5 to strike a balance between the number of potential misalignments in the dataset and the extractiveness of the data. Overall, as ident in Table 1, this filtering process reduced the dataset size by about half. Filtering reduced the MC-CLEAN portion by a larger proportion, indi- cating that human data is less lexically similar as opposed the noisy data.
Multilinguality We use the same 1-1 alignment property between English and other languages in Cochrane abstracts and PLS described above (Sec- tion 3.1.2) to derive the multilingual portion of MC-NOISY from the English data described in Sec- tion 3.2. As shown in Figure 2, the distribution of the articles and sentence pairs across language in the noisy dataset is similar to that of the human- annotated dataset.
Evaluation of Simplification Models
MULTICOCHRANE enables extensive evaluation of multilingual sentence simplification systems simul- tancously across 4 languages. In this section we present the first study of this kind (to our knowl- edge), assessing zero-shot and fine-tuned models (Section 5.1) with both automatic metrics (Sec-

---

tion 5.2) and human evaluation (Section 5.3) for medical text simplification.
train
test
validation
7728 (61194/30032)
22 (395)
5 583)
7643 (59319/29351)
22 (395)
5 (80)
5.1 Models
fr
4709 (35402/11760))
11 (214)
4 (65)
3117 (31246/15514)
(148)
3 (56)
We experiment with zero-shot and fine-tume tune mod- cls, as well as a simplify-then transstta pipeline.
Table 3: Train/test/validation splits used for evaluation.
Both the number of articles and the total number of alignments are displayed (for training: unfiltered (r 0ffiltered 0.5)).
Zero-shot models:
GPT-3 zero-shot. we evaluate zero-shot simpli- fications generated by the GPT-3-davinci 0003 model using a prompt similar to that used in Au- gust et al. (2022), adapted for multilinguality:
System
BLEU
BS
SARI
LCS
GPT3
2.38
0.8774
42.111
0.569
English
Plan-TS. T52ero
8.12
0.8810
39.057
0.340
mTS,-
7.66
0.8785
40.219
0.466
My fifth grader asked me what this sentence means:
mT5y=uoo
8.82
0.8843
39.579
0.379
[TEXT TO SIMPLJFY] rephrased it for him in [LANG], in plain language fifth grader under- stand:
Flan-T5,-o.5
8.70
0.8875
39.526
0.319
GPT3
5,70
0.7412
37.972
0.361
GPT3
3,11
0.7188
41.123
0.561
Spunish
Flan-T5
3.04
0.7100
37,890
0.367
GPT-3 simplify- seaiity-then-translae GPT-3 has been shown to have strong performance on En- glish simplification for medical texts (Shaib et al.. 2023). Thus, evaluate pipeline model first using GPT-3 for English simplification using the above prompt, and then translating the simplified text to other languages with a strong translation model (Google's Cloud Translation API).
mTS,-0
4.97
0.7337
41.224
0,564
mT5
5.18
0.7381
39.522
0.508
Flan-T5.=0.5
5.21
0.7376
40.064
0.486
GPT3zero
4.42
0.7325
38,096
0.380
GPT3
2.82
0.7134
40,941
0.579
French
Mlan-TT5erre
1.63
0.6989
39.452
0.449
mT5,=0
2.35
0.7146
40.412
0.602
mT5,
2.96
0.7157
39.096
0.553
Flan-T5,=o.s
3.89
0.7284
40.989
0.528
Flan-T5 zero-shot. We also evaluate zero-shot Tlan-TTbase (Chung et al., 2022) performance. We used a simple prompt, specifically prepend- ing "Simplify this sentence:" to the input (com- plex) sentence." 5 When simplifying to languages other than English, the prompt is changed to "Simplify this sentence in [LANG]:". Unfortu- nately, we were unable generate simplifica- tions in Farsi using Flan-T5, so we only evalu- ated this system on English, Spanish, and French.
GPT3zero
1.36
0.7103
41.080
0).497
Farsi
GPT3
1.16
0.7094
43.738
0.577
mT5,
2.21
0.7111
43.280
0.622
mT5,=0.5
2.67
0.7139
43.154
0.631
Table 4: Automatic metries for all generations.
Training Setup. We evaluate models on train/tsstvalldaaton splits shown in Table 3. The train split is composed of both the entirety of MC-NOISY and portion of MC-CLEAN while test and validation splits are subsets of MC-CLEAN. Fine-tuned models (mT5, Flan-T5) were trained over single epoch with learning rate of 5e-5 and used the AdamW optimizer. For both mT5 and Flan-T5 (zero-shot and fine-tuned), nucleus sampling with a top-p of 0.95 was used as the decoding strategy. GPT-3 was evaluated using a temperature of 0.7.
Fine-tuned models:
mT5 We fine-tune separate mTss models (Xue al., 2021) on different language pairs: (En-
glish, English), (English, Spanish), (English, French), and (English, Farsi).
Flan-T5 fine-tuned We further evaluate fine- tuned versions of Flan-T5pase for each language pair, following the setup of its zero-shot coun- terpart. This system also failed to generate Farsi outputs.
5.2 Automatic Evaluation
Metrics. Model outputs were evaluated through five automatic evaluation metries: BLEU (Post, 2018) 8), BERTScore (Zhang et al., 2020), SART (Xu et al., 2016), and the Least-Common Subsequence distance metric (LCS) (Bakkelund, 2009) This dis-
'Preliminary experiments showed that August et al. (2022)'s prompt, though worked well on GPT-3, did not yield gond results with Flan-T5.

---

tance metric has an inverse relationship with the actual least common subsequence (lower means more extractive).
System
Fact.
Fluency
Simpl.
mT5,-o
0.87
0.50
0.24
mTS,
0.59
0.58
0.31
We do not include Flesch-Kincaid as met- ric, for reasons stated in Tanprasert and Kauchak (2021), which provides compelling analysis on how easily it can be misinterpreted, and instead opt for human evaluation of simplicity in Section 5.3.
Finglish
Flan-T3,=00s
0.50
0.25
0.20
GPT3
0.33
0.25
0.02
Flan-T5 2erro
0.39
0.18
0.18
Reference
0.25
0.07
0.02
mT5.-o
1.05
0.44
0.50
m05.
0.89
0.46
0.41
Spanish
Flan-TS,=o.55
0.76
0.83
-0.02
Results. Results of these evaluations are pre- sented in Table 4. The data in this table show common trend in that filtering (Section 3.2) largely improves performance, indicating that filters im- prove data quality. A very visible trend in this data, is that across the board, with the exception of the GPT results, English simplifications vastly outper- form those of other languages. This probably due to the vast resource gap between English and the other languages, as well possible bias towards English in the pre-trained mT5-base model.
GPT3 zcro
0.02
0.00
0.90
GPT3
0.44
0.02
0.51
Flan-T5 zero
1.22
1.05
0.78
Reference
0.50
0.02
0.17
mTS..
1,25
0.67
-0.40
mT5
1.3
0.70
-0.03
French
Flan-TS,=0.5
0.97
0.63
0.04
GPT3zcro
0.43
0.27
0.51
GPT3
0.38
0.44
0.01
Flan-T5 zero
1.32
0.49
Reference
0.09
0.04
-0.03
We observe exceedingly low BLEU scores when the simplified version is used as reference, com- pared to the complex version. Since BLEU is lexical-centric, this reveals that models tend to translate rather than simplify. Simplified sentences in the Cochrane dataset frequently featured inser- tions and elaborations, creating at times an infor- mation mismatch between the model outputs and the corresponding reference simplifications. More- over, the style which PLS were written varied greatly (Devaraj et al., 2021).
mT5,-
0.65
0.49
-0.41
Farsi
mT5,-005
0.65
0.59
-0.53
GPT3
0.40
0.14
-0.28
GPT3tram
0.11
0.06
-0,36
Reference 0.87
0.82
-0.51
Table 5: Summary of human evaluation results across all evaluated systems well the reference simplification, showing average factuality (0-2, lower=better), fluency (0-2, lower=better) and simplicity (-2 roversimplify)-2 (too hard)) ratings. Best system performance bolded.
Another attribute that is of importance when judging simplification is the level of extractiveness, how much the output copied the input. Based on the LCS metric, it seems that while GPT-3 pro- duces less extractive outputs than English mT5, for the other languages the opposite is true. We discuss this further in Section 5.3.
extractive generations compared zero-shot multi- lingual simplification. terms of LCS and SARI. this approach resulted in similar scores as those produced by fine-tuned models. While is ap- pears that the simplify-translate approach improved upon the zero-shot multilingual performance for GPT, it is difficult to draw conclusions about its performance relative fine-tuned models using automatic metrics alone.
Extractiveness and poor multilingual simplifica- tions are also evident in the results for zero-shot Flan-T5 generations. For multilingual simplifi- cations, in particular, fine-tuning significantly in- creased the LCS distance while improving BLEU and BERTScore metrics, indicating higher quality and less extractive simplifications. Interestingly, English simplifications generated by both settings have a significantly lower LCS distance compared those generated by other systems. Why this occurs is unclear to and requires further analysis.
5.3 Human Evaluation
We perform human evaluation of the factuality, lin- guistic quality, and simplicity of model outputs. We also evaluate the reference simplifications. 100 outputs were evaluated for each system. While the alignment may be sound, the references may still contain excess deletions inconsistencies with the original text; prior work Devaraj et al. (2022) found similar phenomena with the Newsela and Wiki datasets. Annotators were blinded to the sys-
The simplify-transie tion different languages elicited similar results as its English counterpart and clearly produces less
approach to simplifica-

---

tem ID (or whether they are evaluating the manu- ally aligned pairs from the texts themselves).
in these large language models seems to be not ye surpassing monolingual simplification combined with machine translation. However, as discussed in Section 1, a pipeline system does come with its drawbacks in terms of cost and robustness.
Metries. (1) Factual faithfulness: We measure overall factuality using a 0-2 scale, where 0 indi- cates no factual issues and 2 indicates severe fac- tual issues. (2) Fluency: We rate linguistic fluency of outputs using scale of 0-2, where O indicates fluent output and 2 indicates severe linguistic is- sues. (3) Simplicity: Simplicity is assessed on a scale of -2 to 2, where -2 indicates heavy oversim- plification, O indicates ideal simplification, and 2 indicates heavy mnder-simplification. The specific criteria are further described in Appendix A.
Annotators also evaluated the reference labels of each complex input from the dataset. Overall, with the exception of Farsi, these references were deemed relatively fluent and simplified. However, due insertions and deletions of information that naturally occur in the data, factuality errors were detected by annotators; these findings echo those in Devaraj et al. (2022).
Examples of common errors. One common ror that found in outputs from all evaluated systems were mistranslated acronyms. Medical acronyms often differ from one language an- other. However, many systems are limited in their ability to properly translate this technical knowl- edge. Usually, the acronym is either not translated, or translated incorrectly, often randomly. The fol- lowing example in French was generated by mTT5 fine-tuned on the unfiltered version of MC-NOISY.
Results. The human evaluation results presented in Table 5 agree with many of the automatic met- rics. First, annotators found English outputs from the mT5 models to be more factual and fluent than those from the other languages. Similarly, factu- ality and fluency also improved with the filtered dataset for the most part (as also indicated by the automatic metrics). This probably due to fil- tering removing less extractive sentence pairs from the training set. However, for French and Farsi, fil- tering slightly worsened the factuality and fluency, perhaps due to the fewer number of examples in the data. For simplicity, with the exception of English, filtering also seems to make outputs simpler.
Input complex sentence: Stress and sedentary lifestyle major determinants of cardiovaseular disease (CVD). Output: Le stress le règlement inactif sont des causes importantes de maladies cardiovasculaires (AVC) les plus courants.
Gloss: The stress and inactive regulation are the com- mon important causes cardiovascular disease (AVC)
There is a stark difference between the GPT- 3 and the mT5 outputs. GPT-3 produced signif- icantly more faithful and fluent text while mTT5 outputs were deemed simpler with the exception of English. However, qualitative feedback from anno- tators suggests that non-English GPT-3 outputs are skewed due to severe degree of extractiveness; in some cases, the output is simply a word-for-word translation of the input. This inflates the factuality and fluency scores, but this level of extractiveness is manifest in the simplicity scores; GPT-3 does proper simplification only for the English to En- glish case.
The correct translation of CVD, or cardiovascu- lar disease, in French is MCV, not AVC (French acronym for stroke).
Fluency issues were especially common in out- puts generated by mT5 and Flan-TS. In the case of Flan-T5, in particular, misspellings and incorrect applications of gendered language are quite preva- lent. Consider the following example in Spanish generated by Flan-T5:
Input complex sentence: The rates were 24.5 for the sponge and 10.9 for the diaphragm in the UK trial.
Ouiput: Las tasas de asistencia respiratoria fueron de un 2.4 para la secreción del vejiga y 10,9 para el diafrag- mata.
The trends in the results for Flan-T5 mirror those for GPT-3. When comparing zero-shot generations to fine-tuned generations, it is clear that Flan-T5 is limited in its ability to perform multilingual simpli- fication. Not only are generations less simplified, but they frequently have factual and fluency errors.
Gloss: The rates of respiratory assistance were 2.4 for the secretion of the bladder and 10.9 for the diaphragm.
This example has number of errors. There are many hallucinations that include the insertion and substitution of irrelevant information. However, with respect to fluency, the main issue is the in- correct gender used for the word "vejiga" as well as the misspelling of the word "diafragma" as "di- afragmata". The important thing to note here is that
The simplify-translar approach proved be fairly strong compared to other systems which di- rectly performed multilingual simplification. The current state of direct multilingual simplification

---

most of these errors are localized to technical words or phrases. More work needs to be done to adapt large language models to these niche vocabularies.
Regarding compensation for annotators, all the annotators were paid wage of $15 for every hour of work.
We recognize the dual use of language models and the possibility of factual errors and hallucina- tions in their generations, and we believe that this line of work is necessary both to advance technol- ogy for social good while also acknowledging these risks. The inaccessibility of medical information and health illiteracy are some of the leading rea- sons for real health consequences including more hospital admissions, emergency room visits, and poorer overall health (Berkman et al., 20.1). Mak- ing medical information accessible is one of the best ways to tackle health literacy, and that is the core of what a simplification system is aimed do. Additionally, medical misinformation is one of the most series issues highlighted in the COVID-19 19 pandemic; one contributing reason for this is the lack of health literacy among the general public. By simplifying trustworthy evidence, we hope to empower the public with a keener eye for such mis- information. Factual errors is one of the key aspects studied in this work; we perform thorough eval- uation dissecting issues that can come from these models, especially in multilingual setting. We believe rigorous evaluation, as done in this work, is one of our best tools to demystify language models, and help the community understand the issues at hand. With such understanding, we hope to point to future directions that collectively, we will be able to provide factual, readable, multilingual access to medical texts.
6 Conclusions
This paper introduces the first human-aligned multi- lingual text simplification dataset, MC-CLEAN, for the medical domain for the languages of English, Spanish, French, and Farsi. We further expanded this dataset using automatic alignments to create a much more vast but noisier dataset, MC-NOISY, that could be used for training language models. We also performed an evaluation of multilingual simplification for various systems, testing some of their zero-shot capabilities well their perfor- mance when fine-tuned MC-NOISY.
Limitations
MULTICOCHRANE has a few limitations. Firstly, for a given article, there does not exist consis- tent set of multilingual versions for that article. As exemplified in Figure 2, the number of articles in which there exists multilingual version varies de- pending on the language. This uneven distribution does favor more well resourced languages, with Farsi being a notable exception.
Looking at the wider scope of text simplification, sentence-level text simplification does have a few drawbacks. Especially within the medical domain, contextual information is helpful for generating good simplifications. In MC-CLEAN elaborative information was often left unaligned. Simplifying at a paragraph-level or document-level could have made use of this additional data to generate more useful simplifications.
Acknowledgements
Because of limitations with the amount of com- puting capability and time, the largest models avail- able for Flan-T5 and mT5 were not used for evalu- ation. The models that we used were the best avail- able models that could have evaluated within our reasonable constraints.
We acknowledge Pouya Nekouei for his help with the Farsi language. This research was partially supported by National Science Foundation (NSF) grants UIS2144579,, IIS-2144493 and IIS-21 226333, and by the National Institutes of Health (NTH) un- der the National Library of Medicine (NLM) grant M
Ethical Concerns
This work does not use any full texts of Cochrane reviews. The research use of publicly available abstracts, including both technical abstracts and multi-lingual plain language summaries, is consid- ered fair use. Note that the Cochrane dataset, in similar capacity as ours, has already been released publicly by prior work (Devaraj et al., 2021; Guo et al., 2020).
References
Emil Abrahamsson, Timothy Forni, Maria Skeppstedt, and Maria Kvist. 2014. Medical text simplification using synonym replacement: Adapting assessment of word difficulty to a compounding language. In Proceedings of the 3rd Workshop Predicting and Improving Text Readability for Target Reader Papu- lations (PITR), pages 57-65.

---

Sweta Agrawal and Marine Carpuat. 2019. Controlling text complexity in neural machine translation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNNLA) pages 1549- 1564, Hong Kong, China, Association for Computa- tional Linguistics.
William Coster and David Kauchak. 2011. Simple En- glish Wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 665-669, Portland, Ore- gon, USA. Association for Computational Linguis- tics.
Ashwin Devaraj, lain Marshall, Byron Wallace. and Junyi Jessy Li. 2021. Paragraph-level simplifica- tion of medical texts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4972-4984, Online. Association for Computational Linguisties.
Marco Alfano, Biagio Lenzitti, Giosuè Lo Bosco, Cinzia Muriana, Tommaso Piazza, and Giovanni Vizzini. 2020. Design, development and validation of sys- tem for automatic help to medical text understand- ing. International journal of medical informatics, 138:104109.
Fernando Alva- Mwaa-Macheego, Carolina Scarton, and Lucia Specia. 2020. Data-driven sentence simplification: Survey and benchmark. Computational Linguistics, 46(1):135-187.
Ashwin Devaraj. William Sheffield. Byron Wallace, and Junyi Jessy Li. 2022. Evaluating factuality in text simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 7331-7345, Dublin, Ireland. Association for Computational Lin- guistics.
Mikel Artetxe and Holger Schwenk. 2019. Mas- sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Natalia Grabar and Rémi Cardon. 2018. Clear-simpi corpus for medical french. In Proceedings of the Ist Workshop on Automatic Text Adaptation (ATA), pages 3-9.
Tal August, Lucy Lu Wang. Jonathan Bragg, Marti A. Hearst, Andrew Head, and Kyle Lo. 2022. Paper plain: Making medical research papers approachable to healthcare consumers with natural language pro- cessing.
Yue Guo, Wei Qiu, Gondy Leroy, Sheng Wang, and Trevor Cohen. 2022. Cells: A parallel corpus for biomedical lay language generation. arXiv preprint NxXiv:221.008111
Daniel Bakkelund. 2009. An les-based string metric. In An LCS-based string metric.
Yue Guo, Weijian Qiu, Yizhong Wang, and Trevor A. Cohen. 2020. Automated lay language summa- rization of biomedical scientific reviews. ArXiv, abs/2012.12573.
Chandrayee Basu, Rosni Vasu, Michihiro Yasunaga, and Qian Yang. 2023. Med-casi: Finely annotated dataset and models for controllable simplification of medical texts. In AAAI Conference Artificial Intelligence.
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural CRF model for sentence alignment in text simplification. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 7943-7960, On- line. Association for Computational Linguisties.
Nancy D Berkman, Stacey I. Sheridan, Katrina E. Don- ahue, David J Halpern, and Karen Crotty. 2011. Low health literacy and health outcomes: an up- dated systematic review, Annals of internal medicine, 155(2):97-007.
Πonaa Kickbusch, Jürgen M Pelikan, Franklin Apfel, and Agis D Tsouros. 2013. Health literacy: the solid facts. World Health Organization. Regional Office for Europe.
Yixin Cao, Ruihao Shui, Liangming Pan, Min- Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: new task towards better communi- cation between experts and laymen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1061-1071, On- line. Association for Computational Linguistics.
Joongwon Kim. Mounica Maddela, Reno Kriz, Wei Xu. and Chris Callison-Burch. 2021. BiSECT: Learning to split and rephrase sentences with bitexts. In Pro- ceedings of the 2021 Conference Empirical Meth- ods in Natural Language Processing, pages 6193- 6209, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Rémi Cardon and Natalia Grabar. 2020. French biomed- ical text simplification: When small and precise helps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 710-716.
Nicholas Klochn, Gondy Leroy, David Kauchak, Yang Gu, Sonia Colina. Nicole P Yuan, and Debra Re- vere. 2018. Improving consumer understanding of medical text: Development and validation of new subsimplify algorithm to automatically generate term explanations english and spanish. Journal of medi- cal Internet research, 20(8)e 10779.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. instucttionfiinetueed language models.
2022. Scaling instruction- finetuned arXiv preprint arXiv:2210. 22.0. 1416.

---

Wuwei Lan, Chao Jiang, and Wei Xu. 2021. Neural semi-Markee CRF for monolingual word alignment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Pupers), pages 6815-6828, Online. Association for Computational Linguistics.
Natural Language Processing, pages 10215-10245. Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Michael Ryan, Tarek Naous, and Wei Xu. 2023. Re- visiting non-English text simplification: A unified multilingual benchmark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4898- 4927. Toronto, Canada. Association for Computa- tional Linguisties.
Cheaxi Liu. Dan Wang, Chaojie Liu, Junnan Jiang. Xuemei Wang, Haihong Chen, Xin Ju, and Xinping Zhang. 2020. What is the meaning health literac? a systematic review and qualitative synthesis. Family medicine and community health, 8(2).
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski. and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP). pages 8051-8061 Online. Association for Computa- tional Linguistics.
Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoit Sagot. 2022. MUSS: Multilin- gual unsupervised sentence simplification by mining paraphrases. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 1651-1664, Marseille, France. European Language Resources Association.
Chantal Shaib, Millicent Li, Scbastian Joseph, Iain Marshall, Junyi Jessy Li, and Byron Wallace. 2023, Summarizing, simplifying. and synthesizing medical evidence using GPT-3 (with varying success). In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Pupers), pages 1387-1407, Toronto, Canada. Association for Computational Linguistics.
Gustavo Paetzold, Fernando Alva-Manchego, and Lucia Specia. 2017. MASSAlign: Alignment and anno- tation of comparable documents. In Proceedings of the IJCNLP 2017, System Demonstrations, pages 1-4, Tapei, Taiwan. Association for Computational Linguistics.
Atharva Phatak, David W Savage, Robert Ohle, Jonathan Smith, and Vijay Mago. 2022. Medical text simplification using reinforcement learning (teslea): Deep learning-based text simplification approach. JMIR Medical Informatics, 10(1 :3838095
Advaith Siddharthan. 2014. A survey of research on text simplification. I7I-Innernational Journal of Applied Linguistics, 165(2):259-298.
Neha Srikanth and Junyi Jessy Li. 2021. Elaborative simplification: Content addition and explanation gen- eration in text simplification. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021. pages 1123-5137, Online. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Pupers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Teerapaun Tamprasert and David Kauchak. 2021. Flesch-kincaid is not text simplification evaluation metric. Proceedings of the 1st Workshop on Natu- ral Language Generation, Evaluation, and Metrics (GEM 2021), pages 1-14, Online. Association for Computational Linguistics.
Justus Randolph. 2005. Free-marginal multirater kappa (multirater k [free]): An alternative fleiss' mixe-maagginal multirater kappa. Online submission.
Nils Reimers and Iryna Gurevych. 2019. Sentence-
BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing /EMNLP-IJCNLA pages 9982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space, In Proceedings of the 2019 Conference Empirical Methods in Natural Language Procexsing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNIPP) pages 1342- 1348, Hong Kong. China, Association for Computa- tional Linguistics.
Juliane Ried. 2023. About translation cochrane. https://documenattion.omm www.ainee.ccccrrco.ooooooe. h.splay/////bbotttttaaaaatttttto
Jan Trienes, Jörg Schlötterer, Hans-Ulrich Schildhaus, and Christin Seifert. 2022. Patient- friendly clinical notes: Towards a new text simplification dataset. Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022) pages 19-27, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Sid- dhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin John- son. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in

---

Hoang Van, David Kauchak, and Gondy Leroy. 2020. AutoMeTTS: The autocomplete for medical text sim- plification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1424-1434, Barcelona, Spain (Online). International Committee on Computational Linguisties.
Laurens Van den Bercken, Robert-Jan Sips, and Christoph Lofi. 2019. Evaluating neural text sim- plification in the medical domain. The World Wide Web Conference, pages 1286-3292
Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mo- hit lyyer, and Noah Constant. 2022. Overcoming catastrophic forgetting in zero-shot cross-lingual gen- eration. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages
Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Kristian Woodsend and Mirella Lapata. 2011. Learning simplify sentences with quasi-synchroouus gram- mar and integer programming. In Proceedings of the 2011 I Conference on Empirical Methods in Natural Language Processing, pages 409-420, Edinburgh, Scotland, UK. Association for Computational Lin- guistics.
Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification re- search: New data can help. Transactions of the Asso- ciation for Computational Linguistics, 3:283-297.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burt Buroch. 2016. Optimizing sta- tistical machine translation for text simplification. Transactions the Association for Computational Linguistics, 4:401-415.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5:: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Tianyi Zhang. Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Interna- tional Conference Learning Representations.
Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 584- 594, Copenhagen. Denmark. Association for Compu- tational Linguistics.
Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-hased translation model for sentence simplification. In Proceedings of the 23rd International Conference Computational Lin- guistics (Coling 2010), pages 1353-1361, Beijing, China. Coling 2010 Organizing Committee.

---

A
Human Evaluation Framework
The following is the exact framework that annotators used for evaluating simplification outputs.
Factual (faithful to input)
Choices - 0-2
Give overall rating how factually consistent the model-generatel output is on scale from 0 to 2, where 0 is completely factual, indicates inconsequential factual errors, and indicates severe factual errors.
Example:
Rating 0:
Original: We discovered a high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: We found that many study participants had bad side effects from heart disease treatments.
Rating 1:
Original: We discovered a high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: We discovered some participants had bad side effects from heart disease treatments.
Rating 2:
Original: We discovered high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: We discovered participants had no side effects from heart disease trealments.
Fluency
Choices 002
Give an overall rating on how well the model-generated output follows grammatical rules and appears as natural sentence, where O indicates no fluency issues, indicates superficial issues, and 2 indicates severe fluency issues that obfuscate the meaning of the sentence.
Example:
Rating 1:
Original: We discovered a high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: We found that many study participants had badly side effects from heart disease treatments.
Rating 2:
Original: We discovered a high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: In heart disease treatments, many participants had effects bad on the side.

---

Simplicity
Choices (-2 2)
Rate the level of simplification that occurred from the original English sentence the model-generated output. Note: since the input readability varied, please make judgements independent of the input.
-2: Output is severely oversimplified to the point where the original intent of the original sentence has been lost.
-1: Output is oversimplified, missing some important details. However, the
intent of the original sentence is preserved.
O: Output is idcally simplified. It is readable to the average layman and does not omit important details.
Output should be simplified more and include uncommon technical terms without any elaboration/explaaation
2: Output should be simplified MUCH more. There is little to no change the style of the sentence from the original sentence, and the output difficult for the average layman to understand.
Example:
Rating -2:
Original: We discovered a high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: Study participants had some effects while being treated for some disease.
Rating -1:
Original: We discovered a high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: Study participants had bad effects while being treated for heart disease.
Rating 0:
Original: We discovered a high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: We found that many study participants had bad side effects from heart disease treatments.
Rating 1:
Original: We discovered high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease. Output: We found that many study participants had experienced adverse effects from ischemic heart disease treatments.
Rating 2:
Original: We discovered a high percentage of study participants experienced adverse reactions while being treated for ischemic heart disease.
Output: We discovered high percentage of study participants experienced antagonistic reactions while being remedied for ischemic heart disease.

---

Factuality
Fluency
Simplicity
Combined
0.362
0.549
0.372
0.464
Systems
Table Randolph's kappa on English oulputs evaluated for agreement.
100
Alignment Type
Method A
Method
1-1
36.4%
50.3%
N-1
20.3%
36.6%
1-N
9.4%
10. 1
N-M
33.9%
2.9%
Table Distribution of alignment types in MC-CLEAN for each method used.
Human Evalustion Across all Systems Spantsh
B Human Evaluation Score Distributions
300
Figure 3 show the distributions of the human evalu- ation scores for all evaluated systems.
Human Evaluation Agreement
In addition to evaluating outputs in their respective languages, all human evaluators also evaluated set of 100 examples in English. These were col- lected to estimate how similarly annotators would evaluate outputs.
GIT
To analyze agreement, we use Randolph's kappa (Randolph, 2005), free-marginal version of Fleiss' kappa. These values are presented in Table 7 for the different human evaluation criteria along with one for all of them combined. These results show there is moderate agreement among annotators.
Human Evalustion Acroes all Systems French
D Alignment Analysis
Since annotators were given total freedom to align sentences, many different alignment relationships are present in MC-CLEAN. To quantify the propor- tion of alignments that belong to these alignment relationships, two methods were used for analysis. For future reference, 1-1, N-1, 1-N, and N-M align- ments refer number of complex sentence aligned to the number of simple sentences in an alignment group.
Systemt
500
Method A. This method treats alignments in document edges in bipartite graph, with com- plex and simple sentences vertices. Relation- ships are found by tracking the connected compo- nents in this graph through depth-first search. The type of relationship determined by 2-tuple of the number complex sentences and the number simple sentences in a connected component.
Phanery
Figure 3: These graphs show the distribution of human evaluation scores for each evaluated system in English, Spanish, French, and Farsi.

---

Method
Precision
Recall
F1
threshold 0.3 (selected on the validation set) be "aligned": this is the alignment method used in Xu et al. (2015).
Jaccard
0.737
0.238
0.360
Sentence-BERT
0.437
0.416
0.426
TFIDF
0.372
0.354
0.363
BERTScore
0.468
0.446
0.457
TF-IDF Similarity. We generated TF-IDF vec- tors for every possible pair. Pairs were scored based on the cosine similarity of their embed- dings. We aligned each sentence in the PLS to the most similar sentence in the corresponding complex version. This is the alignment method used in Pactzold al. (2017).
Neural-CRPNiwwan
0.454
0.592
0.514
Neural-CRFecchhnn 0.781
0.466
0.584
Table 9: Precision, Recall, F1 for sentence alignment methods evaluated the manually aligned English por- tion of MC-CLEAN.
This method is extension of Method that specifically targets on breaking up N-M alignment relationships. While Method A counts any connected component as an alignment group. this method requires that each component must be fully connected (every complex sentence is aligned to every simple sentence in the group) well. As such, an N-M alignment group that is not fully connected is broken up into smaller fully connected components.
Method B.
Sentence-BERT Similarity. Similar to TF-IDE, but here we use Sentence-BEil embeddings (Reimers and Gurevych, 2019) instead.
BERTScore. The BERTScore (Zhang et al., 2020) between the complex and simplified sen- tences was calculated for every possible pair, and like for TF-IDF and Sentence-BERT, we consid- ered the highest-scoring pair for each sentence in the PLS to be aligned.
The results of this analysis is presented in Ta- ble 8. The results from Method A can be viewed as a lower bound for 1-1, N-1, and 1-N, alignment types. Method B could produce different results de- pending on how the N-M alignment groups are bro- ken up. It is difficult to know cxactly how to break up alignment group in Method B without know- ing the annotator's intentions, so heuristic methods were used. For these results, the alignment groups were broken down such that simple sentences that were aligned to all complex sentences or vice versa were first grouped together. Then, the rest of the alignments were split into 1-N or N-1 groups.
Neural CRF Alignment. We also evaluate a BERT-based CRF alignment model (Jiang et al., 2020). We test both the original model trained on the Newsela dataset, and model fine-tuned on the training set of MC-CLEAN, The hyperpa- rameters used for fine-tuning the same those used for the original model.
For experiments, we used 395 alignments from 22 MC-CLEAN English articles as the test set, and 84 aligned sentence pairs from 5 articles as the validation set. The rest of the 149 alignments from 74 articles were used for fine-tuning the neural CRF alignment model (Jiang et al., 2020). Table 9 reports the precision, recall, and F'l scores of these methods against reference annotated alignments MC-CLEAN. We observe that the CRF alignment model trained on the Cochrane dataset achieved the best Fl measure; this is unsurprising, as it is the state-of-the-an supervised method for sentence alignment.
E Automatic Alignment Experiments
While automatic sentence alignment is standard in text simplification (Zhang and Lapata, 2017; Xu et al., 2015), we are not aware of evaluation work for sentence alignment algorithms on medical texts. The construction of MC-NOISY allows us to create a ticred version of MULTiCOCHRANE that is large enough for training state-of of fefftheeatt neural text simplification models.
Human Evaluation Correlation
We evaluate series of automatic sentence align- ment methods to determine which approach is the most compatible with the medical data hand to derive the English portion of MC-NOISY.
Table 10 displays the Pearson correlation between automatic metries and the human evaluation scores. Overall the correlation between them is mostly
Jaccard Similarity. The Jaccard Similarity score of every possible sentence pair was cal- culated. We consider pairs that score over a
'For both TF-DF and SentennceBBRRY we tested a version where a similarity threshold is selected; this led to much worse Fl scores, likely due to similarity heing used as an absolute measure as opposed to relative onc.

---

Auto Metric
Fact.
Fluency
Simpl.
Dataset
en-ten
en en-ses
en-*ft
BLEU
-0.124
-0.032
0.037
Simple Complex
MC-NOISY
39.7
39.5
39.4
41.1
Scor
-0.174
-0.174
-0.030
filtering
44.8
44.5
44.4
46.6
English
SARI
-0.078
0.033
0.014
CL-AL
41.6
41.6
41.7
43.2
LCS
0.319
0.144
-0.254
MC-NOISY
36.0
48.0
54.1
52.7
BLEU
-0.190
-0.100
0.017
filtering
32.3
43.6
49.3
48.0
BERTScore
-0.105
-0.083
0.010
Spanish
SARI
-0.037
0.091
-0.367
MC-CLEAN
36.6
47.2
55.0
49.5
LCS
0.196
0.095
-0.95
BLEU
-0.977
-0.108
0.042
Table 12: Statistics for average number of tokens per sentence for complex and simple sentences in MC-
BERTScore
-0.219
-0.086
0.047
French
SARI
0.029
0.035
-0.86
NOISY and MC-CLEAN.
LCS
0.392
0.188
-0.253
BLEU
-0.120
-0.059
0.055
BERT'Score
-0.281
-0.074
0,227
are the results of this hallucinating behavion.
Farsi
SARI
-0.243
0.031
0.286
The following is an example an extreme hal- lucination found among the generated outputs for the English version.
LCS
0.319
0.257
-0.347
Table 10: Pearson correlation between automatic met- rics and human evaluation scores. Human evaluation closer better; BLEU/BERTScore/SAAA higher- = better; LCS: higher more abstractive.
metrics:
Input complex sentence: Randomised controlled trials of tai chi lasting least three months involving healthy adults adults high risk of CVD.
Language
BLEU
BS
SARI
LCS
English
0.25
0,8442
43.677
0.695
Output: Another study investigated effect of physical exercise as treatment for people with carpal tunnel syndrome.
Spanish
0,7055
41.009
0.651
French
0.90
0.6806
41.987
0.695
Farsi
1.00
0.6862
44.204
0.698
While the output does mention a "study" and the use of "physical exercise," which is somewhat related to randomized controlled trial about tai chi, it remains wildly off-topic compared to the original sentence.
Table 11: Automatic metrics for mT5 fine-tuned only on MC-Clean data.
weak. BLEU and BERTScore, not surprisingly, does have weak inverse correlation with factu- ality and fluency measures. The metric with the strongest correlation human scores is LCS, show- ing a moderate positive correlation with factuality (more abstractive sentences tend to contain more factual errors), well as moderate inverse corre- lation with the simplicity score (more abstractive sentences tend to be simpler). The other simplifica- tion metric, SARI, had varied results depending the language.
H Additional Statistics
Table 12 shows the statistics for the average number of tokens per sentence in MULTICOCHRANE.
G Fine-tuning with MC-CLEAN Only
We attempted fine-tuning mT5 with just data from MC-CLEAN. The same training methodology was used, with the only difference being the number of epochs being increased to 5. Overall, due to the significantly lower amount of alignments in MC- CLEAN as compared MC-NOISY, the generated output featured many extreme hallucinations. The automatic metrics for these generations shown in Table 11. The low BLEU score and BERTScore

---

I Notable Examples
L11 Good Simplifications
Below are some examples of good simplifications made by the systems. GPT3 systems in particular were good at explaining medical terms and information present in parentheses. Meanwhile, other systems like Flan-T5-ft (r=0.5) and Flan-T5 (0-shot) had fluency issues and hallucinations even in the best simplifications and had more difficulty with longer input sentences. mTT5 outputs had issues with extractiveness and retained complex medical vocabulary.
Language
System
Input
Output
Gloss
English
GPT3 (0-shot)
We also found improved success rates within two attempts (RR 1.78, 95%
This sentence is talking ahout study that was done compare two dif- ferent types of guidance when using medical tool called catheter. The study found that if ultra- sound guidance used, successful
CI 1.25 to 2.51, 134 catheters, two RCTs, moderate-quality evi- dence) with ultrasound guidance compared with other types of guidance.
it than other types of guid- ance. Out 134 catheters that used, the rate was 1.78 times higher when ultrasound guidance was used. This result reliable because two differ- ent tests were done, evidence was of mod- erate quality.
Spanish
GPT3 (0-shot)
Four RCTs compared ul- trasound with palpation, and compared ultra- sound with Doppler audi- tory assistance.
Cuatro estudios de inves- tigación evaluaron la cfi- cacia de ultrasonido para detectar problemas médicos, comparándolo con usar palpación (to- car con manos) y uno estudios comparó el uso del ultrasonido con ayuda de un aparato Ila- mado Doppler para es- euchar los sonidos del cuerpo.
Four investigative studies evaluated the efficacy of using ultrasound detect medical problems, com- paring using the pal- pation (touching with the hands) one of stud- compared the use the ultrasound with the help an apparatus called Doppler listen the sounds the body.
English
mT5 (r=0.5)
There serious ad- verse events related to ther nitrous oxide bon dioxide pneumoperi- toneum (three studies; participants: very low quality evidence).
There serious side effects related either oxide carbon diox- pneumoperitoneum.
French
m'55 (r=0.5)
Thiospot
effec- thiospots était plus cfti- que le placebo.
Thiospot was effec- tive than the placebo.
tive than placebo (SMD- 2.61, 95% CI -3.76 1.47).
Spanish
GPP3-simp-trans
There
serious ad- events related to ei- ther nitrous oxide bon dioxide pneumoperi- toneum (three studies; 196 participants; very low quality of evidence).
Los tres estudios real- izados 196 personas mostraron que no hubo efectos negativos al respi- gas nitrógeno dióxido de carbono durante pro- cedimiento médico.
The three studies car- ried out with 196 people showed that there negative effects in breath- ing nitrogen gas carbon dioxide during medical procedure.

---

Language
System
Input
Output
Gloss
French
-im--simp-trans
To assess interventions used in the management of all types of melasma: epidermal, dermal, and mixed.
Pour déterminer quels traitements fonctionnent mieux pour les différents types de mélasma (une affection cutanée), le type qui affecte couche supéricure de peau, le type qui affecte la couche sous couche supéricure le type qui affecte deux couches.
In order to determine which treatments work the best for the different types of melasma (a skin afflie- tion). like the type which affects top layer the skin, the type which af- feets the layer under the
top layer and the type which affects the two lay- ers.
Spanish
mT5 (r=0)
For smokeless tobacco use, difference found between the skills- community arm and the control group 42 weeks (RR 0.93, 95% CI 0.67 to 1.30), though a signif- icant difference served between only arm and the control group (RR 0.57, 95% CI 0.39 0.85).
cuanto los fumadores sin cigarrillos, identi- ficaron pocos casos y no huho difcrencias entre grupo de ejercicios grupo control las se- manas, pero se observaron efeetos secundarios entre el grupo de ejercicios y grupo control.
for smokers without cigarettes. few identified and there
differences between the exercise group und the weeks,
trol group but side effects were ob- served between cise group and control group.
oh- skills-
French
mT5 (r=0)
We scarched the Cochrane Stroke Group Trials Reg- (16 January 2018), Cochrane Central Reg- of Controlled Tri- (CENTRAL; 2017, Is- sue 12) the Cochrane Library (January 2018), MEDLINE Ovid (1946 Jamuary 2018). Embase Ovid (1974 to January 2018), Web of Science Core Collection (1950 to January 2018) and three Chinese databases (Jan- uary 2018).
Nous recherché des études complémentaires publićes jusqu' 16 jan- vier 2018.
We researched additional published studies until the January 2018.
Spanish
Flan-T5: Ca--TT--t (r005)
We included seven trials, involving 735 participants, in this review.
incluyeron siete en- sayos, que incluyeron participantes.
Seven trials, involving 735 participants, were in- cluded.
French
Flan-TS-ft (r=0.5)
The quality of the evi- dence was rated high.
La qualité des preuves était élevée.
The quality of the evi- dence was high.
Spanish
Flan-T5 (0-shot)
We searched until January 2016.
También buscamos hasta enero 2016.
We also searched until Jan- uary 2016.
English
Flan-T5 (0-shot)
We searched the Cochrane Kidney and Transplant Specialised Register up to 17 August 2017 through contact with the Infor- mation Specialist using search terms relevant to this review.
Cochrane Kidney and Transplant Specialised Register was searched for to 17 August 2017.
Table
1.2 Fluency
Fluency errors across systems include repetition and grammatical errors, as well as the use of foreign words in the Flan outputs.

---

This category includes examples of fluency errors involving the repetition of single phrase word that confuses the meaning of the output and, in some cases, makes it completely meaningless. This error was present all systems except for the GPT3 (0-shot) and GPT3-simp-trans systems.
Repetition.
Language
System
Input
Output
Gloss
Englishh
mT5 (r=0)
In people with symp- tomatic intracerebral artery stenosis, recurrent stroke significantly reduced RIC (risk ratio (RR) 0.32. 95% confidence interval (CI) 0.12 0.83; trials, 182 participants, low-quality evidence).
Compared other an- tipsychotic drugs, rent stroke was signifi- cantly reduced recur- rent stroke.
English
mTT5 (r=0.5)
Similarly, results sug- gest, but do not confirm, the possibility of posi- tive influence of expertise in the use of ultrasound the first attempt success
A significantly sig- nificantly significant difference in the use of the first attempt success rate.
ultrasound
Spanish
Flan-T5-ft (r=0.5)
We located 10 RCT's in- volving 2961 participating surgeons performing an operation in which the of blunt needles pared to the use of sharp needles.
Se identificaron 10 ECA con 2961 médicos que alizaron una operación que fue realizada agujas y cuando comparó agujas tópicas agujas tópicas con agujas agujas.
We identified 10 RCTs with 2961 doctors who per- formed operation that performed with dles and when topical dles were compared with topical needles and with needles needles.
Spanish
mTT5 (r=0)
Results of the ongoing tri- als will add to the evi- dence hase but additional longer-term, high-quality trials are needed.
Los resultados de los en- sayos curso adicionales corto plazo, adicionales. de alta calidad, se necesi- tan más estudios de alta calidad.
The results of the ongo- ing, additional short-term trials, additional [trials], of high quality. more high- quality studies needed.
French
mT5 (r=0)
Smoking prevalence in Indigenous youth is twice that of the non-Indigenous population. with tobacco experimentation com- mencing age.
La prévalence du tabag- isme chez les jeunes je- unes jeunes développent des problèmes d' éésttmaa à I''' l'avenir présentent une amélioration des symp- tômes.
The prevalence of smok- ing in the youth youth youth developed stomach problems in the future and presents improvement of the symptoms.
early
French
mT5 (r=1).)
The rates were 24.5 for the sponge and 10.9 for the di- aphragm in the UK trial.
Les taux étaient étaient 25 pour le tube de sponge et 10,9 pour le diaphragme au Royaume- Uni.
The rate was was 25 for the sponge tube and 10.9 for the diaphragm in the UK.
Table
Grammar. This category includes examples of grammatical issues in the output sentences, such the overuse of prepositions, spelling errors, and disagreement between articles and nouns. Multiple prepositions appeared sequentially in a way that was incoherent, and there were instances where articles did not agree with nouns in gender or number.

---

Language
System
Input
Output
Gloss
French
mT5 (r=0)
There appears be significant differences in the improvement of tech- nical skills hetween differ- methods of box model training.
En qui concerne modifications de technique manuelle des méthodes de modèles bovés, majorité des participants dans leur étude ont montré pas de différence significative entre les méthodes d'entrainement au modèle d' é'éuudiant de modèle d' e'évaluation en termes d' d'améliortion des compétences infor- matiques pour différentes méthodes e'entranemmen au modble bouchon.
Concerning the modifica- tions of the manual tech- nique of the selection methods bové(?) mod- els, the majority purtici- pants their study didn't show significant differ- ence between ing methods in student model the evaluation model terms improve- ment informatic skills for the different training methods model.
sélection
train-
blockage
Spanish
Flan-TS Ca-n-55-ttoe (r-05)
carbon dioxide, helium, ar- gon, nitrogen, ide, room air) used for establishing pneumoperi- toneum in participants un- dergoing laparoscopic gen- eral abdominal gynacco- logical pelvic surgery.
uso de dióxidos car- honosas, alto carhono, la argón, náusea fármaco nitroso (es decir, flujos náuseos y el fár- maco nitroso).
The use of the carbon diox- ides, the high carbon, the argon, nausea and the nitrous drug (which say, the nausea flows the nitrous drug).
ox-
Spanish
Flan-T5-ft (r=0.5)
Indigenous youth in par- ticular commence smok- ing early age, disproportionate burden dubstance-related morbid- ity mortality exists as result.
probable que mor- bilidad de los nios gesta un término prano o un pas a lener un
It likely that the morhid- ity of the chiren of intake to an early term or worsening health and death a loss treatment.
empeoramiento de salud y muerte debido a una pér- dida del tratamiento.
Table 15
Foreign words (Flan-T5). This category includes examples of fluency errors involving the use of foreign words in the output sentences. For example, Flan-T5 (0-shot) produced a French output that started with the English phrase 'in four studies', and Flan-T5-ft (r=0.5) produced an English output that started with the Spanish word 'otros'. This type of error was mostly found in the Flan-T5-ft (r=0.5) and Flan-T5 (0-shot) systems.
I Language
System
Input
Output
Gloss
French
Flan-T5-ft (r=0.5)
It usually involves restrict- ing blood flow to limbs and then releasing the is- chaemic blood to promote neuroprotective effect.
L'exercice consiste généralement la restric- tion du flux sanguin de la membres des limbs et ensuite à libérer les bloods ischémies pour promouvoir un effet neuroprotecteur.
The excrcise generally in- volves the restriction of the blood flow to the mem- bers of the limbs and then to free the ischemic blood to promote neuroprotec- tive effect.
French
Flan-T5 (0-shot)
In four studies, the use of blunt needles reduced the number of self-reported needle stick injuries with RR of 0.31 (95% CI 0. to 0.68).
In four studies, I' l'utilisation des ne- jéros stru000Fu a réduit autorem-
In four studies, the use of nejéros strru00fu (?) to reduce the autorem- présenté (?) injury to the nejéros (?) of a risk ration of 31, on average from .14 to .68.
I' l'injurage présenté aux nejéros d' une rRR de 0,31, en moyenne de 0,14 à 0,68.
English
Flan-T5-ft (r=0.)
Therefore, other gases have been introduced as al- termatives to carbon diox- ide for establishing pneu- moperitoncum.
Otros such as carbon diox- ide are used as alterna- tive gases for estahlishing pneumoperitoncum.

---

Language
System
Input
Output
Gloss
Spanish
Flan-T5 (0-shot)
Remote ischaemic condi- tioning (RIC) has been de- veloped neuroprotec- tive strategy prevent and treat ischaemic stroke.
EI afecto izquierdo de condicionamiento remoto (RIC) ha sido desarrollado como estrategia neuro- promotiva para prevenir y tratar strokes izquierdo.
The remote conditioning left effect (RIC) has been neutropro-
developed molive to help prevent and left strokes.
Table 16
1.3 Factuality
Factuality enrors displayed by the systems could be broadly categorized into 3 types minor hallucinations, extreme ballucinations, and contradictions.
Minor hallucinations. This category includes examples where the output contained minor details that were not present the input (hallucination). The systems would hallucinate details like numbers, dates, medical terms, incorrect expansions of acronyms, etc. GPT3 systems would also misinterpret the context of words present in the input. In one of the examples, the word 'rates' in the input is referring to pregnancy rates" but GPT3 misinterpreted the word to mean 'cost".
Language
System
Input
Output
Gloss
French
mT5 (r=0)
assess interventions
Cette revue examiné dif- férents type mélanome, la peau, d' reilll et or- ganes, et s e'est améliorée dans années futures.
This review has exam- ined different types melanoma, the skin, the eye and and improved the future years.
management
all types melasma:
epidermal, dermal, mixed.
organs,
Spanish
GPT3 (0-shot)
It usually involves restrict- ing blood flow limbs and then releasing chaemic blood promote neuroprotective effect.
Esta oración está hablando de procedimiento médico Ilamado Prea- condicionamiento Isquémico (IPC).
This sentence talking about medical procedure called Ischemic Precondi- tioning (IPC). The proce- consists temporar- ily cutting off the flow blood certain parts the body and then return- ing the blood to the body. pro-
procedimiento consiste en cortar temporalmente flujo sangre ciertas partes del cuerpo y luego devolver la sangre cuerpo. EI objetivo este procedimiento es proteger el cerebro y sistema nervioso del daño.
The objective cedure is to protect the brain and the nervous sys- tem from damage.
Spanish
mT5 (r=0.5)
We included five RCTs re- porting arterial cannu- lations in paediatric partic- ipants.
incluyeron cinco ensayos controlados aleatorios que informaron 556 cannulaciones arteri- ales en los participantes niños.
randomized con- trolled trials reported 556 arterial cannulations in pediatric participants were included.
French
m'5 (r=0)
The 24.5 for the sponge and 10.9 for aphragm the UK trial.
Les étaient de 18,3 pour le diaphragme de 10,9 pour le diaphragme diaphragme UK l'essai.
18.3 for the diaphragm and 10.9 for the diaphragm with UK di- aphragm the trial.
The
English
m'5 (r=0)
We included three tri- als (involving 371 partic- ipants) in the analysis of the effects of RIC chaemic stroke prevention.
We included three trials with 373 participants and provided
trial with seven trials involving 373 participants.

---

Language
System
Input
Output
Gloss
French
mT5 (r=D)
The adverse events most commonly reported mild and transient such skin irritation, itching, burning, and stinging.
Les effets secondaires les plus fréquemment signalés étaient les plus graves transitoires la irritation cutanée le fuite d' d'eau, brülures, les vertiges vertiges.
The secondary effects most frequently reported the most serious and transitory: skin irritation and the water leak, burns, dizziness and dizziness.
Fnglishh
m'5 (i=0)
We searched the Cochrane Kidney and Transplant Specialised Register up 17 August 2017 through contact with the Infor- mation Specialist using search relevant this review.
We searched scientific databases for studies 27 March 2017.
French
m'15 (r=0.5)
We included seven trials, involving 735 participants, in this review.
Nous n avons trouvé au- cun essai, avec total de 535 participants.
We haven't found trial, with total of 535 partici- pants.
English
Flan-T5-ft (r=0.5)
Our results suggest, but do not confirm, that sible advantage of ultra- sound guidance for the first attempt success rate over other techniques is more pronounced in in- fants and small children than in older children.
Furthermore, ultrasound guidance is more accurate
and safer than alternative techniques for treating first tentative baby ear infec- tion.
English
GPT3 (0-shot)
The rates were 24.5 for the sponge and 10.9 for the di- aphragm in the UK trial.
In the UK trial, cost 24.5 pounds to buy sponge and 10.9 pounds to buy diaphragm.
Table
Extreme hallucinations. This category includes examples of hallucinations that extremely irrele- vant. Information hallucinated in the output sentences has no connection the information present the input. This degree of hallucinations was much more prevalent and extreme in the Flan systems as compared to other systems.
Language
System
Input
Output
Gloss
English
Flan-TS TS (0-shot)
Randomized controlled tri- als based uniform and standard methods needed.
The aim of the experi- ments is to give patients chance perform the periment they want do.
French
Flan-T5-ft (r=0.5)
In acute ischaemic stroke, for people receiving intra- venous thrombolysis, the rate of death depen- deney was significantly in- creased by RIC treatment compared with RIC treatment (RR 2.34; 95% 1.19 4.61; trial, 285 participants, low-quality evidence).
Pour les personnes vic- times d'un AVC nigu, les femmes admissées de greffe avait une probabil- ité aceruc d' d'être victimes de maladie grâce à traitement remmanant par injection d' une coagu- lation récurrente.
For the victims of acute stroke, the women admin- istered (?) the transplant had an increased probabil- of heing victims of the disease thanks manant (?) treatment by injection of recurrent coagulation.

---

Language
System
Input
Output
Gloss
Spanish
mT5 (r=0.5)
In people with carotid stenosis undergoing carotid stenting. there significant differ- in the incidence of ischacmic stroke between participants treated with RIC and non-RIC. (RR 0.22, CI 0.01 4.03; trial, 189 participants, Jow-quality evidence); however the stroke sever- ity (assessed by infarct volume) was significantly lower participants treated with RIC (mean diffcrence (MD) -0.17 ml., 95% CI -0,23 to -0.11; trial, 189 participants, low-quality evidence).
En los pacientes enfer- medad isquémica cuando acortan los brazos los pacientes tratados ENIC y ningún riesgo de caídas.
In the patients with ischemic disease, when shorted in the patients treated with ENIC and no risk of falls.
Table 18
This category includes examples where the output directly contradicts ideas stated in the input. This error was infrequent in the GPT3 systems but was equally prevalent in other systems.
Contradictions.
Language
System
Input
Output
Gloss
French
Flan-T5 (0-shot)
There was considerable heterogeneity between the studies included in this review and studies were small and at some risk bias
Il y a lieu d' éécantillonner des études incluant dans cette étude et les études étaient grandes et en hausse une hausse de la biase.
There a reason to sam- ple the studies included this study and the studies were big und increasing an increase of bias.
English
mT5 (r=0)
There is qualitative evi- dence that oral morphine has much the same effi- cacy as other available opi- oids.
It is not clear if oral mor- phine is effective other available opioids.
French
mTS (r=0.5)
people with acute chaemic stroke, there was significant difference between RIC non-RIC for reducing stroke sever- ity as assessed by the Na- tional Institutes of Health Strke Scale score the final infarct volume (stan- dardised mean difference (SMD) -0.24 mL. 95% CI -1.02 to 0,54; 2 trials, 175 participants, very low qual- ity evidence).
réduisait grav- ischémique, mais essais confondus avaient des preuves de très faible qualité.
reduced the severity of ischemic stroke, but the included trials have evidence very weak quality.
Spanish
mT5 (r=0.5)
Our results suggest, but not confirm, that a pos- sible advantage ultra- sound guidance for the first attempt success rate over other techniques more pronounced in in- fants and small children than in older children.
Los resultados indican que la gufa congráfíca redujo
The results indicate the ultrasound guidance re- duced success rate the broader trial in fants and the young chil- dren in comparison with other techniques.
tasa éxito en- sayo más amplia en los lactantes y los niños pe- queños en comparación con otras técnicas.

---

Language
System
Input
Output
Gloss
English
mT5 (r=0.5)
There is qualitative evi- dence that oral morphine has much the efti- cacy other available opi- oids.
The evidence for this topic is current December 2018 that oral morphine has serious toxicity different opioid.
Table 19
1,4 Simplification
Simplification errors included instances of the system being overly extractive, oversimplification of terms and ideas, and excessive deletion of entire parts of the original text.
Extractive. GPT3 (0-shot) was highly extractive and failed to explain complex medical language in everyday terms that general audience could understand. The system also left in statistics from the input sentence that are not interpretable to the general public. mTS (r=0) exhibited similar patterns with regards to extractiveness.
Language
System
Input
Output
Gloss
Spanish GPT3 (0-shot)
Interventions included:
Esta frase describe un estu- dio que se hizo para com- parar diferentes tratamien- tos para ayudar mejo- rar la circulación san- guínea. Estos tratamien- tos incluycron: terapia con tromboliticos en com- paración con un placebo (1 estudio); dosis baja en comparación con do- sis alta de trombolíticos (I); alteplasa compara- ción con uroquinasa (1): tiempo corto de tromboliti- cos en comparación con tiempo largo (1); terapia con tromboliticos com- paración con despeje de aguja (I); despeje de fibrina com- paración cambio de catéter con alambre (1): cambio de catéter alambre comparación con cambio sin despeje de fibrina con angioplastia (1).
This sentence deseribes a study that was conducted to compare different treat- ments to help improve the blood circulation. These treatments included: ther- apy with thrombolytics in comparison with a placcho (1 study); low dose comparison with high dose of thrombolytics (I); al- teplase compared to uroki- nase (1); short time of thrombolytics compared with long time (1); therapy with thrombolyties com- pared to clearance of tib- rin with needle (1); clear- ance of fibrin in compar- ison with wire catheter change (I); and wire catheter change compar- with change without clearance of fibrin with gioplasty (1).
thrombolytic therapy versus placebo (1 study); low versus high dose thrombolytic therapy (1); alteplase versus urokinase (1); short versus long thrombolytic dwell (1); thrombolytic therapy versus percutancous fihrin sheath stripping (I); fibrin sheath stripping versus over-the-wing catheter exchange (1); and over-the-wire catheter exchange versus exchange with and without angio- plasty sheath disruption (I).
fibrina
Spanish
mT5 (r=0)
On current data there evidence to support physical intervention the use of pharmaceutical agents in the acute setting,
Esta evidencia apoya intervención física sobre administración de agentes farmacológicos la prác- tica aguda.
This evidence does not support the physical inter- vention the adminis- tration of pharmaceutical agents the acute prac- tice.

---

Language
System
Input
Output
Gloss
French
GPT3 (0-shot)
In acute ischaemic stroke, for people receiving intra- venous thrombolysis. the rate of death depen- dency was significantly ereased RIC treatment compared with non-RIC treatment (RR 2.34; 95% 1.19 4.61; trial, participants, low-quality evidence).
En de crise d'ischémie aiguë, pour les personnes qui ont reçu traitement par thrombolyse par voie intraveineuse. le taux de décès été significativement aug- menté par traitement RIC par rapport au ment RIC (RR 2,34; 4,61; essai, 285 participants, preuves faible qualité).
In the case of acute is- chemic crisis, for the peo- ple who have received treatment by intravenous thrombolysis, the rate of death dependence was significantly increased by the RIC treatment parison to non-RL RIC treatment (RR 2.34; 95% to 4.61; trial, participants. evidence weak quality).
dépendance
Table 20
Oversimplifictionn Flan-T5-ft (r=0.5) had issues with excessive deletions, resulting in fragmented sentences that failed to communicate the meaning of the input sentence. Excessive deletion in mT5 (r=0) and m15 (r=0.5) manifested as loss of information that was necessary to preserve the meaning of the input sentence. The resulting outputs were missing critical information, causing them to be generic and vague, such in the Spanish m'T5 (r=0.5) example that discusses the of RIC for ischemic stroke prevention.
Language
System
Input
Output
Gloss
French
Flan-T5-ft (r=0.5)
We rated the quality of ev- idence for all outcomes moderate owing to impre- cision due wide con- fidence intervals, modest sample sizes and limited numbers of events.
La qualité des preuves
The quality of the evi- dence
French
m'T5 (r=0)
Four RCTs compared ul- trasound with palpation, and one compared ultra- sound with Doppler audi- tory assistance.
Cing ECR ont comparé l'échographie avec une palpation.
Five RCTs have compared ultrasounds with palpa- tion.
French
mTT5 (r=0.5)
We included three tri- als (involving 371 partic- ipants) in the analysis of the effects of RIC on is- chaemic stroke prevention.
Les auteurs de la revue ont inclus un total de 371 par- ticipants.
The authors of the review included total of 371 pur- ticipants.
Spanish
mT5 (r=0.5)
The quality the evi- dence was rated as high.
Calidad evidencia
Quality of evidence
Spanish
mTS (r=0.5)
We included three tri- (involving partic- ipants) analysis effects of RIC is- chaemic stroke prevention.
incluyeron ensayos que incluyeron 372 par- ticipantes.
Three trials with 372 par- ticipants were included.
English
m'55 (r=0)
serious ad- verse events related to ther nitrous oxide bon dioxide pneumoperi- toneum (three studies; participants; very quality evidence).
There
There serious ad- verse events.

---

Language
System
Input
Output
Gloss
English
mT5 (r=0.5)
We reran the searches for all databases in March
The search for most sci- entific databases March 2016.
2016, and when up- date the review, will deal with the studies of interest found through this search that await- ing classification.
Table 21
1,5
Other Errors
Repetition of output (GPT3 (0-shot)). The GPT3 (0-shot) system had unique error that exclusively appeared in the French outputs where separate paraphrase was generated in addition to the main output sentence. The system sometimes gave generated output, then repeated the same information in different words using the phrase 'in other words' or simply generating two different versions of simplified output.
Language
System
Input
Output
Gloss
French
GPT3 (0-shot)
All but one trial were at high risk of bias.
Tous les essais sauf un étaient haut risque de bi- ais. En d' autres mots, la plupart des essais présen- taient un risque élevé d' être hiaisés.
All the trials except one were high risk of bias. In other words, the major- ity of the trials presented elevated risk of being biased.
French
GPT3 (0-shot)
We rated the quality of ev- idence for all outcomes as moderate owing to impre- cision due to wide con- fidence intervals, modest sample sizes and limited numbers of events.
Nous avons estimé que qualité de la preuve pour tous les résultats était mod- êréc en raison l'une im- précision due à des inter- valles de confiance larges, des tailles d' E'échantillon modestes et un nom- bre limité d' d'événements. Nous avons estimé que la preuve était très pré- cise parce que les inter- valles de confiance étaient grands, les échantillons pas très grands il y avait peu d'ééénements.
We estimated that the qual- ity of the evidence for all the results was moderate on account of an impre- cision due to large confi- dence intervals, the size of the modest samples and to limited number of events. We estimated that the evi- dence wasn't 't very precise because the confidence in- tervals
big, the not very big samples and there were few events.
Table 22
Acronyms. Across all systems, a frequent issue that arose when simplifying medical text was acronyms. These issues can further broken down into failure to translate into the target output language, halluci- nated acronym generations, and factually incorrect generations.
Acronym translation. Systems failed to translate the input sentence's acronym from English into Spanish or French. GPT3 (0-shot) would explicitly state that the acronym was in English, rather than translating, in some cases.

---

Language
System
Input
Output
Gloss
Spanish
GPT3-simp-trans
To the benefits and harms of RIC for pre- venting ischaemic stroke and for treating people with ischaemic stroke and those at risk for ischaemic stroke.
Esta oración pide que averiguemos si cierto tipo de tratamiento médico (RIC, por sus siglas in- glés) útil para prevenir los accidentes cere- brovasculares, Queremos saber los heneficios de usar RIC son mayores que los riesgos.
This sentence asking find out if certain med- ical treatment (RIC, its acronym English) ful prevent and treat strokes. We want know benefits using RIC greater than the risks.
French
GPT3 (0-shot)
We included randomised controlled trials (RCTs) comparing RIC with sham RIC medical manage- in people with is- chaemic stroke risk of ischaemic stroke.
Nous inclus des sais contrôlés randomisés (RCT) pour comparer le RIC
We have included domly controlled trials (RCT) to compare the RIC with fake RIC medi- cal management in people that suffered stroke risk stroke.
faux RIC gestion médicale chez les personnes ayant subi un accident vasculaire cérébral ou risque d'accident vasculaire cérébral.
French
Flan-T5 (0-shot)
To the benefits and harms of RIC for pre- venting ischaemic stroke and for treating people with ischaemic stroke and those at risk for ischaemic stroke.
évaluer avantages les éléments de la RIC de supprimer le stroke in- sécutif duire les risques lićes l'insécutif.
To evaluate the advantages and the elements of RIC in order tif stroke reduce the risks associ- ated with the insécutif (?).
the insécu- order
ré-
Table 23
Acronym hallucination.
Flan-T5-ft (r=0.5) hallucinated acronyms in the output sentence that do not This occurred across all languages and in both Flan and mT5 systems.
exist in the target language.
Language
System
Input
Output
Gloss
English
Flan-TS-ft (r=05)
In acute ischaemic stroke, people receiving intra- venous thrombolysis, the rate of death depen- dency was significantly in- creased by RIC treatment compared with non-RIC treatment (RR 2.34; 95% 1.19 to 4.61; trial, 285 participants, low-quality evidence).
For acute ischaemic stroke, people receiving intra- venous thrombolysis, the rate of death depen- dency was significant in- creased by RAC treatment compared with non-RAC treatment (low-quality evi- dence).
French
Flan-T5-ft (r=055)
Preclinical studies have suggested RIC may beneficial effects ischaemic stroke patients and those at risk is- chaemic stroke.
Les résultats de cette vue ont suggéré que les BRA pourrail avoir fets bénéfiques dans pa- tients souffrant d' PIO et chez personnes ayant un accident cérébral is- chémique.
The results this review suggested the BRA could have beneficial fects in patients suffering had an ischemic stroke.
people who

---

Language
System
Input
Output
Gloss
French
m55 (r=0.5)
In people with acute is- chaemic stroke, there significant difference between RIC and non-RIC for reducing stroke ity assessed by the Na- tional Institutes of Health Stroke Scale score and the final infaret volume (stan- dardiscd mean difference (SMD) -0.24 mL, CI -1.02 to 0.54; trials, participants, very low qual- ity evidence).
La SEP réduisait la grav- ité du AVC ischémique, mais les essais confondus avaient des preuves de très faible qualité.
The SEP reduced the severity of ischemic stroke, but the included trials have evidence of very weak quality.
Table 24
Acronym factuality. Flan-T5 (0-shot) and mT55 (r=0) produced factuality issues in the output sentences. In the Spanish output example, Flan generated the acronym for AIDS (SIDA), though the input was referring to randomized controlled trials (RCTs), In the French example, mT5 (r=0) generated the acronym for stroke (AVC), but the input sentence refers to cardiovascular disease (CVD).
Language
System
Input
Output
Gloss
Spanish
Flan-T5 (0-shot)
Eight RCTs with 733 women in total that com- pared brief co-incubation and the standard insemi- nation protocol were in- cluded.
Ocho estudios compararon la co-incubación breve con un protocolo de inserción estándar en pacientes con SIDA total.
Eight studies compared the brief co-incubation with a protocol of standard insertion in patients with AIDS in total.
French
mTS (r=0)
Stress and sedentary lifestyle are major deter- minants of cardiovascular disease (CVD).
Le stress et règlement inactif sont des causes im- portantes de maladies car- diovasculaires (AVC) les plus courants.
Stress and inactive regula- tion are the most common important causes of cardio- vascular disease (AVC).
Table 25
Input hallucination (GPT3). The systems GPT3-simp-trans and GPT3 (0-shot) show some minor hallucination by referencing the input sentence in the output. Some output generations would start with "this phrase' or 'this sentence' in talking about the input. This error not found in any of the other systems.

---

Language
System
Input
Output
Gloss
English
GPT3 (0-shot)
There no significant difference between RIC and non-RIC for improv- ing the psychological impairment (SMD -0.37 points, 95% CI -1.15 to 0.41; trial, 26 partici- pants, very low quality evidence) tive impairment (SMD -0.26 points: 95% -0.72 to 0.21; trials, 79 participants, low-quality evidence) in people with acute ischaemic stroke and cerebral small vessel discase.
This sentence is talking about two different groups of people who had strokes: those who received type of treatment called RIC (RIC stands Rehabilita- tion Intervention Cere- brovascular Disease) and those who did not. It talking about how treatment affected their psychological and cogni- tive health, The results showed that there was not much difference hetween the two groups in terms of improving the psychologi- cal and cognitive health of the people with strokes.
cogni-
French
GPT3 (0-shot)
people with carotid stenosis undergoing carotid stenting, there was no significant differ- ence in the incidence of ischaemic stroke between participants treated with RIC and non-RIC (RR 0.22, 95% CT 0.01 to 4.03; I trial, 189 participants, low-quality evidence); however the stroke sever- ity (assessed by infarct volume) was significantly lower in participants treated with RIC (mean difference (MD) -7.17 mL, 95% CI -0.23 to -0.11; I trial, 189 participants, low-quality evidence).
Cette phrase parle d'une étude qui a été faite sur des personnes souffrant de sténose carotide et qui ont été traitées par stent carotidien. L'étude montre qu'il n'y pas de différence significative entre les personnes traitées par RIC (Revascularisa- tion Intracarotidienne) et celles qui ne l'ont pas été, au niveau du risque accident vasculaire cérébral. Cependant, les personnes traitées par RIC ont présenté une sévérité moindre des accidents vasculaires cérébraux (mesurée par le volume d' l'infarctus).
This phrase talks about study that was done on people suffering from carotid stenosis and who were treated by carotid stenting. The study shows that there isn' a significant difference between peo- ple treated by RIC (Revas- cularization Intracarotid) and those who weren't, t, at the level of risk of stroke. However, the peo- ple treated by RIC pre- sented a lessened severity of strokes (measured by the volume of infarction).
Spanish
-PT3-simp-trans
Three trials randomised participants to nitrous oxide pneumoperitoneum (100 participants) or carbon dioxide pneu- moperitoneum (96 participants).
Esta oración describe un experimento científico que probó dos tipos diferentes de tratamientos médicos en dos grupos de personas. El primer grupo tenía 100 personas y se les dio un tratamiento con éxido ni- troso. El segundo grupo tenía 96 personas se les dio un tratamiento con dióxido de carbono.
This sentence describes scientific experiment that tested two different types of medical treatment in two groups of people. The first group had 100 peo- ple and they were given treatment with nitrous ox- ide. The second group had 96 people and they were given a treatment with car- bon dioxide.
Table 26
J ANNO-VIEWER
SIEWER is inspired by a similar annotation tool that was used in Jiang et al. (2020) for correcting crowdsoureed alignment labels used for training and evaluating neural CRF model for aligning sentences. The primary function of this tool is to enable annotators to make alignments efficiently. ANNO-VIEWER also enables annotators to make factuality annotations for alignments. These annotations are largely based from the factuality annotations used in Devaraj et al. (2022), and also includes an additional field annotate for elaborations in a similar manner.
This annotation tool also has the additional functionality of allowing annotators to use existing similarity

---

Alignment Viewer
Browso... Reannotated.o
Anticarty
Downloed
Article79
Insetions:
Deletor:
Subetrution
Caborationi
SWC
Complex
Simple
tackground
Limbor supports an used the ircotment low back pain panaets, to prevent the onnat of loe zase pai farmary
Lumbor susperie jain called araces oe coroety used the provension and teatmment low back pain
provaniian) in prevent
of lon bock owin spiiccdke
bersuet
pat't stukiyy lears the experou treatment and sikk loose due lack pain.
Objectives
Vi rncl recuded saven studkiee on prosention (14-427 people) and right studies treatmentt (120) peopli this reviest
scseet the attrcts of unber supporte for preceetion and traetment of non lack pains.
Prevendor:
titll diftorence
individual
supporte
rsseiee tractraatt thee stucke, 13,005 peoplis
on ffting techtiques tawo stadies, 954 pooplet back pain prwertionn or reduction odk lose people, back toatien about chdtyo of
We updosed the seardh the Cochrane Contral Rto Controlve ind Traie PunMed, EVRASE, and CINAALL Droemser 2006.
adend back school
from
Alignment Viewer
Up
Dlwnload
Article79
Insetion:
Deltion:
Subsitutierd
Babonation
Complex
Simple
Background
NULL
uerierr supporta are used in the treamment lose back pan pateeess to grevent the onse ioe hark pain grrinery preventint
turiaa supports niioo called braces corsens are used
back
low- cro-hack
rixdd
Seourskyy
sapports mmponaed
massage
enothe stuzy peosiid, of lumber cornet
Chijectiven
lacc wapport wao mors offecsive velusing paino the
sharsdeem than onroet sine
To soees the effect lumber tu supoorts for prevention and of speciii low back
Prevention: There
diflerence beoweer
indiiidu's low-lteck
poop of education litinng tadiniquss 954 poop di back pain prevention reduction of sidk leave
os,
upoiandd souch Cartrane Cenrral Regiow
Contiolldd hrin,, PutkWrd IMEAASE and crNWil December
Treme sonficting denur livossted 590 peoplel
2006.
whether bork
hetter than nothing
screeed ieleaat
Figure Screenshots of ANNO-VIEWTR The sentence sorting tool is featured bottom image.
measures to help find sentence alignments faster. When a sentence is selected for alignment, the tool can sort the other document by which sentences are most similar (determined by the automatic similarity measure) to the selected sentence.


Reviews:
[{"review_id": "r2OkEbMlQp", "reviewer": "Reviewer_XHER", "summary": "", "questions": "A. What predictions do you have about the results for non-Indo-European languages, such as Chinese or Japanese, where sentence alignment is more complex? \n\nB. It is mentioned that sentence-level simplification has some limitations, especially in the medical field, where contextual information is important. How could the paragraph or discourse be incorporated into the alignment methodology? \n\nC. How could insights from human evaluations be incorporated into the evaluation metrics of the models?  \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n**Re languages:** yes, exploring the effect of the imbalanced distribution and including more languages are excellent directions for future work!\n\n**Re automatic evaluation metrics:** we absolutely agree! One takeaway from this paper is a call for the development of more robust automatic metrics.\n\n**Re quality degradation in other languages:** we believe that this points to the weakness in LLMs that claim to be multilingual, hence revealing a research gap in the training and evaluation of LLMs.\n\n**Re A:** The amount of data collected for the other languages was significantly lower compared to the amount we collected for the languages we used for evaluation. This, plus potentially noisier alignments with those languages, will likely lead to worse results. However, the other factor is also whether that language is high-resource in the training data of the underlying multilingual LLMs, which is a separate distribution than the amount of data available in Cochrane (e.g., Cochrane has more Farsi data, but that may not be true when the underlying LLM was trained in the general domain). As you mentioned in your review, extending our evaluation to non-Indo-European languages is definitely a worthy future direction. As a first step in this direction however, we decided to focus our evaluation to the languages that had the most data available due to resource and time constraints.\n\n**Re B:** While the CRF method we used for alignment is contextual, we had null-alignments when sentences were deleted from the original text or added to the simplified text. Thus, future efforts for alignments could consider the role of such deletions and elaborations in the discourse, and incorporate those into the alignment itself.\n\n**Re C:** The human evaluation results show that automatic metrics like BLEU and SARI lack the level of understanding required to reliably evaluate simplifications. Metrics for evaluating simplifications need to be able to better assess factuality and simplicity.\n\n\n**Re intellectual property:** the authors have had a fair use discussion with our institution library and copyright office about this: the research use of publicly available abstracts, including both technical abstracts and multi-lingual plain language summaries, is considered fair use. This work does not use any full texts of Cochrane reviews. Note that the Cochrane dataset, in similar capacity as ours, has already been released publicly by prior work (Devaraj et al 2021 “Paragraph Level Simplification of Medical Texts”, Guo et al 2021 “Automated Lay Language Summarization of Biomedical Scientific Reviews”). Our institution library recommends the Creative Commons license for this work. We will clarify this in the paper as well as the repo for data release.\n\n**Re annotator compensation:** All annotators are hired as hourly research assistants by our institution, with a pay of $15/hr. We will add this information to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The most important contributions of the paper:\n\n1. Advancing Multilingual Medical Text Simplification: The paper addresses the limitation of existing work, focusing solely on monolingual text simplification. It presents MULTICOCHRANE, the first parallel dataset for medical text simplification across 4 Indoeuropean languages, English, Spanish, French, and Farsi, with a range of speakers from 80 million to more than 400 million. This advancement enables the direct simplification of complex texts into target languages.\n\n2. Dataset Creation: The authors introduce the MC-CLEAN subset, consisting of 101 technical abstracts with expert-annotated manual alignments in English, then semi-automatically aligned to other languages, totalling around 5,000 sentence pairs across all four languages. They also present the MC-NOISY subset, a larger but noisier dataset created using an automatic sentence alignment model trained on MC-CLEAN, containing approximately 100,000 sentence pairs across languages.\n\n3. Systematic Evaluation of Simplification Models: The authors evaluate various large pre-trained language models in zero-shot and fine-tuned settings on text simplification across four languages.\n\n4. Human Assessments: The study includes human assessments covering simplicity, fluency, and factuality, in addition to automatic evaluations. The correlation between automatic metrics and human assessments is analysed, revealing mostly weak correlations.\n\n5. Identifying Model Performance Differences: The results show that while pre-trained models effectively simplify English, their performance significantly degrades in other languages, leading to factuality errors. Different models, such as GPT-3 (zero-shot) and Flan-T5 (zero-shot), exhibit varying strengths and weaknesses when simplifying to different languages.\n\n6. Availability of Resources: The authors publicly release MULTICOCHRANE, model outputs, and all human judgments collected during the study, motivating future research on multilingual medical text simplification to enhance model performance across those four languages.\n\nIn conclusion, this study contributes to multilingual medical text simplification by introducing a novel and evaluated dataset, providing insights into model performance across languages, and releasing valuable resources for further research and development in this area. However, it also acknowledges certain limitations, such as uneven distribution of multilingual versions for articles and potential drawbacks of sentence-level simplification in the medical domain. The discussion of the results of the evaluation of the models is very relevant and contributes to the final quality of the article.\n\nThe relevance of the appendices should also be mentioned: they convincingly demonstrate the quality of the work.\n", "reasons_to_reject": "There are no strong reasons against the paper, only suggestions for improving the resource and the methodology.\n\n\n* Limitation of Languages: Although MULTICOCHRANE covers several languages (English, Spanish, French, and Farsi), it is acknowledged that there is an uneven distribution of resources, and some languages, such as Farsi, may have less coverage of multilingual articles. This could bias the results towards languages with more resources and affect the representativeness. It would also be interesting to extend the approach to other non-Indo-European languages, such as Chinese and Japanese, to compare results.\n\n* Weakly Correlated Human Evaluation: It is mentioned that the correlation between automatic metrics and human evaluations is mainly weak. This could suggest that the automatic metrics used to evaluate the models do not fully capture the quality of the simplifications, raising questions about the validity and robustness of the conclusions. Let us not forget that human evaluations of the quality of the results are more important.\n\n* Quality Degradation in Other Languages: The results show that pre-trained models perform significantly less when simplifying texts in languages other than English, introducing accuracy errors. This quality degradation could affect the models' practical utility in multilingual applications.\n\nOverall, these problems may suggest that the study has certain limitations, especially in practical application. Still, the quality of the analysis of the evaluations and the methodology employed is beyond doubt.\n", "questions_to_authors": "A. What predictions do you have about the results for non-Indo-European languages, such as Chinese or Japanese, where sentence alignment is more complex? \n\nB. It is mentioned that sentence-level simplification has some limitations, especially in the medical field, where contextual information is important. How could the paragraph or discourse be incorporated into the alignment methodology? \n\nC. How could insights from human evaluations be incorporated into the evaluation metrics of the models?  \n", "paper_topic_and_main_contributions": "The paper falls under the \"new data resources\" topic. \n\nThe main hypothesis emphasises the importance of simplifying complex medical texts to make them understandable for the general public, facilitating access to critical medical information.\n\nText simplification models have emerged to address this issue, automatically transforming complex texts into simpler versions that readers can understand without specific healthcare training. However, these models have focused solely on simplifying texts in a single language. This limits the availability of simplified information in multiple languages, especially in the medical domain, leading to inequalities for those who do not speak English and face barriers to accessing information.\n\nThe paper's primary contribution is presenting MULTICOCHRANE, the first parallel dataset for medical text simplification across multiple languages: English, Spanish, French, and Farsi. MULTICOCHRANE contains aligned sentence pairs sourced from the Cochrane Library of Systematic Reviews, a repository of meta-analyses of treatment effectiveness. These review articles consist of both technical abstracts and plain-language summaries (PLS), from which we derive two subsets of MULTICOCHRANE:\n\n* MC-CLEAN contains 101 technical abstracts with manually aligned sentence pairs by experts.\n* MC-NOISY, a larger but noisier subset created using an automatic sentence alignment model.\n\nThe authors evaluate various pre-trained language models on text simplification in four languages and conduct human evaluations regarding simplicity, fluency, and factuality. The results demonstrate that while pre-trained models effectively simplify English, their performance significantly degrades in other languages, leading to factuality errors. Additionally, the approach of translating English simplifications to other languages is analysed, which, in many cases, can circumvent these issues.\n\nOverall, this resource is novel, multilingual, and thoroughly evaluated, and it can help stimulate further efforts to improve the performance of models across different languages.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The literature review is correct and up to date. \n", "Typos_Grammar_Style_and_Presentation_Improvements": "Overall the paper is well-written and well-structured. \n"}, "extra_text": {"Justification_for_Ethical_Concerns": "Following the Ethics Review questions for papers presenting new datasets, the authors should address these questions in their paper:\n\n1. Describe how intellectual property (copyright, etc) was respected in the data collection process.\n\n2 Describe how crowd workers or other annotators were fairly compensated and how the compensation was determined to be fair."}}, {"review_id": "GSHSs7KmoD", "reviewer": "Reviewer_gRpt", "summary": "", "questions": "Other important missing points\n\n- the copyright status of the source material is not discussed. AFAIK, Wiley allows text mining, but redistributing a dataset built from Cochrane is another thing.  \n\n- The profile of the annotators involved in this work (alignement, evaluation) needs to be described (medical background, language fluency…). For the evaluation, a medical background is expected to check the factuality of the output. Yet, to evaluate the simplicity, one would expect someone with no medical background.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n**Re simplify-then-translate systems:** We do evaluate the simplify-then-translate pipeline using GPT-3 and Google Translate and it does tend to outperform the multilingual pipeline in our evaluation. Our methodology is described in section 5.1 under zero-shot models, and the results can be found in Table 4 and Table 5 with the system name GPT_trans. However, the motivation for preferring the multilingual pipeline over the simplify-then-translate pipeline stands: Simplify-then-translate requires two models, and is more expensive (when scaled up) in terms of cost and compute compared to the multilingual pipeline. The multilingual pipeline also addresses some of the robustness issues associated with machine translation as described in the papers we cited. Ruder et al. (2021) described the poor performance of MT models in low-resource languages, but it also describes poor performance in languages with non-Latin scripts. Vu et al. (2022) discussed how using machine translation along with tasks like cross-lingual summarization leaves significant headroom for improvement. Based on this, we believe the multilingual pipeline has a greater potential for improvement compared to the simplify-then-translate approach. This paper is a step in the direction of solving what we see as an open problem. \n\n**Re atomic unit sentences:** The main reason for this choice was an effort to try to constrain the models to generate sentence-to-sentence simplifications. Accounting for the splitting and the grouping could have caused models trained on these alignments to generate variable length simplifications, which would have been hard to evaluate against gold labels. This combined with the possible complications from using multilingual data, made the sentence-level approach a reasonable starting point from our view. However, as we explain in the limitations section, we do think that working on larger units makes sense.\n\n**Re Copyright status:**  the authors have had a fair use discussion with our institution library and copyright office about this: the research use of publicly available abstracts, including both technical abstracts and multi-lingual plain language summaries, is considered fair use. This work does not use any full texts of Cochrane reviews. Note that the Cochrane dataset, which contains only abstracts, was released publicly by prior work (Devaraj et al 2021 “Paragraph Level Simplification of Medical Texts”, Guo et al 2021 “Automated Lay Language Summarization of Biomedical Scientific Reviews”). Our institution library recommends the Creative Commons license for this work. We will clarify this in the paper as well as the repo for data release.\n\n**Re annotator profile:** The annotators were all fluent in the language they were assigned to annotate. They did not have medical backgrounds, but they did have a strong background in research and in evaluating technical texts. Since factuality was assessed by determining how factually consistent the simplification was to the original input, a medical background was not a necessity. If the annotators could comprehend both the input and the simplification, expert knowledge is not really required to make such an assessment. Annotators were also always encouraged to factcheck anything they were unsure about. We will add this information to the paper in future drafts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Positive points\nThe comparison of the automatic and human-based metrics is enlightening.\nThe proposed resource may be useful for further research. \nThe proposed approach to build the corpus, combining existing techniques and leading to a gold corpus and a silver one, is sound.\n", "reasons_to_reject": "Negative points\n\n- the whole problem is ill-justified. The simple approach consisting in simplifying and then translating (one model to simplify from En to En, and a translation model from En to the target language) is rejected in a footnote, stating that some languages are not correctly dealt with by translation systems. Yet, the whole paper only considers very well covered languages (with the exception of Farsi maybe, although it is not a low resource language), and systems (GPT, T5…) that were trained on mostly English and some major languages.\n\n- Following the previous point: no comparison is made with such simplify-and-then-translate systems. The results of GPT3_zero on En-to-En tend to indicate that translating the output would certainly be a good solution.\n\n- As noted by the authors, taking the sentence as the atomic unit is not satisfying; simplifying may imply splitting or grouping sentences. What is the reason of this choice? Modern models, such as LLM, can process texts with more context than sentence by sentence.", "questions_to_authors": "Other important missing points\n\n- the copyright status of the source material is not discussed. AFAIK, Wiley allows text mining, but redistributing a dataset built from Cochrane is another thing.  \n\n- The profile of the annotators involved in this work (alignement, evaluation) needs to be described (medical background, language fluency…). For the evaluation, a medical background is expected to check the factuality of the output. Yet, to evaluate the simplicity, one would expect someone with no medical background.\n", "paper_topic_and_main_contributions": "This paper deals with text simplification in the medical domain, with the source text and the simplified text being in different languages. The authors propose a new corpus of paired sentences sourced from the Cochrane Library (English to another language among Spanish, French, Farsi).\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\nr is presented as a function (ratio of overlapping tokens…) and a threshold with a fixed value. Give a different name to the 2 objects.\n"}, "extra_text": {"Justification_for_Ethical_Concerns": "The copyright status of the collected/distributed data from the Cochrane library is not discussed in the paper. The authors should simply indicate that they got Wiley’s authorization to build and distribute their corpus."}}, {"review_id": "AaCuG3iQ5r", "reviewer": "Reviewer_dMiC", "summary": "", "questions": "\n1. Page 4, line 240: Please provide agreement also in terms of the Kappa coefficient.\n\n2.- Page 4, lines 266-269: Please better explain this, that even though 60% of the sentences were removed, \"most\" of the information remained. How was this actually evaluated? Further, in lines 289-280, it is strange that the \"simplified\" sentences are actually larger than the original (complex) ones. Usually, longer sentences are more difficult to read than shorter ones. I understad that the simplified text can be longer, but I'd actually expect the sentences to be shorter.\n\n3. I wonder why the authors did not try any LLM speciafically built for the medical (or biomedical) domain, such as SciFive. And I miss details about the hyperparameters that were used in the experiments. Will the source code be made available as well?\n\n* minor comments:\n\n- Page 3, line 16: I wonder if one of the difference to MUSS is that this was not built for the medical domain?\n\n- Page 3, lines 225-231: 25+75=100, not 101 as stated earlier in this section.\n\n- Page 4, line 321: Was \"DP\" previoulsy defined?\n\n- Page 5, line 383: Did you try other values for the threshold?\n\n- Page 6, lines 467-469: I think that this statement is only valid for the BLEU scores.\n\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review!\n\n**Re Question 1:** We have calculated the Krippendorff’s Alpha in regards to your question to be 0.662. We will add this to the paper. \n\n**Re Question 2:** We agree that we worded this particular sentence poorly.  We had meant to emphasize that the simple texts were a more condensed version of the complex texts. As for why simplified sentences may be longer than the complex sentence, this happens usually because technical sentences use jargon for brevity which has to be explained or elaborated upon in the simplified version. This usually involves succinctly explaining a certain disease, treatment, or research method to make the text more comprehensible to a lay reader. \n\n**Re Question 3 & LMs for medical domain:** At the time of our evaluation, we had not come across any medical-specific LLMs that were of comparable performance and had the same level of multilingual capabilities to the LLMs we chose to evaluate.\n\n**Re Question 3 & Hyperparameters:** We do provide hyperparameters for the experiments we performed under section 5.1 under Training Setup. Additionally, the source code will be provided in a github repo along with the dataset.\n \n**Re minor comments:** Thank you for these comments. The 75 was supposed to be a 76. We seemed to have overlooked this mistake, so thank you for catching it. We’ll make sure to fix these in the camera-ready version. As for your comment on page 5, we did assess several threshold values and observed a tradeoff between the noisiness and extractiveness of the resulting data, which is why we settled on the value of 0.5. As for your comment on page 6, we also observed an improvement in the BERTScore. However, we do concede that filtering did have a negative impact of the simplicity measure (SARI and LCS).\n\n**Re ethical questions:** \n\nWe certainly agree that in this domain it necessary to be especially cognizant of the risks posed by  potential factual errors and hallucinations. Nonetheless, following a body of existing work on medical text simplification and/or summarization, we believe that this line of work is important both to advance technology for social good while also acknowledging the risks: (1) Inaccessibility of medical information causes problems in health literacy, which is one of the leading reasons for real health consequences including more hospital admissions, emergency room visits, and poorer overall health. Making medical information accessible is one of the best ways to tackle health literacy, and that is the core of what a simplification system is aimed to do. (2) Medical misinformation is one of the most series issues highlighted in the COVID-19 pandemic; one contributing reason for this is the lack of health literacy among the general public. By simplifying trustworthy evidence, we hope to empower the public with a keener eye for such misinformation. (3) Factual errors is one of the key aspects studied in this work; we perform a thorough evaluation dissecting issues that can come from these models, especially in a multilingual setting. We believe rigorous evaluation, as done in this work, is one of our best tools to demystify language models, and help the community understand the issues at hand. With such understanding, we hope to point to future directions that collectively, we will be able to provide factual, readable, multilingual access to medical texts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The construction of a new resource that will be helpful for text simplification in the medical domain.\n- Comprehensive experiments with the dataset, a good discussion of the results, and a short error analysis.\n- Availability of the dataset and human evaluation.", "reasons_to_reject": "- Lack of experiments with language models which were specifically built for the medical domain.\n- Lack of details about the experiments, e.g., hyperparameters.", "questions_to_authors": "\n1. Page 4, line 240: Please provide agreement also in terms of the Kappa coefficient.\n\n2.- Page 4, lines 266-269: Please better explain this, that even though 60% of the sentences were removed, \"most\" of the information remained. How was this actually evaluated? Further, in lines 289-280, it is strange that the \"simplified\" sentences are actually larger than the original (complex) ones. Usually, longer sentences are more difficult to read than shorter ones. I understad that the simplified text can be longer, but I'd actually expect the sentences to be shorter.\n\n3. I wonder why the authors did not try any LLM speciafically built for the medical (or biomedical) domain, such as SciFive. And I miss details about the hyperparameters that were used in the experiments. Will the source code be made available as well?\n\n* minor comments:\n\n- Page 3, line 16: I wonder if one of the difference to MUSS is that this was not built for the medical domain?\n\n- Page 3, lines 225-231: 25+75=100, not 101 as stated earlier in this section.\n\n- Page 4, line 321: Was \"DP\" previoulsy defined?\n\n- Page 5, line 383: Did you try other values for the threshold?\n\n- Page 6, lines 467-469: I think that this statement is only valid for the BLEU scores.\n\n\n", "paper_topic_and_main_contributions": "\nThe authors constructed the first multiligual simplication corpus for the medical domain. It includes four languages and consist of a clean (manually validated) and noisy (automatically compiled) datasets. Experiments with the datasets have been carried out using zero-shot and fine-tuning. A sample of the generated simplifed texts was manually chaked by exeprts. They provide a good discussion of the results and will make both the datasets and the human evaluation available for download.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Justification_for_Ethical_Concerns": "Since this is a work about text generation (text simplification) for the medical domain, it is important to discuss if the generated text could somehow cause any damages to the user, given that the users are inteded to be lay persons."}}]

Author rebuttal:

Meta-review: Review Summary:
Reviewers agree that the paper advances multilingual medical text simplification, creates a new dataset, and evaluates various pre-trained language models in zero-shot and fine-tuned settings are positive contributions. Some reasons to reject this paper are stated including not using language models specifically trained for the medical-domain, poor justification of approach, poor human evaluation quality, degradation of performance on certain languages, and lack of detailed information regarding experiments. In addition, since this paper presents a generative approach for the medical domain, it raises additional ethics consideration including the intellectual property regulations, annotators compensation, etc

Reasons to Accept:
(1) The paper makes significant contributions to multilingual medical text simplification, introducing the MULTICOCHRANE dataset for four Indo-European languages.
(2) It presents a novel approach to creating the dataset, including MC-CLEAN and MC-NOISY subsets, which can be valuable resources for further research.
(3) The systematic evaluation of simplification models in both zero-shot and fine-tuned settings across multiple languages provides insights into model performance and challenges.
(4) The inclusion of human assessments alongside automatic evaluations, and the analysis of their correlation, enhances the understanding of model performance.
(5) The paper identifies differences in model performance when simplifying to different languages, shedding light on potential quality degradation.
(6) The release of MULTICOCHRANE, model outputs, and human judgments promotes future research in multilingual medical text simplification.

Reasons to Reject:
(1) While there are no strong reasons to reject the paper, some suggestions for improvement include addressing the limitation of uneven language distribution in the dataset, exploring the use of the simplify-and-translate approach for comparison, and considering more context-aware approaches instead of sentence-level simplification. Additionally, providing more experimental details and experiments with domain-specific language models could enhance the paper's quality.

The author rebuttal have tried to address these issues.
Decision: Accept-Main