FACTPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence (2024)

Abstract: Plain language summarization with LLMs can be useful for improving textual accessibility of technical content. But how factual are these summaries in a high-stakes domain like medicine? This paper presents FACTPICO, a factuality benchmark for plain language summarization of medical texts describing randomized controlled trials (RCTs), which are the basis of evidence-based medicine and can directly inform patient treatment. FACTPICO consists of 345 plain language summaries of RCT abstracts generated from three LLMs (i.e., GPT-4, Llama-2, and Alpaca), with fine-grained evaluation and natural language rationales from experts. We assess the factuality of critical elements of RCTs in those summaries: Populations, Interventions, Comparators, Outcomes (PICO), as well as the reported findings concerning these. We also evaluate the correctness of the extra information (e.g., explanations) added by LLMs. Using FACTPICO, we benchmark a range of existing factuality metrics, including the newly devised ones based on LLMs. We find that plain language summarization of medical evidence is still challenging, especially when balancing between simplicity and factuality, and that existing metrics correlate poorly with expert judgments on the instance level. FactPICO and our code is available at https: //github.com/lilywchen/FactPICO.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8437–8464 August 11-16, 2024 ©2024 Association for Computational Linguistics
FactPICO and our code is available at https: //github.com/lilywchen/FactPICO.

heading: 1 Introduction
text: New findings in medicine observed in randomized controlled trials (RCTs) are published in journal articles which describe their design and outcomes. These RCTs “measure the effectiveness of a new intervention or treatment” (Hariton and Locascio, 2018) and are the important basis of evidencebased medicine (Sackett et al., 1996). However,
*Equal Contribution
understanding such articles requires “specific attention outside of general literacy capacities” (Ad Hoc Committee on Health Literacy, 1999), rendering them effectively inaccessible to most (lay) people. Ideally, healthcare providers would stay current on all medical evidence and share relevant findings with patients, but this is impractical due to the volume and growth of the evidence base (Bastian et al., 2010; Marshall et al., 2021).
LLMs may provide a means for lay readers to access such findings by automatically summarizing and simplifying texts into plain language (August et al., 2023; Shaib et al., 2023). Done successfully, this could allow patients to access the most up-todate literature relevant to their healthcare. In turn, this may promote health literacy broadly by disseminating trustworthy information (Thielmann et al., 2023; Cheng et al., 2022). But given the inherent risks to personal health, the factual correctness of such outputs is paramount in this domain.
While Shaib et al. (2023) showed that GPT-3 infrequently introduced outright errors when simplifying RCT abstracts, inaccuracies are occasionally introduced; this ought to be addressed before wide adoption of such technology. Unfortunately, there is no standard evaluation benchmark for factuality on this important medical evidence text simplification task. Consequently, it is unknown whether and to what degree existing automatic factuality evaluation metrics align with human judgments.
We posit that focusing on critical elements in the RCT structure is key for factual medical evidence communication. This work presents FACTPICO, an expert-constructed factuality benchmark for the plain language summarization of technical abstracts describing RCTs.1 FACTPICO is a finegrained benchmark focused on key characteristics
1We focus on abstracts, because they are always publicly accessible, and typically include the key results that would be of interest to individuals.
8437
of trials: Populations (e.g. COVID patients; diabetics), Interventions (e.g. remdesivir), Comparators (e.g. placebo), Outcomes (e.g. 30 day mortality, or pain), as well as Evidence Inference (i.e., whether the intervention yielded a significant difference in the treated group with respect to the outcome; Lehman et al. 2019). PICO is a standard framework to structure clinical questions.
Figure 1 shows an example of FACTPICO annotation. In contrast to standard summaries, plain language summaries have the additional goal of simplifying content for lay readers. This may involve elaboration and explanation of difficult concepts to foster understanding (Srikanth and Li, 2021). Thus, FACTPICO includes a correctness assessment of added content. FACTPICO is also distinct in that it includes expert-written rationales that contextualize the evaluation of these fine-grained characteristics, providing a useful first step in assessing explainable factuality evaluation methods.
FACTPICO includes outputs from a mix of proprietary and open-source LLMs (GPT-4, Llama2-Chat, and Alpaca). Our findings are somewhat less optimistic than prior work in medical (Shaib et al., 2023) and news summarization (Goyal et al., 2022): Factual errors (occasionally important ones) with respect to key RCT elements are introduced by LLMs. One concerning phenomenon is the extent to which models overgeneralize, resulting in problematic information loss (Trienes et al., 2024).
Using FACTPICO, we evaluated a suite of exist-
ing automatic metrics shown to perform well for factuality in summarization (Scialom et al., 2021; Goyal and Durrett, 2021; Fabbri et al., 2022; Zha et al., 2023a; Tang et al., 2023a), as well as newly devised LLM-based evaluations. We find that existing metrics correlate with expert ratings at the system level, but not at the instance level. The best performing metric is an LLM-based approach in which we first identify key RCT elements; this shows that providing models with explicit domain knowledge may help. Analysis of LLM-generated rationales shows that LLMs often provide flawed reasoning when justifying their self-evaluations.
2 FACTPICO Benchmark
The FACTPICO benchmark consists of expert factuality assessments of 345 LLM-generated plain language summaries of 115 RCT abstracts.

heading: 2.1 LLM-based Plain Language Summaries
text: The medical abstracts used in FACTPICO are sourced from the Evidence Inference 2.0 dataset (DeYoung et al., 2020), which contains abstracts and full articles that describe RCTs from PubMed. We use a subset of the abstracts which include annotated spans that state clinical results. We also exclude abstracts that have corresponding humanwritten plain language summaries in the PubMed database, which may have appeared in the pretraining data of the LLMs we are experimenting
with.2 FACTPICO includes 115 abstracts randomly sampled from this filtered subset.
For each abstract, we generate plain language summaries using three different LLMs under zeroshot prompting. This zero-shot setting better emulates how lay users would likely interact with models, compared to few-shot. We use GPT-4 (OpenAI, 2023), Llama-2-Chat (Touvron et al., 2023), and Alpaca (Taori et al., 2023), which resulted in 345 (115× 3) total outputs. In a preliminary investigation, we found that these models seemed qualitatively best at generating plain language summaries. Details about this pilot study, prompt exploration, and the final prompts are in Appendix E.

heading: 2.2 Human Evaluation Framework
text: We evaluate generated summaries using a set of questions addressing factuality as related to the key PICO aspects of RCTs, as well as information added by LLMs during simplification. We ask evaluators to score model outputs and provide natural language rationales to justify their ratings. Annotation interface details are in Appendix P.
PICO Elements. Population, Intervention, Comparator, Outcome (PICO) elements are the key components of an RCT (Richardson et al., 1995). The trial population concerns characteristics of subjects in the trial, including what condition they have, the number of participants, and their demographics. The intervention is the active treatment being assessed for comparative efficacy; the comparator is the control with respect to which this is being compared. Finally, outcomes are those things that are measured to determine results.
Accurate representation of these essential descriptors in the plain language summaries of an RCT is imperative. More explanations and examples are provided in Appendix Table 7. We ask evaluators to provide a rating between 1 to 4 codifying the factuality with respect to each PICO element, respectively: 4: Mentioned and described accurately; 3: Mentioned but described somewhat inaccurately or vaguely; 2: Mentioned but described with severe inaccuracies and/or is missing critical descriptors; 1: Missing. We describe the rating system in greater detail in Appendix A.
Rationales. Evaluators are also asked to provide natural language rationales justifying their chosen
2We check this by querying the Entrez database system and filtering out abstracts which have other abstracts associated with their PubMed ID.
rating. Such rationales may reveal technical justifications for annotations which would be difficult for a layperson to assess (see Figure 2 for examples). More usefully, rationales can express a degree of uncertainty in evaluations. For example, consider the following rationale for an intervention being evaluated as accurately mentioned:
The interventions are clearly described as one group receiving morphine and one triamcinolone, yet their combination with bupivacaine is missing, but doesn’t seem very important.
This rationale indicates that the choice to evaluate the intervention as accurate was not a clear-cut decision. By including rationales in FACTPICO, this complex decision process is documented. We compare LLM rationales and expert rationales in Section 6, and envision future work to dive deeply into explainable factuality measures.
Evidence Inference. We additionally evaluate the evidence inference aspect of LLM-generated plain language summaries, i.e., whether the results and findings concerning PICO are reported factually, as PICO covers what outcomes were measured, it does not account for the corresponding results (e.g., if an intervention was found to outperform a comparator). A challenge here is that most trials will report multiple results, any of which may be conveyed (un)factually in the summary. Thus, we collect factuality assessments at the level of individual results, using findings annotated in the source abstracts from the Evidence Inference 2.0 dataset as reference. Evaluators are asked to determine how well each particular inference is reflected in the plain language summary (4: accurate, 3: vague/slightly inaccurate, 2: inaccurate, 1: not mentioned), with a free-text rationale.
Added Information. Unlike traditional summarization where content addition is seen as extrinsic hallucination (Maynez et al., 2020), plain language summarization often requires the model to explain and elaborate complex concepts (Srikanth and Li, 2021). It is important to verify the correctness of content additions. We ask annotators to highlight addition spans, determine whether they are factual, and justify each rating with a free-text rationale.

heading: 2.3 Annotation
text: FACTPICO was annotated by two senior students in their fifth year of medical school who are highly proficient in English. They are experienced with
data annotation for text simplification and summarization tasks. To ensure high annotation quality, we conducted a training phase which involved annotating a set of summaries as pilot (excluded from FACTPICO). Next, we collected two sets of annotations on 75 summaries (25 from each model). For the first 60 of these summaries, we asked annotators to discuss their responses with respect to the PICO elements. These discussions were intended to resolve any conceptual differences related to PICO annotation (Appendix C). The remaining 15 summaries were evaluated individually. Compensation was 12C per hour, which is standard for fixed-hour contracts with the annotators’ institutions.
Inter-Evaluator Agreement. Table 1 reports agreement on the 15 held-out (undiscussed) subset for PICO elements, as well as agreement on the full set of 75 texts for the Evidence Inference and Added Information questions (all annotated without discussion). We report Randolph’s kappa (Randolph, 2005), a free-marginal version of Fleiss’ kappa.3 With respect to added information, the agreement reported in Table 1 is at the sentence level (for each sentence, whether it is considered as added information). The correctness of added information is reported on the 48 spans both annotators identified as added information. Overall, evaluators showed moderate to high agreement (Artstein and Poesio, 2008). This range of kappa values are expected given the degree of subjectivity inherent to rating the severity of factual errors.

heading: 3 Evaluation and Analysis of Plain Language Summaries
text: Results of the factuality evaluation of the plain language summaries generated by GPT-4, Llama-2, and Alpaca in FACTPICO are presented in Table 3.
3Free-marginal refers to distributions where raters have no prior knowledge as to the quantity of instances that would be assigned each rating, such as is the case with FACTPICO.
We report average ratings for each PICO category and Evidence Inference, as well as the number and percentage of non-factual added information spans identified by either of the evaluators. Since readability is a key goal of plain language summarization, we also report the change in Flesch-Kincaid Grade Level (Kincaid et al., 1975) between the abstract and the summary (∆FK), as well as the ROUGE-L score (Lin, 2004), which quantifies ngram overlap between the generated plain language summary and the original abstract (i.e., high overlap may indicate low readability).
Factuality vs. Simplicity. We observe a clear trade-off between the simplicity of the generated text and its factuality. Plain language summaries generated by Alpaca are rated as more factual across all dimensions, including most of the automatic metrics (Table 5) discussed in Section 5. However, Alpaca is also the most extractive (i.e., heavily relying on deletion), with a near 0.5 ROUGE-L score and an advanced reading level. By contrast, GPT-4 and Llama-2 both simplify by rephrasing, with similar ROUGE-L and reading levels. Yet the plain language summaries generated by GPT-4 and Llama-2 are less factual, with a significant increase in the number of hallucinations (added non-factual information). Comparatively, Llama-2 produced the least factual summaries.
Can LLMs reliably convey critical RCT elements? The fine-grained framework in FACTPICO exposes issues with LLM-generated plain language summaries for medical texts. As shown in Table 3, LLMs can explain the Interventions and Outcomes more accurately, while failing to do so for other information, such as the Populations, Comparators, and Evidence Inference.
Overgeneralization and omissions of such critical elements may distort the findings and conclusions of medical research studies in nuanced but important ways. Consider this example where GPT4 omitted a critical element of RCT in its summary.
Scientists did a study to see if people who know they have a gene that puts them at risk for heart problems would eat healthier. They looked at people’s intake of a vitamin called folate, which is good for heart health. The participants in the study were told if they have the risk gene or not. But in the end, there was no difference in how much folate the people with the risk gene ate compared to those without the risk gene. So, knowing if they have the risk gene didn’t make people eat healthier.
In the original RCT abstract, participants received “either general healthy eating advice or varying lev-
els of personalised nutrition advice”. One of the levels of such personalized advice involved testing for a gene “in relation to cardiovascular health and the importance of a sufficient intake of folate”. But, the generated summary above fails to mention the comparator, which is the general healthy eating advice; instead, it focuses on the overgeneralized intervention, i.e., personalized advice involving genetic variants (without mentioning the personalized nutrition advice). These omissions not only render the simplification incomplete, but they also make understanding the actual result impossible.
Do LLMs generate accurate elaborations? Table 3 also shows a concerning amount of nonfactual additions within the generated simplifications. Examples of these span-level annotations can be found in Table 2. Most of these non-factual additions may misrepresent the original medical text and consequently mislead lay readers. Even GPT-4 produced many plain language summaries (31.3% of all additions added) with such errors, raising questions as to the trustworthiness of LLMs for tasks in high-stakes domains such as medicine, when used by zero-shot prompting (a most common use-case for lay users).

heading: 4 Factuality Evaluation Metrics
text: FACTPICO is a dataset that can be used to assess automatic evaluation methods for plain language summarization of RCT texts. We assess existing methods used for factuality evaluation, as well as the capabilities of LLMs themselves to evaluate factuality. These analyses focus on questions about PICO elements and Evidence Inference; we leave evaluation of the factuality of added information for future work, as this entails fact-checking using external knowledge sources.

heading: 4.1 Factuality Metrics Evaluated
text: We first evaluate a suite of existing automatic factuality metrics shown to be effective in prior work. (1) Dependency-Arc Entailment (DAE) (Goyal and Durrett, 2021) decomposes summaries into smaller entailment tasks at the arc-level to assess their factuality. We use a numeric score by taking the minimum of the probability scores assigned to individual arcs. (2) QuestEval (Scialom et al., 2021) uses a QA-based framework to analyze the factual faithfulness of a summary to the original text. This method scores summaries a 0 if there is no common token and a 1 for an exact match. (3) QAFactEval (Fabbri et al., 2022) is a QA-
based metric that combines various components from other factuality metrics and assigns scores based on the LERC score (Chen et al., 2020). This score, usually used for evaluating reading comprehension answers, ranges from 1 to 5, where 1 is a completely wrong answer and 5 is a perfect answer. (4) AlignScore (Zha et al., 2023b) is an alignment-based method for analyzing factual consistency. The final score assigned is the average of the maximum alignment probabilities between sentences from the summary to context chunks from the original abstract.

heading: 4.2 LLM Evaluators
text: Prior work has also shown that LLMs themselves can be good evaluators for factuality in summarization (Luo et al., 2023; Wang et al., 2023; Tang et al., 2023a; Tian et al., 2023). FACTPICO judgments are finer-grained. Therefore, we prompt LLMs with instructions emulating the questions asked of human evaluators in FACTPICO. In addition to ratings, the LLMs also generate rationales for their scores, which we analyze in Section 6.
The implementation details and the prompts used for this task are in Appendix H. Prior work found that LLM evaluations may be biased in that a sys-
tem may ‘prefer’ its own outputs (Liu et al., 2023), but we did not observe this here (Table 5). Full-Text Evaluation. As input, we provide LLMs the full-text of a complex medical abstract and corresponding LLM-generated plain language summary. We instruct the evaluator LLM to find PICO elements in the text and evaluate them according to the provided criteria. In addition to PICO elements, we also evaluate Evidence Inference outputs; here the reference results span (annotated in prior work) is compared against the full text of the plain language summary. We evaluate four LLMs in this way: GPT-4 (OpenAI, 2023), Llama-2-Chat (7B) (Touvron et al., 2023), Alpaca (7B) (Taori et al., 2023), and Mistral (7B-Instructv0.1; Jiang et al. 2023).
PICO-R Extraction Pipeline. We also evaluate LLM scorers explicitly informed of the PICO elements and results inferred from evidence (PICO-R). We adopt this two-stage pipeline using GPT-4. We first extract PICO-R from both the original abstract and the plain language summary.4 For Evidence
4Initial experiments showed that GPT-4 can more accurately extract PICO elements compared to other LLMs and PICO-tagger models (Nye et al., 2020), especially for plain language summaries.
Inference, this extraction is only necessary from the summary. Next, extracted elements are passed to GPT-4 along with an evaluation prompt.

heading: 5 Factuality Metric Evaluation Results
text: To conduct a meta evaluation of the system factuality metrics, we compute the Kendall’s τb coefficient, Spearman correlation coefficient, and Pairwise Accuracy coefficient (Deutsch et al., 2023) for automatic vs. human evaluations for each of the PICO and evidence inference aspect. We posit that good metrics must correlate well with the most salient elements in a high-stake domain.
Instance-level Results. We present the results for the above metrics in Table 4.5 Across most measures, with the exception of Kendall’s τb, the pipeline combining GPT-4 evaluation with PICO extraction has the highest correlations with the FACTPICO ratings. Thus, decomposing the original evaluation task into separate localization then evaluation steps yield better performance, indicating that LLMs benefit from breaking apart a complex tasks into a series of simpler steps completed separately.
FACTPICO ratings correlated more with dedicated factuality models than LLM evaluators with the exception of GPT-4. In fact, the smaller, opensource LLMs had barely any correlation at all with the FACTPICO ratings. These results show that there is a sizeable gap in an LLMs ability to evaluate generated text compared to its generation capabilities.
System-level Results. LLMs in Table 5, with the exception of Alpaca, produce similar results compared to the human average PICO-R ratings in FACTPICO in Table 3. Alpaca rates plain language summaries low, with averages under the minimum in the scale (1), and so has failed to follow the evaluation instructions. Although Alpaca-generated summaries have higher ratings, these aren’t meaningful as are extractive.
Visual Analysis. For more visual analysis of the system factuality metrics vs human ratings, Figure 2 shows the top three performing systems vs. averaged human scores. The highest performing system, the PICO-R extraction pipeline, has a the most balanced linear pattern. On the other hand,
5The evaluation scale for Llama-2 was reversed because it struggled to follow the original instructions. We described this behavior in greater detail in Appendix K
GPT-4 without PICO-R extraction often rates nonfactual summaries highly. Contrastively, QuestEval, the best performing non-LLM factuality metric, is more cautious and rates high quality summaries lower. We further analyze automatic vs. human rating distributions in Appendix G. These results hint at the potential challenges of factuality assessment brought by the shift in readability.

heading: 6 Preliminary LLM Rationale Analysis
text: We perform a preliminary analysis on LLM rationales, comparing to expert rationales in FACTPICO; we leave a thorough human evaluation of LLM rationales for future work.
Qualitative analysis of a small sample of rationales show that Llama-2 and Mistral are often able to comprehend the medical text but fail in making correct judgments according to the provided instructions. Most commonly, Mistral focuses on the abstract rather than the evaluated summary, while Llama-2 generates long explanations that eventually arrive at the wrong conclusion. Rationales from GPT-4 and its pipelined counterpart usually did not do this. For the most part, rationales made logical sense. However, some rationales were overly generous in it evaluation, ignoring critical errors. We provide examples of such erroneous rationales in Appendix M.
Table 6 shows the BERTScore (Zhang et al., 2020) between expert vs. LLM rationales across all evaluators experimented in this work. Overall, the rationales are dissimilar. The PICO-R extraction pipeline produced rationales most similar to that of humans, with the exception of rationales for evidence inferences, where GPT-4 rationales are slightly more similar than others. Despite explicit prompting, Alpaca frequently did not produce rationales and stopped generation after outputting a
numerical rating. Despite reporting these preliminary results, we advise caution when adopting reference-based metrics for automatic free-text rationale evaluation, as the results may be misleading. For example, we observed that Alpaca rationales tend to be a somewhat arbitrary variation of the rating descriptions (e.g., The interventions were described accurately), and human evaluators often use the words in these descriptions in their rationales (e.g., The intervention is described correctly as an 8 week program of specific exercises), inflating word overlap scores. By contrast, an example Llama-2 rationale (The comparator in the PICO model is the placebo group...) could present the same idea as a human rationale (ex: Comparator (placebo) is mentioned.), but result in a low BERTScore (ex: -0.182) because rationales from Llama-2 tend to (much) longer than those written by humans. Future work developing metrics for rationale correctness should not solely rely on reference-based metrics. We additionally present a length analysis in Appendix L.

heading: 7 Related Work
text: Meta-evaluation of factual consistency metrics in summarization (and related tasks) in the “general domain”, e.g., Wikipedia and news, has garnered considerable attention (Pagnoni et al., 2021; Honovich et al., 2022; Laban et al., 2022; Tang et al., 2023a; Min et al., 2023). However, in addition to focusing on different domains, these existing benchmarks in summarization include primarily older models. Newer LLMs may yield novel error types (or be more factual overall) (Tang et al., 2023a).
Existing summarization factuality benchmarks also fail to generalize to simplification, in which content addition in the form of elaborations or explanations is often necessary (Srikanth and Li,
2021). Broadly, simplification entails substantial language changes that often lead to the text being more general (Li and Nenkova, 2015). Devaraj et al. (2022) evaluated the factuality of automated simplification model outputs at the sentence-level, noticing that content deletion can often lead to factual errors (in contrast to only summarization). Our findings confirm that these errors also exist in plain language summarization; the overgeneralization problem may lead to safety issues in the medical domain.
Shaib et al. (2023) and Tang et al. (2023b) evaluated LLM-generated summaries of medical evidence. Notably, Shaib et al. (2023)’s work included an evaluation of plain language summaries. Our work deepens this analysis with a finer-grained evaluation focusing on critical components of RCTs and medical evidence, covering three LLMs. Our findings call for caution against LLM-generated plain language summaries despite the absence of outright inconsistencies. Pal et al. (2023) presents an analysis of “hallucinations” in medical QA tests, focusing on reasoning rather than factual consistency.
The inclusion of human-written natural language rationales in factuality benchmarks is rare, and there is a paucity of work evaluating these. The FELM benchmark (Chen et al., 2023), an opendomain evaluation of LLM-generated long form texts covering factual knowledge, math, and reasoning included human rationales. Work in LLM critiquing has started to incorporate natural language critiques from both humans (Saunders et al., 2022) and LLMs (Cui et al., 2023; Kim et al., 2024). FACTPICO is the first factuality benchmark of the plain language summarization task that includes expert-generated natural language rationales.

heading: 8 Conclusions
text: We introduced FACTPICO, an expert-annotated benchmark in the domain of evidence-based medicine for evaluating the factuality of plain language summarization with respect to clinically important dimensions. Using FACTPICO, we presented an analysis of factual errors along these fine-grained aspects in LLM-generated plain language summaries. We also presented an analysis of methods to evaluate factuality, including both dedicated factuality models and novel LLM-based methods.

heading: Limitations
text: The process of evaluating texts in FACTPICO was time consuming for human evaluators, requiring close reading of complex, technical language. Consequently, to make the workload manageable, we did not ask evaluators to localize PICO elements (or results regarding these) in texts in plain language summaries. Annotating how these elements are represented overall in the summary would provide more insights in cases where they are vaguely represented and thus required more focus in this evaluation. We automatically localize these elements in the evaluated PICO-R extraction pipeline, but we encourage future work expanding FACTPICO to include human-annotated, span-level annotations.
We primarily evaluated the zero-shot capabilities of LLMs in evaluating the factuality of simplified medical texts. We chose this setting as it best reflects how an end-user—a lay individual— would likely interact with an LLM, as such users are unlikely to provide LLMs with expert-evaluated plain language summaries for few-shot prompting. Future work could explore other LLM evaluation methods methods and use FACTPICO as an evaluation benchmark.
In our meta-evaluation we compared factuality metrics that assess the overall factuality of a text against numerical ratings assessing the fine-grained factuality of key characteristics in these texts. We acknowledge that this is not an equivalent comparison. However, we posit that the aggregation of these assessments should be well-correlated with the overall factuality of RCT texts for them to be useful in this important domain.

heading: Acknowledgments
text: We thank David Heineman for help with the annotation interface, Yao Dou for help with Alpaca,
and Tanya Goyal and Greg Durrett for useful discussions. This research is partially supported by NSF CAREER Award IIS-2145479 and Good Systems,6 a UT Austin Grand Challenge to develop responsible AI technologies. Trienes, Göke and Coers were supported by the Cancer Research Center Cologne Essen (CCCE). Trienes was also supported by the Federal Ministry of Education and Research (BMBF) and by a fellowship within the IFI programme of the German Academic Exchange Service (DAAD). Wallace was supported in this work by the National Institutes of Health (NIH), grant R01LM012086, and by the National Science Foundation (NSF), grant 1750978. Xu is supported in part by NSF awards IIS-2144493 and IIS-2112633.

heading: A Evaluation Guidelines
text: Annotators are shown a “source” input (abstract of a technical paper describing an RCT), along with a plain language summary of it, automatically produced by a model, and asked a series of 7 questions for evaluation.
Question I: Added Information For this part of the question, please highlight words, phrases, or sentences in the output that adds or modifies the information from the original document. Afterwards, answer whether this added/modified information is factual or not factual. Also, provide a rationale for why this is added/modified information and why it is factual or not factual.
Questions II-V (one analogous question for each PICO element): For instance, for population, we ask:
□ The population is mentioned, and described accurately
□ The population is mentioned, but described somewhat inaccurately or vaguely
□ The population is mentioned, but described with severe inaccuracies and/or is missing critical descriptors
□ The population is missing in the model summary
□ N/A
Please provide a rationale for why you chose the answer choice in relation to the abstract and the summary.
Some examples will actually not quite be randomized trials (e.g., they might be observational studies, or a description of a prospective trial not yet run). In these cases, it may be that there is no meaningful population (or intervention, comparator, and outcome). Here you should select the “N/A” option.
Exhaustive Outcomes (Analyzed in Appendix I): Sometimes, the plain language summary may not mention all the outcome measures described in the abstract. The summary may still be considered factual if the omitted measures are non-critical for the experiment and are not mentioned any further in the abstract. However, please do annotate
separately when the plain language summary does exhaustively mention all outcome measures and when it does not.
Question VI: Evidence Inferences You will be presented with a span of text highlighting the inferred result from the experiment presented in the abstract. Based on this span, choose the following based on how this span is presented in the plain language summary:
□ Accurate
□ Vague/Slightly Inaccurate
□ Inaccurate
□ Not mentioned
Please provide a rationale for why you chose the answer choice in relation to the evidence inference span and the summary.

heading: A.1 Additional Questions
text: We also collected information for any additional comments on the generated plain language summary, as well as contradictions that are not covered by the other questions. These annotations are scarce, thus we have not included them in the FACTPICO benchmark.
Additional comments: “asks you to write down any commands you would want the machine to follow if you could interact with it, e.g., “Make it shorter”, “Explain XXX a bit more”, and so on.”
Contradictions: “Here you will be looking for content in the output that contradicts some part of the input. We ask you to annotate both the input and the output for this question. Please provide a rationale for why the content is a contradiction.” The contradictions are analyzed in Appendix J.

heading: B Data Release and License
text: We reused RCT abstracts from the Evidence Inference V2.0 dataset (DeYoung et al., 2020); https:// evidence-inference.ebm-nlp.com, accessed 2024-02- 15). All articles in this dataset are from the PubMed Open Access subset which only includes license terms that allow reuse (https://ncbi.nlm.nih.gov/ pmc/tools/openftlist, accessed 2024-02-15). After discussion with our institutions’ librarian on fair use, we release the annotations in FACTPICO under CC-BY-4.0.
C Improving Agreement Through Discussion
Initially, we had observed low agreement on PICO questions among evaluators on the first 30 doubly annotated documents. Upon deeper analysis of these disagreements, we modified our instructions to be clearer and more detailed. We had asked evaluators to independently re-evaluate the previous annotations as well as doubly annotate 30 more documents. We observed that while agreement had improved significantly overall, evaluators still disagreed substantially on questions regarding population and outcome.
To fix this issue, we facilitated a “soft" discussion between evaluators regarding their annotations. Evaluators were presented with documents in which they had disagreed majorly on any PICO questions from the last 30 documents they evaluated. Then they were asked to come to consensus on how to rate these questions. Afterwards, we asked evaluators to independently reevaluate their previous annotations based on the insights they gained from this discussion. Similarly, they were asked to doubly annotate 15 new documents inde-
pendently. Table 8 shows the resulting agreement through Randolph’s kappa after this discussion. For each question type, the first number is the kappa for the entire set of 75 doubly annotated documents. The second number is the kappa for the set of 15 documents that were annotated independently after the discussion. Both sets of number show moderate to high agreement for these questions. Furthermore, the agreement for this undiscussed set being significantly higher than the agreement for all 75 doubly annotated documents indicates that this discussion method was effective at improving agreement.

heading: D Model Details and Compute
text: We used a High-RAM T4 GPU through Google Colab Pro+ to conduct our experiments.

heading: D.1 Plain Language Summary Generation
text: GPT-4 We used a frequency penalty of 0, presence penalty of 0, temperature of 1, and top p of 1. Llama-2 7B Chat. We set the max new tokens to 4000, did multinomial sampling, temperature of 1, top k of 50, and top p of 1.0. Alpaca (7B). We set the max new tokens to 4000, used greedy decoding, temperature of 1, top k of 50, and top p of 1.0.

heading: D.2 LLM evaluation
text: GPT-4 We used a frequency penalty of 0, presence penalty of 0, temperature of 1, and top p of 1.
Together.AI We ran experiments with Llama-2 7B Chat, Alpaca (7B), and
Mistral-7B-Instruct-v0.1 using the Together.AI API interface. For all models, we set the max new tokens to 256, temperature of 0.6, top k of 90, top p of 0.8, and a repetition penalty of 1.1.

heading: E Zero-shot Prompts for Plain Language
text: Summarization/Simplification

heading: E.1 Preliminary Prompt Exploration and Model Evaluation
text: We conducted a preliminary evaluation in Table 9 for prompt engineering and model exploration on 100 medical abstracts from Shaib et al. (2023). The prompts we tested for are all typical-use prompts, representing how these systems would be usually used by the public. We avoided engineering prompts for the best-case use of these systems for this task, as we did not want to induce a false trust in any of these systems for medical use, which could be potentially harmful. Given this criteria, we evaluated the following prompts:
Paper Plain: (August et al., 2023) My fifth grader asked me what this passage means: [abstract] I rephrased it for him, in plain language a fifth grader can understand:
Short: "My fifth grader asked me what this passage means: [abstract] Help me summarize it for him, in plain language a fifth grader can understand. Make it short."

heading: Summarize:
text: "My fifth grader asked me what this passage means: [abstract] Help me summarize it for him, in plain language a fifth grader can understand."
5th grade: 5th Grade: Paraphrase this passage completely in your own words. Always define words the reader may not know: [abstract]

heading: Complex:
text: "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction: Rewrite the following complex passage in order to make it easier to understand by non-native

heading: ModelPrompt DA FK #tokens
text: speakers of English.
### Input:" [abstract]
###Response:"

heading: Medical:
text: "Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction: Rewrite the following medical abstract in order to make it easier to understand by non-native speakers of English.
### Input:" [abstract]
###Response:"
E.2 FACTPICO Prompts
For GPT-4, we randomly sampled from GPT4Summarize and GPT-4Short. We also used Llama2Paper Plain and AlpacaComplex.

heading: E.3 Explanation of Selection Criteria
text: We primarily selected prompts based on the length of the plain language summaries produced when they are used. They had to be substantially shorter than their corresponding abstracts, qualifying them as plain language summaries. We did not use any form of length control outside of prompt instructions, as we found prompt instructions, such as "Make it short.", are just as effective as other forms
of length control without creating any unwanted artifacts and disfluencies in the text. We did not consider ChatGPT-DA (Wang et al., 2023) to be a reliable measure of factuality. However, it was an effective sanity check in determining the relevance of the summaries to their corresponding abstracts.
We included two prompts to get generations from GPT-4 because both prompts met the above selection criteria and added more diversity in FACTPICO. As a benchmark for factuality evaluation methods, including more diverse outputs will ensure only very robust systems are capable of attaining good results on FACTPICO.

heading: F Analysis of PICO Specific LLM Evaluations
text: We present the attribute-wise results for each LLMbased system we evaluated in Table 10. Similar to the results presented in Table 4, the GPT-4 based systems are more correlated with human ratings than the other LLM evaluators. Interestingly, we do see that for some attributes, in particular Population and Evidence Inference, the full text GPT-4 approach produced more human correlated ratings compared to those produced using the extraction pipeline. This suggests that evaluation of these elements requires more context than that provided in the extracted spans to perform a more accurate evaluation. However, despite this result, in terms of overall factuality, the extraction pipeline is slightly more correlated than the full text approach, suggesting an overall benefit to decomposing a complex task such as this into smaller, simpler steps to be executed sequentially.

heading: G System to Avg. PICO-R Visualizations
text: Figure 4 shows all system evaluations plotted against human evaluations. Figure 3 describes the distribution of averaged FACTPICO ratings alongside those of the automatic evaluation methods. These graphs display Gaussian approximations of the standardized distributions, allowing for easy visual comparison despite differences in scales. We only observe the average of the PICO-R ratings across human and LLM evaluations.
Our primary focus in this analysis is observing how closely the distribution of automaticallyderived factuality scores are to the human evaluations in FACTPICO. Upon visual inspection of Figure 3, we see that GPT-4, Mistral, and AlignScore seem to have the most closely aligned distributions to the average human rating distribution in FACTPICO. In order to quantify this, we also calculated the KL divergence between the distributions of the evaluated metrics and the distribution of the averaged human ratings from FACTPICO as they are presented in Figure 3. These results can be found in Table 11. As previously hypothesized, the three distributions closest to the averaged human distribution are from AlignScore, Mistral, and GPT4, in that order. Another trend that is observable in Figure 3 are the differences in the narrowness of the distributions between some LLM evaluators and dedicated factuality models. Ratings from GPT-4, Mistral, and, to a certain extent, Alpaca typically have narrow distributions, indicating an overall preference towards a single rating. On the other hand, distributions from dedicated factuality models are wider, signaling more variation.
Interestingly, these is a mismatch between wellcorrelated metrics and metrics whose distribution aligns closer to that of the human ratings (Figures 2 and 4). Having a closely aligned distribution does not imply good correlation. Similarly, good correlation, unless it is exceptionally high, does not imply closely aligned distributions. This is the most evident in the case of AlignScore. However, this mismatch does highlight interesting trends. A left or right skew in a score distribution references the “strictness” of the evaluator, with a more leaning skew indicating the evaluators are less strict and vice versa. Through this lens, the human evaluators could be viewed as not as “strict” compared to many of the metrics. A possible explanation of this effect could be that the simplified nature of the text or the accurate elaborations present in the text
could have been confused for factual errors.

heading: H Zero-shot Prompts for LLM Evaluation
text: Here we present the prompts used for LLM evaluation. To find the implementation details see Appendix D.2.

heading: H.1 Post-processing Ratings
text: The ratings produced from the prompts displayed below follow a reverse scale, where lower scores indicate the evaluated text is more factual. For the sake of comparison, we post-process these ratings as 5− Original Rating, flipping the scale so that it follows the rest of the evaluated metrics. Llama2 is the only system that was not post-processed. We present the reasons for why this was done in Appendix K.

heading: H.2 LLM Full-text Evaluation Prompt for PICO Elements
text: The following prompt is provided as a system prompt to the large language model.
You are given an abstract and a summary. <PICOInfo> Find the <PICOElem> in accordance with PICO in both the abstract and the summary and use it rate the summary between 1 to 5.

heading: The ratings are as follows.

heading: 1 - The <PICOElem> is mentioned in the
text: model summary and described accurately. 2 - The <PICOElem> is mentioned in the model summary but described vaguely or somewhat inaccurately. 3 - The <PICOElem> is mentioned in the model summary but described inaccurately or is missing critical descriptors. 4 - The <PICOElem> is missing in the model summary. 5 - N/A
Please provide only the rating and the rationale for the rating. Provide the rating after stating "Rating:".
The model is then queried as follows:

heading: Abstract:
text: <Abstract>

heading: Summary: <Summary>
text: The tag "<PICOInfo>" correspond to the following four descriptors depending on the evaluated PICO element.
Population: Population in PICO describes the type of subjects involved in the trial. Critical descriptors for population include important demographic information and any specific shared conditions. Intervention: Intervention in PICO describes the treatments considered in the trial. Comparator: Comparator in PICO describes the alternative treatment to which the intervention is being compared against. Outcome: Outcome in PICO describes the outcome measures used to determine results of the trial. If the primary outcome measures are not be mentioned, then the summary is critically flawed.
The "<PICOElem>" tag is replaced with the evaluated PICO element name ("population", "intervention", "comparator", "outcome").

heading: H.3 PICO-only LLM Evaluation Prompt
text: The prompt used for PICO-only LLM evaluation is almost identical to the one above. The only change is that the first sentence of the previous prompt ("You are given an abstract and a summary.") is changed to "You are given a list of PICO elements from an abstract and a summary."

heading: H.4 Prompt for Extracting PICO Elements
text: The following prompt is used to extract PICO elements from medical text.

heading: Definition of each PICO element:
text: Population: The types of patients involved in the trial Intervention: The treatments considered Comparator: The alternative treatment to which the intervention is being compared to. Outcome: What is measured. NOT what was found in the study (“result”). For example, if the study finds a drug reduces the duration of headache, the outcome here is just the “duration of headache”, not that it reduced it.
Identify the PICO elements in the following
passage. Pull direct quotes from the passage:

heading: H.5 Evidence Inference Full-Text Prompt
text: In FACTPICO, evaluators analyze if individual evidence inference spans from the abstract are accurately represented in the plain language summary. The LLM evaluation is modeled after this as well, comparing evidence inference spans from the abstract to the full text of the summary. The following is the system prompt used for this evaluation. You are given a result inference span from an abstract, and you are given a summary. A result inference span corresponds to an inferred result in an experiment. Find the corresponding result inference in the summary and use it to rate the summary between 1 to 4.

heading: The ratings are as follows:

heading: 1 - The result inference is mentioned and
text: described accurately. 2 - The result inference is mentioned but is described vaguely or is slightly inaccurate. 3 - The result inference is critically inaccurate. 4 - The result inference is missing in the model summary.
Please provide only the rating and the rationale for the rating. Provide the rating after stating "Rating:".
The model is then queried as follows:
Result Inference Span: <Span from abstract>

heading: Summary: <Summary>

heading: H.6 Evidence Inference Extraction Prompts
text: We used the following prompt to extract evidence inferences from the plain language summary. These extractions are subsequently compared against evidence inferences from the abstract for the evaluations. The extraction prompt is as follows:

heading: An result inference span corresponds to an inferred
text: result in an experiment.
Identify result inference spans in the following passage. Pull direct quotes from the passage:
Here is the prompt for the evaluation itself:
You are given a result inference span from both an abstract and a summary. A result span corresponds to an inferred result in an experiment. Use the result inference spans from the abstract and the summary to rate the summary between 1 to 4.

heading: The ratings are as follows:

heading: 1 - The result inference is mentioned and
text: described accurately. 2 - The result inference is mentioned but is described vaguely or is slightly inaccurate. 3 - The result inference is critically inaccurate. 4 - The result inference is missing in the model summary.
Please provide only the rating and the rationale for the rating. Provide the rating after stating "Rating:".
The model is then queried as follows:

heading: Abstract: <Abstract Evidence Inference Span>

heading: Summary: <Extracted Evidence Inferences from Summary>

heading: I Exhaustive Outcomes
text: The outcome element in RCTs may often be represented through multiple measures, some of which may not be critical for the experiment. The omission of these non-critical outcome measures in plain language summaries usually does not impact its factuality. In FACTPICO we also asked evaluators to determine whether all outcome measures are exhaustively mentioned in the plain language summary as a separate tag exhaustive. This enables us to keep track of when these omissions occur without tying them to the factuality evaluation. The addition of this annotation also enabled better agreement on outcome annotations. This was one of the factors that led to better agreement as discussed in Section C. We also calculated agreement for exhaustive annotations through Randolph’s kappa
and report it to be 0.44, which signifies moderate agreement. These collected annotations will also be included in the released data.
We define Contradictions as pieces of information in the plain language summary that meaningfully disagree with the input abstract. In addition to the characteristics described in the main paper, we also ask evaluators to find spans in both the original abstract and the plain language summary that contradict each other and provide a rationale. The results are presented in Table 12, following similar trends to that found for other dimensions in Table 3.
We observed poor agreement overall for this question, largely due to “fuzzy” nature of this task. We define fuzziness as the difficulty to exhaustive distinguish such characteristics within a text. Table 13 shows a fuzzy, contradiction example which demonstrates how certain exaggerations may not be detected by the reader because it may be considered acceptable to use as part of the simplified language style. Annotators with different levels of discernment will likely suffer from low agreement for these types of annotations due to this phenomenon (Trienes et al., 2024). These collected annotations will also be included in the released data.

heading: K Llama-2 Correlation
text: The negative correlations under the Flip column of Table 14 demonstrate the worser performance of Llama-2 scores that are flipped (post-processed to be 5-x) compared to its original ratings. Out of all LLM evaluators, this occurrence is unique to Llama-2 . We attribute this to Llama-2’s struggles to remain consistent and faithful to the rating system defined in the prompt. Through analyzing its rationales, we discovered that Llama-2 often ranks summaries it explains as accurate with high numbers, which was opposite to the instructions.
This Llama-2 rationale assigns a rating of 4 to a summary that it describes as accurate and sufficiently detailed, which is the opposite of the rating system provided in the prompt.
"The population in the abstract and summary is described accurately and with sufficient detail. The authors mention the type of subjects involved in the trial, including their demographics (age, sex, and duration of HIV infection) and any specific shared conditions (abacavir use). The summary provides a brief overview of the study population, including their baseline characteristics and the results of the intervention.The summary accurately reflects the information provided in the abstract and provides a clear and concise summary of the study population. The authors use descriptive language to highlight the key findings related to the study population, such as the slight improvement in endothelial function after maraviroc intensification. The summary does not contain any inaccuracies or vagueness that could affect the reader’s understanding of the study population. Therefore, I would rate the summary as a 4, indicating that the population is described accurately and with sufficient detail."
Additionally, the following rationale demonstrates the hallucinations that Llama-2 makes about the rating scale, as it assumes a ranking order of 1-5 with 5 being the most accurate. Not only is 5 out of the range of the rating options given in the prompt, but also this scale is inconsistent with Llama-2’s other evaluation scale of using 4 as its highest rating.
"The population in the abstract and summary is described as "drug users" or "people using illicit opiates," which is a broad and vague term that does not provide critical demographic information. The summary does not provide enough detail about the specific characteristics of the population, such as age, gender, ethnicity, or comorbidities. Additionally, the summary does not indicate whether the population was selected randomly or if there were any exclusion criteria.
To rate the summary, I would give it a score of 4 because it does not provide sufficient information about the population involved in the study. A rating of 5 would indicate that the summary provides detailed and accurate information about the population, including critical demographic information and any specific shared conditions."
This analysis explains why non-flipped Llama-2 ratings were more correlated with human evaluation. Llama-2 has a tendency to not follow rating instructions and rate more accurate summaries with higher numbers, defeating the purpose of flipping its ratings in post-processing. Furthermore, Table 5 shows how Llama-2 (non-flipped) is capable of rating plain language summaries similarly to human annotators in Table 3. If we flipped Llama-2 ratings, the average ratings for Llama-2 would have given opposite results. Therefore, we report the non-flipped Llama-2 evaluations throughout this paper to most accurately depict Llama-2’s intended ratings.

heading: L Rationale Length Analysis
text: Table 15 shows the average number of tokens for rationales in FACTPICO and those generated by LLMs. As discussed in Section 6, Llama-2 produced the longest rationales. GPT-4, Mistral, and GPT-4 pipelined with PICO-R extraction generated rationales with similar lengths. Among the LLMs, Alpaca produced the shortest rationales. However, overall, expert-written rationales in FACTPICO have the shortest lengths. This is largely because evaluators tend to justify themselves as concisely as possible, especially for easy evaluation instances,
such as when an element is clearly mentioned accurately or clearly missing.

heading: M LLM Rationale Errors
text: Table 16 shows several examples of erronous rationales generated by various LLMs. These rationales illustrate several patterns of errors exhibited when these systems generate rationales. The examples from Mistral show the tendency to “forget” to evaluate the summary as described in Section 6. Mistral here either completely ignores the input summary or it passes off text from the abstract as belonging to the summary, as exemplified by the very first example. In addition to this issue, we also observe that Mistral generates illogical and contradictory language in its rationales (ex 3).
The examples from Llama-2 exemplify the length of its rationales as well as its tendency to use irrelevant ideas in its justifications. For example, it may acknowledge factual information from the summary (“While the summary mentions the age range of the patients (adults) and the type of surgery (elective)”), but draws invalid conclusions (“it does not provide additional demographic information”).
Rationale examples from GPT-4 here ignores critical factual errors in the summary. For example, the first GPT-4 example shows that the model acknowledges that the summary left critical characteristics missing (“Although the summary doesn’t specifically mention ‘adult’ patients or the type of hospital”), but claims that “this doesn’t seem to be crucial information”, downplaying its effect during evaluation.
We also included some examples from Alpaca, showing its rationales tend to be very similar to the the rating descriptions provided through the
prompt, and exceptions to this tend be nonsensical and illogical text.

heading: N Full Text of Abstract in Figure 1
text: Due to space constraints, Figure 1 substituted the full abstract used during evaluation with a link. The full text of this abstract is included below:
NicVAX®, a nicotine vaccine (3’AmNic-rEPA), has been clinically evaluated to determine if higher antibody concentrations are associated with higher smoking abstinence rates and if doses and frequency of administration are associated with increased antibody response. This randomized, double-blinded, placebo-controlled multicenter clinical trial (N=301 smokers) tested 200 and 400 µg doses administered 4 or 5 times over 6 months compared to placebo. 3’AmNic-rEPA recipients with the highest serum anti-nicotine antibody response (top 30% by AUC) were significantly more likely to attain 8 weeks continuous abstinence from weeks 19 through 26 than the placebo recipients (24.6% vs. 12.0%, p=0.024, OR=2.69, 95% CI, 1.14–6.37). The 5 injection 400 µg dose regimen had the greatest antibody response and had significantly higher abstinence rates than placebo. This study demonstrates proof-of-concept that 3’AmNic-rEPA elicits antibodies to nicotine and is associated with higher continuous abstinence rates, justifying its further development as a treatment for nicotine dependence.
Human Rationale: The population of the study is not mentioned clearly at all. LLM Rationale [Mistral]: The population in the abstract and the summary is described accurately. The population includes 16 eyes of 16 patients with corneal neovascularization (NV) secondary to herpetic keratitis, graft rejection, chemical burn, pemphigoid, and recurrent ulcer. The patients were randomly assigned to receive 2.5 mg of bevacizumab or 1 mg of ranibizumab through subconjunctival and intrastromal routes. The researchers followed up with the patients for one month after the injections and calculated the corneal NV areas using Image J software. The results showed that bevacizumab injection resulted in a more effective and stable regression of corneal NV compared to ranibizumab injection.

heading: O Notable Examples
text: In this section, we show several examples of interest from FACTPICO. We present relevant snippets from the abstract and the appendix and show the rating and rationale given by evaluators.

heading: O.1 Obvious Errors
text: In the following table, we show examples of factual errors that were easily identified by the expert evaluators. They stood out as clear examples of the summary misrepresenting the information in the abstract.

heading: O.2 Subtle Errors
text: In the following table, we show examples where the error is harder to detect at a first glance. These errors disguise themselves as appearing factual, but a closer reading of the text would reveal factual inconsistencies.

heading: O.3 Edge Cases
text: Here we show examples that evaluators felt were "edge cases." In other words, there is some subjectivity involved in rating these examples according to the FACTPICO framework.

heading: P Evaluation Interface
text: FACTPICO evaluations were collected through the Thresh platform (Heineman et al., 2023). This platform allows for the creation of a customizable interface for any text generation task, supporting a variety of question types and text span highlighting. The platform is hosted as a website and the customizable interface is encoded in YAML.
We modify the original Thresh interface to support integration with the Google Drive API. 7 This integration enables annotators to securely retrieve their saved evaluations while having a copy of their evaluations stored in a shared Google drive. This facilitates a straightforward organization of evaluation files in an easily accessible, shared location.
Figure 5 shows both the initial state of the interface and the state after annotations have been completed. The initial state shows almost all questions evaluators would have to answer as "edit annotations." However, span-level annotations have to be created by adding an edit, selecting the type of edit (i.e. Added
7This version of Thresh is available as a fork of the original project through this link: https://github.com/SebaJoe/ thresh
Information), and highlighting text. Figure 6 shows how evaluators can select spans in text to evaluate. Evaluators answer questions by clicking the pencil icon next to each edit. Figure 7 shows how questions are displayed for evaluators to answer.

heading: Future Work
text: We automatically localize these elements in the evaluated PICO-R extraction pipeline, but we encourage future work expanding FACTPICO to include human-annotated, span-level annotations. We primarily evaluated the zero-shot capabilities of LLMs in evaluating the factuality of simplified medical texts. We chose this setting as it best reflects how an end-user—a lay individual— would likely interact with an LLM, as such users are unlikely to provide LLMs with expert-evaluated plain language summaries for few-shot prompting. Future work could explore other LLM evaluation methods methods and use FACTPICO as an evaluation benchmark. In our meta-evaluation we compared factuality metrics that assess the overall factuality of a text against numerical ratings assessing the fine-grained factuality of key characteristics in these texts. We acknowledge that this is not an equivalent comparison. However, we posit that the aggregation of these assessments should be well-correlated with the overall factuality of RCT texts for them to be useful in this important domain