Venue: NAACL 2025

# Automatic Evaluation of Healthcare LLMs Beyond Question-Answering

Anna Arias-Duart\({}^{\dagger 1}\), Pablo Agustin Martin-Torres\({}^{\dagger 1}\), Daniel Hinjos\({}^{1}\),

**Pablo Bernabeu-Perez\({}^{1}\), Lucia Urcelay Ganzabal\({}^{3}\), Marta Gonzalez Mallo\({}^{1}\), Ashwin Kumar Gururarajan\({}^{1}\), Enrique Lopez-Cuena\({}^{1}\), Sergio Alvarez-Napagao\({}^{1,2}\), Dario Garcia-Gasulla\({}^{1}\)**

###### Abstract

Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality of responses but lack expressiveness. Open-ended capture the model's capacity to produce discourse responses but are harder to assess for correctness. These two approaches are commonly used, either independently or together, though their relationship remains poorly understood. This work is focused on the healthcare domain, where both factuality and discourse matter greatly. It introduces a comprehensive, multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics. Findings include blind spots and overlaps in current methodologies. As an updated sanity check, we release a new medical benchmark --CareQA--, with both open and closed variants. Finally, we propose a novel metric for open-ended evaluations --Relaxed Perplexity-- to mitigate the identified limitations.

+ Equal contribution. \({}^{1}\) Barcelona Supercomputing Center (BSC)

\({}^{2}\) Universitat Politecnica de Catalunya (UPC)-BarcelonaTech

\({}^{3}\) Independent Researcher (formerly affiliated with BSC)

## 1 Introduction

The growing use of large language models (LLMs) in public domains, such as healthcare, shows promise for improving global quality of life He et al. (2025). At the same time, the reliability and evaluation of LLMs in such sensitive topics requires extreme caution due to the potential impact on people's rights and well-being.

LLM evaluation today is approached through various perspectives, which consider different types of LLM assessment: automatic evaluation (scalable and factual), user evaluation (utility and usability) Chiang et al. (2024), and expert evaluation (support and coherence) Chen et al. (2023). While each of these evaluation perspectives serves distinct roles that contribute to a holistic assessment, automatic evaluation remains the most prevalent one due to its lack of dependency on human effort.

Within automatic evaluation, there are two types of tests. Those which include closed-ended responses Bedi et al. (2024), namely multiple-choice question answering (MCQA), and those which have open-ended responses Dada et al. (2024). Close-ended MCQA validation enables the automatic verification of response factuality, but it does not reflect the complex nature of real world situations (_e.g._, clinical settings Hager et al. (2024); Zhou et al. (2023)). As such, MCQA alone often fails to identify critical short-comings of model performance Li et al. (2024); Umapathi et al. (2023); Ahmad et al. (2023); Pezeshkpour and Hirschka (2023); Alazhrani et al. (2024); Zheng et al. (2023).

To incorporate a broader range of tasks relevant to the medical field Dada et al. (2024); Kanithi et al. (2024), one typically has to rely on open-ended answers. That is, reference responses are not the only valid outputs. Since these cannot be completely assessed for actuality without human expert supervision, approximate measures based on n-grams and model perplexity remain in place, which limits the reliability of these evaluations Kamalloo et al. (2023).

Efforts have been dedicated to analyze the relation between automatic evaluations and either user or expert evaluations, showing a lack of direct correspondence Fleming et al. (2024); Nimah et al. (2023). This is explained by the difference in the model features these assess (_e.g._, factuality vs usability vs support capacity), pointing at their complementary nature. Nonetheless, a similar analysis within the family of automatic evaluations is still pending; a study of the relations between open-ended and close-ended benchmarks and metrics, to understand which of these tests should be used, and when. For that purpose, we focus on the healthcare domain, providing the following contributions

* A correlation-based, empirical analysis of open-ended and close-ended tasks, benchmarks, and metrics.
* A novel medical benchmark (CareQA) featuring both closed- and open-ended formats for the verification of our findings.
* A new metric for open-ended evaluations (Relaxed Perplexity) which fills a gap identified in existing methodologies.

## 2 Methodology

This study considers four different close-ended healthcare tasks, which include nine different datasets (_e.g._, MedQA). These are all assessed using the accuracy metric. At the same time, six open-ended tasks are studied, based on nine distinct datasets (_e.g._, MedText). In this case, eleven different metrics are extracted. Further details are shown in Table 1. To assess the consistency within tasks, datasets and metrics, this work considers up to 12 different open LLMs, both specifically tuned for healthcare and general purpose, motivated by previous work Shoham and Rappoport (2024); Kaniithi et al. (2024).

### CareQA: A Novel Benchmark

Updated benchmarks are necessary to prevent both data drift (as human knowledge evolves), and data contamination (as training data crawling efforts scale). To validate the integrity and consistency of existing tests, this work introduces a new benchmark for automatic evaluation, CareQA, available in both closed-ended and open-ended formats.

CareQA originates from the Spanish Specialised Healthcare Training (MIR) exams by the Spanish _Ministry of Health_. The close-ended version is a MCQA including 5,621 QA pairs across six categories: medicine, nursing, biology, chemistry, psychology, and pharmacology, sourced from the 2020 to 2024 exam editions. CareQA is available in both English and Spanish, with the translation performed using GPT-4.

The open-ended version (English only) was created by rephrasing the questions from the close-ended version using the Qwen2.5-72B-Instruct model. After the rephrasing process, the number of

\begin{table}
\begin{tabular}{c c l l} \multicolumn{3}{l}{**Close-ended**} \\ \hline
**Tasks** & **Metrics** & **Datasets** & \\ \hline Multiple choice & & \(\cdot\)MedMCQA et al. (2022) & \(\cdot\)PubMedGA et al. (2019) \\  & & \(\cdot\)MedQA et al. (2020b) & \(\cdot\)MMLU et al. (2020a) \\  & & & \(\cdot\)CareQA-Close & \\ Prescriptions & ” & \(\cdot\)Prescription & \\ writing & & & \\ Medical text & ” & \(\cdot\)Medical Text for classification Schopf et al. (2023) & \\ classification & & & \(\cdot\)Medical Transcriptions & \\ Relation extraction & ” & BioRED Luo et al. (2022) & \\ \hline \end{tabular} 
\begin{tabular}{c c l} \multicolumn{3}{l}{**Open-ended**} \\ \hline Open-ended & BLEUURI, ROUGE, & \(\cdot\)MedDialog Raw Zeng et al. (2020) \\ medical questions & BERTScore, MoverScore, Promeheus, Perplexity & MEDIQA2019 Ben Abacha et al. (2019) \\ \hline Making diagnosis & & \\ and treatment & & \(\cdot\)’ & \(\cdot\)MedText \\ recommendations & & & \\ Clinical note-taking & & \(\cdot\)’ & MTS-Dialog Ben Abacha et al. (2023) \\ \hline Medical & & \(\cdot\)’ & MCIBench Yim et al. (2023) \\ factuality & \(+\text{Relaxed Perplexity}\) & \(\cdot\)OLAPH Jeong et al. (2024) \\ Summarization & ” & \\  & & \(+\text{FI-RadGraph}\) & \\ Question & ” & \(\cdot\)’ & \(\cdot\)Meddialog Qsumm Zeng et al. (2020) \\ entailment & & & \\ \end{tabular}
\end{table}
Table 1: This table presents the tasks implemented in this paper. The first column specifies the different tasks. The second details the metrics used (ROUGE includes ROUGE1, ROUGE2 and ROUGEL, and Perplexity includes Bits per Byte, Byte Perplexity, and Word Perplexity). The third column outlines the benchmarks used for each task.

 

suitable questions was reduced to 3,730 QA pairs. This set retains the same categories as the closed-ended version.

To ensure the validity of both the translations and rephrasing, 10 annotators conducted a manual review of a total of 360 samples, each reviewed by at least three evaluators. This process achieved a confidence level of 95% and a margin of error of 5% approximately.

The translation results were positive, with all three evaluators agreeing on 83.1% of the questions as correct. Based on this, we considered the translation to be of good quality. However, the percentage of rephrased QA pairs labeled as correct by the three evaluators was 65.8%.

To address this, we conducted a second iteration incorporating feedback from human reviewers. The main issue identified was that while the rephrased answers might differ from the ground truth, they could still be considered valid. As a result, a new rephrasing iteration was carried out, explicitly prompting the model to account for this nuance, and questions with multiple valid answers were excluded. This led to the removal of 961 samples, leaving the final CareQA (open-ended) dataset with 2,769 QA pairs. Consequently, the percentage of correct labels increased to 73.6%. See Appendix A for further details.

### Metrics

For close-ended evaluations, the metric of choice is accuracy. In contrast, for open-ended queries, there is a variety of metrics which provide different insights into model performance. This work considers eleven of those, which are sorted into four distinct categories:

* **N-gram based metrics** evaluate the overlap of n-grams between the generated and reference answers. This category includes: ROUGE1, ROUGE2, ROUGEL and BLEU.
* **Semantic similarity metrics** evaluate the semantic similarity between the generated text and reference text, often leveraging embeddings or deep learning models. This includes: BERTScore, BLEURT and MoverScore.
* **Perplexity metrics** assess the predictive capabilities of the model by measuring how well it can predict a sequence of words. This includes: Word Perplexity, Bits per Byte and Byte Perplexity.
* **LLM-judge**: In this category we use the Prometheus Kim et al. (2024) model to grade responses based on specific scoring criteria.

## 3 Experimentation

### Correlation of open-ended vs close-ended

The first experiment conducted studies the correlation between open-ended and close-ended tasks, as detailed in Table 1. Specifically, we compare the weighted average accuracy from the various MCQA benchmarks against all other close-ended and open-ended tasks and metrics. Figure 1 presents the results for the smaller models.

Of all close and open-ended tasks, only clinical note-taking correlates positively with MCQA, and even in this case, correlation is rather weak. In contrast, summarization, question entailment and the remaining close-ended benchmarks correlate negatively with MCQA, except for Med Transcriptions. The rest show a generalized lack of correlation. The negative correlation could be explained by the lack of medical expertise needed for summarizing and entailing (as information is available in the input), and by the diverse nature of close-ended tasks. At metric level, all open alternatives correlate very weakly with MCQA, except for Perplexity, for which we observe a slight correlation. These findings illustrate the relevance of the benchmarks chosen for evaluation, as well as the complementary nature of MCQA, when considering other tasks like summarization or clinical note-taking. Further details in Appendix B.1.

### Correlation of open-ended benchmarks

The previous section locates open-ended tasks with a variable degree of correlation with close-ended tasks (_e.g._, clinical note-taking, summarization). Let us now analyze correlations within the open-ended category. Details on this are shown in Appendix B.3.

Notably, no consistently high correlation is observed for any benchmark or task. This suggests that each benchmark measures distinct aspects of model performance. This is the case even for benchmarks tackling the same task (_e.g._, ACI-Bench and MTS-Dialog), illustrating the importance of benchmark source (_i.e._, who crafted the benchmark and in which context). This underscores the need for specialized evaluations for downstream tasks, as generalization cannot be assumed 



### Correlation of open-ended metrics

To assess whether the metrics used in the open evaluation are correlated among themselves, and to simplify future analyses for practitioners, we conduct a correlation analysis for each of the metrics detailed in SS2.2 across all implemented open-ended benchmarks (more details in Appendix B.2).

This analysis identifies three distinct clusters of highly correlated metrics. The first cluster includes the perplexity metrics, (_i.e._, Word Perplexity, Bits per Byte, and Byte Perplexity) all of which show a correlation above 0.96 across all analyzed benchmarks. Noticeably, these metrics are all based on probabilistic prediction (perplexity) and information efficiency (Bits per Byte). The results obtained from Prometheus (an LLM judge) can be considered a distinct cluster of evaluation, illustrating how an external model provides a different and rather unique perspective on model performance. Finally, the third cluster includes all n-gram-based metrics, together with semantic similarity metrics (_i.e._, BERTScore, BLEURT, and MoverScore). A strong correlation among these metrics is consistently observed across benchmarks, which can be attributed to their shared focus on content and overall text quality.

### Metrics resilience to rephrasing

A limitation of open-ended evaluations is their sensitivity to rewording. Let us now analyze the different metrics under this open setup, to better understand their reliability. To do so, the model's output are rephrased, and evaluation recomputed. Six rephrased versions are produced using Qwen2.5-72B-Instruct.

Results show that most n-gram-based metrics (_i.e._, ROUGE1, ROUGE2, ROUGEL and BLEU) are resilient to rephrasing. This difference may arise because these metrics rely on surface-level word matching, making them less sensitive to phrasing changes as long as the core vocabulary remains intact. _i.e._, in healthcare texts, key terms like 'diagnosis,' treatment,' or medication names often stay consistent, allowing these metrics to maintain a high overlap. In contrast, Prometheus (LLM judge) is the most affected by rewording, which is reasonable considering that, for this evaluation, correct punctuation and formatting in the answers greatly improve scores. This metric is followed by BLEURT and BERTScore (model similarity based) as the least resilient. More details can be found in Appendix C.1.

### Metrics self-consistency

Another issue that affects LLM evaluation, particularly on the open-ended setup, is the lack of self-consistency across model runs for some widespread sampling strategies, such as top_p and top_k. To evaluate its impact on open-ended evaluation, we generate and evaluate 11 responses for each prompt in CareQA-Open using top_p sampling, \(p=0.9\). Results can be seen in Figure 2. We observe that among n-gram metrics, BLEU and ROUGE2 are the most self consistent. BLEURT and Prometheus (LLM judge) are the less consistent. Perplexity metrics are perfectly self-consistent. More details can be found in Appendix C.2.

## 4 Relaxed Perplexity: A novel metric

By being optimized for next token prediction on the ground truth, LLM's are optimized for perplexity.

Figure 1: Correlation between the weighted average accuracy from the MCQA benchmarks and all other close-ended and open-ended tasks and metrics. These results correspond to the smaller models.



However, as seen before, this does not necessarily entail good performance on open or close-ended downstream tasks. Additionally, perplexity can be greatly impacted by instruct-tuning and alignment techniques Lee et al. (2024). On the other hand, it has been widely noted that models are more likely to arrive at the correct answer after outputting intermediate tokens, commonly known as chain of thought (CoT) Suzgun et al. (2022); Wang et al. (2023), and that this happens even without specific CoT prompting Wang and Zhou (2024). However, perplexity fails to capture this improvement, and can be negatively impacted by the presence of intermediate tokens.

To evaluate factuality in open-ended benchmarks, with no dependence on confounders or exact formulation while accounting for the potential benefits of intermediate tokens, we propose Relaxed Perplexity. Given a _question_ and a _target_, we wish to estimate

\[\mathbb{P}(target\sim\text{model}\mid question)=\] \[=\mathbb{P}(A_{0})+\ldots+\mathbb{P}(A_{n}\mid B_{n})\]

that is, the probability that the target is sampled from the model given the prompt, at any time in the completion. We denote the events \(A_{n}\equiv\{target\sim\text{model}(question+seq_{n})\}\) and \(B_{n}\equiv\{seq_{n}\sim\text{model}(question)\}\) for any \(seq_{n}\) of \(n\) tokens that comes from the model before the target. We can estimate \(\mathbb{P}(A_{n}\mid B_{n})\) as

\[\mathbb{P}(A_{n}\mid B_{n})\approx\mathbb{P}(A_{n}\mid seq^{i_{1}}_{n})+ \ldots+\mathbb{P}(A_{n}\mid seq^{i_{r}}_{n})\]

for the \(\ell\) more likely \(n\)-token sequences sampled from the model given _question_, because the events \(seq^{i}_{n}\) and \(seq^{j}_{n}\) are mutually exclusive. In this notation, \(\mathbb{P}(seq^{i_{r}}_{n}):=\mathbb{P}(seq^{i_{r}}_{n}\sim\text{model}(question))\). Using this, we can define Relaxed Perplexity as

Relaxed-Perplexity( \[target,question,model\] ) \[=\exp\left(-\frac{1}{n+len(target)}\sum_{i=0}^{n}logP(A_{i}\mid B _{i})\right)\]

This allows to evaluate correctness in the model's answers probability distribution, with no regard for the exact formulation. Further, for a given prompt and fixed sampling parameters, the metric is perfectly self consistent. We thus test it with the Olaph Jeong et al. (2024) medical factuality dataset. In contrast to Perplexity, we observe that Relaxed Perplexity assigns higher scores to models fine-tuned on healthcare datasets. More details on the mathematical formulation, implementation and results of Relaxed Perplexity can be found in Appendix D.

## 5 Conclusions

This study finds very weak correlations between close-ended and open-ended benchmarks. These results highlight the complementary roles of close-ended and open-ended approaches, and the limited insights provided by individual tests. It thus advocates for broader evaluation setups. Even within open-ended benchmarks targeting the same task (_e.g._, ACI-Bench and MTS-Dialog), no consistently high correlations were found. This indicates that different benchmarks assess distinct model capabilities, underscoring the significance of the benchmark's design.

The analysis of evaluation metrics for open-ended benchmarks identified three distinct clusters that are particularly relevant for assessing medical models: (1) perplexity-based metrics, (2) n-gram-based metrics combined with semantic similarity metrics, and (3) LLM-as-a-judge metrics. Notably, none of these clusters showed strong correlations with the close-ended MCQA evaluation. Additionally, differences in resilience to answer rephrasing and self-consistency were observed, due to the distinct ways these metrics are computed.

The findings highlight the importance of selecting appropriate benchmarks and evaluation metrics designed for specific tasks. In this regard, the introduced CareQA benchmark, featuring both closed- and open-ended formats, serves as a sanity check of existing tests, while the proposed Relaxed Perplexity metric fills a gap in evaluation by focusing on factuality and being resistant to exact formulations in an open-ended setting.

Figure 2: Mean variance distributions across different runs and averaged across models using the CareQA-Open dataset. Closer to 0 means more self-consistent.

 



## 6 Limitations

Since this study is based on specific models, the findings may not generalize to other LLM architectures. Additionally, the quality and diversity of the datasets used for evaluation are limited, meaning these benchmarks may not fully capture the performance of LLMs across the broader healthcare landscape. While metrics and benchmarks can indicate how well LLMs perform on certain tasks, they may not reflect the complexities of integrating LLMs into real-world healthcare practices.

In evaluating the models, we observed that applying the model's chat template to MCQA tasks led to decreased performance, whereas open-ended evaluations showed improvement. To ensure a fair comparison between open-ended and MCQA evaluations, we maintained the same configuration across both categories and did not apply the model's chat template to any of the evaluations.

Regarding the new benchmark introduced, although subject matter experts created the original exam materials, which underwent public scrutiny, CareQA has not been subjected to formal bias assessment. Consequently, it may not adequately represent the full spectrum of medical knowledge or encompass all possible patient demographics. Furthermore, although a human review was performed on the open-ended version, it has not undergone thorough evaluation by healthcare experts, raising the possibility of errors or biases introduced by the LLM used to rephrase the questions. Therefore, we advise users to exercise caution when interpreting and generalizing the results.

All experiments are conducted on English benchmarks (except for the Spanish version of CareQA), and generalization to other languages has not been considered. To enable reproducibility, all resources are made available. CareQA is accessible on Hugging Face1 and all new tasks are accessible in the original _lm-evaluation-harness_ framework2.

Footnote 1: [https://huggingface.co/datasets/HPAI-BSC/CareQA](https://huggingface.co/datasets/HPAI-BSC/CareQA)

Footnote 2: [https://github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)

## Acknowledgements

This work is supported by Anna Arias Duart, Pablo Agustin Martin Torres and Daniel Hinjos Garcia fellowships within the "Generciion D" initiative, Red.es, Ministerio para la Transformacion Digital y de la Funcion Publica, for talent atraction C005/24-ED CV1). Funded by the European Union NextGenerationEU funds, through PRTR.

We also acknowledge the computational resources provided by the FinisTerrae III, Leonardo, and MareNostrum 5 supercomputers. We are particularly grateful to the Operations department at BSC for their technical support.

Lastly, we sincerely thank Jordi Bayarri-Planas, Atia Cortes, Orlando Montenegro and Oscar Molina for their valuable time and feedback during the human evaluation process.

## References

* A. Ahmad et al. (2023)Muammad Aurangzeb Ahmad, Ilker Yaramis, and Taposh Dutta Roy. 2023. Creating trustworthy llms: Dealing with hallucinations in healthcare ai. arXiv preprint arXiv:2311.01463. Cited by: SS1.
* A. Mota (2024)Llama 3 model card. External Links: Link Cited by: SS1.
* N. Alzahrani, H. Abdullah Alyanya, Y. Alnumayleh, S. Alrashed, S. Alsubaie, Y. Almunshaykeh, F. Mirza, N. Altotaibi, N. Altwairesh, A. Alowisheq, et al. (2024)When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781. Cited by: SS1.
* S. Bedi, Y. Liu, L. Orr-Ewing, D. Dash, S. Koyejo, A. Callahan, J. A. Fries, M. Wornow, A. Swaminathan, L. Soleymani Lehmann, et al. (2024)A systematic review of testing and evaluation of healthcare applications of large language models (llms). medRxiv. Note: pp. 2024-04 Cited by: SS1.
* A. Ben Abacha, C. Shivade, and D. Demner-Fushman (2019)Overview of the mediqa 2019 shared task on textual inference, question entailment and question answering. In ACL-BioNLP 2019, Cited by: SS1.
* A. Ben Abacha, W. Yim, Y. Fan, and T. Lin (2023)An empirical study of clinical note generation from doctor-patient encounters. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, pp. 2291-2302. External Links: Document, Link Cited by: SS1.
* Q. Chen, J. Du, Y. Hu, V. Kuttichi Keloth, X. Peng, K. Raja, R. Zhang, Z. Lu, and H. Xu (2023)Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv preprint arXiv:2305.16326. Cited by: SS1.
* W. Chiang, L. Zheng, Y. Sheng, A. Nikolas Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E Gonzalez, et al. (2024)Chatbot Arena: an Open

Platform for Evaluating LLMs by Human Preference. In _Forty-first International Conference on Machine Learning_.
* Christophe et al. (2024) Clement Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. 2024. Med42-v2: A suite of clinical llms. _arXiv preprint arXiv:2408.06142_.
* Dada et al. (2024) Amin Dada, Marie Bauer, Amanda Butler Contreras, Osman Alperen Koras, Constantin Marc Seibold, Kaleb E Smith, and Jens Kleesiek. 2024. Clue: A clinical language understanding evaluation for llvms. _arXiv preprint arXiv:2404.04067_.
* Pal et al. (2022) Ankit Pal et al. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In _Proceedings of the Conference on Health, Inference, and Learning_, volume 174 of _Proceedings of Machine Learning Research_, pages 248-260. PMLR.
* Hendrycks et al. (2020a) Dan Hendrycks et al. 2020a. Measuring Massive Multitask Language Understanding. In _International Conference on Learning Representations_.
* Jin et al. (2020b) Di Jin et al. 2020b. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. _arXiv preprint arXiv:2009.13081_.
* Jin et al. (2019) Qiao Jin et al. 2019. PubMedQA: A Dataset for Biomedical Research Question Answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2567-2577.
* Fleming et al. (2024) Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jimdal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al. 2024. Medalign: A clinician-generated dataset for instruction following with electronic medical records. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 22021-22030.
* Frick et al. (2024) Evan Frick, Peter Jin, Tianle Li, Karthik Ganesan, Jian Zhang, Jiantao Jiao, and Banghua Zhu. 2024. Athene-70b: Redefining the boundaries of post-training for open models.
* Hager et al. (2024) Paul Hager, Friederike Jungmann, Robbie Holland, Kunal Bhagat, Inga Hubrecht, Manuel Knauer, Jakob Vielhauer, Marcus Makowski, Rickmer Braren, Georgios Kaissis, et al. 2024. Evaluation and mitigation of the limitations of large language models in clinical decision-making. _Nature medicine_, 30(9):2613-2622.
* He et al. (2025) Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. 2025. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. _Information Fusion_, page 102963.
* Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. Camels in a changing climate: Enhancing lm adaptation with tubu 2. _Preprint_, arXiv:2311.10702.
* Jeong et al. (2024) Minbyul Jeong, Hyeon Hwang, Chanwoong Yoon, Taewhoo Lee, and Jaewoo Kang. 2024. Olaph: Improving factuality in biomedical long-form question answering. _arXiv preprint arXiv:2405.12701_.
* Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. _Scientific data_, 3(1):1-9.
* Kamalloo et al. (2023) Ehsan Kamalloo, Nouha Diziri, Charles Clarke, and Davood Rafiei. 2023. Evaluating open-domain question answering in the era of large language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5591-5606.
* Kanith et al. (2024) Praveen K Kanith, Clement Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, and Shadab Khan. 2024. Medic: Towards a comprehensive framework for evaluating llms in clinical applications. _arXiv preprint arXiv:2409.07314_.
* Kim et al. (2024) Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minojon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. _arXiv preprint arXiv:2405.01535_.
* Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cordy Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In _Proceedings of the 29th Symposium on Operating Systems Principles_, pages 611-626.
* Lee et al. (2024) Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. _arXiv preprint arXiv:2401.01967_.
* Li et al. (2024) Wangyeu Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. Can multiple-choice questions really be useful in detecting the abilities of llms? _arXiv preprint arXiv:2403.17752_.
* Luo et al. (2022) Ling Luo, Po Ting Lai, Chih-Hsuan Wei, Cecilia N Arighi, and Zhiyong Lu. 2022. Biored: a rich biomedical relation extraction dataset. _Briefings in Bioinformatics_, 23(5):bbac282.
* Nimah et al. (2023) Iftitahu Nimah, Meng Fang, Vlado Menkovski, and Mykola Pechenizkiy. 2023. Nilg evaluation metrics beyond correlation analysis: An empirical metric 

preference checklist. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1240-1266.
* Pezeshkpour and Hruschka [2023] Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions. _arXiv preprint arXiv:2308.11483_.
* Schopf et al. [2023] Tim Schopf, Daniel Braun, and Florian Matthes. 2023. Evaluating unsupervised text classification: Zero-shot and similarity-based approaches. In _Proceedings of the 2022 6th International Conference on Natural Language Processing and Information Retrieval_, NLIPR '22, page 6-15, New York, NY, USA. Association for Computing Machinery.
* Shoham and Rappoport [2024] Ofir Ben Shoham and Nadav Rappoport. 2024. Medconceptsqa-open source medical concepts qa benchmark. _arXiv preprint arXiv:2405.07348_.
* Suzgun et al. [2022] Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. _arXiv preprint arXiv:2210.09261_.
* Team [2024] Gemma Team. 2024. Gemma.
* Umapathi et al. [2023] Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. 2023. Med-halit: Medical domain hallucination test for large language models. _arXiv preprint arXiv:2307.15343_.
* Vijayakumar et al. [2016] Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural sequence models. _arXiv preprint arXiv:1610.02424_.
* Wang et al. [2023] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023. Towards understanding chain-of-thought prompting: An empirical study of what matters. In _The 61st Annual Meeting Of The Association For Computational Linguistics_.
* Wang and Zhou [2024] Xuezhi Wang and Denny Zhou. 2024. Chain-of-thought reasoning without prompting. _arXiv preprint arXiv:2402.10200_.
* Yang et al. [2024] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. _arXiv preprint arXiv:2407.10671_.
* Yim et al. [2023] Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. Acibench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. _Nature Scientific Data_, 10.
* Young et al. [2024] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianquun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. _arXiv preprint arXiv:2403.04652_.
* Zeng et al. [2020] Guangtao Zeng, Wemian Yang, Zeqian Ju, Yue Yang, Sicheng Wang, Ruisi Zhang, Meng Zhou, Jiaqi Zeng, Xiangyu Dong, Ruoiu Zhang, et al. 2020. Meddialog: Large-scale medical dialogue datasets. In _Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)_, pages 9241-9250.
* Zheng et al. [2023] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. In _The Twelfth International Conference on Learning Representations_.
* Zhou et al. [2023] Hongjian Zhou, Boyang Gu, Xinyu Zou, Yiru Li, Sam S Chen, Peilin Zhou, Junling Liu, Ying Hua, Chengfeng Mao, Xian Wu, et al. 2023. A survey of large language models in medicine: Progress, application, and challenge. _arXiv preprint arXiv:2311.05112_.

 

Novel Benchmarks

### CareQA (close-ended)

CareQA is a novel benchmark for evaluating healthcare Large Language Models (LLMs) through multiple-choice question answering. CareQA was created by collecting exam materials in PDF format from the official Spanish government website. These documents were automatically parsed and then underwent post-processing to ensure data quality. This process involved removing 23 inaccurately parsed instances and excluding officially impugned questions. To enhance global accessibility, the original Spanish questions were translated into English using GPT-4.

Each CareQA sample contains metadata including a numeric exam identifier, full question text, four answer options, correct answer, exam year, and specialization category. The dataset is available in both Spanish and English, facilitating cross-lingual research. Examples of CareQA samples are provided in Figure 3 and Table 3.

While CareQA shares its source with HeadQA in the Spanish Specialised Healthcare Training (MIR) exams, there is no overlap between the datasets. CareQA expands upon its predecessor, covering the years 2020 to 2024 and comprising 5,621 questionnaire reset pairs, compared to HeadQA's 2,742 test pairs from 2013 to 2017. The dataset's composition is illustrated in Figure 5, showing the category distribution by year to reveal potential temporal trends in exam content.

Table 4 presents additional information about the dataset, including the total number of questions per category, the longest and average question and answer lengths (in tokens), and the overall vocabulary size. This comprehensive overview of CareQA's structure and content demonstrates its potential as a valuable resource for evaluating and improving healthcare-focused language models.

### CareQA (open-ended)

We developed the open-ended dataset by adapting the existing closed-ended CareQA dataset through the expansion of the English set. The first step was to filter out questions that contained terms such as "incorrect", "except", "false", "not correct", or "NOT", as these terms indicate that the questions focus on identifying incorrect answers among the provided options. After this filtering, we rephrased the remaining questions into an open-ended format using the Qwen2.5-72B-Instruct model, specifically instructing it to only rephrase questions that could be effectively transformed. This process excluded questions that explicitly ask for incorrect options or require a selection from the provided answers. We employed two different prompts for rephrasing, followed by a selection process to determine the best-rephrased version or to discard the question if neither was suitable.

Initially, the close-ended CareQA contained 5,621 QA pairs, but after the rephrasing process, the number of suitable questions for the open-ended version was reduced to 3,730 QA pairs. This new dataset retains the same categories as the closed-ended version, including medicine, nursing, biology, chemistry, psychology, and pharmacology.

Based on feedback from the human review (detailed in SSA.3), a second iteration of rephrasing was conducted, as illustrated in Figure 4. In this phase, the model was instructed to validate only questions that could be answered exclusively using the ground truth, ensuring there were no alternative correct answers. As a result, 961 questions were removed, reducing the CareQA (open-ended) dataset to a total of 2,769 QA pairs.

Figure 6 illustrates the distribution of these 2,769 QA pairs in the open-ended version and examples of QA pairs from both the close-ended and open-ended versions of the CareQA dataset are shown in Table 5. Both datasets are publicly available3.

Footnote 3: [https://huggingface.co/datasets/HPAI-BSC/CareQA](https://huggingface.co/datasets/HPAI-BSC/CareQA)

### Human evaluation

To validate the translations performed by GPT-4 for the English version of CareQA, as well as the rephrasing process executed by Qwen2.5-72B-Instruct for the open-ended CareQA, a human eval

Figure 3: CareQA example from Medicine category.

 

uation was conducted with 10 human evaluators, including 5 authors of this article.

We selected a total of 260 QA pairs for evaluation, covering both translation and rephrasing. This sample size ensures a confidence level of 95% with a margin of error of 5% for translation and 5.73% for rephrasing. Each question was evaluated by at least three evaluators.

The results are shown in Table 2 and correspond to the percentages of correct answers labeled by at least one evaluator, by two evaluators, and by all three evaluators. For both translation and rephrasing, the percentage of questions labeled as correct by at least one evaluator is high (98.6% for translation and 96.1% for rephrasing). However, when considering the cases where all three evaluators agreed on the correctness of the QA pair, the percentages drop: 83.1% for translation and 65.8% for rephrasing (first iteration).

For translation, the agreement percentage was considered sufficiently high, and the English dataset was deemed valid. In contrast, for the open-ended rephrasing version, the agreement rate was not high enough, so a second iteration of rephrasing, as explained in the previous section, was carried out. After removing invalid questions, the percentage of correct answers increased, see third column of Table 2. After this second iteration, the open dataset was also considered valid. The final agreement of both tasks grouped per category can be seen in Figure 7.

\begin{table}
\begin{tabular}{l c c c} \hline
**Agreement** & **Translation (\%)** & **Rephrasing (\%)** \\  & & **Iter 1** & **Iter 2** \\ \hline Correct (1/3) & 98.6 & 96.1 & 98.1 \\ Correct (2/3) & 96.7 & 85.8 & 92.8 \\ Correct (3/3) & 83.1 & 65.8 & 73.6 \\ Interrater & 84.4 & 69.7 & 75.5 \\ \hline \end{tabular}
\end{table}
Table 2: Evaluation results for translation and rephrasing. The first row shows the percentage of correct samples tagged by at least one evaluator. The second row refers to samples tagged as correct by two evaluators. The third row indicates samples labeled as correct by all three evaluators. The last row shows the agreement rate among the three evaluators.

Figure 4: Iterations with human evaluators to create the CareQA dataset in English, including both open and closed versions.

 



## Appendix A

\begin{table}
\begin{tabular}{c c c c c c c}  & \multicolumn{5}{c}{**CareQA**} \\ \hline  & **QA Pairs** & **Max Q tokens** & **Avg Q tokens** & **Max A tokens** & **Avg A tokens** & **Vocab** \\ \hline Medicine & 857 & 202 & 48.57 & 43 & 9.65 & 9626 \\ Nursing & 923 & 96 & 24.61 & 70 & 12 & 9113 \\ Pharmacology & 969 & 147 & 18.94 & 56 & 8.51 & 7906 \\ Biology & 966 & 51 & 12.82 & 48 & 6.6 & 6300 \\ Psychology & 962 & 208 & 22.60 & 67 & 9.92 & 7573 \\ Chemistry & 944 & 81 & 16.88 & 47 & 8.2 & 6022 \\ \hline \end{tabular}
\end{table}
Table 4: CareQA (close-ended) dataset statistics, where Q and A represents the Question and Answer respectively.

Figure 5: Category distribution per Category and Year (CareQA close-ended)

Figure 6: Category distribution per Category and Year (CareQA open-ended).

Figure 7: Correctness distribution per Category CareQA (open-ended).

 



## Appendix A

\begin{table}
\begin{tabular}{p{142.3pt} p{85.4pt} p{76.9pt} p{108.1pt}} \hline  & **Close-ended** & **Open-ended** & **Category** \\ \hline
**Question** & The best way to estimate the relative strength of hydrogen bonds between the molecules of halogen hydrides, H-X, is by measuring: & What is the best way to estimate the relative strength of hydrogen bonds between the molecules of halogen hydrides, H-X? & Chemistry \\ \cline{2-3}
**Answer** & The enthalpies of vaporization & The enthalpies of vaporization. & \\ \hline
**Question** & Taking into account the general principles regarding the minimum interval between the non-simultaneous administration of vaccines, identify the minimum interval between 2 attenuated vaccines: & What is the minimum interval the non-simultaneous administration of two attenuated vaccines, according to general principles? & Nursing \\ \cline{2-3}
**Answer** & Four weeks. & Four weeks. & \\ \hline
**Question** & We evaluated in the emergency room an adult person who is irritable, yawning, complaining of muscle pain and cramps. They are nauseous and have notable tearing. The pupils are dilated. Which of the following is the most probable diagnosis? & An adult patient presents to the emergency room with irritability, yawning, muscle pain and cramps, nausea, notable tearing, and dilated pupils. What is the most probable diagnosis based on these symptoms? & Medicine \\ \hline
**Answer** & Opioid abstinence. & Opioid abstinence. & \\ \hline \end{tabular}
\end{table}
Table 5: Examples of QA pairs: On the left, the close-ended version from CareQA, and on the right, the open-ended version.





## Appendix B Correlations

### Correlations between MCQA and Elo results

We perform a correlation analysis on the performance results of the medical MCQA benchmarks listed in Table 1. Additionally, we include Elo scores from the Chatbot Arena4, a crowdsourcing platform that collects pairs of model-generated answers in response to user prompts, where the user selects the winning model based on their criteria.

Footnote 4: [https://lmarena.ai/](https://lmarena.ai/)

We conducted a correlation analysis using both small and medium models. The small models used for the correlation shown in Figure 8 are as follows: gemma-2-9b-it Team (2024), Meta-Llama-3.1-8B-InstructAI@Meta, 2024), Mistral-7B-Instructv0.2, Mistral-7B-Instruct-v0.3, Phi-3-mini-4k-instruct, Phi-3-medium-4k-instruct, Qwen1.5-7B-Chat, Starling-LM-7B-beta and Yi-1.5-9B-Chat. And the medium models used in Figure 9 are as follows: Athene-70B(Frick et al., 2024), tulu-2-dpo-70b(Ivison et al., 2023), Yi-1.5-34B-Chat, gamma-2-27b-it, Llama-3.1-70B-Instruct, Mixtral-8x7B-Instruct-v0.1, Owen2-72B-InstructYang et al., 2024), and WizardLM-70B-V1.0

From this analysis, we found that MedQA, MedMCQA, Care0A, and MMLU are highly correlated with one another. However, PubMedQA exhibits a noticeably lower correlation with the other medical benchmarks, particularly in smaller models.

Regarding the Elo scores, we observe a moderate correlation with the MCQA benchmarks, with the correlation being significantly stronger for larger models. This is likely due to larger models' ability to produce more coherent responses. Non-expert evaluators, such as those in the Elo scoring system, may favor responses that are well-structured and fluent, even if they lack precise medical accuracy. As a result, this preference for more polished answers could lead to a higher correlation with MCQA performance.

Figure 8: Comparison of correlations between MCQA benchmarks and ELO results for small models.

Figure 9: Comparison of correlations between MCQA benchmarks and ELO results for medium models.





### Correlation between metrics

In this correlation analysis, we fix the open-ended benchmark and examine the correlations across the various computed metrics. Figure 10, presents the correlation matrix for the benchmark focused on making diagnosis and treatment recommendations, highlighting the three clusters of metrics identified in the paper. This correlation matrix was also computed for the rest of benchmarks revealing three similar clusters. The matrices were computed using the following models: BioMistral-MedMNX, JSL-MedLlama-3-8B-v2.0, Phi-3-mini-4k-instruct, Mistral-7B-Instruct-v0.3, Qwen2-7B-Instruct Yang et al. (2024), Llama3-Med42-8B Christophe et al. (2024), Meta-Llama-3.1-8B-InstructAI@Meta (2024) Yi-1.5-9B-Chat Young et al. (2024), Phi-3-medium-4k-instruct, Yi-1.5-34B-Chat Young et al. (2024), Mixtral-8x7B-Instruct-v0.1.

### Correlations of benchmarks

In this correlation analysis we study the relationships between specific metrics across all the opened benchmarks implemented. As stated in the paper, no consistent high correlation was observed among all metrics for any benchmark or task. Examples of these correlation matrices are shown in Figures 11 and 12. The models used to generate these correlation matrices are the same as those described in the Appendix B.2.

## Appendix C Resilience to rephrasing and self-consistency

### Resilience

As described earlier, we conducted this experiment by rephrasing the model outputs six times and recomputing the metrics. We used both Qwen2.5-72B-Instruct and Meta-Llama-70B-Instruct with the following _system_prompt_: "You are a helpful rephrasing assistant. Rephrase the prompt provided without changing its original meaning, but do not try to address or answer it in any case."

We run the script 5 times on recorded model answers with top_p sampling to obtain several rephrasings of each answer. After manual inspection, the outputs of Qwen2.5-72B-Instruct were deemed of higher quality.

Figure 13 shows the mean variance across all runs for the MEDIQA2019 dataset. Before plotting, we scale variances by dividing by the max interval (max value - min value) in each column. Figures 14 and 15 present the variance distributions for two specific models. Figure 14 displays the results for the Phi-3-mini-4k-instruct model, while Figure 15

Figure 11: Correlations of BERTScore across benchmarks.

Figure 12: Correlation of Prometheus scores across benchmarks.

Figure 10: This correlation matrix illustrates the relationships among the different open-ended metrics used to evaluate the benchmark for diagnosis and treatment recommendations. Three distinct clusters of metrics are identified: (1) perplexity metrics, (2) n-gram and semantic similarity metrics, and (3) Prometheus metrics.



shows the results for the Yi-1.5-9B-Chat model.

In Figure 13 we can observe three different clusters: rouge metrics (low mean-variance, low meta-variance), bleu and moverscore (low mean-variance, medium meta-variance) and bert_score, bleurt, prometheus (high mean variance, high meta-variance).

### Self-consistency

As described earlier, we conducted this experiment by prompting models with each question in CareQA-Open for a number of repetitions (\(r\)). We fix \(r=11\). Sampling parameters used where top_p = 0.9 and temperature = 1. We compute variances per prompt, and then average across models. Results can be seen in Figure 2. Besides, we compute the coefficient of variation, defined for prompt \(p\) as:

\[CV(p)=\frac{1}{\mu_{P}}\sqrt{\frac{\sum_{i}(x_{i}-\mu_{P})^{2}}{N}}\]

Then we average across models, and plot the \(CV\) distribution for all prompts in CareQA-Open. Results can be seen in Figure 16. From this computation we remove the BLEURT metric, for it can take negative values.

## Appendix D Novel Metric: Relaxed Perplexity

As mentioned before, we define RelaxedPerplexityas

Relaxed-Perplexity(\(target,question,model\)) = \(=\exp\left(-\dfrac{1}{n+len(target)}\sum_{i=0}^{n}logP(A_{i}\mid B_{ i})\right)\) for events \(A_{n}\equiv\{target\sim\text{model}(question+seq_{n})\}\) and \(B_{n}\equiv\{seq_{n}\sim\text{model}(question)\}\).

Figure 14: Mean variance distributions across different rephrasings using the Phi-3-mini-4k-instruct model and the MEDIQA2019 dataset. Each metric is represented by a different color.

Figure 13: Mean variance distributions across different rephrasings and models using the MEDIQA2019 dataset. Each metric is represented by a different color.

Figure 15: Mean variance distributions across different rephasings using the Yi-1.5-9B-Chat model and the MEDIQA2019 dataset. Each metric is represented by a different color.

 

That is, \(A_{n}\) is the event that target is sampled from the model inputted with \(question+seq_{n}\), for any \(seq_{n}\) of \(n\) tokens.

Thus, in order to compute \(\mathbb{P}(A_{n}\mid B_{n})\) we need to take into account the probability distribution of all \(n\)-token model answers when the input is \(question\), which is extremely costly (with computational time exponential in \(n\)). In fact, by the law of total probability we would have

\[\begin{split}\mathbb{P}(A_{n}\mid B_{n})&\;\mathbb{ P}(B_{n})=\mathbb{P}(A_{n}\mid seq^{1}_{n})\;\mathbb{P}(seq^{1}_{n})+\cdots\\ &\qquad+\mathbb{P}(A_{n}\mid seq^{q^{n}}_{n})\;\mathbb{P}(seq^{ q^{n}}_{n})\end{split}\]

\(q\) being the size of the vocabulary. This holds because the events \(seq^{i}_{n}\) and \(seq^{j}_{n}\) are mutually exclusive. In this notation, \(\mathbb{P}(seq^{i_{L}}_{n}):=\mathbb{P}(seq^{i_{L}}_{n}\sim\text{model}( question))\), and also \(\mathbb{P}(B_{n})=\;\mathbb{P}(\cup_{i}seq^{i_{L}}_{n})\).

However, given that almost all this combinations of tokens contribute with negligible probabilities to the sum, we can estimate the above quantity as

\[\begin{split}\mathbb{P}(A_{n}\mid B_{n})&\approx \mathbb{P}(A_{n}\mid seq^{i_{1}}_{n})\;\mathbb{P}(seq^{i_{1}}_{n})+\ldots\\ &\qquad+\mathbb{P}(A_{n}\mid seq^{i_{L}}_{n})\;\mathbb{P}(seq^{ i_{L}}_{n})\end{split}\]

for the \(\ell\) more likely \(n\)-token sequences sampled from the model given _question_, which can be computed efficiently using beam search, diverse beam search (Vijayakumar et al., 2016) or top_p sampling.

Notice that also \(\mathbb{P}(B_{n})=1\) unless stop tokens appeared before in the completion, and then the value decreases for big \(n\). In our implementation, where \(max\_tokens\in[128,256]\), stop tokens rarely appear and so we estimate \(\mathbb{P}(B_{n})\approx 1\).

Now, there is an issue with this formulation. We noticed that, since \(\mathbb{P}(seq^{i}_{n})\) is the joint probability of all tokens in the sequence, as \(n\) grows this value collapses very quickly. In fact, among the \(\ell\) most likely sequences, we may bound

\[\frac{1}{c_{n}}\leq\mathbb{P}(seq^{i}_{n})\leq\frac{1}{d_{n}}\]

for constants \(c_{n}\) and \(d_{n}\) that only depend on \(n\) (for example, take the average and max prob of sequences of that length respectively; also, notice \(d_{n}\leq n\) ). And thus we may take

\[\begin{split}\mathbb{P}(A_{n}\mid B_{n})&\approx\\ &\frac{c_{n}+d_{n}}{2c_{n}d_{n}}\left(\mathbb{P}(A_{n}\mid seq^{ i_{1}}_{n})+\ldots+\mathbb{P}(A_{n}\mid seq^{i_{l}}_{n})\right.\end{split}\]

This effectively assigns more value to the target appearing earlier in the completion, benefiting models that do not verbose and biasing comparisons without adding real value, for this constant does not depend on the target. In order to deal with this, we _skew_ the models distribution with respect to length by multiplying with the inverse of the constant, and end up with the final approximation:

\[\mathbb{P}(A_{n}\mid B_{n})\approx\mathbb{P}(A_{n}\mid seq^{i_{1}}_{n})+\ldots+ \mathbb{P}(A_{n}\mid seq^{i_{l}}_{n})\]

Notice this step may be omitted depending on the evaluation goal.

Relaxed Perplexity is specifically designed to evaluate factuality in the answers, with no regard for the exact formulation. We thus test it with the OLAPH (Jeong et al., 2024) dataset, and note that for more effective evaluation of other open-ended benchmarks, some preprocessing of the ground truths must be carried out.

For our experiments we use top-p sampling, selecting the \(\ell\in\{5,10\}\) best sentences in a search space of \(s\in\{10,100\}\). We observe similar results with all combinations, and so fix \(\ell=5\) and \(s=10\) for better performance.

Figure 17: Correlation between OLAPH - Relaxed Perplexity and the rest of benchmarks.

 

We add another hyperparameter, which we denote as _stride_, for better efficiency. Instead of computing \(\sum_{i=0}^{n}logP(A_{i}~{}|~{}B_{i})\) we compute \(\sum_{i=0,i+\textit{stride}}^{n}logP(A_{i}~{}|~{}B_{i})\), which we find to be as effective. We select \(stride\in\{8,16\}\).

The implementation is built using vllm5, which provides tools for efficient LLM inference Kwon et al. (2023). It remains as future work to implement Relaxed Perplexity with beam search.

Footnote 5: [https://github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)

### Connection with cross-entropy

The exponent of perplexities can be understood as a cross-entropy. Generally, it corresponds to the bits required to encode the correct answer using the model's distribution. In the case of Relaxed Perplexity we have:

\[\text{H}(q,P)=-\sum_{i=0}^{n}logP(A_{i}~{}|~{}B_{i})\]

This is the cross entropy between two distributions, \(q\) and \(P\), where \(q\) is the delta distribution of the target appearing in the correct position, and \(P\) the model's distribution. Thus, this could be understood as the bits required to encode the correct answer _anywhere_ in the completion (up to \(n\) steps), using the model's (skewed) distribution.

See Table 6 for an example usage to evaluate model factuality on healthcare benchmarks. Here, we report Relaxed-CrossEntropy instead of Relaxed Perplexity.

## Appendix E Evaluation Results

Figure 18: Ranking results for all models on the OLAPH medical factuality dataset for all metrics. The top position is ranked as 1 and the lowest as 11. Different models are represented in distinct colors. It can be seen there is low agreement across metrics.

 



\begin{table}
\begin{tabular}{l|c c c|c c c|c c c} \hline \multirow{2}{*}{**Model**} & \multicolumn{6}{c|}{**Global Non-taking**} & \multicolumn{5}{c|}{**Multi-tasking**} \\  & \multicolumn{3}{c|}{**ACTBench**} & \multicolumn{3}{c|}{**MTTS Eclipse**} & \multicolumn{3}{c|}{**OLAF**} \\  & **Rink per byte: i.e.** & **Byte Payolsky j** & **Word Popularity j** & **Bink per byte: i.e.** & **Byte Payolsky j** & **Word Popularity j** & **Bink per byte: i.e.** & **Byte Payolsky j** & **Word Popularly j** \\ \hline BioExtond Multi-MEDEX & 0.049 & 1,577 & 13,584 & 1,099 & 2,093 & 12,827 & 0.447 & 1,358 & 7,108 \\ JSL MedLism-348-2.0+2.0 & 0.370 & 1,648 & 21,725 & 1,099 & 2,143 & 108,188 & 0.532 & 1,407 & 9,078 \\ Liumium-36402-3b & 0.485 & 1,399 & 8,357 & 1,060 & 2,085 & 133,416 & 0.450 & 1,366 & 7,211 \\ Meta-Lums-31-19-known & - & - & - & 0.984 & 1,978 & 939,346 & 2,230 & 4,601 & 18984857 \\ Meta-Lums-31-19-known & 0.621 & 1,559 & 14,681 & 1,074 & 2,105 & 142,221 & 2,181 & 4,593 & 1451300702 \\ Muon-78-knowes-0.3 & 0.956 & 1,852 & 13,628 & 1,653 & 2,074 & 120,876 & 0.438 & 1,355 & 6,888 \\ Mutant-8C-from



\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c} \hline \hline
**Model** & \multicolumn{20}{c|}{**Question Entailment**} & \multicolumn{6}{c|}{**Open-ended Medical Questions**} & \multicolumn{3}{c}{**Treatment recommendations**} \\  & \multicolumn{1}{c|}{**MedDialog Qsumm**} & \multicolumn{1}{c|}{**MedDiago Raw**} & \multicolumn{1}{c|}{**MedDiago Raw**} & \multicolumn{1}{c|}{**MediQA2019**} & \multicolumn{1}{c|}{**Careo4-Open**} & \multicolumn{1}{c|}{**MedText**} & \multicolumn{1}{c|}{**Prometheus**} & \multicolumn{1}{c|}{**Prometheus**} & \multicolumn{1}{c}{**Treatent recommendation**} \\ \hline BioMistral-MedMNX & 0.163 \(\pm\) 0.005 & 0.330 \(\pm\) 0.016 & 0.273 \(\pm\) 0.027 & 0.240 \(\pm\) 0.007 & 0.297 \(\pm\) 0.009 \\ JSL-MedLlama-3-8Bv-v2.0 & 0.087 \(\pm\) 0.004 & 0.298 \(\pm\) 0.017 & 0.365 \(\pm\) 0.031 & 0.302 \(\pm\) 0.008 & 0.172 \(\pm\) 0.008 & 0.172 \(\pm\) 0.008 \\ Llama3-Med42-SB & 0.241 \(\pm\) 0.007 & 0.213 \(\pm\) 0.016 & 0.157 \(\pm\) 0.024 & 0.105 \(\pm\) 0.005 & 0.130 \(\pm\) 0.008 & 0.150 \(\pm\) 0.008 & 0.150 \(\pm\) 0.008 \\ Meta-Llama-3-1-70B-Instruct & 0.314 \(\pm\) 0.007 & 0.342 \(\pm\) 0.016 & 0.313 \(\pm\) 0.026 & 0.313 \(\pm\) 0.007 & 0.281 \(\pm\) 0.





\begin{table}
\begin{tabular}{l|c c c c c c c} \hline \hline
**Model** & \multicolumn{6}{c}{**Medical factuality**} \\  & \multicolumn{6}{c}{**OLAPH**} \\  & **BERTScore**\(\uparrow\) & **BLEU**\(\uparrow\) & **BLEURT**\(\uparrow\) & **MoverScore**\(\uparrow\) & **ROUGE1**\(\uparrow\) & **ROUGE2**\(\uparrow\) & **ROUGEL**\(\uparrow\) \\ \hline BioMistral-MedMNX & 0.864 \(\pm\) 0.001 & 0.022 \(\pm\) 0.002 & -0.557 \(\pm\) 0.014 & 0.555 \(\pm\) 0.001 & 0.211 \(\pm\) 0.004 & 0.058 \(\pm\) 0.002 & 0.166 \(\pm\) 0.003 \\ JSL-MedLlama-3-SB-v2.0 & 0.868 \(\pm\) 0.001 & 0.031 \(\pm\) 0.003 & -0.544 \(\pm\) 0.019 & 0.558 \(\pm\) 0.002 & 0.230 \(\pm\) 0.005 & 0.071 \(\pm\) 0.004 & 0.183 \(\pm\) 0.005 \\ Llama3-Medd2-8B & 0.876 \(\pm\) 0.001 & 0.024 \(\pm\) 0.002 & -0.387 \(\pm\) 0.015 & 0.567 \(\pm\) 0.001 & 0.239 \(\pm\) 0.005 & 0.069 \(\pm\) 0.004 & 0.185 \(\pm\) 0.005 \\ Meta-Llama-3.1-70B-Instruct & 0.866 \(\pm\) 0.001 & 0.021 \(\pm\) 0.002 & -0.538 \(\pm\) 0.017 & 0.559 \(\pm\) 0.001 & 0.225 \(\pm\) 0.005 & 0.064 \(\pm\) 0.004 & 0.178 \(\pm\) 0.005 \\ Meta-Llama-3.1-8B-Instrct & 0.845 \(\pm\) 0.001 & 0.009 \(\pm\) 0.001 & -0.792 \(\pm\) 0.015 & 0.538 \(\pm\) 0.001 & 0.166 \(\pm\) 0.004 & 0.038 \(\pm\) 0.002 & 0.129 \(\pm\) 0.003 \\ Mistral-7B-Instruct-v0.3 & 0.886 \(\pm\) 0.001 & 0.056 \(\pm\) 0.005 & -0.285 \(\pm\) 0.022 & 0.581 \(\pm\) 0.002 & 0.293 \(\pm\) 0.008 & 0.110 \(\pm\) 0.006 & 0.240 \(\pm\) 0.007 \\ Mixstral-8x7B-Instruct-v0.1 & 0.810 \(\pm\) 0.003 & 0.000 \(\pm\) 0.000 & -1.148 \(\pm\) 0.015 & 0.501 \(\pm\) 0.001 & 0.081 \(\pm\) 0.004 & 0.003 \(\pm\) 0.001 & 0.067 \(\pm\) 0.003 \\ Phi-3-medium-4k-instruct & 0.880 \(\pm\) 0.002 & 0.047 \(\pm\) 0.005 & -0.369 \(\pm\) 0.022 & 0.574 \(\pm\) 0.002 & 0.274 \(\pm\) 0.007 & 0.096 \(\pm\) 0.006 & 0.221 \(\pm\) 0.007 \\ Phi-3-mini-4k-instrct & 0.867 \(\pm\) 0.002 & 0.025 \(\pm\) 0.003 & -0.494 \(\pm\) 0.022 & 0.559 \(\pm\) 0.002 & 0.220 \(\pm\) 0.007 & 0.063 \(\pm\) 0.004 & 0.177 \(\pm\) 0.006 \\ Qwen2-7B-Instruct & 0.876 \(\pm\) 0.001 & 0.033 \(\pm\) 0.003 & -0.349 \(\pm\) 0.014 & 0.570 \(\pm\) 0.001 & 0.250 \(\pm\) 0.005 & 0.076 \(\pm\) 0.003 & 0.200 \(\pm\) 0.004 \\ Yi-1.5-34B-Chat & 0.879 \(\pm\) 0.001 & 0.041 \(\pm\) 0.003 & -0.371 \(\pm\) 0.016 & 0.570 \(\pm\) 0.002 & 0.269 \(\pm\) 0.006 & 0.092 \(\pm\) 0.004 & 0.216 \(\pm\) 0.005 \\ Yi-1.5-9B-Chat & 0.878 \(\pm\) 0.001 & 0.037 \(\pm\) 0.002 & -0.349 \(\pm\) 0.012 & 0.569 \(\pm\) 0.001 & 0.253 \(\pm\) 0.004 & 0.083 \(\pm\) 0.003 & 0.203 \(\pm\) 0.004 \\ \hline \end{tabular}
\end{table}
Table 15: Medical factuality results.





\begin{table}
\begin{tabular}{l|c c c c c c c c c} \hline \hline
**Model** & \multicolumn{8}{c}{**Open-ended medical questions**} \\  & & \multicolumn{8}{c}{**Care04-Open**} \\  & **BERTScore**\(\uparrow\) & **BLEU**\(\uparrow\) & **BLEU**\(\uparrow\) & **MEURT**\(\uparrow\) & **MoveScore**\(\uparrow\) & **ROUGEL**\(\uparrow\) & **ROUGEL**\(\uparrow\) & **ROUGEL**\(\uparrow\) \\ \hline BioMistral-MedMNX & 0.816 ± 0.002 & 0.002 ± 0.000 & -1.329 ± 0.009 & 0.492 ± 0.001 & 0.066 ± 0.002 & 0.017 ± 0.001 & 0.058 ± 0.002 \\ JSL-MedLlama-3-8Bvz0 & 0.827 ± 0.001 & 0.003 ± 0.000 & -1.234 ± 0.009 & 0.493 ± 0.001 & 0.069 ± 0.002 & 0.019 ± 0.001 & 0.060 ± 0.002 \\ Llama3-Med42-8B & 0.293 ± 0.010 & 0.002 ± 0.001 & -1.441 ± 0.010 & 0.503 ± 0.001 & 0.030 ± 0.002 & 0.006 ± 0.001 & 0.027 ± 0.002 \\ Meta-Llama-3.1-70B-Instruct & 0.660 ± 0.007 & 0.005 ± 0.001 & -1.283 ± 0.010 & 0.508 ± 0.001 & 0.096 ± 0.003 & 0.031 ± 0.002 & 0.087 ± 0.003 \\ Meta-Llama-3.1-8B-Instruct & 0.761 ± 0.004 & 0.002 ± 0.000 & -1.496 ± 0.007 & 0.485 ± 0.001 & 0.049 ± 0.001 & 0.013 ± 0.001 & 0.042 ± 0.001 \\ Mistral-7B-Instructor-v0.3 & 0.841 ± 0.002 & 0.004 ± 0.001 & -1.212 ± 0.026 & 0.501 ± 0.003 & 0.109 ± 0.008 & 0.037 ± 0.006 & 0.098 ± 0.008 \\ Mikstral-8x/7B-Instructor-v0.1 & 0.768 ± 0.010 & 0.008 ± 0.001 & -1.140 ± 0.022 & 0.515 ± 0.003 & 0.126 ± 0.007 & 0.040 ± 0.004 & 0.114 ± 0.007 \\ Phi-3-medium-4k-instruct & 0.814 ± 0.003 & 0.005 ± 0.001 & -1.276 ± 0.010 & 0.499 ± 0.001 & 0.089 ± 0.003 & 0.028 ± 0.001 & 0.077 ± 0.002 \\ Phi-3-mini-4k-instruct & 0.684 ± 0.008 & 0.003 ± 0.001 & -1.277 ± 0.010 & 0.500 ± 0.001 & 0.064 ± 0.002 & 0.016 ± 0.001 & 0.054 ± 0.002 \\ Qwen2-7B-Instruct & 0.755 ± 0.005 & 0.003 ± 0.000 & -1.229 ± 0.008 & 0.496 ± 0.001 & 0.067 ± 0.002 & 0.018 ± 0.001 & 0.057 ± 0.001 \\ Yi-1.5-34B-Chat & 0.809 ± 0.003 & 0.005 ± 0.001 & -1.186 ± 0.008 & 0.496 ± 0.001 & 0.078 ± 0.002 & 0.024 ± 0.001 & 0.067 ± 0.002 \\ Yi-1.5-9B-Chat & 0.831 ± 0.001 & 0.004 ± 0.000 & -1.180 ± 0.008 & 0.491 ± 0.001 & 0.079 ± 0.002 & 0.023 ± 0.001 & 0.066 ± 0.002 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Results for CareQA-Open.

\begin{table}
\begin{tabular}{l|c



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} \hline \hline
**Model** & \multicolumn{10}{c}{**Case-ended**} \\  & **MehlCGQ3** & **MehlQA** & **CarvedQion** (w) & **CarvedQion** (w) & **millimajor** & **PubMedQA** & **Tad Text Classification** & **Med Transactions** & **RinkED** & **MMLU** , \\ \hline BioMarki-WebDRX & 0.495 \(\pm\) 0.008 & 0.515 \(\pm\) 0.014 & 0.657 \(\pm\) 0.006 & 0.546 \(\pm\) 0.007 & 0.547 \(\pm\) 0.006 & 0.776 \(\pm\) 0.019 & 0.272 \(\pm\) 0.011 & 0.356 \(\pm\) 0.007 & 0.216 \(\pm\) 0.013 & 0.678 \(\pm\) 0.003 \\ FSL-McLlama-38b-7.20 & 0.633 \(\pm\) 0.008 & 0.617 \(\pm\) 0.014 & 0.672 \(\pm\) 0.006 & 0.572 \(\pm\) 0.007 & 0.684 \(\pm\) 0.006 & 0.742 \(\pm\) 0.020 & 0.191 \(\pm\) 0.010 & 0.361 \(\pm\) 0.007 & 0.254 \(\pm\) 0.004 & 0.739 \(\pm\) 0.0305 \\ Llama-38d-62ab-88 & 0.603 \(\pm\) 0.008 & 0.629 \(\pm\) 0.014 & 0.683 \(\pm\) 0.006 & 0.573 \(\pm\) 0.007 & 0.682 \(\pm\) 0.006 & 0.772 \(\pm\) 0.019 & 0.328 \(\pm\) 0.011 & 0.537 \(\pm\) 0.007 & 0.203 \(\pm\) 0.013 & 0.735 \(\pm\) 0.013 \\ Meta-Llama-3.1-72bg-Internal & 0.722 \(\pm\) 0.007 & 0.798 \(\pm\) 0.011 & 0.857 \(\pm\) 0.005 & 0.825 \(\pm\) 0.005 & 0.764 \(\pm\) 0.005 & 0.880 \(\pm\) 0.018 & 0.145 \(\pm\) 0.003 & 0.381 \(\pm\) 0.007 & 0.515 \(\pm\) 0.006 & 0.8711 \(\pm\) 0.0236 \\ Meta-Llama-3.1-8b-Insortx & 0.593 \(\pm\) 0.008 & 0.637 \(\pm\) 0.013 & 0.700 \(\pm\) 0.006 & 0.592 \(\pm\) 0.007 & 0.658 \(\pm\) 0.006 & 0.752 \(\pm\) 0.019 & 0.161 \(\pm\) 0.003 & 0.534 \(\pm\) 0.007 & 0.232 \(\pm\) 0.013 & 0.762 \(\pm\) 0.003 \\ Main-Llama-3.0-8b-Insortx & 0.402 \(\pm\) 0.008 & 0.523 \(\pm\) 0.014 & 0.607 \(\pm\) 0.007 & 0.529 \(\pm\) 0.007 & 0.658 \(\pm\) 0.006 & 0.774 \(\pm\) 0.019 & 0.178 \(\pm\) 0.001 & 0.356 \(\pm\) 0.007 & 0.388 \(\pm\) 0.015 & 0.661 \(\pm\) 0.0045 \\ Meta-Soft-Nat-10.50 & 0.549 \(\pm\) 0.008 & 0.516 \(\pm\) 0.014 & 0.712 \(\pm\) 0.006 & 0.688 \(\pm\) 0.006 & 0.652 \(\pm\) 0.006 & 0.796 \(\pm\) 0.018 & 0.277 \(\pm\) 0.011 & 0.544 \(\pm\) 0.007 & 0.332 \(\pm\) 0.015 & 0.796 \(\pm\) 0.0024 \\ Ph-3-semix-4k-innordx & 0.653 \(\pm\) 0.007 & 0.556 \(\pm\) 0.014 & 0.769 \(\pm\) 0.006 & 0.718 \(\pm\) 0.006 & 0.661 \(\pm\) 0.006 & 0.732 \(\pm\) 0.018 & 0.048 \(\pm\) 0.002 & 0.365 \(\pm\) 0.007 & 0.251 \(\pm\) 0.004 & 0.823 \(\pm\) 0.0275 \\ Ph-3-semix-4k-insortx & 0.572 \(\pm\) 0.008 & 0.537 \(\pm\) 0.014 & 0.701 \(\pm\) 0.008 & 0.585 \(\pm\) 0.007 & 0.684 \(\pm\) 0.006 & 0.752 \(\pm\) 0.019 & 0.192 \(\pm\) 0.003 & 0.367 \(\pm\) 0.007 & 0.282 \(\pm\) 0.014 & 0.739 \(\pm\) 0.0321 \\ Quenz-29-Intreval & 0.551 \(\pm\) 0.008 & 0.570 \(\pm\) 0.014 & 0.689 \(\pm\) 0.006 & 0.652 \(\pm\) 0.006 & 0.596 \(\pm\) 0.006 & 0.742 \(\pm\) 0.020 & 0.225 \(\pm\) 0.011 & 0.363 \(\pm\) 0.007 & 0.197 \(\pm\) 0.013 & 0.737 \(\pm\) 0.003 \\ Yt-15-34B-Chat & 0.575 \(\pm\) 0.008 & 0.614 \(\pm\) 0.014 & 0.733 \(\pm\) 0.006 & 0.652 \(\pm\) 0.006 & 0.652 \(\pm\) 0.006 & 0.774 \(\pm\) 0.019 & 0.301 \(\pm\) 0.012 & 0.345 \(\pm\) 0.007 & 0.543 \(\pm\) 0.016 & 0.796 \(\pm\) 0.0289 \\ Yi-15-59B-Char & 0.488 \(\pm\) 0.008 & 0.515 \(\pm\) 0.014 & 0.650 \(\pm\) 0.006 & 0.587 \(\pm\) 0.007 & 0.546 \(\pm\) 0.006 & 0.774 \(\pm\) 0.019 & 0.227 \(\pm\) 0.011 & 0.330 \(\pm\) 0.007 & 0.537 \(\pm\) 0.016 & 0.200 \(\pm\) 0.0239 \\ \hline \end{tabular}
\end{table}
Table 19: Summarization results.

\begin{table}
\begin{tabular}{l|c c c c c c c c} \hline \hline
**Model** & \multicolumn{10}{c}{**Summarization**} \\  & **F1-RadGraph**\(\uparrow\) & **BERTScore**\(\uparrow\) & **BLEU**\(\uparrow\) & **BLEURT**\(\uparrow\) & **MoveScore**\(\uparrow\) & **ROUGEL**\(\uparrow\) & **ROUGEL**\(\downarrow\) \\ \hline BioMixtral-MedMNX & 0.089 \(\pm\) 0.001 & 0.837 \(\pm\) 0.000 & 0.009 \(\pm\) 0.000 & -0.796 \(\pm\) 0.003 & 0.551 \(\pm\) 0.000 & 0.130 \(\pm\) 0.001 & 0.031 \(\pm\) 0.001 & 0.110 \(\pm\) 0.001 \\ JSL-MedLDA-38b-v2.0 & 0.079 \(\pm\) 0.002 & 0.841 \(\pm\) 0.000 & 0.014 \(\pm\) 0.001 & -0.780 \(\pm\) 0.005 & 0.556 \(\pm\) 0.001 & 0.143 \(\pm\) 0.002 & 0.041 \(\pm\) 0.001 & 0.124 \(\pm\) 0.002 \\ Liama3-Med42-8B & 0.093 \(\pm\) 0.002 & 0.843 \(\pm\) 0.000 & 0.013 \(\pm\) 0.001 & -0.729