RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
The growing use of large language models (LLMs) in public domains, such as healthcare, shows promise for improving global quality of life He et al. ( 2025 ) . At the same time, the reliability and evaluation of LLMs in such sensitive topics requires extreme caution due to the potential impact on people’s rights and well-being.

LLM evaluation today is approached through various perspectives, which consider different types of LLM assessment: automatic evaluation (scalable and factual), user evaluation (utility and usability) Chiang et al. ( 2024 ) , and expert evaluation (support and coherence) Chen et al. ( 2023 ) . While each of these evaluation perspectives serves distinct roles that contribute to a holistic assessment, automatic evaluation remains the most prevalent one due to its lack of dependency on human effort.

Within automatic evaluation, there are two types of tests. Those which include closed-ended responses Bedi et al. ( 2024 ) , namely multiple-choice question answering (MCQA), and those which have open-ended responses Dada et al. ( 2024 ) . Close-ended MCQA validation enables the automatic verification of response factuality, but it does not reflect the complex nature of real world situations ( e.g. , clinical settings Hager et al. ( 2024 ); Zhou et al. ( 2023 ) ). As such, MCQA alone often fails to identify critical short-comings of model performance Li et al. ( 2024 ); Umapathi et al. ( 2023 ); Ahmad et al. ( 2023 ); Pezeshkpour and Hruschka ( 2023 ); Alzahrani et al. ( 2024 ); Zheng et al. ( 2023 ) .

To incorporate a broader range of tasks relevant to the medical field Dada et al. ( 2024 ); Kanithi et al. ( 2024 ) , one typically has to rely on open-ended answers. That is, reference responses are not the only valid outputs. Since these cannot be completely assessed for factuality without human expert supervision, approximate measures based on n-grams and model perplexity remain in place, which limits the reliability of these evaluations Kamalloo et al. ( 2023 ) .

Efforts have been dedicated to analyze the relation between automatic evaluations and either user or expert evaluations, showing a lack of direct correspondence Fleming et al. ( 2024 ); Nimah et al. ( 2023 ) . This is explained by the difference in the model features these assess ( e.g. , factuality vs usability vs support capacity), pointing at their complementary nature. Nonetheless, a similar analysis within the family of automatic evaluations is still pending; a study of the relations between open-ended and close-ended benchmarks and metrics, to understand which of these tests should be used, and when. For that purpose, we focus on the healthcare domain, providing the following contributions:

A correlation-based, empirical analysis of open-ended and close-ended tasks, benchmarks, and metrics.

A novel medical benchmark ( CareQA ) featuring both closed- and open-ended formats for the verification of our findings.

A new metric for open-ended evaluations ( Relaxed Perplexity ) which fills a gap identified in existing methodologies.
Methodology
This study considers four different close-ended healthcare tasks, which include nine different datasets ( e.g. , MedQA). These are all assessed using the accuracy metric. At the same time, six open-ended tasks are studied, based on nine distinct datasets ( e.g. , MedText). In this case, eleven different metrics are extracted. Further details are shown in Table 1 . To assess the consistency within tasks, datasets and metrics, this work considers up to 12 different open LLMs, both specifically tuned for healthcare and general purpose, motivated by previous work Shoham and Rappoport ( 2024 ); Kanithi et al. ( 2024 ) .
CareQA : A Novel Benchmark
Updated benchmarks are necessary to prevent both data drift (as human knowledge evolves), and data contamination (as training data crawling efforts scale). To validate the integrity and consistency of existing tests, this work introduces a new benchmark for automatic evaluation, CareQA , available in both closed-ended and open-ended formats.

CareQA originates from the Spanish Specialised Healthcare Training (MIR) exams by the Spanish Ministry of Health . The close-ended version is a MCQA including 5,621 QA pairs across six categories: medicine, nursing, biology, chemistry, psychology, and pharmacology, sourced from the 2020 to 2024 exam editions. CareQA is available in both English and Spanish, with the translation performed using GPT-4.

The open-ended version (English only) was created by rephrasing the questions from the close-ended version using the Qwen2.5-72B-Instruct model. After the rephrasing process, the number of suitable questions was reduced to 3,730 QA pairs. This set retains the same categories as the closed-ended version.

To ensure the validity of both the translations and rephrasing, 10 annotators conducted a manual review of a total of 360 samples, each reviewed by at least three evaluators. This process achieved a confidence level of 95% and a margin of error of 5% approximately.

The translation results were positive, with all three evaluators agreeing on 83.1% of the questions as correct. Based on this, we considered the translation to be of good quality. However, the percentage of rephrased QA pairs labeled as correct by the three evaluators was 65.8%.

To address this, we conducted a second iteration incorporating feedback from human reviewers. The main issue identified was that while the rephrased answers might differ from the ground truth, they could still be considered valid. As a result, a new rephrasing iteration was carried out, explicitly prompting the model to account for this nuance, and questions with multiple valid answers were excluded. This led to the removal of 961 samples, leaving the final CareQA (open-ended) dataset with 2,769 QA pairs. Consequently, the percentage of correct labels increased to 73.6%. See Appendix A for further details.
Metrics
For close-ended evaluations, the metric of choice is accuracy. In contrast, for open-ended queries, there is a variety of metrics which provide different insights into model performance. This work considers eleven of those, which are sorted into four distinct categories:

N-gram based metrics evaluate the overlap of n-grams between the generated and reference answers. This category includes: ROUGE1, ROUGE2, ROUGEL and BLEU.

Semantic similarity metrics evaluate the semantic similarity between the generated text and reference text, often leveraging embeddings or deep learning models. This includes: BERTScore, BLEURT and MoverScore.

Perplexity metrics assess the predictive capabilities of the model by measuring how well it can predict a sequence of words. This includes: Word Perplexity, Bits per Byte and Byte Perplexity.

LLM-judge : In this category we use the Prometheus Kim et al. ( 2024 ) model to grade responses based on specific scoring criteria.
Correlation of open-ended vs close-ended
The first experiment conducted studies the correlation between open-ended and close-ended tasks, as detailed in Table 1 . Specifically, we compare the weighted average accuracy from the various MCQA benchmarks against all other close-ended and open-ended tasks and metrics. Figure 1 presents the results for the smaller models.

Of all close and open-ended tasks, only clinical note-taking correlates positively with MCQA, and even in this case, correlation is rather weak. In contrast, summarization, question entailment and the remaining close-ended benchmarks correlate negatively with MCQA, except for Med Transcriptions. The rest show a generalized lack of correlation. The negative correlation could be explained by the lack of medical expertise needed for summarizing and entailing (as information is available in the input), and by the diverse nature of close-ended tasks. At metric level, all open alternatives correlate very weakly with MCQA, except for Perplexity, for which we observe a slight correlation. These findings illustrate the relevance of the benchmarks chosen for evaluation, as well as the complementary nature of MCQA, when considering other tasks like summarization or clinical note-taking. Further details in Appendix B.1 .
Correlation of open-ended benchmarks
The previous section locates open-ended tasks with a variable degree of correlation with close-ended tasks ( e.g. , clinical note-taking, summarization). Let us now analyze correlations within the open-ended category. Details on this are shown in Appendix B.3 .

Notably, no consistently high correlation is observed for any benchmark or task. This suggests that each benchmark measures distinct aspects of model performance. This is the case even for benchmarks tackling the same task ( e.g. , ACI-Bench and MTS-Dialog), illustrating the importance of benchmark source ( i.e. , who crafted the benchmark and in which context). This underscores the need for specialized evaluations for downstream tasks, as generalization cannot be assumed.
Correlation of open-ended metrics
To assess whether the metrics used in the open evaluation are correlated among themselves, and to simplify future analyses for practitioners, we conduct a correlation analysis for each of the metrics detailed in § 2.2 across all implemented open-ended benchmarks (more details in Appendix B.2 ).

This analysis identifies three distinct clusters of highly correlated metrics. The first cluster includes the perplexity metrics, ( i.e. , Word Perplexity, Bits per Byte, and Byte Perplexity)
all of which show a correlation above 0.96 across all analyzed benchmarks. Noticeably, these metrics are all based on probabilistic prediction (perplexity) and information efficiency (Bits per Byte). The results obtained from Prometheus (an LLM judge) can be considered a distinct cluster of evaluation, illustrating how an external model provides a different and rather unique perspective on model performance. Finally, the third cluster includes all n-gram-based metrics, together with semantic similarity metrics ( i.e. , BERTScore, BLEURT, and MoverScore). A strong correlation among these metrics is consistently observed across benchmarks, which can be attributed to their shared focus on content and overall text quality.
Metrics resilience to rephrasing
A limitation of open-ended evaluations is their sensitivity to rewording. Let us now analyze the different metrics under this open setup, to better understand their reliability. To do so, the model’s output are rephrased, and evaluation recomputed. Six rephrased versions are produced using Qwen2.5-72B-Instruct .

Results show that most n-gram-based metrics ( i.e. , ROUGE1, ROUGE2, ROUGEL and BLEU) are resilient to rephrasing. This difference may arise because these metrics rely on surface-level word matching, making them less sensitive to phrasing changes as long as the core vocabulary remains intact. i.e. , in healthcare texts, key terms like ‘diagnosis,’ ‘treatment,’ or medication names often stay consistent, allowing these metrics to maintain a high overlap. In contrast, Prometheus (LLM judge) is the most affected by rewording, which is reasonable considering that, for this evaluation, correct punctuation and formatting in the answers greatly improve scores. This metric is followed by BLEURT and BERTScore (model similarity based) as the least resilient. More details can be found in Appendix C.1 .
Metrics self-consistency
Another issue that affects LLM evaluation, particularly on the open-ended setup, is the lack of self-consistency across model runs for some widespread sampling strategies, such as top_p and top_k. To evaluate its impact on open-ended evaluation, we generate and evaluate 11 responses for each prompt in CareQA -Open using top_p sampling, p=0.9 . Results can be seen in Figure 2 . We observe that among n-gram metrics, BLEU and ROUGE2 are the most self consistent. BLEURT and Prometheus (LLM judge) are the less consistent. Perplexity metrics are perfectly self-consistent. More details can be found in Appendix C.2 .
Relaxed Perplexity : A novel metric
By being optimized for next token prediction on the ground truth, LLM’s are optimized for perplexity. However, as seen before, this does not necessarily entail good performance on open or close-ended downstream tasks. Additionally, perplexity can be greatly impacted by instruct-tuning and alignment techniques Lee et al. ( 2024 ) . On the other hand, it has been widely noted that models are more likely to arrive at the correct answer after outputting intermediate tokens, commonly known as chain of thought (CoT) Suzgun et al. ( 2022 ); Wang et al. ( 2023 ) , and that this happens even without specific CoT prompting Wang and Zhou ( 2024 ) . However, perplexity fails to capture this improvement, and can be negatively impacted by the presence of intermediate tokens.

To evaluate factuality in open-ended benchmarks, with no dependence on confounders or exact formulation while accounting for the potential benefits of intermediate tokens, we propose Relaxed Perplexity . Given a question and a target , we wish to estimate

\begin{split}\mathbb{P}(target\simmodel\mid question)=\ =\mathbb{P}(A_{0})+\ldots+\mathbb{P}(A_{n}\mid B_{n})\end{split}

that is, the probability that the target is sampled from the model given the prompt, at any time in the completion. We denote the events A_{n}\equiv\{target\sim\text{model}(question+seq_{n})\} and B_{n}\equiv\{seq_{n}\sim\text{model}(question)\} for any seq_{n} of n tokens that comes from the model before the target. We can estimate \mathbb{P}(A_{n}\mid B_{n}) as

\mathbb{P}(A_{n}\mid B_{n})\approx\mathbb{P}(A_{n}\mid seq^{i_{1}}_{n})+\ldots+\mathbb{P}(A_{n}\mid seq^{i_{\ell}}_{n})

for the \ell more likely n -token sequences sampled from the model given question , because the events seq^{i}_{n} and seq^{j}_{n} are mutually exclusive. In this notation, \mathbb{P}(seq_{n}^{i_{\ell}}):=\mathbb{P}(seq_{n}^{i_{\ell}}\sim\text{model}(question)) . Using this, we can define Relaxed Perplexity as

\begin{split}Relaxed-Perplexity(target,question,model)=\ =\exp\left(-\frac{1}{n+len(target)}\sum_{i=0}^{n}logP(A_{i}\mid B_{i})\right)\end{split}

This allows to evaluate correctness in the model’s answers probability distribution, with no regard for the exact formulation. Further, for a given prompt and fixed sampling parameters, the metric is perfectly self consistent. We thus test it with the Olaph Jeong et al. ( 2024 ) medical factuality dataset. In contrast to Perplexity, we observe that Relaxed Perplexity assigns higher scores to models fine-tuned on healthcare datasets. More details on the mathematical formulation, implementation and results of Relaxed Perplexity can be found in Appendix D .
Conclusions
This study finds very weak correlations between close-ended and open-ended benchmarks. These results highlight the complementary roles of close-ended and open-ended approaches, and the limited insights provided by individual tests. It thus advocates for broader evaluation setups. Even within open-ended benchmarks targeting the same task ( e.g. , ACI-Bench and MTS-Dialog), no consistently high correlations were found. This indicates that different benchmarks assess distinct model capabilities, underscoring the significance of the benchmark’s design.

The analysis of evaluation metrics for open-ended benchmarks identified three distinct clusters that are particularly relevant for assessing medical models: (1) perplexity-based metrics, (2) n-gram-based metrics combined with semantic similarity metrics, and (3) LLM-as-a-judge metrics. Notably, none of these clusters showed strong correlations with the close-ended MCQA evaluation. Additionally, differences in resilience to answer rephrasing and self-consistency were observed, due to the distinct ways these metrics are computed.

The findings highlight the importance of selecting appropriate benchmarks and evaluation metrics designed for specific tasks. In this regard, the introduced CareQA benchmark, featuring both closed- and open-ended formats, serves as a sanity check of existing tests, while the proposed Relaxed Perplexity metric fills a gap in evaluation by focusing on factuality and being resistant to exact formulations in an open-ended setting.
Limitations
Since this study is based on specific models, the findings may not generalize to other LLM architectures. Additionally, the quality and diversity of the datasets used for evaluation are limited, meaning these benchmarks may not fully capture the performance of LLMs across the broader healthcare landscape. While metrics and benchmarks can indicate how well LLMs perform on certain tasks, they may not reflect the complexities of integrating LLMs into real-world healthcare practices.

In evaluating the models, we observed that applying the model’s chat template to MCQA tasks led to decreased performance, whereas open-ended evaluations showed improvement. To ensure a fair comparison between open-ended and MCQA evaluations, we maintained the same configuration across both categories and did not apply the model’s chat template to any of the evaluations.

Regarding the new benchmark introduced, although subject matter experts created the original exam materials, which underwent public scrutiny, CareQA has not been subjected to formal bias assessment. Consequently, it may not adequately represent the full spectrum of medical knowledge or encompass all possible patient demographics. Furthermore, although a human review was performed on the open-ended version, it has not undergone thorough evaluation by healthcare experts, raising the possibility of errors or biases introduced by the LLM used to rephrase the questions. Therefore, we advise users to exercise caution when interpreting and generalizing the results.

All experiments are conducted on English benchmarks (except for the Spanish version of CareQA ), and generalization to other languages has not been considered. To enable reproducibility, all resources are made available. CareQA is accessible on Hugging Face 1 1 1 https://huggingface.co/datasets/HPAI-BSC/CareQA and all new tasks are accessible in the original lm-evaluation-harness framework 2 2 2 https://github.com/EleutherAI/lm-evaluation-harness .
Acknowledgements
This work is supported by Anna Arias Duart, Pablo Agustin Martin Torres and Daniel Hinjos García fellowships within the “Generación D” initiative, Red.es , Ministerio para la Transformación Digital y de la Función Pública, for talent atraction (C005/24-ED CV1). Funded by the European Union NextGenerationEU funds, through PRTR.

We also acknowledge the computational resources provided by the FinisTerrae III, Leonardo, and MareNostrum 5 supercomputers. We are particularly grateful to the Operations department at BSC for their technical support.

Lastly, we sincerely thank Jordi Bayarri-Planas, Atia Cortés, Orlando Montenegro and Òscar Molina for their valuable time and feedback during the human evaluation process.
A.1 CareQA (close-ended)
CareQA is a novel benchmark for evaluating healthcare Large Language Models (LLMs) through multiple-choice question answering. CareQA was created by collecting exam materials in PDF format from the official Spanish government website. These documents were automatically parsed and then underwent post-processing to ensure data quality. This process involved removing 23 inaccurately parsed instances and excluding officially impugned questions. To enhance global accessibility, the original Spanish questions were translated into English using GPT-4.

Each CareQA sample contains metadata including a numeric exam identifier, full question text, four answer options, correct answer, exam year, and specialization category. The dataset is available in both Spanish and English, facilitating cross-lingual research. Examples of CareQA samples are provided in Figure 3 and Table 3 .

While CareQA shares its source with HeadQA in the Spanish Specialised Healthcare Training (MIR) exams, there is no overlap between the datasets. CareQA expands upon its predecessor, covering the years 2020 to 2024 and comprising 5,621 question-answer test pairs, compared to HeadQA’s 2,742 test pairs from 2013 to 2017.
The dataset’s composition is illustrated in Figure 5 , showing the category distribution by year to reveal potential temporal trends in exam content.

Table 4 presents additional information about the dataset, including the total number of questions per category, the longest and average question and answer lengths (in tokens), and the overall vocabulary size. This comprehensive overview of CareQA ’s structure and content demonstrates its potential as a valuable resource for evaluating and improving healthcare-focused language models.
A.2 CareQA (open-ended)
We developed the open-ended dataset by adapting the existing closed-ended CareQA dataset through the expansion of the English set. The first step was to filter out questions that contained terms such as "incorrect", "except", "false", "not correct", or "NOT", as these terms indicate that the questions focus on identifying incorrect answers among the provided options. After this filtering, we rephrased the remaining questions into an open-ended format using the Qwen2.5-72B-Instruct model, specifically instructing it to only rephrase questions that could be effectively transformed. This process excluded questions that explicitly ask for incorrect options or require a selection from the provided answers. We employed two different prompts for rephrasing, followed by a selection process to determine the best-rephrased version or to discard the question if neither was suitable.

Initially, the close-ended CareQA contained 5,621 QA pairs, but after the rephrasing process, the number of suitable questions for the open-ended version was reduced to 3,730 QA pairs. This new dataset retains the same categories as the closed-ended version, including medicine, nursing, biology, chemistry, psychology, and pharmacology.

Based on feedback from the human review (detailed in § A.3 ), a second iteration of rephrasing was conducted, as illustrated in Figure 4 . In this phase, the model was instructed to validate only questions that could be answered exclusively using the ground truth, ensuring there were no alternative correct answers. As a result, 961 questions were removed, reducing the CareQA (open-ended) dataset to a total of 2,769 QA pairs.

Figure 6 illustrates the distribution of these 2,769 QA pairs in the open-ended version and examples of QA pairs from both the close-ended and open-ended versions of the CareQA dataset are shown in Table 5 . Both datasets are publicly available 3 3 3 https://huggingface.co/datasets/HPAI-BSC/CareQA .
A.3 Human evaluation
To validate the translations performed by GPT-4 for the English version of CareQA , as well as the rephrasing process executed by Qwen2.5-72B-Instruct for the open-ended CareQA , a human evaluation was conducted with 10 human evaluators, including 5 authors of this article.

We selected a total of 260 QA pairs for evaluation, covering both translation and rephrasing. This sample size ensures a confidence level of 95% with a margin of error of 5% for translation and 5.73% for rephrasing. Each question was evaluated by at least three evaluators.

The results are shown in Table 2 and correspond to the percentages of correct answers labeled by at least one evaluator, by two evaluators, and by all three evaluators. For both translation and rephrasing, the percentage of questions labeled as correct by at least one evaluator is high (98.6% for translation and 96.1% for rephrasing). However, when considering the cases where all three evaluators agreed on the correctness of the QA pair, the percentages drop: 83.1% for translation and 65.8% for rephrasing (first iteration).

For translation, the agreement percentage was considered sufficiently high, and the English dataset was deemed valid. In contrast, for the open-ended rephrasing version, the agreement rate was not high enough, so a second iteration of rephrasing, as explained in the previous section, was carried out. After removing invalid questions, the percentage of correct answers increased, see third column of Table 2 . After this second iteration, the open dataset was also considered valid. The final agreement of both tasks grouped per category can be seen in Figure 7 .