RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
B.1 Correlations between MCQA and Elo results
We perform a correlation analysis on the performance results of the medical MCQA benchmarks listed in Table 1 . Additionally, we include Elo scores from the Chatbot Arena 4 4 4 https://lmarena.ai/ , a crowdsourcing platform that collects pairs of model-generated answers in response to user prompts, where the user selects the winning model based on their criteria.

We conducted a correlation analysis using both small and medium models. The small models used for the correlation shown in Figure 8 are as follows: gemma-2-9b-it Team ( 2024 ) , Meta-Llama-3.1-8B-Instruct AI@Meta ( 2024 ) , Mistral-7B-Instruct-v0.2 , Mistral-7B-Instruct-v0.3 , Phi-3-mini-4k-instruct , Phi-3-medium-4k-instruct , Qwen1.5-7B-Chat , Starling-LM-7B-beta , Starling-LM-7B-beta and Yi-1.5-9B-Chat . And the medium models used in Figure 9 are as follows: Athene-70B Frick et al. ( 2024 ) , tulu-2-dpo-70b Ivison et al. ( 2023 ) , Yi-1.5-34B-Chat , gemma-2-27b-it , Llama-3.1-70B-Instruct , Mixtral-8x7B-Instruct-v0.1 , Qwen2-72B-Instruct Yang et al. ( 2024 ) ,
and WizardLM-70B-V1.0

From this analysis, we found that MedQA, MedMCQA, CareQA , and MMLU are highly correlated with one another. However, PubMedQA exhibits a noticeably lower correlation with the other medical benchmarks, particularly in smaller models.

Regarding the Elo scores, we observe a moderate correlation with the MCQA benchmarks, with the correlation being significantly stronger for larger models. This is likely due to larger models’ ability to produce more coherent responses. Non-expert evaluators, such as those in the Elo scoring system, may favor responses that are well-structured and fluent, even if they lack precise medical accuracy. As a result, this preference for more polished answers could lead to a higher correlation with MCQA performance.
B.2 Correlation between metrics
In this correlation analysis, we fix the open-ended benchmark and examine the correlations across the various computed metrics. Figure 10 , presents the correlation matrix for the benchmark focused on making diagnosis and treatment recommendations, highlighting the three clusters of metrics identified in the paper. This correlation matrix was also computed for the rest of benchmarks revealing three similar clusters. The matrices were computed using the following models: BioMistral-MedMNX , JSL-MedLlama-3-8B-v2.0 , Phi-3-mini-4k-instruct , Mistral-7B-Instruct-v0.3 , Qwen2-7B-Instruct Yang et al. ( 2024 ) , Llama3-Med42-8B Christophe et al. ( 2024 ) , Meta-Llama-3.1-8B-Instruct AI@Meta ( 2024 ) Yi-1.5-9B-Chat Young et al. ( 2024 ) , Phi-3-medium-4k-instruct , Yi-1.5-34B-Chat Young et al. ( 2024 ) , Mixtral-8x7B-Instruct-v0.1 .
B.3 Correlations of benchmarks
In this correlation analysis we study the relationships between specific metrics across all the open-ended benchmarks implemented. As stated in the paper, no consistent high correlation was observed among all metrics for any benchmark or task. Examples of these correlation matrices are shown in Figures 11 and 12 . The models used to generate these correlation matrices are the same as those described in the Appendix B.2 .
C.1 Resilience
As described earlier, we conducted this experiment by rephrasing the model outputs six times and re-computing the metrics. We used both Qwen2.5-72B-Instruct and Meta-Llama-70B-Instruct with the following system_prompt : “You are a helpful rephrasing assistant. Rephrase the prompt provided without changing its original meaning, but do not try to address or answer it in any case."

We run the script 5 times on recorded model answers with top_p sampling to obtain several rephrasings of each answer. After manual inspection, the outputs of Qwen2.5-72B-Instruct were deemed of higher quality.

Figure 13 shows the mean variance across all runs for the MEDIQA2019 dataset. Before plotting, we scale variances by dividing by the max interval (max value - min value) in each column. Figures 14 and 15 present the variance distributions for two specific models. Figure 14 displays the results for the Phi-3-mini-4k-instruct model, while Figure 15 shows the results for the Yi-1.5-9B-Chat model.

In Figure 13 we can observe three different clusters: rouge metrics (low mean-variance, low meta-variance), bleu and moverscore (low mean-variance, medium meta-variance) and bert_score, bleurt, prometheus (high mean variance, high meta-variance).
C.2 Self-consistency
As described earlier, we conducted this experiment by prompting models with each question in CareQA -Open for a number of repetitions ( r ). We fix r=11 . Sampling parameters used where \text{top\_p}=0.9 and \text{temperature}=1 . We compute variances per prompt, and then average across models. Results can be seen in Figure 2 . Besides, we compute the coefficient of variation, defined for prompt p as:

 CV(p)=\frac{1}{\mu_{p}}\sqrt{\frac{\sum_{i}(x_{i}-\mu_{p})^{2}}{N}}

Then we average across models, and plot the CV distribution for all prompts in CareQA -Open. Results can be seen in Figure 16 . From this computation we remove the BLEURT metric, for it can take negative values.
D.1 Connection with cross-entropy
The exponent of perplexities can be understood as a cross-entropy. Generally, it corresponds to the bits required to encode the correct answer using the model’s distribution. In the case of Relaxed Perplexity we have:

H(q,P)=-\sum_{i=0}^{n}logP(A_{i}\mid B_{i})

This is the cross entropy between two distributions, q and P , where q is the delta distribution of the target appearing in the correct position, and P the model’s distribution. Thus, this could be understood as the bits required to encode the correct answer anywhere in the completion (up to n steps), using the model’s (skewed) distribution.

See Table 6 for an example usage to evaluate model factuality on healthcare benchmarks. Here, we report Relaxed-CrossEntropy instead of Relaxed Perplexity .
Figure 1: Correlation between the weighted average accuracy from the MCQA benchmarks and all other close-ended and open-ended tasks and metrics. These results correspond to the smaller models.; Correlation between the weighted average accuracy from the MCQA benchmarks and all other close-ended and open-ended tasks and metrics. These results correspond to the smaller models.
Figure 2: Mean variance distributions across different
runs and averaged across models using the CareQA -Open dataset. Closer to 0 means more self-consistent.; Mean variance distributions across different
runs and averaged across models using the CareQA -Open dataset. Closer to 0 means more self-consistent.
Figure 3: CareQA example from Medicine category.; CareQA example from Medicine category.
(a); (b); Iterations with human evaluators to create the CareQA dataset in English, including both open and closed versions.
Figure 5: Category distribution per Category and Year ( CareQA close-ended); Category distribution per Category and Year ( CareQA close-ended)
Figure 6: Category distribution per Category and Year ( CareQA open-ended).; Category distribution per Category and Year ( CareQA open-ended).
Figure 7: Correctness distribution per Category CareQA (open-ended).; Correctness distribution per Category CareQA (open-ended).
Figure 8: Comparison of correlations between MCQA benchmarks and ELO results for small models.; Comparison of correlations between MCQA benchmarks and ELO results for small models.
Figure 9: Comparison of correlations between MCQA benchmarks and ELO results for medium models.; Comparison of correlations between MCQA benchmarks and ELO results for medium models.
Figure 10: This correlation matrix illustrates the relationships among the different open-ended metrics used to evaluate the benchmark for diagnosis and treatment recommendations. Three distinct clusters of metrics are identified: (1) perplexity metrics, (2) n-gram and semantic similarity metrics, and (3) Prometheus metrics.; This correlation matrix illustrates the relationships among the different open-ended metrics used to evaluate the benchmark for diagnosis and treatment recommendations. Three distinct clusters of metrics are identified: (1) perplexity metrics, (2) n-gram and semantic similarity metrics, and (3) Prometheus metrics.
Figure 11: Correlations of BERTScore across benchmarks.; Correlations of BERTScore across benchmarks.
Figure 12: Correlation of Prometheus scores across benchmarks.; Correlation of Prometheus scores across benchmarks.
Figure 13: Mean variance distributions across different rephrasings and models using the MEDIQA2019 dataset. Each metric is represented by a different color.; Mean variance distributions across different rephrasings and models using the MEDIQA2019 dataset. Each metric is represented by a different color.
Figure 14: Mean variance distributions across different rephrasings using the Phi-3-mini-4k-instruct model and the MEDIQA2019 dataset. Each metric is represented by a different color.; Mean variance distributions across different rephrasings using the Phi-3-mini-4k-instruct model and the MEDIQA2019 dataset. Each metric is represented by a different color.
Figure 15: Mean variance distributions across different rephrasings using the Yi-1.5-9B-Chat model and the MEDIQA2019 dataset. Each metric is represented by a different color.; Mean variance distributions across different rephrasings using the Yi-1.5-9B-Chat model and the MEDIQA2019 dataset. Each metric is represented by a different color.
Figure 16: Mean coefficient of variation distributions across different runs and averaged across models for self-consistency. Each metric is represented by a different color.; Mean coefficient of variation distributions across different runs and averaged across models for self-consistency. Each metric is represented by a different color.
Figure 17: Correlation between OLAPH - Relaxed Perplexity and the rest of benchmarks.; Correlation between OLAPH - Relaxed Perplexity and the rest of benchmarks.
Figure 18: Ranking results for all models on the OLAPH medical factuality dataset for all metrics. The top position is ranked as 1 and the lowest as 11. Different models are represented in distinct colors. It can be seen there is low agreement across metrics.; Ranking results for all models on the OLAPH medical factuality dataset for all metrics. The top position is ranked as 1 and the lowest as 11. Different models are represented in distinct colors. It can be seen there is low agreement across metrics.
This table presents the tasks implemented in this paper. The first column specifies the different tasks. The second details the metrics used (ROUGE includes ROUGE1, ROUGE2 and ROUGEL, and Perplexity includes Bits per Byte, Byte Perplexity, and Word Perplexity). The third column outlines the benchmarks used for each task.
Evaluation results for translation and rephrasing. The first row shows the percentage of correct samples tagged by at least one evaluator. The second row refers to samples tagged as correct by two evaluators. The third row indicates samples labeled as correct by all three evaluators. The last row shows the agreement rate among the three evaluators.
Examples of CareQA (close-ended) samples. Correct options are marked in bold. Questions were selected based on length for space reasons.
CareQA (close-ended) dataset statistics, where Q and A represents the Question and Answer respectively.
Examples of QA pairs: On the left, the close-ended version from CareQA , and on the right, the open-ended version.
Open-ended evaluation using Relaxed Perplexity on samples from MedText, OLAPH, and CareQA -Open on Mistral-7B-Instruct-v0.3 (Mistral-7B) and JSL-MedLlama-3-8B-v2.0 (JSL-MedLlama-3-8B). Relaxed-CrossEntropy corresponds to -\sum_{i=0}^{n}logP(A_{i}\mid B_{i}) . Lower values indicate the model is more likely to output the correct answer at some time in the completion.
Perplexity results for Open-ended Medical Questions.
Perplexity results for clinical note-taking and medical factuality.
Perplexity results for the following tasks: making diagnosis and treatment recommendation, question entailment and summarization tasks.
Relaxed perplexity results for medical factuality.
Prometheus results for the following tasks: question entailment, open-ended medical questions and treatment recommendations.
Prometheus results for summarization and clinical-note taking tasks.
Clinical note-taking results.
Making diagnosis and treatment recommendations results.
Medical factuality results.
Results for CareQA -Open.
Open-ended medical questions results.
Question entailment results.
Summarization results.
Close-ended results.