## 5 Evaluation  
Evaluating the quality of diagnoses generated by LLMs is a challenging task, especially in the sensitive domain of healthcare [26]. Evaluation of long-form generations from LLMs remains a nascent area of work [42]. In this study, we developed an approach to automatic evaluation using a multifaceted, iterative design methodology. We leveraged the capabilities of the LLMs themselves to assess the quality of the generated diagnoses. Specifically, we used GPT-4o [[[https://openai.com/index/hello-gpt-4o/?ref=blog.roboflow.com](https://openai.com/index/hello-gpt-4o/?ref=blog.roboflow.com)]([https://openai.com/index/hello-gpt-4o/?ref=blog.roboflow.com](https://openai.com/index/hello-gpt-4o/?ref=blog.roboflow.com))] [43]accessed 30 August 2024, an external LLM, as the evaluator. We chose GPT-4o as the evaluator for several reasons. First, GPT-4o is known for its advanced capabilities in understanding and generating coherent, contextually relevant text, making it well-suited for evaluating the quality of diagnostic outputs. Second, its extensive training in a diverse range of tasks and data ensures that it can provide robust and reliable evaluations across various metrics. Third, using an external evaluator helps mitigate potential biases that might arise from using the same model for both diagnosis and evaluation. This model evaluates the diagnoses generated by other LLMs, which we refer to as Diagnostic LLMs. We designed the evaluation pairwise to compare two possible answers for a given clinical case. This style of evaluation is a common paradigm for the evaluation and tuning of LLMs [44; 45]. The pairwise evaluation rubric instructs the evaluator to compare the model answers with the diagnosis by healthcare specialists to determine if the Diagnostic LLMs show high capacities across our four metrics: (a) precision, (b) recall, (c) completeness, and (d) qualitative assessment:  
1. Precision: Does the generated diagnosis correctly identify the condition described in the clinical case? 2. Recall: Does the generated diagnosis include all relevant conditions described in the clinical case? 3. Completeness: Does the generated diagnosis provide a comprehensive description of the patient's condition, including any relevant details or comorbidities? 4. Qualitative assessment: We assessed the overall quality of the generated diagnosis, including its clarity, coherence, and relevance to the clinical case. To perform the automatic evaluation, we used a set of pre-defined prompts to query the evaluator and elicit their assessments of the generated diagnoses. The general structure of the prompts and configurations used for evaluation are summarized in Figure 1. Text between '<>' represents substitutions that are made to the prompt at inference time, for example, adding the name of the metric evaluated. Each prompt is based on the same structure represented in Figure 1, where we replace the name of the metric ('<metric>') and the scale provided ('<scale>'). The prompt instructs the evaluator to compare the predicted diagnoses to the true diagnosis using a scale from 0 to 100 and provide a brief explanation for the score. Examples are also provided to help the evaluator understand the scale and the metric being evaluated. We opted to use LLMs for this evaluation because traditional precision and recall metrics fall short of capturing the nuanced complexities of our datasets, which contain a multitude of possible diagnoses. These metrics are binary and do not account for the varying degrees of accuracy among different diagnoses. For instance, two diagnoses may be semantically similar yet distinct, leading to misleading evaluations when using strict precision and recall. In contrast, LLMs can convert these diagnoses into embeddings, capturing the semantic relationships and subtle differences between them. By measuring nuanced values between 0 and 100, LLMs provide a more granular and accurate assessment,  
Figure 1: Template of the evaluator prompt for diagnoses evaluation.