## 2 Methodology
### LLM Evaluation Design  
In order to assess the performance of LLMs in extracting patient characteristics from free-text eligibility criteria descriptions, it is crucial to establish a comprehensive and multifaceted evaluation framework. The evaluation of LLMs in this context is essential for several reasons:  
1. Ensuring accuracy and reliability: Extracting patient characteristics from free-text descriptions requires a high level of accuracy to support clinical decision-making and research. Evaluating LLMs helps ensure that the extracted information is reliable and can be trusted for downstream applications [23]. 2. Assessing generalizability: LLMs should be able to perform well across a diverse range of eligibility criteria descriptions, covering various therapeutic areas, study designs, and patient populations. A comprehensive evaluation framework allows us to assess the generalizability of LLMs and identify potential limitations or biases [24]. 3. Facilitating model selection and improvement: By evaluating multiple LLMs using a standardized framework, we can compare their performance, identify strengths and weaknesses, and select the most suitable models for specific tasks [23]. Moreover, the evaluation results can guide the development and fine-tuning of LLMs to enhance their performance in extracting patient characteristics [25; 26]. To address these objectives, we propose an evaluation framework that incorporates multiple metrics, each capturing different aspects of LLM performance (Figure 2). The rationale for using multiple metrics is to provide a holistic assessment of LLMs, considering factors such as accuracy, relevance, consistency, and coherence [25]. By combining these metrics, we aim to obtain a more comprehensive understanding of LLM performance and make informed decisions regarding their deployment in clinical research settings. The evaluation metrics included in our framework are as follows:  
1. Human Validation: This metric involves comparing the LLMs' outputs with ground truth answers provided by human experts for specific criteria such as BMI, age, gender, platelets count, and white blood cell count [27]. Human validation ensuresthat the LLMs' performance is benchmarked against the gold standard of human judgment [27]. 2. F1-Score, Precision, and Recall: These metrics, derived from the BERTScore methodology [28], assess the LLMs' ability to generate outputs that match the ground truth data. By computing the similarity between tokens in the LLM output and the ground truth, we quantify each model's accuracy and completeness in extracting relevant information. 3. Semantic Textual Similarity: Using the Sentence Transformer library with the \"all-mpnet-base-v2\" model, we measure the semantic similarity between the ground truth data and the LLM output. This metric captures the semantic proximity of sentences, providing insights into the model's capacity to generate coherent and meaningful outputs [26; 29], and is inspired by the DeepEval framework [30]. 4. Factual Consistency: Factual consistency, a critical metric inspired by the DeepEval framework [30], was used to evaluate the alignment of the LLMs' outputs with the input data. This metric is particularly important in the clinical domain, where inaccuracies could have serious consequences for patient care and research integrity [26]. In our study, factual consistency was assessed by employing string-matching algorithms. These algorithms compared key terms and phrases extracted by the LLMs, such as age, gender, and other eligibility-related criteria, against their counterparts in the input data. This approach ensured that the extracted terms matched the intended meanings in the clinical context, reducing the risk of misinterpretation or error. The choice of string-matching algorithms was informed by their ability to handle synonyms, abbreviations, and variations in phrasing common in free-text clinical descriptions.