## 4 Discussion
These results have practical implications for medical educators, indicating that while AI tools like ChatGPT-3.5 can support question generation, they should be complemented with expert oversight to ensure the development of questions that engage critical thinking. Future research should explore ways to enhance the model's capacity to generate questionsthat align with theories of cognitive complexity, such as Bloom's Taxonomy, which emphasizes the importance of creating questions that test analysis, synthesis, and evaluation. By doing so, we can better integrate AI-generated content into medical education in a manner that promotes deeper learning and prepares students for clinical practice. **Proposed Evaluation Framework for Future Implementation of AI in Exam Generation. **  
Given the results of our study, which identified several shortcomings of ChatGPT-3.5, including hallucinations and the tendency to produce oversimplified questions, it is crucial to adopt a comprehensive, multi-stage evaluation framework for the future implementation of AI in exam question generation. This framework aims to enhance the reliability, accuracy, and educational value of AI-generated content, particularly for high-stakes assessments like the USMLE. We suggest the following steps:  
* Initial automated screening: Implement AI-based tools such as the Google Knowledge Graph API, the Wolfram Alpha API, or the MedGPT to perform an initial quality check of the generated content. This stage would identify potential inaccuracies, inconsistencies, or superficial questions that do not meet established standards, flagging them for further review. * Expert review process: Involve a panel of three to five domain-specific experts, such as pharmacologists and medical educators, to thoroughly assess the generated questions. This expert review would evaluate the medical accuracy, clinical relevance, level appropriateness, and adherence to item-writing standards (such as the NBME guidelines), ensuring that the questions align with the depth and complexity required for medical exams [11]. * Integration of specialized medical knowledge bases: Use external databases such as UpToDate, PubMed, or other clinical guidelines to cross-reference the generated content for factual accuracy. Incorporating these knowledge bases would help validate medical accuracy and ensure that the AI's knowledge aligns with the latest developments in current clinical practice. * Iterative feedback loop: Create an iterative process where expert feedback is used to refine the AI-generated questions. This would involve adjusting prompts, question structures, or explanations, followed by re-evaluation until the content reaches a satisfactory level of quality and complexity. * Comparison using bootstrapping methods: Recent studies suggested applying bootstrapping techniques to evaluate exam questions [31]. Bootstraping could be used to compare the psychometric characteristics of AI-generated questions with instructor-written ones, analyzing factors such as the size of confidence intervals, reliability, and question difficulty. * Pilot testing with medical students: Conduct small-scale pilot testing of AI-generated questions with medical students. This step would assess the perceived difficulty and educational value of the questions in real exam conditions, providing direct feedback on how well the questions prepare students for actual exams like the USMLE. By adopting this multi-stage evaluation framework, future AI-generated questions can be fine-tuned to not only meet NBME standards but also foster critical thinking and higher-order reasoning among medical students. This approach addresses the limitations of current AI models while maximizing their potential as an educational tool. However, a key consideration for future research is the emphasis on generating questions that test higher-level medical knowledge rather than relying predominantly on basic recall.