###### Abstract  
ChatGPT by OpenAI is an AI model designed to generate human-like responses based on diverse datasets. Our study evaluated ChatGPT-3.5's capability to generate pharmacology multiple-choice questions adhering to the NBME guidelines for USMLE Step exams. The initial findings show ChatGPT's rapid adoption and potential in healthcare education and practice. However, concerns about its accuracy and depth of understanding prompted this evaluation. Using a structured prompt engineering process, ChatGPT was tasked to generate questions across various organ systems, which were then reviewed by pharmacology experts. ChatGPT consistently met the NBME criteria, achieving an average score of 13.7 out of 16 (85.6%) from expert 1 and 14.5 out of 16 (90.6%) from expert 2, with a combined average of 14.1 out of 16 (88.1%) (Kappa coefficient = 0.76). Despite these high scores, challenges in medical accuracy and depth were noted, often producing \"pseudo vignettes\" instead of in-depth clinical questions. ChatGPT-3.5 shows potential for generating NBME-style questions, but improvements in medical accuracy and understanding are crucial for its reliable use in medical education. This study underscores the need for AI models tailored to the medical domain to enhance educational tools for medical students. ChatGPT; AI; NBME; pharmacology; medical education +
Footnote †: journal: _arxiv.org_  
0000-0001-6000-0000-0000-0000-0000-0000-0000-0000-0000-0000-0000-00000-0000-0000-0000-0000-00000-00000-00000-00000-00000-00000-00000-000000-00000-00000  
Previous studies have assessed the performance of ChatGPT in USMLE Step exams using the NBME and AMBOSS datasets [9]. It was found that [PERSON] answered the NBME Step 1 exam with 64.4 percent of questions correct, the NBME Step 2 exam with 57.8 percent of questions correct, the AMBOSS Step 1 exam with 44 percent of questions correct, and the AMBOSS Step 2 exam with 42 percent of questions correct [9]. The ability of [PERSON] to perform with a passing grade of over 60 percent in the NBME Step 1 exam prompted our interest in assessing its ability of ChatGPT to generate NBME-style pharmacology multiple-choice questions that meet the NBME Item-Writing Guidelines [10; 11]. Previous research has assessed the ability of ChatGPT to take an Ophthalmic Knowledge Assessment Program (OKAP) exam, which is created for ophthalmology residents [12]. The study found that ChatGPT performed at the level of a first-year resident, providing insight into its capabilities and knowledge base. ChatGPT offers promising benefits in scientific research, healthcare practice, and healthcare education [13]. In scientific research, it has been found to be a useful tool in academic research and writing [14]. In healthcare practice, ChatGPT shows the potential to assist in documentation, disease risk prediction, improving health literacy, and enhancing diagnostics [15; 16]. In healthcare education, ChatGPT has been found to be beneficial in its ability to pass exams such as the USMLE Step exams and ophthalmology residency examinations [12; 13]. ChatGPT shows promising potential in healthcare and can be a useful tool for medical education [17]. Additionally, ChatGPT has been found to be beneficial in the rapid generation of clinical vignettes, which could help reduce costs for healthcare students [18]. However, there are concerns that ChatGPT can potentially provide inaccurate information and references [19]. With the plethora of resources available to students during Step preparation, the process can be both time-consuming and overwhelming to students. If ChatGPT-generated questions are medically accurate and abide by the NBME guidelines, they can be a valuable resource for medical students [11]. This study aimed to evaluate ChatGPT-3.5's capability to generate NBME-style pharmacology multiple-choice questions, adhering to the NBME Item-Writing Guidelines [11]. Our objective was to generate recommendations for enhancing and fine-tuning ChatGPT-generated NBME questions for medical education.