# Assessing ChatGPT's Reliability in Endodontics: Implications for AI-Enhanced Clinical Learning
## 4 Discussion
This suggests that the question format--whether closed-clinical or multiple-choice--may significantly impact the perceived effectiveness of AI models as educational tools. Further research is needed to evaluate ChatGPT's performance across different question types. Evidence from other domains also indicates that the reliability of ChatGPT-4 varies by discipline, with reported accuracy rates of 71.7% in oral surgery [26] and only 25.6% in prosthodonics [37]. Similarly, [PERSON] et al. [46] found varying performances of ChatGPT-4o across different areas of the Polish Final Dentistry Examination, with lower accuracy in orthodontics (52.63%) and higher in prosthodonics (80%). These findings suggest that the effectiveness of ChatGPT may depend on the specific domain of expertise. Regarding repeatability, this study found that ChatGPT-4 exhibited a range from moderate to near-perfect agreement, while ChatGPT-4o ranged from substantial to near-perfect agreement. Thus, ChatGPT-4o demonstrated superior repeatability compared to its predecessor. Although the literature directly comparing repeatability between these versions is scarce, several studies have reported improved performance of ChatGPT-4 over ChatGPT-3.5 [34,36]. In this study, ChatGPT-4 achieved 100% repeatability on 7 questions, whereas ChatGPT-4o achieved this level on 11 questions, highlighting its greater consistency in response generation. These findings are noteworthy, as repeatability is a critical metric in assessing the performance of LLMs. Although the literature in this area remains limited [34], comparable studies have reported similar repeatability ranges in oral surgery [26] and lower ranges in prosthodonics [37]. The observed trend of improved repeatability in newer versions--such as ChatGPT-4o compared to ChatGPT-4--highlights the importance of further exploring this metric across various fields of knowledge and question formats. Repeatability is essential for educational contexts, as it ensures that learners receive dependable feedback when exploring similar content repeatedly. Despite this improvement, the variability in reliability underscores the necessity of expert supervision to validate AI-generated content, particularly in clinical education. As described in previous studies [37], the performance of LLMs can vary significantly when faced with highly specialized tasks. In this context, endodontics is a discipline with particular complexities that challenge the ability of generalist LLMs to provide appropriate responses: the intensive use of technical terminology, reliance on official clinical protocols, and the need for therapeutically precise answers. These characteristics may partly explain the limitations observed in terms of reliability  
Although this study did not focus on directly optimizing model performance, such as through specific training, our results highlight the importance of exploring such strategies in future research. These could include prompt engineering techniques tailored to clinical tasks or supervised adaptation to specialized educational contexts. A limitation of this study is the small number of questions evaluated (n = 30), which may restrict the wider application of the results to the entire field of endodontics. However, this methodology was selected based on previous studies [21; 26; 37], which used limited sets of clinical questions carefully selected and validated by experts. Additionally, this methodological design was strengthened by repeating each question 30 times per model, which generated 1800 responses individually evaluated by specialists, thus providing a solid foundation for the reliability and repeatability analyses. Another limitation of the study could be the evaluation scale used. In this study, a quantitative evaluation based on a three-level scale was chosen, which is suitable for assessing brief responses to closed-clinical questions.