# Accuracy, consistency, and contextual understanding of large language models in restorative dentistry and endodontics

Claire Lafourcade, Olivia Kerouedan, Benoît Ballester, Raphaël Richert

<figure><img src="image_1.png" /></figure>

## To cite this version:

Claire Lafourcade, Olivia Kerouedan, Benoît Ballester, Raphaël Richert. Accuracy, consistency, and contextual understanding of large language models in restorative dentistry and endodontics. Journal of Dentistry, 2025, 157, pp.105764. 10.1016/j.jdent.2025.105764. hal-05069452

HAL Id: hal-05069452

https://hal.science/hal-05069452v1

Submitted on 26 Jan 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

Check for
updates

# Accuracy, consistency, and contextual understanding of large language models in restorative dentistry and endodontics

Claire Lafourcade<sup>a,b</sup>, Olivia Kérourédan<sup>a,b,c</sup> (ID), Benoit Ballester<sup>d,e</sup>,
Raphael Richert<sup>f,g,h,*</sup> (ID)

## ARTICLE INFO

**Keywords:**

Dental education

Endodontics

Restorative dentistry

Medical informatics

Artificial intelligence

Natural Language Processing

## ABSTRACT

**Objective:** This study aimed to evaluate and compare the performance of several large language models (LLMs) in the context of restorative dentistry and endodontics, focusing on their accuracy, consistency, and contextual understanding.

**Methods:** The dataset was extracted from the national educational archives of the *Collège National des Enseignants en Odontologie Conservatrice* (CNEOC) and includes all chapters from the reference manual for dental residency applicants. Multiple-choice questions (MCQs) were selected following a review by three independent academic experts. Four LLMs were assessed: ChatGPT-3.5, ChatGPT-4 (OpenAI), Claude-3 (Anthropic), and Mistral 7B (Mistral AI). Model accuracy was determined by comparing responses with expert-provided answers. Consistency was measured through robustness (the ability to provide identical responses to paraphrased questions) and repeatability (the ability to provide identical responses to the same question). Contextual understanding was evaluated based on the model's ability to categorise questions correctly and infer terms from definitions. Additionally, accuracy was reassessed after providing the LLMs with the relevant full course chapter.

**Results:** A total of 517 MCQs and 539 definitions were included. ChatGPT-4 and Claude-3 demonstrated significantly higher accuracy and repeatability than Mistral 7B, with ChatGPT-4 showing the greater robustness. Advanced LLMs displayed high accuracy in presenting dental content, although performance varied on closely related concepts. Supplying course chapters generally improved response accuracy, though inconsistently across topics.

**Conclusion:** Even the most advanced LLMs, such as ChatGPT-4 and Claude 3, achieve moderate performance and require cautious use due to inconsistencies in robustness. Future studies should focus on integrating validated content and refining prompt engineering to enhance the educational and clinical utility of LLMs.

**Clinical Significance:** The findings underscore the potential of advanced LLMs and context-based prompting in restorative dentistry and endodontics.

## 1. Introduction

Advancements in artificial intelligence (AI) have unlocked new possibilities for synthesising and processing information. Among these advancements, large language models (LLMs), especially following the

transformative release of ChatGPT-3, have emerged as valuable tools in various fields, including medical education [1–5]. Trained on extensive datasets, these models achieve "contextual understanding" by interpreting language within the appropriate context and generating human-like responses. By leveraging attention mechanisms, LLMs

<footnote><sup>a</sup> UFR des Sciences Odontologiques, Université de Bordeaux, Bordeaux, France</footnote>

<footnote><sup>b</sup> CHU de Bordeaux, Pôle de Médecine et Chirurgie bucco-dentaire, Bordeaux, France</footnote>

<footnote><sup>c</sup> UMR 1026 BioTis INSERM, Université de Bordeaux, Bordeaux, France</footnote>

<footnote><sup>d</sup> Assistance Publique Des Hôpitaux de Marseille, Marseille, France</footnote>

<footnote><sup>e</sup> Aix Marseille Univ, Inserm, IRD, SESSTIM, Sciences Economiques & Sociales de la Santé & Traitement de l'Information Médicale, ISSPAM, Marseille, France</footnote>

<footnote><sup>f</sup> Faculté d'Odontologie Université Lyon 1, Lyon, France</footnote>

<footnote><sup>g</sup> INSA Lyon, CNRS, LaMCoS, UMR5259, Villeurbanne, France</footnote>

<footnote><sup>h</sup> Hospices Civils de Lyon, PAM Odontologie, Lyon, France</footnote>

<footnote>* Corresponding author at: Faculté d'Odontologie Université Lyon 1, Lyon, France.
E-mail address: raphael.richert@insa-lyon.fr (R. Richert).</footnote>
selectively focus on key parts of the input, producing contextually
appropriate replies without directly memorising prior data [6]. How-
ever, this ability to generalise can sometimes create an illusion of fa-
miliarity with specialised content [7].

In medicine, ChatGPT-3 has demonstrated high performance on general medical tests, such as the United States Medical Licensing Exam (USMLE), where it meets or exceeds the 60 % accuracy threshold [3,8]. It has also achieved up to 83 % accuracy on the MMLU (Massive Multitask Language Understanding), leading medical students to use these chatbots for studying [9], though this trend has raised concerns about appropriate usage [10–12]. Despite such strong performance in general medical tests, LLM accuracy often declines in more specialised areas. For instance, on the MedQA, accuracy dropped to 67 %, with certain topics and chapters scoring below 30 % [13,14]. Similar discrepancies are noted in ophthalmology, where LLMs perform well in glaucoma management but exhibited weaker results with retina-related questions [15] and in dentistry, where they handle general dental knowledge accurately [16] but show reduced accuracy on complex questions in endodontics [17].

In oncology, LLM errors have been shown to affect patient health in 63.6 % of cases, with 18 % classified as highly impactful [18]. This raises concerns in fields such as endodontics, where misdiagnoses could have significant implications for oral healthcare [19]. Such discrepancies, often termed “hallucinations”, may be linked to limitations in LLM context windows and the diverse, sometimes unverified, datasets used for training [7,20]. Despite the increasing scale of LLMs, a fundamental limitation also persists: reliance on broad, internet-sourced knowledge that often lacks validation and specificity for specialised fields such as restorative dentistry and endodontics [21–23].

Recently, techniques such as Retrieval-Augmented Generation (RAG) have been developed to integrate external, and domain-specific content into LLMs, providing access to more relevant information without additional model retraining [24]. In medicine, RAG has improved question-answering by up to 18 % and reduced hallucinations [24,25]. However, RAG faces challenges in maintaining response consistency, and ensuring the quality and relevance of external data [24, 26]. Additionally, the limited context window further complicates the integration of extensive or complex materials, making prompt engineering essential for effective outcomes [24,26]. Evaluating the contextual understanding of these models with dental curricula is crucial to ensure they support students and practitioners in navigating and comprehending their course content.

We hypothesise that H1a) advanced LLMs will show higher performances compared to simple LLMs in restorative dentistry and endodontics; and H1b) accuracy of these LLMs on multiple-choice questions (MCQs) could be enhanced by preceding questions with appropriate and validated lecture material. The aim of this study was to evaluate the performance of various LLMs in the context of restorative dentistry and endodontics, focusing on their accuracy, consistency, and contextual understanding.

## 2. Material and methods

### 2.1. Dataset

The dataset used herein was extracted from the national educational archives of the *Collège National des Enseignants en Odontologie Con-servatrice* (CNEOC). It includes all chapters of the reference manual for dental residency applicants in restorative dentistry and endodontics [22] as well as a comprehensive set of MCQs designed for third-year students; it has been co-written by a panel comprising 79 national ed-cucators. The MCQs encompass various topics such as emergencies, restorative procedure, and disinfections practices. A series of MCQs was selected after a meticulous review process performed by three inde-pendent academic experts to ensure clarity and accuracy. Each MCQ of this dataset was then categorised based on its relevance to specific

chapters of the reference manual. Categories with less than ten questions as well as questions containing images or radiographs were excluded. The final set of questions was stored in a collaborative worksheet using Microsoft Excel (Microsoft Corporation, Redmond, WA, USA).

### 2.2. Generated responses

The responses to the MCQs were evaluated using four advanced LLMs: ChatGPT-3 and ChatGPT-4 (OpenAI, San Francisco, CA, USA), Claude 3 (Anthropic, San Francisco, CA, USA), and Mistral 7B (Mistral AI, Paris, France). The queries were submitted through the respective application programming interfaces (APIs) from the same access point using a standardised prompt format with interaction scripts written in Python 3.9 to ensure consistency and repeatability. For MCQs, each prompt was carefully crafted to present the question and its formatted choices in a clear and consistent manner (Table S1). The responses generated by the LLMs were then automatically parsed using Python scripts to extract the letters corresponding to each choice (A to E), indicating whether the choice was deemed correct or incorrect. This study follows a benchmarking approach, where the performance of each LLM is compared across standardised tasks using the same dataset. Subsequently, an expert reviewed all raw responses to ensure their accuracy compared to the established answer. The Python scripts used for this study are available for review at the GitHub repository: https://github.com/richertra/drechatbot.

### 2.3. Accuracy and consistency

The performance of each LLM on the MCQs was evaluated based on several metrics to determine their accuracy and consistency (Fig. 1).

The model responses were compared with the answers provided by expert reviewers, using a scoring system in which responses could earn 5 (no error), 3 (one error), 1 (two errors) or 0 points (more than two errors). Accuracy was then computed as the mean score across all MCQs and expressed as a percentage [3].

Consistency was evaluated using two key metrics: repeatability and robustness.

* Repeatability was defined as the ability of the models to provide identical responses to the same question when asked twice.

* Robustness was defined as the ability of the models to provide identical responses to paraphrased versions of the same question, assessing the model's ability to process variations in question phrasing. This paraphrasing was performed manually by the expert coordinator of the final dataset. Both metrics were computed as the percentage of consistent responses across all MCQs.

In addition to these assessments, the proximity of embeddings was assessed to better understand how LLMs process questions and generate answer. An embedding is a numerical representation of a text, created by the model to capture its meaning. Embeddings were herein generated using the text-embedding-ada-002 model from OpenAI for each MCQ, expert's correct answer, and LLM's generated response. The cosine similarity between the embeddings of each MCQ and expert's correct answer was calculated to measure the proximity between them, and similarly, between each MCQ and LLM's generated response [27]. A high cosine similarity means that the two texts are closely related in meaning, while a low similarity indicates they are more distinct.

### 2.4. Contextual understanding

Contextual understanding was evaluated using two approaches: classification of questions and definition of terms.

The LLM was asked to classify each MCQ in the accurate category; the prompt provided the LLM with the question and a list of the existing categories, asking it to identify the most accurate (Table S1). The
<figure><img src="image_5.png" /><figcaption>**Fig. 1.** Methodological framework for evaluating large language models (LLM) with the three main stages: a) pretreatment of the dataset (selection and annotation of multiple-choice questions (MCQs) by experts), b) evaluation of the LLMs using their respective Application Programming Interfaces (APIs), and c) extraction of metrics to assess the performance of the LLMs.</figcaption></figure>

response of the LLM was defined as correct when it appropriately matched the predefined category of the question.

In addition, a second set of questions was created by selection of endodontic terms from the glossary provided by the Association of American Endodontists (AAE) for undergraduate third-year students [23]. This set was designed to test the models' ability to understand the specific endodontic terminology. Each prompt asked the model to infer the correct terms based on their definitions (Table S1). Both metrics were computed as the percentage of consistent responses across all questions. Similarly, embeddings were generated using the text-embedding-ada-002 model from OpenAI for each definition. These embeddings were calculated for each definition, the expert-defined term, and the LLM-inferred term. The cosine similarity between the embeddings of each definition and correct term was calculated, and similarly, between the definition and the inferred term [27]. The proximity of embeddings was also analysed using a visualisation of embedding values (heatmap) and the t-SNE method (Fig. 2) [28].

To further assess the models' effectiveness in comprehensively integrating data, the response accuracy to the MCQs was also evaluated after providing the LLMs with the relevant full course chapter, according to the category of each MCQ. The queries were executed through the

platforms' APIs using context-based prompting that included a specific chapter of the dataset directly related to the question's category, followed by the MCQ. The prompt format required the LLM to identify the correct answers from options A to E, indicating whether each option was true or false (Table S1). Similarly, embeddings were generated using the text-embedding-ada-002 model from OpenAI for each MCQ, expert's correct answer, and LLM's generated response. The cosine similarity between the embeddings of each MCQ and full course chapter was calculated to measure the proximity between them [27].

## 2.5. Statistical analysis

All answers were stored in an Excel spreadsheet (Microsoft Corporation) and descriptive analytics were conducted using Python scripts alongside Python's statistical libraries. Normality was evaluated using the Shapiro-Wilk test. For ordinal data, such as accuracy, a Friedman test was used as a global test to assess differences in performance among the LLMs, with the significance level set at α=0.05. Post hoc comparisons were conducted using the Wilcoxon Signed-Rank Test with Bonferroni correction (adjusted α=0.0083). For binary paired data, such as repeatability, robustness, and the capacity to define terms or categories,

<figure><img src="image_6.png" /><figcaption>**Fig. 2.** Workflow for evaluating contextual understanding of endodontic terms by large language models with a) data extraction: the glossary of endodontic terms is extracted from a PDF document using the python library PyPDF2; b) data pre-treatment: data is cleaned and structured in an Excel sheet; c) inference generation: definitions are sent to ChatGPT via an API prompt to generate inferred terms; d) embedding generation: each definition, term, and inferred term is converted into a vector using the text-embedding-ada-002 model; and e) similarity calculation: cosine similarity is used to measure the alignment between vectors: a higher cosine similarity indicates a closer relationship in meaning between two texts.</figcaption></figure>
Cochran's Q test was used as a global test, followed by post hoc comparisons using McNemar's test with Bonferroni correction.

## 3. Results

The final dataset included 517 MCQs (Fig. S1) as well as the definitions of 539 endodontic terms considered representative of third-year undergraduate students' level. The Shapiro-Wilk test demonstrated that all the data were not normally distributed (p < 0.00001, Table S2).

The accuracy of answers of ChatGPT-4 (63 %), Claude 3 (68 %) and ChatGPT-3.5 (51 %) was significantly greater than that of Mistral 7B (45 %; p < 0.001). When analysing the incorrect responses of ChatGPT-4, for 30 % of the MCQs, their embedding was closer to that of the inferred response, as compared to that of the expert's response. Of note, for 70 % of the MCQs the difference between their embedding and that of ChatGPT-4 was lower than 5 % (Table S3). Similarly, the repeatability of answers of ChatGPT-4 (100 %) and Claude 3 (97 %) was significantly higher than that of Mistral 7B (77 %; p < 0.0083, for both). Regarding robustness of the answers provided, that of ChatGPT-4 (72 %) was significantly greater than those of Mistral 7B (50 %; p < 0.001) and ChatGPT-3.5 (63 %; p < 0.001).

No significant global difference was observed in the capacity of the models to classify questions into the correct category, as indicated by the Cochran's Q test (p = 0.22). Descriptively, the capacity of Mistral 7B (31 %) appeared lower compared to ChatGPT-3.5 (89 %), ChatGPT-4 (89 %), and Claude 3 (98 %). When analysing the incorrect responses of ChatGPT-4, for 33 % of the MCQs, their embedding was closer to that of the inferred response, as compared to that of the expert's response. Of note, for 56 % of the MCQs the difference between their embedding and that of ChatGPT-4 was lower than 5 % (Table S3). Some questions from the same category were distributed across different clusters (Fig. S2).

The capacity of Mistral 7B (56 %) to define terms was significantly lower than that of those of ChatGPT-4 (77 %; p < 0.0083, Fig. 3, Table 1). When analysing the incorrect responses of ChatGPT-4, for 52 % of the definitions, their embedding was closer to that of the inferred

response, as compared to that of the expert's response. Of note, for 21 % of the definitions the difference between their embedding and that of ChatGPT-4 was lower than 5 % (Table S3). This proximity of embeddings could be visualised through the contextual clustering (Fig. 4).

The accuracy of answers of ChatGPT-4 with context-based prompting was significantly greater compared to without, for questions on restorations and obturation (p < 0.05). Although lower, no statistically significant difference was observed in the accuracy of answers of ChatGPT-4 with context-based prompting compared to without, for questions on medical status as well as shaping and cleaning (p = 0.61 and p = 0.44, respectively, Table 2). When analysing the incorrect responses of ChatGPT-4, for 68 % of the MCQs, their embedding was closer to that of another chapter, as compared to that of the provided full course chapter. Of note, for 32 % of the MCQs the difference between their embedding and that of another chapter was lower than 5 % (Table S3).

## 4. Discussion

Early-career students and practitioners often feel overwhelmed by the vast medical knowledge required in clinical settings [29–32]. To address this challenge, many could use LLMs for reliable contextual understanding [12,33,34]. In the present study, LLMs demonstrated strong performance in linking questions to relevant chapters and defining terms, highlighting their potential to guide students toward key sections of their lectures and assist in retrieving terms from partial definitions.

Our findings also confirm that recent, advanced LLMs demonstrate greater accuracy than older models [2,35], likely due to specific architectural and training differences [36,37]. First, larger training datasets and increased model parameters have been associated with improved generalisation, particularly in zero-shot and few-shot learning settings, enabling better performance across a variety of tasks [36]. Additionally, very large context windows have been reported to improve performance [37]. For instance, the extensive token context window (>100k tokens) of Claude 3 is substantially larger than that of Mistral 7B, which is

<figure><img src="image_7.png" /><figcaption>**Fig. 3.** Bar chart representation of model performance with a) accuracy: performance in answering multiple-choice questions, b) robustness: ability to provide consistent responses to paraphrased versions of the same question, c) classification: capacity to correctly assign MCQs to predefined categories and d) definition: capacity to infer terms from their definitions.</figcaption></figure>
<table><caption>**Table 1**

Comparative performance metrics of large language models across key evaluation domains.</caption><thead><tr><th>Model</th><th>Accuracy (% [CI])</th><th>Repeatability (% [CI])</th><th>Robustness (% [CI])</th><th>Classification (% [CI])</th><th>Definition (% [CI])</th></tr></thead><tbody><tr><td>ChatGPT-3.5</td><td>51.3* [45.8-54.9]</td><td>92.0 [87.4-97.2]</td><td>62.7 [58.2-68.6]</td><td>89.0 [85.0-91.2]</td><td>76.0 [68.1-83.2]</td></tr><tr><td>ChatGPT-4</td><td>63.0* [62.7-78.7]</td><td>100.0* [93.5-100.0]</td><td>71.6* [55.9-84.3]</td><td>89.4 [76.3-94.9]</td><td>77.0* [63.7-85.9]</td></tr><tr><td>Mistral AI 7B</td><td>44.5 [39.4-48.8]</td><td>77.4 [73.5-81.4]</td><td>50.0 [45.6-55.1]</td><td>31.0 [25.9-34.5]</td><td>56.0 [43.0-68.6]</td></tr><tr><td>Claude 3</td><td>67.7* [50.0-92.8]</td><td>97.0* [87.0-99.6]</td><td>63.0 [51.0-74.9]</td><td>98.1 [85.5-98.8]</td><td>78.0 [65.1-86.8]</td></tr></tbody></table>

Results are expressed as mean percentages with 95 % Wilson confidence intervals. Asterisks (*) indicate statistical significance (p < 0.0083). Accuracy was significantly higher for ChatGPT-4, Claude 3, ChatGPT 3.5 compared to Mistral 7B (p < 0.001), without significant differences observed between ChatGPT-4, Claude 3, and ChatGPT-3.5. Repeatability was significantly higher for ChatGPT-4 and Claude 3 compared to Mistral 7B (p < 0.0083), but no significant differences were observed between ChatGPT-3.5 and the other models. Robustness was significantly higher for ChatGPT-4 compared to Mistral 7B and ChatGPT-3.5 (p < 0.001), while Claude 3 showed no significant differences with other models. Finally, the ability to define terms was significantly higher for ChatGPT-4 compared to Mistral 7B (p < 0.0083), with no other significant differences among the remaining models.

<figure><img src="image_8.png" /><figcaption>**Fig. 4.** Contextual analysis of terms of the American Association of Endodontists (AAE) glossary using the text-embedding-ada-002 model with a) a heatmap visualisation of the three closest terms of the glossary and two randomly distant terms; and b) cluster analysis after dimensionality reduction with t-SNE [28]. Colours correspond to the 8 clusters identified by the KMeans algorithm. The coloured areas denote the Voronoi regions associated with each cluster, and the crosses indicate the centroids of the clusters.</figcaption></figure>

limited to a 4k-token window.

Overall, performances on accuracy appear however restricted, with at least a quarter of questions still answered incorrectly. This aligns with

previous studies [38] and may be due to inherent LLM limitations in interpreting and processing complex questions, as reported by Suarez et al. [17]. In the present study, errors frequently arose from the
<table><caption>**Table 2**

Impact of course chapter access on Language Model Performance.</caption><thead><tr><th>Category</th><th>Without course chapter (% [CI])</th><th>With course chapter (% [CI])</th><th>p-value</th></tr></thead><tbody><tr><td>Emergencies</td><td>43.6 [34.8-51.8]</td><td>52.5 [43.4-60.5]</td><td>0.07</td></tr><tr><td>Medical status</td><td>60.3 [42.0-75.6]</td><td>57.2 [38.5-71.6]</td><td>0.61</td></tr><tr><td>Dental Caries Management</td><td>59.0 [40.3-75.7]</td><td>73.4 [53.9-86.2]</td><td>0.08</td></tr><tr><td>Non-Carious Lesions</td><td>80.6 [51.7-93.7]</td><td>91.3 [64.6-98.5]</td><td>0.70</td></tr><tr><td>Restorations*</td><td>50.2 [39.2-61.0]</td><td>65.9 [54.9-75.5]</td><td><strong>0.03</strong></td></tr><tr><td>Pulp and periapical pathologies</td><td>49.0 [25.5-73.2]</td><td>58.1 [32.9-80.1]</td><td>0.38</td></tr><tr><td>Shaping and Cleaning</td><td>66.3 [51.0-78.8]</td><td>61.8 [46.2-74.7]</td><td>0.44</td></tr><tr><td>Obturation*</td><td>44.6 [32.9-57.9]</td><td>69.2 [56.6-80.1]</td><td><strong>0.01</strong></td></tr></tbody></table>

Results are expressed as mean with [CI] 95 % Wilson confidence interval. As-terisks (*) indicate statistical significance (p < 0.05), with p-values highlighted in bold between performance without and with course chapter.

proximity of embeddings between correct and incorrect answers. Additionally, while context-based prompting generally improved the accuracy of LLMs for most MCQs, it unexpectedly failed to enhance, and even reduced, accuracy for certain chapters, such as those on shaping. The errors may again be attributed to the model's difficulty in distinguishing closely related specialised concepts and identifying relevant information within training datasets that are inherently broad [36,37]. These findings also suggest that merely providing additional information to the LLM is insufficient to improve performance and may even mislead the model [20,36]. While the size of training datasets has grown for advanced LLMs, evidence of parallel improvements in data quality and curation within specialised domains, such as restorative dentistry and endodontics, remains limited. Taking into account ongoing advancements in techniques such as reinforcement learning and RAG, our findings emphasize the importance of first establishing benchmarks to evaluate LLMs in specialised domains. This, therefore highlights the critical role of dental practitioners, educators, and researchers in collaborating to develop smaller, yet more robust, domain-specific language models such as Camembert-bio in medicine [39-42].

Another issue relies in the potential use of LLMs by students; their routine use could differ significantly from the specific questions evaluated herein, highlighting the importance of assessing performance across diverse contexts. It is noteworthy that the robustness of the models was herein relatively limited, as responses were inconsistent when questions were paraphrased. This inconsistency may pose a challenge for students attempting to integrate knowledge. The repeatability of the LLMs also raised concerns; high performances and consistency were reported in previous studies in endodontics [17], while our results indicate that consistency may still be insufficient for educational purposes, except for the most recent LLMs. It is of particular importance to note that this consistency was evaluated using closed-ended questions. In contrast, studies have shown that LLMs achieved an acceptable level of reliability on open-ended questions in endodontics, [43] but lower reliability in prosthodontics [44]. These differences might be due to heterogeneous evaluation protocols and terminologies, underscoring the need for clear guidelines to evaluate LLMs in education [12]. Given these findings, if LLMs are to be used into educational applications, focusing on the most recent LLMs and using closed-ended questions may reduce inconsistency risks.

Despite the potential benefits for dental education, it is of particular importance to note that the prompts used in the present study were carefully crafted to ensure optimal responses through prompt engineering. This level of precision might however not be maintained by students or novice users without dedicated lectures [45,46]. Such training is particularly crucial, as users may otherwise rely excessively on LLMs for convenience, without seeking feedback from educators. This dependency could undermine critical thinking skills and foster biases such as overconfidence [12]. Students have indeed demonstrated reduced performance when deprived of access to tools like ChatGPT,

after becoming reliant on their use [47]. These observations underscore the need for clear guidelines to govern the education and clinical use of LLMs as well as to assess their performance [10]. Achieving a consensus, however, remains a complex challenge. Even at the level of terminology, a lack of standardisation is notable, with metrics such as reading comprehension [18,24,48] or logical reasoning [10,49] being variably employed to assess contextual understanding as the number of studies evaluating LLMs is increasing.

## 5. Conclusions

Even the most advanced LLMs, including ChatGPT-4 and Claude 3, achieve moderate performance in the field of dental education. They must notably be used cautiously due to inconsistencies in robustness. Regarding ChatGPT-4, context-based prompting improved accuracy, though results varied across chapters. Overall, the models achieved promising levels of contextual understanding, suggesting relevance in this area. However, further studies are needed to evaluate their potential in real educational settings, especially given their limitations in distinguishing closely related concepts.

## Declaration of generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the authors used Le Chat (Mistral AI) to improve language and readability, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication

## CRediT authorship contribution statement

**Claire Lafourcade:** Writing – review & editing, Writing – original draft, Validation, Resources, Methodology, Data curation, Conceptualization. **Olivia Kérourédan:** Writing – review & editing, Writing – original draft, Validation, Methodology, Investigation, Data curation, Conceptualization. **Benoit Ballester:** Writing – review & editing, Writing – original draft, Validation, Software, Resources, Conceptualization. **Raphael Richert:** Writing – review & editing, Writing – original draft, Validation, Software, Resources, Methodology, Data curation, Conceptualization.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgements

The authors thank the Collège National des Enseignants en Odontologie Conservatrice (CNEOC), and the contributors to the reference manual for dental residency applicants in restorative dentistry and endodontics for their acceptance in sharing the data. The authors also thank Shanez Haouari (DRS, Hospices Civils de Lyon, Lyon, France) for help in manuscript preparation.

## Supplementary materials

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.jdent.2025.105764.

## References

[1] C.W. Safranek, A.E. Sidamon-Eristoff, A. Gilson, D. Chartash, The role of large language models in medical education: applications and implications, JMIR Med. Educ. 9 (2023) 1–12, https://doi.org/10.2196/50945.
[2] H. Song, Y. Xia, Z. Luo, H. Liu, Y. Song, X. Zeng, T. Li, G. Zhong, J. Li, M. Chen, G. Zhang, B. Xiao, Evaluating the performance of different large language models on health consultation and patient education in urolithiasis, J. Med. Syst. 47 (2023) 125, https://doi.org/10.1007/s10916-023-02021-3.

[3] T.H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, M. Elepaño, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo, V. Tseng, Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models, PLOS Digit. Heal. 2 (2023), https://doi.org/10.1371/journal. pdig.0000198.

[4] H. Huang, O. Zheng, D. Wang, J. Yin, Z. Wang, S. Ding, H. Yin, C. Xu, R. Yang, Q. Zheng, B. Shi, ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model, Int. J. Oral Sci. 15 (2023) 1–13, https://doi.org/10.1038/s41368-023-00239-y.

[5] K. Giannakopoulos, A. Kavadella, A.A. Salim, V. Stamatopoulos, E.G. Kaklamanos, Evaluation of the performance of generative AI large language models ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: comparative mixed methods study, J. Med. Internet Res. 25 (2023), https://doi.org/10.2196/51580.

[6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Adv. Neural Inf. Process. Syst. (2017) 5999–6009, https://doi.org/10.48550/arXiv.1706.03762.

[7] Y. Zhu, J.R.A. Moniz, S. Bhargava, J. Lu, D. Piravirumal, S. Li, Y. Zhang, H. Yu, B. H. Tseng, Can large language models understand context?, in: EACL 2024 - 18th Conf. Eur. Chapter Assoc. Comput. Linguist. Find. EACL 2024, 2024, pp. 2004–2018, https://doi.org/10.48550/arXiv.2402.00858.

[8] A. Gilson, C.W. Safranek, T. Huang, V. Socrates, L. Chi, R.A. Taylor, D. Chartash, How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment, JMIR Med. Educ 9 (2023) 1–9, https://doi.org/10.2196/45312.

[9] S.M.I. Alkhaaldi, C.H. Kassab, Z. Dimassi, L. Oyoun Alsoud, M. Al Fahim, C. Al Hageh, H. Ibrahim, Medical student experiences and perceptions of ChatGPT and artificial intelligence: cross-sectional study, JMIR Med. Educ. 9 (2023) e51302, https://doi.org/10.2196/51302.

[10] M. Büttner, U. Leser, L. Schneider, F. Schwendicke, Natural language processing: chances and challenges in dentistry, J. Dent. 141 (2024) 104796, https://doi.org/10.1016/j.dent.2023.104796.

[11] F. Eggmann, R. Weiger, N.U. Zitzmann, M.B. Blatz, Implications of large language models such as ChatGPT for dental medicine, J. Esthet. Restor. Dent. 35 (2023) 1098–1102, https://doi.org/10.1111/jerd.13046.

[12] S.E. Uribe, I. Maldupa, A. Kavadella, M. El Tantawi, A. Chaurasia, M. Fontana, R. Marino, N. Innes, F. Schwendicke, Artificial intelligence chatbots and large language models in dental education: worldwide survey of educators, Eur. J. Dent. Educ. (2024) 1–12, https://doi.org/10.1111/eje.13009.

[13] K. Singhal, S. Azizi, T. Tu, S.S. Mahdavi, J. Wei, H.W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, A. Babiker, N. Schärli, A. Chowdhery, P. Mansfield, D. Demner-Fushman, B. Agüera y Arcas, D. Webster, G.S. Corrado, Y. Matias, K. Chou, J. Gottweis, N. Tomasev, Y. Liu, A. Rajkomar, J. Barral, C. Semturs, A. Karthikesalingam, V. Natarajan, Large language models encode clinical knowledge, Nature 620 (2023) 172–180, https://doi.org/10.1038/s41586-023-06291-2.

[14] W.H.K. Chiu, W.S.K. Ko, W.C.S. Cho, S.Y.J. Hui, W.C.L. Chan, M.D. Kuo, Evaluating the diagnostic performance of large language models on complex multimodal medical cases, J. Med. Internet Res. 26 (2024) 1–4, https://doi.org/10.2196/53724.

[15] A.S. Huang, K. Hirabayashi, L. Barna, D. Parikh, L.R. Pasquale, Assessment of a large language model's responses to questions and cases about glaucoma and Retina management, JAMA Ophthalmol 142 (2024) 371–375, https://doi.org/10.1001/jamaophthalmol.2023.6917.

[16] A. Danesh, H. Pazouki, K. Danesh, F. Danesh, A. Danesh, The performance of artificial intelligence language models in board-style dental knowledge assessment: a preliminary study on ChatGPT, J. Am. Dent. Assoc. 154 (2023) 970–974, https://doi.org/10.1016/j.adaj.2023.07.016.

[17] A. Suárez, V. Díaz-Flores García, J. Algar, M. Gómez Sánchez, M. Llorente de Pedro, Y. Freire, Unveiling the ChatGPT phenomenon: evaluating the consistency and accuracy of endodontic question answers, Int. Endod. J. 57 (2023) 1–6, https://doi.org/10.1111/iej.13985.

[18] J.B. Longwell, I. Hirsch, F. Binder, G.A. Gonzalez Conchas, D. Mau, R. Jang, R. G. Krishnan, R.C. Grant, Performance of large language models on medical oncology examination questions, JAMA Netw. Open. 7 (2024) e2417641, https://doi.org/10.1001/jamanetworkopen.2024.17641.

[19] E.J. Dogramaci, G. Rossi-Fedele, Patient-related outcomes and Oral health-related quality of life in endodontics, Int. Endod. J. 56 (2023) 169–187, https://doi.org/10.1111/iej.13830.

[20] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, T. Liu, A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions, Trans. Assoc. Comput. Linguist. (2023), https://doi.org/10.48550/arXiv.2311.05232.

[21] A. Baaij, C. Kruse, J. Whitworth, F. Jarad, Undergraduate curriculum guidelines for endodontology, Int. Endod. J. (2024) 1–14, https://doi.org/10.1111/iej.14064.

[22] R. Devillard, O. Romieu, R. Arbab-Chirani, D. Seux, E. Mortier, Dentisterie restauratrice et endontie: référentiel pour l'internat, Information Dentaire (2021).

[23] AAE special committee of full-time educators, Glossary of Endodontic Terms 2016, 2015.

[24] C. Zakka, A. Chaurasia, R. Shad, W. Hiesinger, Almanac: knowledge-grounded language models for clinical medicine, N. Engl. J. Med. (2023) 1–23, https://doi.org/10.48550/arXiv.2303.01229.

[25] G. Xiong, Q. Jin, Z. Lu, A. Zhang, Benchmarking retrieval-augmented generation for medicine, (2024), http://arxiv.org/abs/2402.13178.

[26] M. Jeong, J. Sohn, M. Sung, J. Kang, Improving medical reasoning through retrieval and self-reflection with retrieval-augmented large language models, Bioinformatics 40 (2024) i119–i129, https://doi.org/10.1093/bioinformatics/bte4238.

[27] H. Steck, C. Ekanadham, N. Kallus, Is cosine-similarity of embeddings really about similarity?, in: in: WWW 2024 Companion - Companion Proc. ACM Web Conf., 2024, pp. 887–890, https://doi.org/10.1145/3589335.3651526.

[28] L. van der Maaten, G. Hinton, Visualizing data using t-SNE, J. Mach. Learn. Res. 9 (2008) 2579–2605, https://doi.org/10.48550/arXiv.2005.14165.

[29] J.Q. Young, J. Van Merriënboer, S. Durning, O. Ten Cate, Cognitive Load Theory: implications for medical education: AMEE Guide No. 86, Med. Teach. 36 (2014) 371–384, https://doi.org/10.3109/0142159X.2014.889290.

[30] B. Buddeberg-Fischer, R. Klaghofer, M. Stamm, J. Siegrist, C. Buddeberg, Work stress and reduced health in young physicians: prospective evidence from Swiss residents, Int. Arch. Occup. Environ. Health 82 (2008) 31–38, https://doi.org/10.1007/s00420-008-0303-7.

[31] F. de Souza Ferreira, I. Barros, T. da Costa Neves, J.M. Pazos, P.P.N.S. Garcia, Stress amongst dental students in the transition from preclinical training to clinical training: a qualitative study, Eur. J. Dent. Educ. 27 (2023) 568–574, https://doi.org/10.1111/eje.12842.

[32] B.S. Malau-Aduli, A. Lee, F. Alele, M. Adu, A. Drovandi, K. Jones, Preclinical to clinical transition experiences of dental students at an Australian Regional University, Eur. J. Dent. Educ. 26 (2022) 182–196, https://doi.org/10.1111/eje.12687.

[33] J.W. Ayers, A. Poliak, M. Dredze, E.C. Leas, Z. Zhu, J.B. Kelley, D.J. Faix, A. M. Goodman, C.A. Longhurst, M. Hogarth, D.M. Smith, Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum, JAMA Intern. Med. (2023), https://doi.org/10.1001/jamainternmed.2023.1838.

[34] A. Suárez, J. Jiménez, M. Llorente de Pedro, C. Andreu-Vázquez, V. Díaz-Flores García, M. Gómez Sánchez, Y. Freire, Beyond the scalpel: assessing ChatGPT's potential as an auxiliary intelligent virtual assistant in oral surgery, Comput. Struct. Biotechnol. J. 24 (2024) 46–52, https://doi.org/10.1016/j.csbj.2023.11.058.

[35] S. Yamaguchi, M. Morishita, H. Fukuda, K. Muraoka, T. Nakamura, I. Yoshioka, I. Soh, K. Ono, S. Awano, Evaluating the efficacy of leading large language models in the Japanese national dental hygienist examination: a comparative analysis of ChatGPT, Bard, and Bing Chat, J. Dent. Sci. 5 (2024), https://doi.org/10.1016/j.csbj.2024.02.019.

[36] T.B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D.M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: Adv. Neural Inf. Process. Syst., 2020.

[37] N.F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang, Lost in the middle: how language models use long contexts, Trans. Assoc. Comput. Linguist. 12 (2024) 157–173, https://doi.org/10.1162/tac1_00638.

[38] R.C.W. Chau, R.T.C. Hsung, C. McGrath, E.H.N. Pow, W.Y.H. Lam, Accuracy of artificial intelligence-designed single-molar dental prostheses: a feasibility study, J. Prosthet. Dent. 9 (2023) 1–7, https://doi.org/10.1016/j.prostdent.2022.12.004.

[39] K. Huang, J. Altosaar, R. Ranganath, ClinicalBERT: modeling clinical notes and predicting hospital readmission, (2019), https://doi.org/10.48550/arXiv.1904.05342.

[40] R. Touchent, L. Romary, E. de La Clergerie, CamemBERT-bio: leveraging continual pre-training for cost-effective models on French biomedical data, in: 2024 Jt. Int. Conf. Comput. Linguist. Lang. Resour. Eval. Lr. 2024 - Main Conf. Proc, 2024, pp. 2692–2701, https://doi.org/10.48550/arXiv.2306.15550.

[41] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, H. Poon, Domain-specific language model pretraining for biomedical natural language processing, ACM Trans. Comput. Healthc 3 (2022) 1–24, https://doi.org/10.1145/3458754.

[42] J. Shen, N. Tenenholtz, J.B. Hall, D. Alvarez-Melis, N. Fusi, Tag-LLM: repurposing general-purpose LLMs for specialized domains, Proc. Mach. Learn. Res. 235 (2024) 44759–44773.

[43] H.M.- Rahimi, S. Amirhossein, M. Amin, P. Omid, Validity and reliability of artificial intelligence chatbots as public sources of information on endodontics, Int. Endod. J. 57 (2023) 305–314, https://doi.org/10.1111/iej.14014.

[44] Y. Freire, A. Santamaría Laorden, J. Orejas Pérez, M. Gómez Sánchez, V. Díaz-Flores García, A. Suárez, ChatGPT performance in prosthodontics: assessment of accuracy and repeatability in answer generation, J. Prosthet. Dent. 131 (2024) 1–6, https://doi.org/10.1016/j.prost dent.2024.01.018.

[45] L. Giray, Prompt engineering with ChatGPT: a guide for academic writers, Ann. Biomed. Eng. 51 (2023) 2629–2633, https://doi.org/10.1007/s10439-023-03272-4.

[46] B. Meskó, Prompt engineering as an important emerging skill for medical professionals: tutorial, J. Med. Internet Res. 25 (2023), https://doi.org/10.2196/50638.
[47] M. Abbas, F.A. Jam, T.I. Khan, Is it harmful or helpful? Examining the causes and consequences of generative AI usage among university students, Int. J. Educ. Technol. High. Educ. 21 (2024), https://doi.org/10.1186/s41239-024-00444-7.

[48] C. Peng, X. Yang, A. Chen, K.E. Smith, N. PourNejatian, A.B. Costa, C. Martin, M. G. Flores, Y. Zhang, T. Magoc, G. Lipori, D.A. Mitchell, N.S. Ospina, M.M. Ahmed, W.R. Hogan, E.A. Shenkman, Y. Guo, J. Bian, Y. Wu, A study of generative large

language model for medical research and healthcare, Npj Digit. Med. 6 (2023), https://doi.org/10.1038/s41746-023-00958-w.

[49] M. Sallam, ChatGPT utility in Healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns, Healthcare 11 (2023) 887, https://doi.org/10.3390/healthcare11060887.