# Validation of a generative artificial intelligence tool for the critical appraisal of articles on the epidemiology of mental health: Its application in the Middle East and North Africa

Cheima Moussa, Sarah Altayyar, Marion Vergonjeanne, Thibaut Gelle, Pierre-Marie Preux

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Cheima Moussa, Sarah Altayyar, Marion Vergonjeanne, Thibaut Gelle, Pierre-Marie Preux. Validation of a generative artificial intelligence tool for the critical appraisal of articles on the epidemiology of mental health: Its application in the Middle East and North Africa. Journal of Epidemiology and Population Health, 2025, 73 (2), pp.202990. 10.1016/j.jeph.2025.202990. hal-05076835

HAL Id: hal-05076835

https://unilim.hal.science/hal-05076835v1

Submitted on 21 May 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
Original article

# Validation of a generative artificial intelligence tool for the critical appraisal of articles on the epidemiology of mental health: Its application in the Middle East and North Africa

<figure><img src="image_3.png" /></figure>

Cheima Moussaª, Sarah Altayyarª, Marion Vergonjeanneª,ᵇ, Thibaut Gelleª,*, Pierre-Marie Preuxª,ᵇ

## ARTICLE INFO

**Article History:**

Received 29 January 2025

Accepted 11 March 2025

Available online 3 April 2025

**Keywords:**

Artificial intelligence

ChatGPT

Critical appraisal

Mental health

MENA

## ABSTRACT

Mental health disorders have a high disability-adjusted life years in the Middle East and North Africa. This rise has led to a surge in related publications, prompting researchers to use AI tools like ChatGPT to reduce time spent on routine tasks. Our study aimed to validate an AI-assisted critical appraisal (CA) tool by comparing it with human raters.

We developed customized GPT models using ChatGPT-4. These models were tailored to evaluate studies using the Newcastle-Ottawa Scale (NOS) or the Jadad Scale in one model, while another model evaluated STROBE or CONSORT guidelines.

Our results showed a moderate to good agreement between human CA and our GPTs for the NOS for cohort, case control and cross-sectional studies and for the Jadad scale, with an ICC of 0.68 [95 %CI: 0.24–0.82], 0.69 [95 %CI: 0.31–0.88], 0.76 [95 %CI: 0.47–0.90] and 0.84 [95 %CI: 0.57–0.94] respectively. There was also a moderate to substantial agreement between the two methods for STROBE in cross sectional, cohort, case control studies, and for CONSORT in trial design, with a K of 0.63 [95 %CI: 0.56–0.70], 0.57 [95 %CI: 0.47–0.66], 0.48 [95 %CI: 0.38–0.50] and 0.70 [95 %CI: 0.63–0.77] respectively. Our custom GPT models produced hallucinations in 6.5 % and 4.9 % of cases, respectively. Human raters took an average of 19.6 ± 4.3 min per article, whereas our customized GPTs took only 1.4.

ChatGPT could be a useful tool for handling repetitive tasks yet its effective application relies on the critical expertise of researchers.

© 2025 The Author(s). Published by Elsevier Masson SAS. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/)

## 1. Introduction

Artificial intelligence (AI) is advancing rapidly in healthcare due to its ability to analyze vast amounts of data and provide insights that support evidence-based clinical decision-making [1].

More recently, large language models (LLMs) have shown substantial promise due to their ability to learn and adapt to various linguistic patterns without additional training [2]. In the realm of Natural Language Processing (NLP), these advanced models, with millions of parameters, have demonstrated efficacy in clinical settings, such as diagnosing medical conditions [3] and answering health-related inquiries [4,5]. Furthermore, they have been applied to extract information from articles, efficiently generate essays, and

draft review papers with minimal input [6–9]. In some instances, LLMs have even achieved near-human performance, such as passing national medical licensing exams [10–12].

OpenAI's advancements in LLMs, such as the release the more advanced GPT-4 in March 2023, have further expanded the capabilities of these models. GPT-4 is noted for its improved reliability, creativity, and ability to handle more complex instructions than its predecessor [13]. These models fall within the broader category of Generative AI Systems (GAIS), which can produce diverse outputs, from text to images and videos, for various purposes [14].

AI systems are designed to enhance speed and accuracy across various fields, including mental health care. Studies have shown significant reductions in anxiety, stress, and depression through the use of mental health mobile applications [15]. In 2019, one in eight people in the world had a mental disorder, with anxiety and depressive disorders being the most common [16]. In the Middle East and North

<footnote>ª Inserm U1094, IRD UMR270, University Limoges, CHU Limoges, EpiMaCT - Epidemiology of Chronic Diseases in Tropical Zone, Institute of Epidemiology and Tropical Neurology, Limoges, France</footnote>

<footnote>ᵇ CHU Limoges, Clinical Data and Research Center CDCR, Limoges, France</footnote>

<footnote>* Corresponding author.
E-mail address: thibaut.gelle1@ird.fr (T. Gelle).</footnote>
Africa (MENA) region, these disorders also ranked high in terms of disability-adjusted life years [17]. The rapid growth of psychiatric literature presents a challenge for researchers, as producing high-quality systematic reviews or even simple critical appraisals (CA) of articles remains time-consuming due to the need to manually screen thousands of studies [18]. While LLMs like ChatGPT-4 offer considerable promise, they remain susceptible to errors and hallucinations, which occur when the model generates inaccurate information and highlights the need for ongoing research into the reliability and transparency of AI systems [19].

Given this context, our study aimed to validate an AI-assisted method for CA in epidemiological studies and clinical trials. We compared quality scores from the Newcastle-Ottawa Scale (NOS) and Jadad scale between human appraisals and AI-assisted appraisals, to evaluate the agreement on study recommendations using the Strengthening Standards of Reporting of Observational Studies in Epidemiology (STROBE) and the Consolidated Standards of Reporting Trials (CONSORT) guidelines. Additionally, we assessed the time saved by using AI and examined the frequency of hallucinations in AI-generated outputs.

# 2. Materials and methods

## 2.1. Study type

We conducted a validation study utilizing AI for the critical appraisal of articles in the epidemiology of mental health disorders within the MENA region.

## 2.2. Study period and material

The study was carried out between January 2024 and June 2024, using ChatGPT-4 by OpenAI®. ChatGPT-4 was selected for its extensive customization options, advanced language comprehension, and generation capabilities. During the study this version was accessible through a paid subscription [13].

## 2.3. Development of customized GPT models

Due to variations in article quality assessment tools, two distinct GPT models were developed. The first model was designed to use the NOS [20] and Jadad scale [21], to assess the quality of observational studies and randomized controlled trials (RCTs), respectively. The second model was developed to complete the STROBE [22] and CONSORT [23] guidelines, which aim to enhance the reporting quality of observational studies and clinical trials.

Utilizing GPT Builder, a tool within ChatGPT-4 that allows users to create customized models tailored to specific tasks, these models were programmed to incorporate specific instructions. Imperative commands such as "you must" were employed to enforce strict adherence to predefined steps, clarify what should be avoided, and direct the model's behavior during assessments.

The knowledge base of our GPT models was supplemented with the following documents:

* First model: The NOS and Jadad scale, along with their manuals, to enable the GPT to make informed decisions and apply best practices in article evaluation. Additional documents outlined restrictions on studies that should not be evaluated by this model.

* Second model: The STROBE and CONSORT guidelines, supplemented with specific instructions from their respective manuals [22,23].

### 2.3.1. Prompt input

The input of prompts has been adapted to suit the specifics of each model using an online prompt generator. The specific prompts utilized for the two models are presented in the appendix A.

### 2.3.2. Custom ChatGPT disabling

To ensure unbiased interaction, it was necessary to disable any customize ChatGPT settings during the main phase of the appraisal. This step ensured that our custom GPTs performance was not influenced by prior customizations during critical evaluations. Fig. 1 illustrates the details of this process.

<figure><img src="image_4.png" /><figcaption>Fig. 1. Disabling of customize ChatGPT.</figcaption></figure>
### 2.3.3. Improvement process

An initial refinement phase involved testing the GPT models with a range of articles of varying quality. This experimental approach allowed for adjustments to optimize accuracy and relevance in article evaluation. Adjustments primarily consisted of simplifying items that the GPT model either misunderstood or incorrectly evaluated based on the respective tools and guidelines.

### 2.3.4. Pre-validation process

A pre-validation phase was conducted prior to the final validation of the GPT models. One researcher tested the Jadad scale model by evaluating RCTs, while another applied the NOS model to articles that had been previously evaluated from a previous systematic review.

### 2.3.5. Data source

In our study, we randomly selected scientific articles on PubMed. These articles addressed pathologies identified by the World Health Organization as prevalent worldwide, including depression, anxiety, and schizophrenia [16], and covered MENA region as defined by the World Bank [24]. The research equation was formulated to change with each country, focusing on specific MeSH terms related to the pathologies studied: ("Depression"[MeSH Terms] OR "Depressive Disorder"[MeSH Terms] OR "depressive disorder, major"[MeSH Terms] OR "Depression"[All Fields] OR "Anxiety"[MeSH Terms] OR "Anxiety Disorders"[MeSH Terms] OR "Anxiety"[All Fields] OR "Schizophrenia"[MeSH Terms] OR "Schizophrenia"[All Fields]) AND ("NAME OF THE COUNTRY"[All Fields]). Appendix B shows examples of the research equation with three MENA countries.

### 2.3.6. Random sampling process

The sample selection process aimed to include one article per pathology, with three pathologies per country. A range of study designs (cross-sectional, cohort, case-control, and clinical trials (RCTs)) were considered, resulting in three different types of studies per country and a total of 63 articles for the entire MENA region. Selection was performed using an online random number generator (www.randomizer.org), and repeated until articles of different study designs and pathologies were obtained.

### 2.3.7. Inclusion and non-inclusion criteria

Inclusion criteria specified articles published between 2013 and 2023 correlating with the release of the Diagnostic and Statistical Manual of mental disorders the fifth edition (DSM-5) [25]. The articles had to be in English, aligning with the common language used in GAIS, which often contain training datasets where English comprising about 30% of the data, while French accounts for approximately 13% [19]. The research focused on various study types related to anxiety, depression and schizophrenia in MENA region.

## 2.4. Validation process

The CA process was organized into two primary phases:

a). Initially, a researcher independently conducted the CA using the NOS and Jadad scales and checklists. Subsequently, a second researcher systematically repeated the evaluation. They subsequently convened to compare their assessments, engaging in a thorough discussion to resolve any discrepancies in the allocation of stars or points. Through this deliberative process, a single final score was assigned. Since consensus was reached through discussion, we did not specifically calculate inter-rater agreement. The time spent during this phase was recorded.

b). The CA was then conducted using our customized GPT models. Time was again recorded, along with the proportion of hallucinations (inaccurate outputs) generated by the models. All results

generated by our GPT models were obtained exclusively from their initial responses.

## 2.5. Statistical analysis

Data were summarized through descriptive analyses, where qualitative variables were described with numbers and proportions, while quantitative variables were presented as means ± standard deviations. The geographical mapping of selected articles was conducted using QGIS, version 2.14.20. The intraclass correlation coefficient (ICC) and Bland-Altman plots were utilized to assess the agreement of scales' scores between the two methods: poor: if ICC (0.50, moderate: if ICC was between 0.50 and 0.75, good: if ICC was between 0.75 and 0.90, and excellent: if ICC ≥ 0.90 [26]). The degree of agreement between the two methods, for the STROBE and CONSORT guidelines, was assessed using the Cohen's unweighted kappa coefficient (K) poor if K < 0.00, slight: if K was between 0.00 and 0.20, fair: if K was between 0.21 and 0.40, moderate: if K was between 0.41 and 0.60, substantial: if K was between 0.61 and 0.80, and almost perfect: if K was between 0.81 and 1.00 [27]. Data analysis was conducted using JASP software, version 0.18.1.

# 3. Results

Our first primary results were the obtaining of the customized GPT models which are freely accessible using the links provided below:

The GPT model for the NOS and Jadad scales are available through this link: https://chat.openai.com/g/g-RoYn8j4aC-study-evaluator

The GPT model for the STROBE and CONSORT are available through this link: https://chat.openai.com/g/g-4157bxu3R-study-analyzer

### 3.1. Characteristics of studies

After randomization, selection according to the availability of articles and to our inclusion criteria, we obtained a total of 61 articles. A total of 19 cross-sectional studies, 13 cohort studies, 16 case-control studies, and 13 RCTs were included. Fig. 2 illustrates the distribution of these articles across MENA countries.

### 3.2. Evaluating GPT models in CA

During the validation process, several hallucinations were observed in the customized GPT models. The model designed for the NOS and Jadad scales experienced hallucinations in 4 cases (6.5%), including incorrect score calculations (2 instances), misapplication of criteria from another scale (1 instance), and failure to correctly identify the study type (1 instance). Similarly, the model designed for the STROBE and CONSORT guidelines had 3 hallucinations (4.9%), where the study type was incorrectly identified, and items from the checklist were either missed or added incorrectly in 2 instances. When the GPT model incorrectly identified a study as cross-sectional, it was excluded from the NOS and STROBE analyses, reducing the number of cross-sectional studies from 19 to 18.

The mean time spent on manual CA per article was 19.6 ± 4.3 min, compared to 1.4 ± 1.0 min per article using the customized GPT models.

## 3.3. Agreement between manual and customized GPT CA for the Jadad scale

The correlation analysis between human and GPT scoring demonstrated good reliability, with an ICC of 0.84 [95%CI: 0.57–0.94] for the Jadad scale. The Bland-Altman plot (Fig. 3 a) showed a mean
<figure><img src="image_5.png" /><figcaption>Fig. 2. Mapping of the selected study type in the MENA.</figcaption></figure>

difference of -0.15 [95 %CI:0.57-0.26], indicating that the GPT scores were slightly overestimated compared to the human scores. Most data points fell within ±1.96 standard deviations of the mean difference, indicating strong consistency between manual and GPT scoring.

### 3.4. Agreement between manual and customized GPT CA of articles for the NOS adapted to cross sectional studies

A good agreement was observed between human and GPT scoring for cross-sectional studies, with an ICC of 0.76 [95 %CI: 0.47-0.90]. The Bland-Altman plot (Fig. 3 b) showed an average difference of -0.33 [95 %CI:0.92, 0.25], suggesting that GPT scores were slightly higher than human scores. Most measurements were within ±1.96 standard deviations, supporting a good agreement between the two methods.

### 3.5. Agreement between manual and customized GPT CA for the NOS for cohort studies

For cohort studies, a moderate agreement was observed with an ICC of 0.68 [95 %CI: 0.24-0.89]. Most cohort studies were rated higher by GPT by one or more points on the NOS scale (total of 9 points). The Bland-Altman plot (Fig. 3 c) indicated an average

difference of -1.92 [95 %CI:2.38, -1.46], revealing that GPT scores were generally overestimated compared to human scores.

### 3.6. Agreement between manual and customized GPT CA for the NOS for case control studies

Good agreement was observed between human and GPT scoring for case-control studies, with an ICC of 0.69 [95 %CI: 0.31-0.88]. The Bland-Altman plot (Fig. 3 d) revealed an average difference of -0.50 [95 %CI:1.01-0.01], indicating a slight overestimation by GPT. Most data points were within ±1.96 standard deviations, suggesting a good level of agreement between the two methods.

### 3.7. Agreement between manual and customized GPT evaluations across STROBEs and CONSORT guidelines

Our findings indicate moderate to good agreement between human and GPT CAs. For the STROBE guidelines applied to cohort and case-control studies, moderate agreement was observed with K values of 0.57 and 0.48, respectively. For STROBE in cross-sectional studies and CONSORT in RCTs, good agreement was observed, with K values of 0.63 and 0.70, respectively, as shown in Table 1.
<figure><img src="image_6.png" /><figcaption>**Fig. 3.** Bland-Altman plot of human CA versus customized GPT CA for the NOS and Jadad scale (n = 61<sup>a</sup>). <sup>a</sup>Studies with the same differences are overlapped. The mean difference of Bland and Altman plot was measured according to this formula (Total human rater scores-Total customize GPT rater scores).</figcaption></figure>

## 4. Discussion

Acknowledging the variability in the availability of different types of studies by pathology and country, as well as the variability of the studied pathologies themselves, the randomization protocol has been designed to be flexible. If, after multiple attempts at randomization, a specific study type or pathology remains not found, we opted for an alternative study type that can fulfil the same research objective or address the same pathology under a different study type.

### 4.1. Role of ChatGPT in preliminary assessment

Our findings showed a significant reduction in time spent on CA, from an average of 19.6 min per article for manual CA to just 1.4 min using GPT models and this is only its performance without the verification of human rater. This suggests that ChatGPT can offer substantial time savings during early stages of article screening, potentially reducing the manual workload by up to 65 % [6].

<table><caption>**Table 1**
The agreement between the completion of STROBE and CONSORT guidelines between human and customized GPT.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th rowspan="2"></th><th colspan="2">ChatGPT-4</th><th rowspan="2">Total</th><th rowspan="2">Unweighted<br>kappa</th><th colspan="2">95% CI</th></tr><tr><th>0</th><th>1</th><th>Lower</th><th>Upper</th></tr></thead><tbody><tr><td rowspan="3">STROBE cross sectional</td><td rowspan="3">(1) Human Rater</td><td>0<sup>a</sup></td><td>126</td><td>57</td><td>183</td><td rowspan="3">0.63</td><td rowspan="3">0.56</td><td rowspan="3">0.70</td></tr><tr><td>1<sup>b</sup></td><td>30</td><td>363</td><td>393</td></tr><tr><td>Total</td><td>156</td><td>420</td><td>576</td></tr><tr><td rowspan="3">STROBE cohort</td><td rowspan="3">(1) Human Rater</td><td>0</td><td>54</td><td>27</td><td>81</td><td rowspan="3">0.57</td><td rowspan="3">0.47</td><td rowspan="3">0.66</td></tr><tr><td>1</td><td>31</td><td>330</td><td>361</td></tr><tr><td>Total</td><td>85</td><td>357</td><td>442</td></tr><tr><td rowspan="3">STROBE case control</td><td rowspan="3">(1) Human Rater</td><td>0</td><td>73</td><td>43</td><td>116</td><td rowspan="3">0.48</td><td rowspan="3">0.38</td><td rowspan="3">0.50</td></tr><tr><td>1</td><td>54</td><td>358</td><td>412</td></tr><tr><td>Total</td><td>127</td><td>401</td><td>528</td></tr><tr><td rowspan="3">CONSORT</td><td rowspan="3">(1) Human Rater</td><td>0</td><td>127</td><td>38</td><td>165</td><td rowspan="3">0.70</td><td rowspan="3">0.63</td><td rowspan="3">0.77</td></tr><tr><td>1</td><td>25</td><td>291</td><td>316</td></tr><tr><td>Total</td><td>152</td><td>329</td><td>481</td></tr></tbody></table>

<footnote><sup>a</sup> 0 was given when the item of the guidelines was not mentioned in the study evaluated.</footnote>

<footnote><sup>b</sup> 1 was given when the item of the guidelines was mentioned in the study evaluated.</footnote>
While many studies have leveraged ChatGPT for tasks such as extracting information and summarizing findings [18], our results indicate that it lacks the capacity to assess study quality thoroughly. Tasks requiring deeper cognitive processes or critical reasoning, such as those involved in the NOS assessment, remain beyond the current capabilities of AI models like ChatGPT [28]. LLMs follow user prompts to produce answers, but a thorough and complete assessment still requires human expertise [29].

ChatGPT can also serve as a third reader to resolve disagreements between human evaluators. Recent studies have shown that human-AI collaboration yields the best results, with AI-assisted ratings achieving up to 96% accuracy for Preferred Reporting Items for Systematic Reviews and Meta-Analyses and 95% for A MeaSurement Tool to Assess systematic Reviews, compared to 89% for human-only ratings [30].

## 4.2. Effectiveness with simplified tools such as the Jadad scale

ChatGPT proved more effective when paired with simpler assessment tools like the Jadad scale, which focuses on straightforward criteria such as randomization, double-blinding, and dropout reporting in RCTs. Our results showed substantial agreement between GPT and human CA with an ICC of 0.86. Similar findings were observed in studies using NLP frameworks to automatically identify Jadad scale components with high accuracy [31].

## 4.3. Importance of vigilance and human expertise

While our customized ChatGPT aids in CA by justifying its responses, it remains crucial that researchers maintain vigilance to ensure the rigor of evaluations [14].

Although our customized GPT models showed moderate agreement with human assessments, they were particularly sensitive to high-quality studies but struggled with weaker research. This limitation is due to the lack to specificity [18]. A two-stage process could enhance efficiency: ChatGPT would filter thousands of articles based on predefined critical criteria, identifying those worthy of further evaluation from the stages of research. This would allow researchers to focus more attention on assessing the methodology of articles that pass the initial screening, ultimately streamlining the process while maintaining accuracy. As such, ChatGPT can serve as a valuable tool for assistance, with the caution that researchers should always request supporting evidence for every answer it provides.

## 4.4. Hallucinations of ChatGPT

Previous studies have shown that human reviewers have an error rate of 7-11 % in abstract screening during systematic reviews, suggesting that automated tools could assist in standardizing this process [6]. However, ChatGPT has also demonstrated susceptibility to hallucinations, where it generates partially correct or incorrect information, particularly when dealing with open-ended tasks [19]. During our validation, despite the structured environment, our GPT models still produced a certain number of hallucinations, highlighting the need for careful oversight.

## 4.5. Ethics and transparency

As AI becomes more integrated into research, maintaining ethical standards and transparency is paramount. French authorities, via the National Commission for Information Technology and Liberties, emphasize privacy protection and compliance with the General Data Protection Regulation in AI development [32]. Additionally, the French Academy of Medicine calls for AI use to be subject to human supervision and clear regulatory frameworks to protect intellectual property and prioritize cybersecurity [14]. Additionally, responsible

application of standard reporting guidelines is essential for LLMs like ChatGPT to avoid inconsistencies in AI use across scientific communities [33].

## 4.6. Strengths and limitations

**Strengths:** Our use of customized GPT models for CA, integrating well-established tools like the NOS, Jadad scale, STROBE, and CONSORT guidelines, enhanced precision and reliability in our findings.

**Limitations:** ChatGPT-4 struggles with interpreting figures and tables and is limited to specific study designs such as cross-sectional, cohort, case-control studies, and RCTs. Future research should adapt validated scales to accommodate systematic reviews and meta-analyses for comprehensive assessments. Additionally, our evaluation was restricted to mental health studies, which may limit generalizability. The stability of ChatGPT's performance is a critical factor that requires further exploration in future studies. It is essential to evaluate potential response variability and emphasize the importance of repeated analyses to ensure the reliability of AI-assisted critical appraisals.

# 5. Perspectives

Since ChatGPT has demonstrated its capabilities across various domains and tasks, optimizing its performance to enhance both efficiency and relevance is necessary. This includes the following:

* Adapting ChatGPT to new fields that employ distinct methodologies;

* Since ChatGPT-4 requires a paid subscription, future research could explore the viability of free or open-source AI models to enhance accessibility, especially in resource-limited settings;

* Incorporating alternative scoring scales for observational studies;

* Developing customized models/platforms to address specific tasks in the research process, such as assessing the internal validity of studies, detecting bias, or synthesizing research results;

* Effective interdisciplinarity requires the invention of a new interdisciplinary profession: "AI designer" in healthcare [14].

# 6. Conclusion

Our study indicates that the use of automated tools like ChatGPT is feasible and offers strong evidence of its effectiveness for CA of articles. However, its implementation must be paired with the critical expertise of researchers, particularly for complex tasks such as assessing study quality. While technology can streamline certain processes, it cannot replace the detailed judgment and nuanced understanding that experts bring to the evaluation process.

# Declaration of competing interest

The authors of this submission declare no conflict of interest.

# Funding

This work was funded by the Association for the Development of Biostatistics, Medical Informatics and Epidemiology, Limoges, France.

# Supplementary materials

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.jeph.2025.202990.
## References

[1] Chen M, Decary M. Artificial intelligence in healthcare: an essential guide for health leaders. Healthc Manage Forum 2020;33:10–8. doi: 10.1177/0840470419873123.

[2] Mahbub M, Dams GM, Srinivasan S, Rizy C, Danciu I, Trafton J, et al. Leveraging large language models to extract information on substance use disorder severity from clinical notes: a zero-shot learning approach 2024. https://doi.org/10.48550/arXiv.2403.12297.

[3] Kusunose K, Kashima S, Sata M. Evaluation of the accuracy of ChatGPT in answering clinical questions on the Japanese Society of Hypertension Guidelines. Circ J 2023;87:1030–3. doi: 10.1253/circj.CJ-23-0308.

[4] Zúniga Salazar G, Zúniga D, Vindel CL, Yoong AM, Hincapie S, Zúniga AB, et al. Efficiency of AI chats to determine an emergency: a comparison between OpenAI's ChatGPT, Google Bard, and Microsoft Bing AI Chat. Cureus 2023. doi: 10.7759/cur-eus.45473.

[5] Gan RK, Uddin H, Gan AZ, Yew YY, González PA. ChatGPT's performance before and after teaching in mass casualty incident triage. Sci Rep 2023;13:20350. doi: 10.1038/s41598-023-46986-0.

[6] Tran V-T, Gartlehner G, Yaacoub S, Boutron I, Schwingshackl L, Stadelmaier J, et al. Sensitivity, specificity and avoidable workload of using a large language models for title and abstract screening in systematic reviews and meta-analyses 2023. https://doi.org/10.1101/2023.12.15.23300018.

[7] Woelfle T, Hirt J, Janiaud P, Kappos L, Ioannidis JPA, Hemkens LG. Benchmarking Human-Al collaboration for common evidence appraisal tools 2024: 2024. https://doi.org/10.1101/2024.04.21.24306137.

[8] Huang J, Tan M. The role of ChatGPT in scientific communication: writing better scientific review articles. Am J Cancer Res 2023;13:1148–54.

[9] Liao W, Liu Z, Dai H, Xu S, Wu Z, Zhang Y, et al. Differentiating ChatGPT-generated and Human-written medical texts: quantitative study. JMIR Med Educ 2023;9: e48904. doi: 10.2196/48904.

[10] Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 2023;9:e45312. doi: 10.2196/45312.

[11] Sahin MC, Sozer A, Kuzucu P, Turkmen T, Sahin MB, Sozer E, et al. Beyond human in neurosurgical exams: ChatGPT's success in the Turkish neurosurgical society proficiency board exams. Comput Biol Med 2024;169:107807. doi: 10.1016/j.compbiomed.2023.107807.

[12] Torres-Zegarra BC, Rios-Garcia W, Ñaña-Cordova AM, Arteaga-Cisneros KF, Chalco XCB, Ordonez MAB, et al. Performance of ChatGPT, Bard, Claude, and Bing on the peruvian National Licensing Medical Examination: a cross-sectional study. J Educ Eval Health Prof 2023;20:30. doi: 10.3352/jeehp.2023.20.30.

[13] OpenAI. GPT-4 OpenAI's most advanced system, producing safer and more useful responses 2023. https://openai.com/index/gpt-4/, (Accessed May 2, 2024).

[14] Nordlinger B, Kirchner C, De Fresnoye O. Rapport 24-03. Systèmes d'IA générative en santé : enjeux et perspectives. Bull Académie Natl Médecine 2024;208:536–47. doi: 10.1016/j.banm.2024.03.005.

[15] Rathbone AL, Prescott J. The use of mobile apps and SMS messaging as physical and mental health interventions: systematic review. J Med Internet Res 2017;19: e295. doi: 10.2196/jmir.7740.

[16] Word Health Organiztion. Mental disorders. World Health Organ 2022 https://www.who.int/news-room/fact-sheets/detail/mental-disorders (Accessed May 11, 2024).

[17] Effatpanah M, Nakhostin-Ansari A, Gorgani F, Khotbehssara ST, Seifi S, Nasoori H, et al. Burden and epidemiology of mental disorders in the Middle East and North Africa from 1990 to 2019: findings from the Global Burden of disease study. Balk Med J 2024;121–9. doi: 10.4274/balkanmedj.galenos.2024.2023-11-55.

[18] Tran V-T, Gartlehner G, Yaacoub S, Boutron I, Schwingshackl L, Stadelmaier J, et al. Sensitivity and specificity of using GPT-3.5 turbo models for title and abstract screening in systematic reviews and meta-analyses. Ann Intern Med 2024;M23-3389. doi: 10.7326/M23-3389.

[19] Nordlinger B, Kirchner C, de Fresnoye O. Generative ai systems in healthcare: challenges and prospects. Paris, France: Académie Nationale de Médecine; 2024.

[20] Wells G, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ott Hosp Res Inst 2021 https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp (Accessed May 6, 2024).

[21] Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17:1–12. doi: 10.1016/0197-2456(95)00134-4.

[22] Vandenbroucke JP, Poole C, Schlesselman JJ, Egger M. Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. PLoS Med 2007;4.

[23] Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869. doi: 10.1136/bmj.c869.

[24] Middle East and North Africa. World Bank Group Data 2023. https://data.world-bank.org/country/ZQ (Accessed May 11, 2024).

[25] Diagnostic and statistical manual of mental disorders: DSM-5TM, 5th ed. Arlington, VA, US: American Psychiatric Publishing, Inc.; 2013. https://doi.org/10.1176/appi books.9780890425596.

[26] Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016;15:155–63. doi: 10.1016/j.jcm.2016.02.012.

[27] Landis JR, Koch GG. The measurement of Observer agreement for categorical data. Biometrics 1977;33:159. doi: 10.2307/2529310.

[28] Lucas HC, Upperman JS, Robinson JR. A systematic review of large language models and their implications in medical education. Med Educ 2024 medu.15402. doi: 10.1111/medu.15402.

[29] Gartlehner G, Kahwati L, Hilscher R, Thomas I, Kugley S, Crotty K, et al. Data extraction for evidence synthesis using a large language model: a proof-of-concept study. Res Synth Methods 2024;15:576–89. doi: 10.1002/jrsm.1710.

[30] Woelfle T, Hirt J, Janiaud P, Kappos L, Ioannidis JPA, Hemkens LG (2024). Benchmarking Human-Al collaboration for common evidence appraisal tools. https://doi.org/10.1101/2024.04.21.24306137.

[31] Casy T, Grasseau A, Charras A, Rouvière B, Pers J-O, Foulquier N, et al. Assessing the robustness of clinical trials by estimating Jadad's score using artificial intelligence approaches. Comput Biol Med 2022;148:105851. doi: 10.1016/j.compbiomed.2022.105851.

[32] The Commission Nationale de l'Informatique et des Libertés (2024). Artificial intelligence: the CNIL is continuing its work to develop innovative and privacy-protective AI. https://www.cnil.fr/fr/developpement-des-systemes-dia-les-recommandations-de-la-cnil-pour-respecter-le-rgpd. (accessed May 13, 2024).

[33] Van Dis EAM, Bollen J, Zuidema W, Van Rooij R, Bockting CL. ChatGPT: five priorities for research. Nature 2023;614:224–6. doi: 10.1038/d41586-023-00288-7.