Aurametrix/dermatology_longitudinal huggingface dataset:
Dataset: Seven questions related to HS were developed with the help of HS patient associations. All questions were asked to both AI systems on the same day (December 20, 2023) in French for practicality. The ChatGPT and Bard responses were independently assessed by 18 hS experts. All experts independently evaluated all responses using a 5-point Likert scale (1: strongly agree; 2: agree; 3: neutral; 4: disagree; 5: strongly disagree).
Reference (2024): Ezanno AC, Fougerousse AC, Pruvost-Balland C, Maccari F, Fite C; ResoVerneuil. AI in Hidradenitis Suppurativa: Expert Evaluation of Patient-Facing Information. Clin Cosmet Investig Dermatol. 2024 Nov 2;17:2459-2464. doi: 10.2147/CCID.S478309. PMID: 39507766; PMCID: PMC11539865.

Evaluation of Bard (Google's Family of LLMs) — Hidradenitis suppurativa (HS), patient education, French
Condition/subject: Hidradenitis suppurativa (French)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 14.00% (sample size: 126)
Comments: Responses by ChatGPT to 6 of the 7 queries were deemed �appropriate� (86%) by the experts, whereas the response to one question (14%) was rated inappropriate (Q6 �What treatments are available?�). Responses by Bard to 1 of the 7 queries were deemed �appropriate� (14%) by the experts (Q5 �Which doctor do I need to see?�), whereas responses to 86% (n=6), were rated inappropriate.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Hidradenitis suppurativa (HS), patient education, French
Condition/subject: Hidradenitis suppurativa (French)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 86.00% (sample size: 126)
Comments: Responses by ChatGPT to 6 of the 7 queries were deemed �appropriate� (86%) by the experts, whereas the response to one question (14%) was rated inappropriate (Q6 �What treatments are available?�). Responses by Bard to 1 of the 7 queries were deemed �appropriate� (14%) by the experts (Q5 �Which doctor do I need to see?�), whereas responses to 86% (n=6), were rated inappropriate.
Dataset: 10 pharmacological repurposing questions considered relevant to the treatment of herpes zoster virus infection
Reference (2024): Daungsupawong H, Wiwanitkit V. Exploring Drug Repurposing Through Artificial Intelligence: A Novel Approach to Treating Herpes Zoster Infection. Mustansiriya Medical Journal. 2024 Jan 1;23(1):29-33.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Herpes zoster infection, commonly known as shingles; medical drug repurposing
Condition/subject: Herpes zoster (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 10)
Comments: All questions about reusing medications to treat herpes zoster can be answered by ChatGPT
Dataset: 3 macroscopic images (900 � 1100 pixels; 96-dpi resolution) of melanomas (malignant) and melanocytic nevi (benign) obtained from the publicly available and validated MClass-D data set [6], Dermnet NZ, and dermatology textbooks 
Reference (2024): Cirone K, Akrout M, Abid L, Oakley A. Assessing the Utility of Multimodal Large Language Models (GPT-4 Vision and Large Language and Vision Assistant) in Identifying Melanoma Across Different Skin Tones. JMIR Dermatol. 2024 Mar 13;7:e55508. doi: 10.2196/55508. PMID: 38477960; PMCID: PMC10973960.

Evaluation of LLAVA (Large Language and Vision Assistant) (LLAVA) — Melanoma detection from macroscopic images
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 45.00% (sample size: 3)

Evaluation of ChatGPT-4 (GPT series) — Melanoma detection from macroscopic images
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 85.00% (sample size: 3)
Dataset: 184 multiple-choice questions from the October 2021 to September 2023 American Academy of Dermatology quizzes, which grant dermatology continuing medical education credits
Reference (2024): Chen, M.L.; Cai, Z. Ran; Kim, J.; Novoa, R.; Barnes, L.A.; Beam, A.; Linos, E.  140 Performance and risk of harm of a large language model on dermatology continuing medical education questions 2024. Journal of Investigative Dermatology, Volume 144, Issue 8, S25

Evaluation of GPT-4 (GPT series) — Multilogical reasoning
Condition/subject: Multilogical reasoning (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 71.40% (sample size: 21)
Comments: Dermatology Continuing Medical Education

Evaluation of GPT-4 (GPT series) — Continuing Education, multiple choice questions
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.40% (sample size: 184)
Comments: Dermatology Continuing Medical Education

Evaluation of GPT-4 (GPT series) — Simple reasoning
Condition/subject: Simple reasoning (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 81.20% (sample size: 85)
Comments: Dermatology Continuing Medical Education

Evaluation of GPT-4 (GPT series) — Knowledge Recall
Condition/subject: Knowledge Recall (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 82.10% (sample size: 78)
Comments: Dermatology Continuing Medical Education

Evaluation of GPT-4 (GPT series) — No omission of relevant content
Condition/subject: No omission of relevant content (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 85.00% (sample size: 30)
Comments: Dermatology Continuing Medical Education

Evaluation of GPT-4 (GPT series) — Correct Reasoning 
Condition/subject: Logical reasoning (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 91.70% (sample size: 30)
Comments: Dermatology Continuing Medical Education

Evaluation of GPT-4 (GPT series) — No Harm Evidence
Condition/subject: No Harm Evidence (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 93.30% (sample size: 30)
Comments: Dermatology Continuing Medical Education

Evaluation of GPT-4 (GPT series) — Correct Comprehension
Condition/subject: Comprehension (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 95.00% (sample size: 30)
Comments: Dermatology Continuing Medical Education

Evaluation of GPT-4 (GPT series) — No Bias Evidence
Condition/subject: No Bias Evidence (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 30)
Comments: Dermatology Continuing Medical Education
Dataset: 184 multiple-choice questions from the October 2021 to September 2023 American Academy of Dermatology quizzes, which grant dermatology continuing medical education credits
Reference (2024): Chen, M.L.; Cai, Z. Ran; Kim, J.; Novoa, R.; Barnes, L.A.; Beam, A.; Linos, E.  140 Performance and risk of harm of a large language model on dermatology continuing medical education questions 2024. Journal of Investigative Dermatology, Volume 144, Issue 8, S25

Evaluation of GPT-4 (GPT series) — Hair Shedding Medication
Condition/subject: Hair Shedding (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: Fail (sample size: 1)
Comments: Failure to recommend decreasing the oral isotretinoin dose after new-onset diffuse hair shedding four months after initiating isotretinoin (Dermatology Continuing Medical Education )

Evaluation of GPT-4 (GPT series) — Ecchymosis due to Child Abuse and Neglect
Condition/subject: Ecchymosis (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: Fail (sample size: 1)
Comments: Failure to recommend follow-up for a child where possible abuse-related ecchymosis could not be ruled out (Dermatology Continuing Medical Education )
Dataset: 18 topics across 3 categories (dermatitis, alopecia, dyspigmentation) (total 54 handouts) 
Reference (2024): Chang CT, Ticknor IL, Spinelli JA, Bhatia BK, Marwaha S, Mirmirani P, Seidler AM, Man JR, McCleskey PE. Comparison of large language models in generating patient handouts for the dermatology clinic: A blinded study. JAAD Int. 2024 Feb 21;15:152-154. doi: 10.1016/j.jdin.2024.02.010. PMID: 38571697; PMCID: PMC10988028.

Evaluation of Bard (Google's Family of LLMs) — Dermatitis (atopic dermatitis, seborrheic dermatitis, psoriasis, intertrigo, venous stasis dermatitis, and asteatotic dermatitis)
Condition/subject: Atopic dermatitis (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 78.67% (sample size: 36)

Evaluation of Bard (Google's Family of LLMs) — Dermatology handouts: Understandability at a sixth grade reading level
Condition/subject: Understandability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 81.50% (sample size: 108)

Evaluation of Bard (Google's Family of LLMs) — Dyspigmentation (melasma, postinflammatory hyperpigmentation, vitiligo, progressive macular hypomelanosis, lichen planus pigmentosus, and exogenous ochronosis)
Condition/subject: Dyspigmentation (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 82.83% (sample size: 36)

Evaluation of Bard (Google's Family of LLMs) — Alopecia (telogen effluvium, alopecia areata, traction alopecia, central centrifugal cicatricial alopecia, lichen planopilaris, and androgenic alopecia)
Condition/subject: Central centrifugal cicatricial alopecia (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 83.00% (sample size: 36)

Evaluation of GPT 3.5, ChatGPT (GPT series) — Alopecia (telogen effluvium, alopecia areata, traction alopecia, central centrifugal cicatricial alopecia, lichen planopilaris, and androgenic alopecia)
Condition/subject: Central centrifugal cicatricial alopecia (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 86.50% (sample size: 36)

Evaluation of BingAI (GPT-4/Prometheus) (GPT series) — Alopecia (telogen effluvium, alopecia areata, traction alopecia, central centrifugal cicatricial alopecia, lichen planopilaris, and androgenic alopecia)
Condition/subject: Central centrifugal cicatricial alopecia (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 87.08% (sample size: 36)

Evaluation of GPT 3.5, ChatGPT (GPT series) — Dermatitis (atopic dermatitis, seborrheic dermatitis, psoriasis, intertrigo, venous stasis dermatitis, and asteatotic dermatitis)
Condition/subject: Atopic dermatitis (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 88.00% (sample size: 36)
Dataset: 18 topics across 3 categories (dermatitis, alopecia, dyspigmentation) (total 54 handouts) 
Reference (2024): Chang CT, Ticknor IL, Spinelli JA, Bhatia BK, Marwaha S, Mirmirani P, Seidler AM, Man JR, McCleskey PE. Comparison of large language models in generating patient handouts for the dermatology clinic: A blinded study. JAAD Int. 2024 Feb 21;15:152-154. doi: 10.1016/j.jdin.2024.02.010. PMID: 38571697; PMCID: PMC10988028.

Evaluation of BingAI (GPT-4/Prometheus) (GPT series) — Dermatology handouts: Understandability at a sixth grade reading level
Condition/subject: Understandability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 88.31% (sample size: 108)

Evaluation of BingAI (GPT-4/Prometheus) (GPT series) — Dermatitis (atopic dermatitis, seborrheic dermatitis, psoriasis, intertrigo, venous stasis dermatitis, and asteatotic dermatitis)
Condition/subject: Atopic dermatitis (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 88.83% (sample size: 36)

Evaluation of BingAI (GPT-4/Prometheus) (GPT series) — Dyspigmentation (melasma, postinflammatory hyperpigmentation, vitiligo, progressive macular hypomelanosis, lichen planus pigmentosus, and exogenous ochronosis)
Condition/subject: Dyspigmentation (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 89.00% (sample size: 36)

Evaluation of GPT 3.5 (GPT series) — Dermatology handouts: Understandability at a sixth grade reading level
Condition/subject: Understandability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 89.53% (sample size: 108)

Evaluation of GPT 3.5, ChatGPT (GPT series) — Dyspigmentation (melasma, postinflammatory hyperpigmentation, vitiligo, progressive macular hypomelanosis, lichen planus pigmentosus, and exogenous ochronosis)
Condition/subject: Dyspigmentation (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 94.08% (sample size: 36)
Dataset: 8 PILs generated by ChatGPT where a BAD-produced PIL was not available. AI-generated PILs were reviewed by eight dermatologists and five nonmedical reviewers using a five-point Likert scale to assess document structure, accuracy, relevance and utility. 
Reference (2024): Callum Verran, Michael R Ardern-Jones, Ella Seccombe, BT23 Artificial intelligence generation of patient information leaflets can have a useful role in the dermatology clinic, British Journal of Dermatology, Volume 191, Issue Supplement_1, July 2024, Page i199, https://doi.org/10.1093/bjd/ljae090.420

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Dermatological Medications - Patient Information Leaflets (PILs)
Condition/subject: Medications (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 12.50% (sample size: 8)
Comments: While 64.6% of condition-related PILs were considered acceptable for patient distribution, this was only 12.5% for medication-related PILs. Dermatologist responses varied based on PIL subject and were positive for structure, relevance and accuracy in 98%, 88% and 65% of responses, respectively, for condition-related PILs, whereas positive responses were seen in only 81%, 38% and 25% for medication-related PILs.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Dermatological Conditions, Common - Patient Information Leaflets (PILs)
Condition/subject: Common conditions (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 64.60% (sample size: 8)
Comments: While 64.6% of condition-related PILs were considered acceptable for patient distribution, this was only 12.5% for medication-related PILs. Dermatologist responses varied based on PIL subject and were positive for structure, relevance and accuracy in 98%, 88% and 65% of responses, respectively, for condition-related PILs, whereas positive responses were seen in only 81%, 38% and 25% for medication-related PILs.
Dataset: 184 multiple-choice questions from the October 2021 to September 2023 quizzes of the American Academy of Dermatology, which grants board-certified dermatologists continuing medical education credits for certification maintenance. The date range for eligible questions was chosen on the basis of the ChatGPT-4 training data cutoff of September 2021 during the time of question input.
Reference (2024): Cai, Zhuo Ran et al. Assessment of Correctness, Content Omission, and Risk of Harm in Large Language Model Responses to Dermatology Continuing Medical Education Questions 2024  Journal of Investigative Dermatology, Volume 144, Issue 8, 1877 - 1879

Evaluation of GPT-4 (GPT series) — Infantile Hemangioma (benign vascular tumor)
Condition/subject: Infantile Hemangioma (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 50.00% (sample size: 4)

Evaluation of GPT-4 (GPT series) — Nutritional Dermatoses
Condition/subject: Nutritional dermatoses (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 63.60% (sample size: 11)

Evaluation of GPT-4 (GPT series) — Physical Injury or Ulcers
Condition/subject: injury (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 70.00% (sample size: 10)

Evaluation of GPT-4 (GPT series) — Multilogical reasoning
Condition/subject: Multilogical reasoning (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 71.40% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Pathophysiology
Condition/subject: Pathophysiology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 71.40% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Nonmelanoma skin cancer
Condition/subject: Nonmelanoma skin cancer (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 16)

Evaluation of GPT-4 (GPT series) — Other inflammatory
Condition/subject: Inflammation (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 77.10% (sample size: 48)

Evaluation of GPT-4 (GPT series) — Risk factors
Condition/subject: Risk factors (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 79.50% (sample size: 39)

Evaluation of GPT-4 (GPT series) — Image-based questions
Condition/subject: Images, Diagnostics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.00% (sample size: 10)

Evaluation of GPT-4 (GPT series) — Continuing Education, multiple choice questions
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.40% (sample size: 184)
Comments: Dermatology Continuing Medical Education
Dataset: 184 multiple-choice questions from the October 2021 to September 2023 quizzes of the American Academy of Dermatology, which grants board-certified dermatologists continuing medical education credits for certification maintenance. The date range for eligible questions was chosen on the basis of the ChatGPT-4 training data cutoff of September 2021 during the time of question input.
Reference (2024): Cai, Zhuo Ran et al. Assessment of Correctness, Content Omission, and Risk of Harm in Large Language Model Responses to Dermatology Continuing Medical Education Questions 2024  Journal of Investigative Dermatology, Volume 144, Issue 8, 1877 - 1879

Evaluation of GPT-4 (GPT series) — Simple reasoning
Condition/subject: Simple reasoning (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 81.20% (sample size: 85)

Evaluation of GPT-4 (GPT series) — Diagnosis, CME
Condition/subject: Diagnostics, Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 81.80% (sample size: 55)

Evaluation of GPT-4 (GPT series) — Knowledge Recall
Condition/subject: Knowledge Recall (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 82.10% (sample size: 78)

Evaluation of GPT-4 (GPT series) — Treatment
Condition/subject: Therapy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 82.60% (sample size: 69)

Evaluation of GPT-4 (GPT series) — Autoimmune/CTD
Condition/subject: Autoimmune Disorders, Connective Tissue Disease (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 84.20% (sample size: 19)

Evaluation of GPT-4 (GPT series) — Pigmented (benign)
Condition/subject: Pigmentary disorders, Benign Lesions (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 85.70% (sample size: 7)

Evaluation of GPT-4 (GPT series) — Pigmented (malignant)
Condition/subject: Pigmentary disorders, Cancers (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 85.70% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Alopecia
Condition/subject: Alopecia (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 86.20% (sample size: 29)

Evaluation of GPT-4 (GPT series) — Cutaneous Lymphoma
Condition/subject: Cutaneous lymphoma (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 87.50% (sample size: 8)

Evaluation of GPT-4 (GPT series) — Skin Infections
Condition/subject: Skin infections (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 11)
Dataset: 492 dermatology-related questions from Amboss (online education company that provides resources such as content and questions for board exams and continuing medical education curriculums accredited through the Accreditation Council for Continuing Medical Education. It provides a question bank that prepares medical students for USMLE and National Board of Medical Examiners (NBME) examinations. It has an internal metric for measuring the difficulty of a question based on the number of students who answer it correctly.)
Reference (2023): Behrmann J, Hong EM, Meledathu S, Leiter A, Povelaitis M, Mitre M. Chat generative pre-trained transformer�s performance on dermatology-specific questions and its implications in medical education. Journal of Medical Artificial Intelligence. 2023 Sep 30;6.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 2.5, Clinical knowledge, level 5, most difficult questions
Condition/subject: Clinical dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 13.00% (sample size: 8)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 3.4 (Examinations, Level 4 questions)
Condition/subject: Challenging dermatology diagnostics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 15.00% (sample size: 20)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 1.5 Basic Science in Dermatology, level 5, most difficult questions
Condition/subject: Basic dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 20.00% (sample size: 5)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 2.3, Clinical knowledge, level 3
Condition/subject: Clinical dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 25.00% (sample size: 48)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 3.5 (Examinations, most difficult Level 5 questions)
Condition/subject: Expert-level dermatology diagnostics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 29.00% (sample size: 7)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 1 with images, USMLE
Condition/subject: Basic dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 30.00% (sample size: 53)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 1.4 Basic Science in Dermatology, level 4
Condition/subject: Basic dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 32.00% (sample size: 19)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 2.4, Clinical knowledge, level 4
Condition/subject: Clinical dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 32.00% (sample size: 28)
Dataset: 492 dermatology-related questions from Amboss (online education company that provides resources such as content and questions for board exams and continuing medical education curriculums accredited through the Accreditation Council for Continuing Medical Education. It provides a question bank that prepares medical students for USMLE and National Board of Medical Examiners (NBME) examinations. It has an internal metric for measuring the difficulty of a question based on the number of students who answer it correctly.)
Reference (2023): Behrmann J, Hong EM, Meledathu S, Leiter A, Povelaitis M, Mitre M. Chat generative pre-trained transformer�s performance on dermatology-specific questions and its implications in medical education. Journal of Medical Artificial Intelligence. 2023 Sep 30;6.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 3.3 (Examinations, Level 3 questions)
Condition/subject: Advanced Dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 33.00% (sample size: 42)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 2 Clinical Knowledge in Dermatology , with images
Condition/subject: Clinical dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 36.00% (sample size: 50)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Dermatology questions with images, USMLE
Condition/subject: Certification, Images (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 36.00% (sample size: 154)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 1.3 Basic Science in Dermatology, level 3
Condition/subject: Basic dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 37.00% (sample size: 59)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 2 Clinical Knowledge in Dermatology (more applied clinical scenarios, requiring the examinee to demonstrate diagnostic skills, clinical reasoning, and knowledge of treatment protocols in dermatology): ability to apply medical knowledge, skills, and understanding of clinical science essential for the provision of patient care under supervision with emphasis on health promotion and disease prevention
Condition/subject: Clinical dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 38.00% (sample size: 171)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 1.2 Basic Science in Dermatology, level 2
Condition/subject: Basic dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 39.00% (sample size: 56)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 2 Clinical Knowledge in Dermatology, without images
Condition/subject: Clinical dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 39.00% (sample size: 121)
Dataset: 492 dermatology-related questions from Amboss (online education company that provides resources such as content and questions for board exams and continuing medical education curriculums accredited through the Accreditation Council for Continuing Medical Education. It provides a question bank that prepares medical students for USMLE and National Board of Medical Examiners (NBME) examinations. It has an internal metric for measuring the difficulty of a question based on the number of students who answer it correctly.)
Reference (2023): Behrmann J, Hong EM, Meledathu S, Leiter A, Povelaitis M, Mitre M. Chat generative pre-trained transformer�s performance on dermatology-specific questions and its implications in medical education. Journal of Medical Artificial Intelligence. 2023 Sep 30;6.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 2.2, Clinical knowledge, level 2
Condition/subject: Clinical dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 40.00% (sample size: 42)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 1 Basic Science in Dermatology, USMLE: foundational science aspects of dermatology, such as the pathology of skin diseases, pharmacology of dermatological treatments, and basic physiological processes related to skin function. Assessment of understanding and ability to apply important concepts of the sciences basic to the practice of medicine, with special emphasis on principles and mechanisms underlying health, disease, and modes of therapy.
Condition/subject: skin disease (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 41.00% (sample size: 160)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Dermatology part of USMLE exam
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 41.00% (sample size: 492)
Comments: ChatGPT answered fewer correct dermatology-specific questions when compared to its overall performance on the USMLE (41% and 60%, respectively)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 3 with images (Dermatological Examinations)
Condition/subject: Images (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 43.00% (sample size: 51)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Dermatology questions without images, USMLE
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 44.00% (sample size: 338)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 1 without images, USMLE
Condition/subject: Basic dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 46.00% (sample size: 107)
Dataset: 492 dermatology-related questions from Amboss (online education company that provides resources such as content and questions for board exams and continuing medical education curriculums accredited through the Accreditation Council for Continuing Medical Education. It provides a question bank that prepares medical students for USMLE and National Board of Medical Examiners (NBME) examinations. It has an internal metric for measuring the difficulty of a question based on the number of students who answer it correctly.)
Reference (2023): Behrmann J, Hong EM, Meledathu S, Leiter A, Povelaitis M, Mitre M. Chat generative pre-trained transformer�s performance on dermatology-specific questions and its implications in medical education. Journal of Medical Artificial Intelligence. 2023 Sep 30;6.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 3 Examinations,  integration of dermatological knowledge into broader medical practice. Step 3 assesses ability to apply medical knowledge and understanding of biomedical and clinical science essential for the unsupervised practice of medicine
Condition/subject: Integration (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 46.00% (sample size: 161)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 3 without images (Dermatological Examinations)
Condition/subject: Diagnostics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 47.00% (sample size: 110)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 3.2 (Examinations, Level 2 questions)
Condition/subject: Intermediate Dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 55.00% (sample size: 44)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 2.1, Clinical knowledge, easiest questions: level 1
Condition/subject: Clinical dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 58.00% (sample size: 45)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 3.1 (Examinations, easiest Level 1 questions)
Condition/subject: Basic dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 65.00% (sample size: 48)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Step 1.1 Basic Science in Dermatology, level 1 questions (easiest)
Condition/subject: Basic dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 67.00% (sample size: 21)
Dataset: generation of American Board of Dermatology Applied Exam (ABD-AE)-style questions from continuing medical education articles from the Journal of the American Board of Dermatology
Reference (2023): Ayub I, Hamann D, Hamann CR, Davis MJ. Exploring the Potential and Limitations of Chat Generative Pre-trained Transformer (ChatGPT) in Generating Board-Style Dermatology Questions: A Qualitative Analysis. Cureus. 2023 Aug 18;15(8):e43717. doi: 10.7759/cureus.43717. PMID: 37638266; PMCID: PMC10450251.

Evaluation of GPT-3.5, ChatPDF (GPT series) — ABD-AE-style question generation (American Board of Dermatology Applied Exam)
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 40.00% (sample size: 40)
Comments: The evaluation encompassed three essential dimensions: accuracy, complexity, and clarity. Of the 40 questions, only 16 (40%) questions created using ChatGPT 3.5 were accurate and at an appropriate level of complexity for a trainee studying for ABD-AE

Evaluation of GPT-3.5, ChatPDF (GPT series) — Disorders of hyperpigmentation. Part I. Pathogenesis and clinical features of common pigmentary disorders
Condition/subject: Pigmentary disorders (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: FAIL (sample size: 1)
Comments: The evaluation encompassed three essential dimensions: accuracy, complexity, and clarity. Of the 40 questions, only 16 (40%) questions created using ChatGPT 3.5 were accurate and at an appropriate level of complexity for a trainee studying for ABD-AE
Dataset: 195 randomly drawn exchanges from reddit's /AskDocs (a subreddit with approximately 474?000 members where users can post medical questions and verified health care professional volunteers submit answers), ie, a unique member�s question and unique physician�s answer, during October 2022. The original question, including the title and text, was retained for analysis, and the physician response was retained as a benchmark response. Only physician responses were studied because it was expected that physicians� responses are generally superior to those of other health care professionals or laypersons
Reference (2023):

Evaluation of GPT-3.5, ChatGPT (GPT series, GPT-3.5, ChatGPT) — Hard Lump Under the Skin of the Penis
Condition/subject: Pityriasis versicolor (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 100.00% (sample size: 1)
Comments: 100% preferred the chatbot: 5.00 mean quality score (chatbot) 3.33 mean quality score (physician); 3.33 mean empathy score (chatbot) 1.67 mean empathy score (physician)
Dataset: Fifty images randomly selected from the Fitzpatrick 17k dataset, a publicly available online collection of clinical images labelled with the appropriate diagnoses and skin types based on the Fitzpatrick scoring system. Half of the images selected represented darker skin tones, Fitzpatrick IV-VI, and the other half represented lighter skin tones, Fitzpatrick I-II. 
Reference (2024): Akuffo-Addo E, Samman L, Munawar L, Akbik M, Kokikian N, Wescott R, Wu JJ. Assessing GPT-4's diagnostic accuracy with darker skin tones: underperformance and implications. Clin Exp Dermatol. 2024 Sep 18;49(10):1244-1245. doi: 10.1093/ced/llae158. PMID: 38696699.

Evaluation of GPT-4V (GPT series) — Image-based diagnostics, Dark skin
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 12.00% (sample size: 25)
Comments: GPT-4 exhibited better performance in providing the correct diagnosis for 77 lighter skin tones (44%, n=11/25) compared to darker skin tones (12%, n=3/25),

Evaluation of GPT-4V (GPT series) — Image-based Diagnostics. Common skin leisons, Top Diagnosis
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 28.00% (sample size: 50)
Comments: Overall, GPT-4 correctly diagnosed the condition in 28% of the images (n=14/50), while the correct diagnosis was included in its list of top differentials for 48% of the images (n=24/50)

Evaluation of GPT-4V (GPT series) — Image-based diagnostics, Light skin
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 44.00% (sample size: 25)
Comments: GPT-4 exhibited better performance in providing the correct diagnosis for 77 lighter skin tones (44%, n=11/25) compared to darker skin tones (12%, n=3/25),

Evaluation of GPT-4V (GPT series) — Image-based Diagnostics. Common skin leisons , Differential Diagnosis
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 48.00% (sample size: 50)
Comments: Overall, GPT-4 correctly diagnosed the condition in 28% of the images (n=14/50), while the correct diagnosis was included in its list of top differentials for 48% of the images (n=24/50)
Dataset: Atopic dermatitis case report written by ChatGPT-4 turbo (AI) compared with the same case report written by two human experts (both with MD and PhD degrees), and written by humans assisted with AI (COM). The survey assessed authorship, ranked their preference, and graded 13 quality criteria for each text. 
Reference (2025): Mara Giavina Bianchi, Andrew D�adario, Pedro Giavina Bianchi, Birajara Soares Machado, Rosana Agondi, Stephanie K.A. Almeida, Wandilson Xavier Alves Junior, Larissa M. Armelin, Marcelo Vivolo Aun, Nat�lia Bordignon, Karla Boufleur, Felipe B. Brunheroto, Elisabeth A. Callegaro, Paula Lazaretti M. Castro, Herberto Jose Chong-Neto, Mariana D. Dall�Osto, Julia Abou Dias, Viviane Heintze Ferreira, Andr� Luiz Oliveira Feodrippe, Livia G. Fonseca, Clycia M. Garcia, Bruna H. Giavina-Bianchi, Ekaterini Goudouris, Danilo Gois Gon�alves, Debora D. Hernandes, Malek Imad, Larissa S. Izabel, Lucas Cau� Jacintho, Carolina Khouri-Panzarin, Fabio Kuschnir, Maria Beatriz P�dua Lima, Amanda I. Lopes, Larissa Nathalia Mac�do N�brega Lopes, Alice Rocha de Magalh�es, Eli Mansour, Ana Karolina B.B. Marinho, Vivian S. Martimiano, Pedro H. Milori, Antonio Marcondes Mutarelli, Guilherme Paes Gon�alves Nogueira, Beatriz K.T. Oguido, Bruna S. Alarcon de Oliveira, Emerson Costa de Oliveira, Georgia A. Padulla, Let�cia D�Ordaz Lhano Santos, Micaelly Samara Meneses Santos, Emanuel Sarinho, Marcela Schoen, Brian Lucas A. Sousa, Eduardo Magalh�es de Souza-Lima, Beatriz C. Todt, Najla Braz da Silva Vaz, Three versions of an atopic dermatitis case report written by humans, artifi

Evaluation of GPT-4-Turbo (GPT series) — Title for atopic dermatitis case report, total score
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes. — Professional Education
Validity/accuracy: 84.00% (sample size: 48)
Comments: AI received the highest grade for the title (4.3 on a 5-point star rating scale (1 = terrible, 5 = excellent)), slightly outperforming humans (4.2) and humans collaborating with AI (4.2) .  The average grade was calculated based on the 48 individual scores provided by the participants.48 individuals consisted of STU (Students): 17 participants; RES (Residents): 12 participants; EXP (Experts): 19 participants. Participants favored AI-assisted versions (AI and COM) over HUM (P < .001), with COM receiving the highest quality scores. COM and AI achieved 83.8% and 84.3% reduction in writing time, respectively, compared to HUM, while showing 13.9% (P < .001) and 11.1% improvement in quality (P < .001), respectively. However, experts assigned the lowest score for the references of the AI manuscript.

Evaluation of GPT-4-Turbo (GPT series) — Title for atopic dermatitis case report, score from students
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes. — Professional Education
Validity/accuracy: 84.00% (sample size: 17)
Comments: 17 students
Dataset: Atopic dermatitis case report written by ChatGPT-4 turbo (AI) compared with the same case report written by two human experts (both with MD and PhD degrees), and written by humans assisted with AI (COM). The survey assessed authorship, ranked their preference, and graded 13 quality criteria for each text. 
Reference (2025): Mara Giavina Bianchi, Andrew D�adario, Pedro Giavina Bianchi, Birajara Soares Machado, Rosana Agondi, Stephanie K.A. Almeida, Wandilson Xavier Alves Junior, Larissa M. Armelin, Marcelo Vivolo Aun, Nat�lia Bordignon, Karla Boufleur, Felipe B. Brunheroto, Elisabeth A. Callegaro, Paula Lazaretti M. Castro, Herberto Jose Chong-Neto, Mariana D. Dall�Osto, Julia Abou Dias, Viviane Heintze Ferreira, Andr� Luiz Oliveira Feodrippe, Livia G. Fonseca, Clycia M. Garcia, Bruna H. Giavina-Bianchi, Ekaterini Goudouris, Danilo Gois Gon�alves, Debora D. Hernandes, Malek Imad, Larissa S. Izabel, Lucas Cau� Jacintho, Carolina Khouri-Panzarin, Fabio Kuschnir, Maria Beatriz P�dua Lima, Amanda I. Lopes, Larissa Nathalia Mac�do N�brega Lopes, Alice Rocha de Magalh�es, Eli Mansour, Ana Karolina B.B. Marinho, Vivian S. Martimiano, Pedro H. Milori, Antonio Marcondes Mutarelli, Guilherme Paes Gon�alves Nogueira, Beatriz K.T. Oguido, Bruna S. Alarcon de Oliveira, Emerson Costa de Oliveira, Georgia A. Padulla, Let�cia D�Ordaz Lhano Santos, Micaelly Samara Meneses Santos, Emanuel Sarinho, Marcela Schoen, Brian Lucas A. Sousa, Eduardo Magalh�es de Souza-Lima, Beatriz C. Todt, Najla Braz da Silva Vaz, Three versions of an atopic dermatitis case report written by humans, artifi

Evaluation of GPT-4-Turbo (GPT series) — Title for atopic dermatitis case report, score from residents
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes. — Professional Education
Validity/accuracy: 90.00% (sample size: 12)
Comments: 12 residents

Evaluation of GPT-4-Turbo (GPT series) — Title for atopic dermatitis case report, score from expert dermatologists
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes. — Professional Education
Validity/accuracy: 78.00% (sample size: 19)
Comments: 19 experts

Evaluation of GPT-4-Turbo (GPT series) — Short Title for atopic dermatitis case report, Total
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes. — Professional Education
Validity/accuracy: 84.00% (sample size: 48)
Comments: AI outperformed humans in short title grading. The average grade was calculated based on the 48 individual scores provided by the participants.48 individuals consisted of STU (Students): 17 participants; RES (Residents): 12 participants; EXP (Experts): 19 participants. Participants favored AI-assisted versions (AI and COM) over HUM (P < .001), with COM receiving the highest quality scores. COM and AI achieved 83.8% and 84.3% reduction in writing time, respectively, compared to HUM, while showing 13.9% (P < .001) and 11.1% improvement in quality (P < .001), respectively. However, experts assigned the lowest score for the references of the AI manuscript.
Dataset: Atopic dermatitis case report written by ChatGPT-4 turbo (AI) compared with the same case report written by two human experts (both with MD and PhD degrees), and written by humans assisted with AI (COM). The survey assessed authorship, ranked their preference, and graded 13 quality criteria for each text. 
Reference (2025): Mara Giavina Bianchi, Andrew D�adario, Pedro Giavina Bianchi, Birajara Soares Machado, Rosana Agondi, Stephanie K.A. Almeida, Wandilson Xavier Alves Junior, Larissa M. Armelin, Marcelo Vivolo Aun, Nat�lia Bordignon, Karla Boufleur, Felipe B. Brunheroto, Elisabeth A. Callegaro, Paula Lazaretti M. Castro, Herberto Jose Chong-Neto, Mariana D. Dall�Osto, Julia Abou Dias, Viviane Heintze Ferreira, Andr� Luiz Oliveira Feodrippe, Livia G. Fonseca, Clycia M. Garcia, Bruna H. Giavina-Bianchi, Ekaterini Goudouris, Danilo Gois Gon�alves, Debora D. Hernandes, Malek Imad, Larissa S. Izabel, Lucas Cau� Jacintho, Carolina Khouri-Panzarin, Fabio Kuschnir, Maria Beatriz P�dua Lima, Amanda I. Lopes, Larissa Nathalia Mac�do N�brega Lopes, Alice Rocha de Magalh�es, Eli Mansour, Ana Karolina B.B. Marinho, Vivian S. Martimiano, Pedro H. Milori, Antonio Marcondes Mutarelli, Guilherme Paes Gon�alves Nogueira, Beatriz K.T. Oguido, Bruna S. Alarcon de Oliveira, Emerson Costa de Oliveira, Georgia A. Padulla, Let�cia D�Ordaz Lhano Santos, Micaelly Samara Meneses Santos, Emanuel Sarinho, Marcela Schoen, Brian Lucas A. Sousa, Eduardo Magalh�es de Souza-Lima, Beatriz C. Todt, Najla Braz da Silva Vaz, Three versions of an atopic dermatitis case report written by humans, artifi

Evaluation of GPT-4-Turbo (GPT series) — Short Title for atopic dermatitis case report, STU
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes. — Professional Education
Validity/accuracy: 82.00% (sample size: 17)
Comments: 17 students

Evaluation of GPT-4-Turbo (GPT series) — Short Title for atopic dermatitis case report, RES
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes. — Professional Education
Validity/accuracy: 86.00% (sample size: 12)
Comments: 12 residents

Evaluation of GPT-4-Turbo (GPT series) — Short Title for atopic dermatitis case report, EXP
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes. — Professional Education
Validity/accuracy: 86.00% (sample size: 19)
Comments: 19 experts