Aurametrix/dermatology_longitudinal huggingface dataset:
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of ChatGPT-4 (GPT series) — Syphilis/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Syphilis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 5)
Comments: Accuracy Rate: 100%; Differential Diagnosis: 27% CC, 55% MC, 18% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — Tinea Corporis/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Tinea corporis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 5)
Comments: Accuracy Rate: 60%  3/5; Differential Diagnosis: 0% CC, 45% MC, 55% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — Melanoma/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 5)
Comments: Accuracy Rate: 100%; Differential Diagnosis: 27% CC, 55% MC, 18% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — Viral exanthem - Workup Completely or Mostly Correct
Condition/subject: Viral exanthem (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — PsO, psoriasis - Workup Completely or Mostly Correct
Condition/subject: Psoriasis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — BCC, basal cell carcinoma - Workup Completely or Mostly Correct
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of ChatGPT-4 (GPT series) — Syphilis - Workup Completely or Mostly Correct
Condition/subject: Syphilis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — CTCL, cutaneous T-cell lymphoma - Workup Completely or Mostly Correct
Condition/subject: cutaneous T-cell lymphoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — Tinea Corporis - Workup Completely or Mostly Correct
Condition/subject: Tinea corporis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — Melanoma    - Workup Completely or Mostly Correct
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — Viral exanthem - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Viral exanthem (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)

Evaluation of ChatGPT-4 (GPT series) — Atopic Dermatitis - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)

Evaluation of ChatGPT-4 (GPT series) — PsO, psoriasis - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Psoriasis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of ChatGPT-4 (GPT series) — BCC, basal cell carcinoma - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)

Evaluation of ChatGPT-4 (GPT series) — Syphilis - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Syphilis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)

Evaluation of ChatGPT-4 (GPT series) — Tinea Corporis - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Tinea corporis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)

Evaluation of ChatGPT-4 (GPT series) — Melanoma    - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 11)

Evaluation of GPT-4, ChatGPT (GPT series) — PsO, psoriasis diagnosis
Condition/subject: Psoriasis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: Completely or Mostly Correct Workup

Evaluation of GPT-4, ChatGPT (GPT series) — Melanoma diagnosis/ChatGPT4 Dx matching top dermatologist Dx
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: Completely or Mostly Correct Workup

Evaluation of GPT-4, ChatGPT (GPT series) — BCC, basal cell carcinoma  diagnosis/ChatGPT4 Dx matching top dermatologist Dx
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: Completely or Mostly Correct Workup

Evaluation of GPT-4, ChatGPT (GPT series) — Viral exanthem  diagnosis/ChatGPT4 Dx matching top dermatologist Dx
Condition/subject: Viral exanthem (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: Completely or Mostly Correct Workup
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of GPT-4, ChatGPT (GPT series) — Syphilis  diagnosis/ChatGPT4 Dx matching top dermatologist Dx
Condition/subject: Syphilis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: Completely or Mostly Correct Workup

Evaluation of ChatGPT-4 (GPT series) — Atopic Dermatitis
Condition/subject: skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: N/A - the paper skipped this in Table 2; did not give percentage of correct diagnosis (sample size: 1)
Dataset: 20 questions derived from 5 high-impact peer-reviewed research publications by the corresponding authors
Reference (2023): Gravel J, D�Amours-Gravel M, Osmanlliu E. Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions. medRxiv. 2023:2023�03 Supplemental mterial:https://ars.els-cdn.com/content/image/1-s2.0-S2949761223000366-mmc1.pdf  Now peer-reviewed: Gravel J, D�Amours-Gravel M, Osmanlliu E. Learning to fake it: limited responses and fabricated references provided by ChatGPT for medical questions. Mayo Clinic Proceedings: Digital Health. 2023 Sep 1;1(3):226-34.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Eczema Self-management (What is the effectiveness of online behavioural interventions to support eczema self-management?)
Condition/subject: Eczema (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 60.00% (sample size: 1)
Comments: Score used as proxy for accuracy: On a scale of 0 to 100% where 0 signifies that the references had no relationship with the study question and 100% is for the three most pertinent references for this topic, how would you rate the references provided? Factual error: minor

Evaluation of GPT-3.5, ChatGPT (GPT series) — Wound Care: addressing concerns related to skin breakdown and pressure ulcers in the context of awake prone positioning in non-intubated adults with hypoxemic respiratory failure
Condition/subject: ulcers (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 70.00% (sample size: 1)
Comments: Score used as proxy for accuracy: On a scale of 0 to 100% where 0 signifies that the references had no relationship with the study question and 100% is for the three most pertinent references for this topic, how would you rate the references provided? Factual error: No
Dataset: Responses to frequently asked questions from the HS Foundation (HSF), HS Patient Guide (HSPG), and ChatGPT-3.5, along with HS-related websites (Google, Yahoo, and Bing were searched using the term �hidradenitis suppurativa�). The top 50 web pages from each search engine were reviewed, of which, 55 met inclusion criteria for further analysis.
Reference (2024): Gawey L, Dagenet CB, Tran KA, Park S, Hsiao JL, Shi V. Readability of Information Generated by ChatGPT for Hidradenitis Suppurativa. JMIR Dermatol. 2024 Aug 14;7:e55204. doi: 10.2196/55204. PMID: 39141908; PMCID: PMC11358659.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Hidradenitis Suppurativa, Readability of Information
Condition/subject: Hidradenitis suppurativa (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 28.70% (sample size: Not stated)
Comments: Flesch Reading Ease Score: 28.7 (indicates difficult readability; on the scale of 100). - Grade Level: 7-9 grades above recommended
Dataset: Questions based on author's original research
Reference (2024): Gabashvili, Language Models in Dermatology: A Systematic Review of Quantitative Evaluations with Meta-analysis (this work)

Evaluation of Perplexity (Perplexity) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of Copilot (GPT series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of ChatGPT 3.5 (GPT series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Hallucination: "PATM stands for Psychogenic Attributed Misattribution."

Evaluation of Google Bard (linked to the author's account) (Google family of LLMs) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 75.00% (sample size: 3)
Comments: Correctly defined PATM and cited an authentic source, but included additional hallucinated references.

Evaluation of Yi-Lightning (01.AI) (01.AI series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 3)
Comments: Defined PATM as "Poor Hygiene, Anxiety, and Trapped Microbes," though correctly associated it with TMAU.

Evaluation of Gemini 1.5 (linked to the author's account) (Google family of LLMs) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Highly accurate responses due to familiarity with the author's publications.

Evaluation of Gemini 1.5 (anonymously assessed via LLM arena) (Google family of LLMs) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Defined PATM as "Post-Acute Tolerance to Mycobacteria."

Evaluation of ChatGPT O1 Mini (anonymously assessed via LLM arena) (GPT series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Hallucination: Defined PATM as either "Pervasive Auditory-Tactile Misophonia" or "People against me."

Evaluation of ChatGPT O1 Preview (GPT series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Provided accurate responses consistently.
Dataset: Questions based on author's original research
Reference (2024): Gabashvili, Language Models in Dermatology: A Systematic Review of Quantitative Evaluations with Meta-analysis (this work)

Evaluation of Chatgpt-4o  (anonymously assessed via LLM arena) (GPT series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Provided accurate responses consistently.

Evaluation of Grok-2-2024-08-13  (anonymously assessed via LLM arena) (Grok series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Provided accurate responses consistently.

Evaluation of GLM-4-9B (Zhipu AI) (Zhipu AI family of models) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Provided accurate responses consistently.

Evaluation of Mistral-large (Mistral series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Hallucination: Defined PATM as "Partially Attenuated Thrush Mycopathy," but correctly identified links with Candida.

Evaluation of LLAMA-3.1 (LLaMA Series by Meta) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Hallucination: Defined PATM as "Periodic Acne, Tiredness, Malaise, and Migraines," but correctly identified potential links with SIBO.

Evaluation of Claude 3.5 Sonnet (Claude series) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: PASS (sample size: 3)
Comments: acknowledged unfamiliarity with the term requesting additional information rather than risking hallucination.

Evaluation of Gemini-1.5-flash-8b-001 (Google family of LLMs) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Hallucination: Defined PATM as "Post-Acne Telangiectasia and Maules," associating it with emotional distress from acne scarring and telangiectasia.

Evaluation of test-LLM1 (assessed via LLM arena: lmarena.ai)) (unnamed LLM from lmarena.ai) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Hallucination: Defined PATM as "Psychogenic Alopecia with Trichodynia and Multiple hairs" (also known as Psychogenic Hair Plucking or Trichotillomania with alopecia).
Dataset: Questions based on author's original research
Reference (2024): Gabashvili, Language Models in Dermatology: A Systematic Review of Quantitative Evaluations with Meta-analysis (this work)

Evaluation of test-LLM2 (assessed via LLM arena: lmarena.ai)) (unnamed LLM from lmarena.ai) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Hallucination: Defined PATM as "People with Auditory, Tactile, and Motion Sensations."

Evaluation of deepseek-v3 (DeepSeek's models) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Provided accurate responses consistently.

Evaluation of gemini-2.0-flash-thinking-exp-1219 (Google family of LLMs) — PATM ("People Allergic To Me")
Condition/subject: PATM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Provided accurate responses consistently.
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of Copilot (GPT series) — Image-based diagnostics
Condition/subject: Skin of Color (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 0.00% (sample size: 1)

Evaluation of Gemini (Google's Family of LLMs) — Image-based diagnostics
Condition/subject: Skin biology and research (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 0.00% (sample size: 1)

Evaluation of Gemini (Google's Family of LLMs) — Image-based diagnostics
Condition/subject: Psychodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 0.00% (sample size: 2)

Evaluation of GPT-4o (GPT series) — Image-based diagnostics
Condition/subject: Psychodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 0.00% (sample size: 2)

Evaluation of Gemini (Google's Family of LLMs) — Image-based diagnostics
Condition/subject: Skin surgery (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 40.00% (sample size: 5)

Evaluation of Claude 3.5-Sonnet (Claude series) — Image-based diagnostics
Condition/subject: Psychodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 50.00% (sample size: 2)

Evaluation of Claude 3.5-Sonnet (Claude series) — Image-based diagnostics
Condition/subject: Skin surgery (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of GPT-4o (GPT series) — Image-based diagnostics
Condition/subject: Skin surgery (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of Gemini (Google's Family of LLMs) — Image-based diagnostics
Condition/subject: Skin oncology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 63.00% (sample size: 16)

Evaluation of Copilot (GPT series) — Image-based diagnostics
Condition/subject: Cutaneous allergy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 67.00% (sample size: 6)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of Gemini (Google's Family of LLMs) — Image-based diagnostics
Condition/subject: Photodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 67.00% (sample size: 3)

Evaluation of Gemini (Google's Family of LLMs) — Image-based diagnostics
Condition/subject: Infectious disorders (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 67.00% (sample size: 9)

Evaluation of Gemini (Google's Family of LLMs) — Image-based diagnostics
Condition/subject: Pediatric Dermatology, Genetics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 73.00% (sample size: 15)

Evaluation of Claude 3.5-Sonnet (Claude series) — Image-based diagnostics
Condition/subject: Skin oncology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 16)

Evaluation of Gemini (Google, Mountain View, CA, USA) (Google's Family of LLMs) — Dermatology, Specialty Certificate Examination (SCE)
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 100)
Comments: The accuracies for Claude-3.5 Sonnet, Copilot, Gemini, ChatGPT-4o, and Perplexity were 87, 88, 75, 90, and 87, respectively (p = 0.023). Each of the 100 questions and their respective five multiple-choice answers were inputted into LLMs individually. The responses of each LLM were recorded and compared against the standard answer. On July 24, 2024, Four multiple-choice options included photographic reference material and images were uploaded in their original resolution alongside the clinical text/question in the same prompt.

Evaluation of Perplexity (Perplexity) — Formulation and systemic therapy
Condition/subject: Therapy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 8)

Evaluation of Perplexity (Perplexity) — Skin oncology
Condition/subject: Skin oncology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 16)

Evaluation of Copilot (GPT series) — Skin surgery
Condition/subject: Skin surgery (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.00% (sample size: 5)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of Copilot (GPT series) — Paediatrics and genetics
Condition/subject: Pediatric Dermatology, Genetics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.00% (sample size: 15)

Evaluation of Perplexity (Perplexity) — Skin surgery
Condition/subject: Skin surgery (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.00% (sample size: 5)

Evaluation of Perplexity (Perplexity) — Paediatrics and genetics
Condition/subject: Pediatric Dermatology, Genetics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.00% (sample size: 15)

Evaluation of Claude 3.5-Sonnet (Claude series) — Cutaneous allergy
Condition/subject: Cutaneous allergy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 83.00% (sample size: 6)

Evaluation of Gemini (Google's Family of LLMs) — Cutaneous allergy
Condition/subject: Cutaneous allergy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 83.00% (sample size: 6)

Evaluation of GPT-4o (GPT series) — Cutaneous allergy
Condition/subject: Cutaneous allergy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 83.00% (sample size: 6)

Evaluation of Perplexity (Perplexity) — Cutaneous allergy
Condition/subject: Cutaneous allergy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 83.00% (sample size: 6)

Evaluation of Gemini (Google's Family of LLMs) — General dermatology
Condition/subject: General dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 86.00% (sample size: 22)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of Claude-3.5 Sonnet (Anthropic, San Francisco, CA, USA) (Claude series) — Dermatology, Specialty Certificate Examination (SCE)
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 87.00% (sample size: 100)
Comments: The accuracies for Claude-3.5 Sonnet, Copilot, Gemini, ChatGPT-4o, and Perplexity were 87, 88, 75, 90, and 87, respectively (p = 0.023). Each of the 100 questions and their respective five multiple-choice answers were inputted into LLMs individually. The responses of each LLM were recorded and compared against the standard answer. On July 24, 2024, Four multiple-choice options included photographic reference material and images were uploaded in their original resolution alongside the clinical text/question in the same prompt.

Evaluation of Perplexity (Perplexity AI, San Francisco, CA, USA) (Perplexity) — Dermatology, Specialty Certificate Examination (SCE)
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 87.00% (sample size: 100)
Comments: The accuracies for Claude-3.5 Sonnet, Copilot, Gemini, ChatGPT-4o, and Perplexity were 87, 88, 75, 90, and 87, respectively (p = 0.023). Each of the 100 questions and their respective five multiple-choice answers were inputted into LLMs individually. The responses of each LLM were recorded and compared against the standard answer. On July 24, 2024, Four multiple-choice options included photographic reference material and images were uploaded in their original resolution alongside the clinical text/question in the same prompt.

Evaluation of Claude 3.5-Sonnet (Claude series) — Formulation and systemic therapy
Condition/subject: Therapy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 8)

Evaluation of Copilot (GPT series) — Formulation and systemic therapy
Condition/subject: Therapy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 8)

Evaluation of Copilot (GPT series) — Dermatopathology
Condition/subject: Dermatopathology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 8)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of Copilot (GPT series) — Skin oncology
Condition/subject: Skin oncology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 16)

Evaluation of Copilot (Microsoft, Redmond, WA, USA) (GPT series) — Dermatology, Specialty Certificate Examination (SCE)
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 100)
Comments: The accuracies for Claude-3.5 Sonnet, Copilot, Gemini, ChatGPT-4o, and Perplexity were 87, 88, 75, 90, and 87, respectively (p = 0.023). Each of the 100 questions and their respective five multiple-choice answers were inputted into LLMs individually. The responses of each LLM were recorded and compared against the standard answer. On July 24, 2024, Four multiple-choice options included photographic reference material and images were uploaded in their original resolution alongside the clinical text/question in the same prompt.

Evaluation of Gemini (Google's Family of LLMs) — Dermatopathology
Condition/subject: Dermatopathology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 8)

Evaluation of GPT-4o (GPT series) — Formulation and systemic therapy
Condition/subject: Therapy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 8)

Evaluation of GPT-4o (GPT series) — Dermatopathology
Condition/subject: Dermatopathology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 8)

Evaluation of GPT-4o (GPT series) — Skin oncology
Condition/subject: Skin oncology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 88.00% (sample size: 16)

Evaluation of Claude 3.5-Sonnet (Claude series) — Infectious disease
Condition/subject: Infectious disorders (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 89.00% (sample size: 9)

Evaluation of Perplexity (Perplexity) — Infectious disease
Condition/subject: Infectious disorders (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 89.00% (sample size: 9)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of GPT-4o, ChatGPT-4o (OpenAI, San Francisco, CA, USA) (GPT series) — Dermatology, Specialty Certificate Examination (SCE)
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 90.00% (sample size: 100)
Comments: The accuracies for Claude-3.5 Sonnet, Copilot, Gemini, ChatGPT-4o, and Perplexity were 87, 88, 75, 90, and 87, respectively (p = 0.023). Each of the 100 questions and their respective five multiple-choice answers were inputted into LLMs individually. The responses of each LLM were recorded and compared against the standard answer. On July 24, 2024, Four multiple-choice options included photographic reference material and images were uploaded in their original resolution alongside the clinical text/question in the same prompt.

Evaluation of Claude 3.5-Sonnet (Claude series) — General dermatology
Condition/subject: General dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 91.00% (sample size: 22)

Evaluation of Claude 3.5-Sonnet (Claude series) — Paediatrics and genetics
Condition/subject: Pediatric Dermatology, Genetics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 93.00% (sample size: 15)

Evaluation of Copilot (GPT series) — General dermatology
Condition/subject: General dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 95.00% (sample size: 22)

Evaluation of GPT-4o (GPT series) — General dermatology
Condition/subject: General dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 95.00% (sample size: 22)

Evaluation of Perplexity (Perplexity) — General dermatology
Condition/subject: General dermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 95.00% (sample size: 22)

Evaluation of Claude 3.5-Sonnet (Claude series) — Skin of colour
Condition/subject: Skin of color (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of Claude 3.5-Sonnet (Claude series) — Skin biology and research
Condition/subject: Skin biology and research (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of Claude 3.5-Sonnet (Claude series) — Dressings and wound care
Condition/subject: Wounds, Dressings (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of Claude 3.5-Sonnet (Claude series) — Photodermatology
Condition/subject: Photodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of Claude 3.5-Sonnet (Claude series) — Genitourinary medicine
Condition/subject: Genitourinary medicine (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of Claude 3.5-Sonnet (Claude series) — Dermatopathology
Condition/subject: Dermatopathology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 8)

Evaluation of Copilot (GPT series) — Skin biology and research
Condition/subject: skin biology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of Copilot (GPT series) — Dressings and wound care
Condition/subject: Wounds (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of Copilot (GPT series) — Psychodermatology
Condition/subject: Psychodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 2)

Evaluation of Copilot (GPT series) — Photodermatology
Condition/subject: Photodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of Copilot (GPT series) — Genitourinary medicine
Condition/subject: Genitourinary (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of Copilot (GPT series) — Infectious disease
Condition/subject: skin infections (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 9)

Evaluation of Gemini (Google's Family of LLMs) — Skin of colour
Condition/subject: Skin of color (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of Gemini (Google's Family of LLMs) — Dressings and wound care
Condition/subject: Wounds (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of Gemini (Google's Family of LLMs) — Genitourinary medicine
Condition/subject: Genitourinary (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of Gemini (Google's Family of LLMs) — Formulation and systemic therapy
Condition/subject: Therapy (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 8)

Evaluation of GPT-4o (GPT series) — Skin of colour
Condition/subject: Skin of color (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of GPT-4o (GPT series) — Skin biology and research
Condition/subject: Skin biology and research (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of GPT-4o (GPT series) — Dressings and wound care
Condition/subject: Wounds, Dressings (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of GPT-4o (GPT series) — Photodermatology
Condition/subject: Photodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-4o (GPT series) — Genitourinary medicine
Condition/subject: Genitourinary medicine (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of GPT-4o (GPT series) — Image analysis of exam questions
Condition/subject: Images, Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 4)
Comments: The low number of questions involving images means that it is not possible to infer their overall performance as image analysis tools. Current iterations of LLMs are at least able to incorporate image analysis as part of their reasoning. Without the images, LLMs attempted to offer a description of each of the available multiple-choice answers, including the epidemiology and macroscopic description of the lesion. Four image-based questions were assessed, with two to five LLMs answering correctly for each of the questions. Only Claude and Copilot provided a caveat on the potential for AI-generated content being incorrect.

Evaluation of GPT-4o (GPT series) — Infectious disease
Condition/subject: Infectious disorders (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 9)

Evaluation of GPT-4o (GPT series) — Paediatrics and genetics
Condition/subject: Pediatric Dermatology, Genetics (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 15)

Evaluation of Perplexity (Perplexity) — Skin of color
Condition/subject: Skin of color (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of Perplexity (Perplexity) — Skin biology and research
Condition/subject: Skin biology and research (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of Perplexity (Perplexity) — Dressings and wound care
Condition/subject: Wounds, Dressings (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)

Evaluation of Perplexity (Perplexity) — Psychodermatology
Condition/subject: Psychodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 2)

Evaluation of Perplexity (Perplexity) — Photodermatology
Condition/subject: Photodermatology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)
Dataset: 100 multiple-choice questions, including 4 image-based questions, extracted from the Specialty Certificate Examination (SCE) in Dermatology sample questions provided by the Membership of the Royal Colleges of Physicians of the United Kingdom. These questions are considered an accurate representation of the actual SCE in Dermatology exam.
Reference (2024): Fan KS, Fan KH. Dermatological Knowledge and Image Analysis Performance of Large Language Models Based on Specialty Certificate Examination in Dermatology. Dermato. 2024 Sep 30;4(4):124-35. https://doi.org/10.3390/dermato4040013

Evaluation of Perplexity (Perplexity) — Genitourinary medicine
Condition/subject: Genitourinary medicine (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of Perplexity (Perplexity) — Dermatopathology
Condition/subject: Dermatopathology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 8)