Aurametrix/dermatology_longitudinal huggingface dataset:
Dataset: 20 dermatologic diseases out of 60 arbitrarily chosen diseases (19 ophthalmologic, 20 dermatologic, and 21 orthopedic)  - dermatology information (therapy recommendation) generated by LLM ranked by 3 specialist physicians
Reference (2023): Wilhelm TI, Roos J, Kaczmarczyk R. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. J Med Internet Res. 2023 Oct 30;25:e49324. doi: 10.2196/49324. PMID: 37902826; PMCID: PMC10644179.

Evaluation of gpt-3.5-turbo (GPT series) — actinic keratosis: NOT harmfulness
Condition/subject: Actinic Keratosis (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — chronic urticaria: NOT harmfulness
Condition/subject: Chronic urticaria (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — pyoderma gangrenosum: NOT harmfulness
Condition/subject: Pyoderma gangrenosum (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — shingles: NOT harmfulness
Condition/subject: Shingles (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — fungal nail infections: NOT harmfulness
Condition/subject: Fungal nail infections (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — pityriasis versicolor: NOT harmfulness
Condition/subject: Pityriasis versicolor (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — tinea corporis: NOT harmfulness
Condition/subject: Tinea corporis (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity
Dataset: 20 dermatologic diseases out of 60 arbitrarily chosen diseases (19 ophthalmologic, 20 dermatologic, and 21 orthopedic)  - dermatology information (therapy recommendation) generated by LLM ranked by 3 specialist physicians
Reference (2023): Wilhelm TI, Roos J, Kaczmarczyk R. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. J Med Internet Res. 2023 Oct 30;25:e49324. doi: 10.2196/49324. PMID: 37902826; PMCID: PMC10644179.

Evaluation of gpt-3.5-turbo (GPT series) — impetigo: NOT harmfulness
Condition/subject: Impetigo (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — erythrasma: NOT harmfulness
Condition/subject: Erythrasma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — dermatomyositis: NOT harmfulness
Condition/subject: Dermatomyositis (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — scleroderma: NOT harmfulness
Condition/subject: Scleroderma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — morphea: NOT harmfulness
Condition/subject: Morphea (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — atopic eczema: NOT Falsness
Condition/subject: Atopic eczema (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — psoriasis: NOT Falsness
Condition/subject: Psoriasis (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity
Dataset: 20 dermatologic diseases out of 60 arbitrarily chosen diseases (19 ophthalmologic, 20 dermatologic, and 21 orthopedic)  - dermatology information (therapy recommendation) generated by LLM ranked by 3 specialist physicians
Reference (2023): Wilhelm TI, Roos J, Kaczmarczyk R. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. J Med Internet Res. 2023 Oct 30;25:e49324. doi: 10.2196/49324. PMID: 37902826; PMCID: PMC10644179.

Evaluation of gpt-3.5-turbo (GPT series) — an abscess: NOT Falsness
Condition/subject: Abscess (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — discoid lupus erythematosus: NOT Falsness
Condition/subject: Discoid lupus erythematosus (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — a clavus: NOT Falsness
Condition/subject: Clavus (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — warts: NOT Falsness
Condition/subject: Warts (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — basal cell carcinoma: NOT Falsness
Condition/subject: Basal Cell Carcinoma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — malignant melanoma: NOT Falsness
Condition/subject: Melanoma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — actinic keratosis: NOT Falsness
Condition/subject: Actinic Keratosis (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity
Dataset: 20 dermatologic diseases out of 60 arbitrarily chosen diseases (19 ophthalmologic, 20 dermatologic, and 21 orthopedic)  - dermatology information (therapy recommendation) generated by LLM ranked by 3 specialist physicians
Reference (2023): Wilhelm TI, Roos J, Kaczmarczyk R. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. J Med Internet Res. 2023 Oct 30;25:e49324. doi: 10.2196/49324. PMID: 37902826; PMCID: PMC10644179.

Evaluation of gpt-3.5-turbo (GPT series) — chronic urticaria: NOT Falsness
Condition/subject: Chronic urticaria (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — pyoderma gangrenosum: NOT Falsness
Condition/subject: Pyoderma gangrenosum (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — shingles: NOT Falsness
Condition/subject: Shingles (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — fungal nail infections: NOT Falsness
Condition/subject: Fungal nail infections (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — pityriasis versicolor: NOT Falsness
Condition/subject: Pityriasis versicolor (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — tinea corporis: NOT Falsness
Condition/subject: Tinea corporis (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — impetigo: NOT Falsness
Condition/subject: Impetigo (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity
Dataset: 20 dermatologic diseases out of 60 arbitrarily chosen diseases (19 ophthalmologic, 20 dermatologic, and 21 orthopedic)  - dermatology information (therapy recommendation) generated by LLM ranked by 3 specialist physicians
Reference (2023): Wilhelm TI, Roos J, Kaczmarczyk R. Large Language Models for Therapy Recommendations Across 3 Clinical Specialties: Comparative Study. J Med Internet Res. 2023 Oct 30;25:e49324. doi: 10.2196/49324. PMID: 37902826; PMCID: PMC10644179.

Evaluation of gpt-3.5-turbo (GPT series) — erythrasma: NOT Falsness
Condition/subject: Erythrasma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — dermatomyositis: NOT Falsness
Condition/subject: Dermatomyositis (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — scleroderma: NOT Falsness
Condition/subject: Scleroderma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity

Evaluation of gpt-3.5-turbo (GPT series) — morphea: NOT Falsness
Condition/subject: Morphea (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Comments: Score derived from rankings of 3 specialist physicians on a scale 1 - 5, converted to percentages of accuracy/validity
Dataset: 10 Patient information leaflets (PILs) generated using ChatGPT
Reference (2024): Verran C. Artificial intelligence-generated patient information leaflets: a comparison of contents according to British Association of Dermatologists standards. Clin Exp Dermatol. 2024 Jun 25;49(7):711-714. doi: 10.1093/ced/llad461. PMID: 38169318.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Readability of Patient Information Leaflets (PILs)
Condition/subject: Common skin conditions, Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 65.00% (sample size: 10)
Comments: 65% accuracy rated positively for condition-related PILs
Dataset: 5 clinical images were selected from the Danish web atlas, Danderm, depicting various common and rare skin conditions
Reference (2024): Nielsen JP, Gr�nh�j C, Skov L, Gyldenl�ve M. Usefulness of the large language model ChatGPT (GPT?4) as a diagnostic tool and information source in dermatology. JEADV Clinical Practice. 2024.

Evaluation of GPT-4 (GPT series) — Image-based diagnostics, Clinical images
Condition/subject: Common and rare skin conditions (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 40.00% (sample size: 15)
Comments: Rating for  relevance, accuracy, and depth (average 2 one the scale of 1 to 5)
Dataset: 240 questions from the Japanese National Nurse Examinations set from 2019 to 2023, including 6 questions in dermatology subject area
Reference (2023): Taira K, Itaya T, Hanada A. Performance of the Large Language Model ChatGPT on the National Nurse Examinations in Japan: Evaluation Study. JMIR Nurs. 2023 Jun 27;6:e47305. doi: 10.2196/47305. PMID: 37368470; PMCID: PMC10337249.

Evaluation of GPT-3.5 (GPT series) — Dermatology subject area in the Japanese National Nurse Examinations
Condition/subject: Certification (Japanese)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 50.00% (sample size: 6)
Comments: ChatGPT had a lower percentage of correct answers in some areas, such as pharmacology, social welfare, related law and regulations, endocrinology/metabolism, and dermatology, and a higher percentage of correct answers in the areas of nutrition, pathology, hematology, ophthalmology, otolaryngology, dentistry and dental surgery, and nursing integration and practice. For Basic Knowledge Questions: Average accuracy was 75.1% (SD 3%); for  General Questions: Average accuracy was 64.5% (SD 5%)
Dataset: 36 Medical records: Clinical information from 36 patients selected from 90 consecutive patients referred to a single dermatology emergency clinic between June and December 202
Reference (2024): Stoneham S, Livesey A, Cooper H, Mitchell C. ChatGPT versus clinician: challenging the diagnostic capabilities of artificial intelligence in dermatology. Clin Exp Dermatol. 2024 Jun 25;49(7):707-710. doi: 10.1093/ced/llad402. PMID: 37979201.

Evaluation of GPT-4 (GPT series) — Diagnosis from clinical history and cutaneous signs recorded by nonspecialists
Condition/subject: Diagnostics, clinical scenarios (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 39.00% (sample size: 36)
Comments: ChatGPT made a correct primary diagnosis 56% of the time (n = 20). Using the clinical history and cutaneous signs recorded by nonspecialists, it was able to make a correct diagnosis 39% of the time (n = 14). This was similar to the diagnostic rate of nonspecialists (36%; n = 13), but it was much lower than that of dermatologists (83%; n = 30). There was no differential offered by referring sources 28% of the time (n = 10), unlike ChatGPT, which provided a differential diagnosis 100% of the time.

Evaluation of GPT-4 (GPT series) — Primary Diagnosis from Clinical Records with clinical history and examination data obtained by a dermatologist
Condition/subject: Primary Diagnosis, Clinical scenarios (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 56.00% (sample size: 36)
Comments: ChatGPT made a correct primary diagnosis 56% of the time (n = 20). Using the clinical history and cutaneous signs recorded by nonspecialists, it was able to make a correct diagnosis 39% of the time (n = 14). This was similar to the diagnostic rate of nonspecialists (36%; n = 13), but it was much lower than that of dermatologists (83%; n = 30). There was no differential offered by referring sources 28% of the time (n = 10), unlike ChatGPT, which provided a differential diagnosis 100% of the time.

Evaluation of GPT-4 (GPT series) — Differential diagnosis
Condition/subject: Differential diagnosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 36)
Comments: ChatGPT made a correct primary diagnosis 56% of the time (n = 20). Using the clinical history and cutaneous signs recorded by nonspecialists, it was able to make a correct diagnosis 39% of the time (n = 14). This was similar to the diagnostic rate of nonspecialists (36%; n = 13), but it was much lower than that of dermatologists (83%; n = 30). There was no differential offered by referring sources 28% of the time (n = 10), unlike ChatGPT, which provided a differential diagnosis 100% of the time.
Dataset: 300 dermatology-related questions. 150 multiple-choice questions from DermQbank inputted into ChatGPT in December 2023 and then again in July 2024. Of these, 83 were text-only questions and 67 had associated images. An additional 150 questions inputted in 2024 made a total of 300 different questions where 169 were text-only and 133 had associated images.
Reference (2024): Smith L, Hanna R, Hatch L, Hanna K. Computer Vision Meets Large Language Models: Performance of ChatGPT 4.0 on Dermatology Boards-Style Practice Questions. SKIN The Journal of Cutaneous Medicine. 2024 Sep 15;8(5):1815-21.

Evaluation of GPT-4, December 2023 (GPT series) — Dermapathology questions for board examination preparation from 2023, with images
Condition/subject: Dermatopathology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 22.20% (sample size: 9)
Comments: Two questions (2023) and six questions (2024) were double coded as having both clinical and dermatopathology images

Evaluation of GPT-4, December 2023 (GPT series) — Dermatology questions for board examination preparation from 2023, with images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 47.80% (sample size: 67)

Evaluation of GPT-4, December 2023 (GPT series) — Clinical dermatology and dermoscopy questions for board examination preparation, with images
Condition/subject: Images, Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 50.90% (sample size: 57)
Comments: Two questions (2023) and six questions (2024) were double coded as having both clinical and dermatopathology images

Evaluation of GPT-4, July 2024 (GPT series) — Dermapathology questions for board examination preparation, with images
Condition/subject: Dermatopathology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 58.80% (sample size: 17)
Comments: Two questions (2023) and six questions (2024) were double coded as having both clinical and dermatopathology images

Evaluation of GPT-4, July 2024 (GPT series) — Clinical dermatology and dermoscopy questions for board examination preparation from 2024, with images
Condition/subject: Images, Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 59.60% (sample size: 57)
Comments: Two questions (2023) and six questions (2024) were double coded as having both clinical and dermatopathology images

Evaluation of GPT-4, December 2023 (GPT series) — Dermatology questions for board examination in 2023, with or without images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 65.30% (sample size: 150)
Dataset: 300 dermatology-related questions. 150 multiple-choice questions from DermQbank inputted into ChatGPT in December 2023 and then again in July 2024. Of these, 83 were text-only questions and 67 had associated images. An additional 150 questions inputted in 2024 made a total of 300 different questions where 169 were text-only and 133 had associated images.
Reference (2024): Smith L, Hanna R, Hatch L, Hanna K. Computer Vision Meets Large Language Models: Performance of ChatGPT 4.0 on Dermatology Boards-Style Practice Questions. SKIN The Journal of Cutaneous Medicine. 2024 Sep 15;8(5):1815-21.

Evaluation of GPT-4, December 2023 (GPT series) — Dermatology questions for board examination preparation, with images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 65.30% (sample size: 150)

Evaluation of GPT-4, July 2024 (GPT series) — Dermatology questions for board examination preparation from 2024, with images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 65.70% (sample size: 67)

Evaluation of GPT-4, July 2024 (GPT series) — Dermapathology questions for board examination preparation from 2024, with images
Condition/subject: Dermatopathology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 66.70% (sample size: 9)
Comments: Two questions (2023) and six questions (2024) were double coded as having both clinical and dermatopathology images

Evaluation of GPT-4, July 2024 (GPT series) — Dermatology questions for board examination preparation, with images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 67.70% (sample size: 133)
Comments: Of the aggregate 300 question data, ChatGPT answered 232 questions correctly (77.3%). ChatGPT performed significantly better with text-only questions than with questions that included images (85.2%(144/169)vs 67.7%(90/133),P<.001). Of image-based questions, ChatGPT performed better with clinical image questions than with dermatopathology questions (69.0%(78/133) vs. 58.8% (10/17),P=.40),but this difference was not statistically significant. performing above the 46thpercentile of PGY-4 question bank users

Evaluation of GPT-4, July 2024 (GPT series) — Dermatology questions for board examination preparation from 2023 and 2024, with images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 67.70% (sample size: 133)
Dataset: 300 dermatology-related questions. 150 multiple-choice questions from DermQbank inputted into ChatGPT in December 2023 and then again in July 2024. Of these, 83 were text-only questions and 67 had associated images. An additional 150 questions inputted in 2024 made a total of 300 different questions where 169 were text-only and 133 had associated images.
Reference (2024): Smith L, Hanna R, Hatch L, Hanna K. Computer Vision Meets Large Language Models: Performance of ChatGPT 4.0 on Dermatology Boards-Style Practice Questions. SKIN The Journal of Cutaneous Medicine. 2024 Sep 15;8(5):1815-21.

Evaluation of GPT-4, July 2024 (GPT series) — Clinical dermatology and dermoscopy questions for board examination preparation, with images
Condition/subject: Images, Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 69.00% (sample size: 113)
Comments: Of the aggregate 300 question data, ChatGPT answered 232 questions correctly (77.3%). ChatGPT performed significantly better with text-only questions than with questions that included images (85.2%(144/169)vs 67.7%(90/133),P<.001). Of image-based questions, ChatGPT performed better with clinical image questions than with dermatopathology questions (69.0%(78/133) vs. 58.8% (10/17),P=.40),but this difference was not statistically significant. performing above the 46thpercentile of PGY-4 question bank users

Evaluation of GPT-4, July 2024 (GPT series) — Dermatology questions for board examination in 2024, with or without images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 73.30% (sample size: 150)

Evaluation of GPT-4, July 2024 (GPT series) — Dermatology questions for board examination, with or without images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 77.30% (sample size: 300)

Evaluation of GPT-4, July 2024 (GPT series) — Dermatology questions for board examination preparation, no images
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 85.20% (sample size: 169)
Comments: Of the aggregate 300 question data, ChatGPT answered 232 questions correctly (77.3%). ChatGPT performed significantly better with text-only questions than with questions that included images (85.2%(144/169)vs 67.7%(90/133),P<.001). Of image-based questions, ChatGPT performed better with clinical image questions than with dermatopathology questions (69.0%(78/133) vs. 58.8% (10/17),P=.40),but this difference was not statistically significant. performing above the 46thpercentile of PGY-4 question bank users
Dataset: 13 questions about cancer that are common points of confusion among the public (per NCI web page �Common Cancer Myths and Misconceptions� 
Reference (2023): Skyler B Johnson, Andy J King, Echo L Warner, Sanjay Aneja, Benjamin H Kann, Carma L Bylund, Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information, JNCI Cancer Spectrum, Volume 7, Issue 2, April 2023, pkad015, https://doi.org/10.1093/jncics/pkad015

Evaluation of GPT-3.5 (December 2022), ChatGPT (GPT series) — General Skin Cancer Information
Condition/subject: Skin cancer (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 100.00% (sample size: 1)
Comments: NCI FKG was found to be 8.8 (content is written at approximately an 8th to 9th-grade reading level accessible to a broad audience, including the general public with middle school education. C FKG = 12.8 The content is written at nearly a 13th-grade level, which corresponds to the first year of college, more suitable for audiences with higher education backgrounds.
Dataset: 89 publicly available sample questions from the Dermatology specialty certificate examination.
Reference (2023): Shetty M, Ettlinger M, Lynch M. GPT-4, an artificial intelligence large language model, exhibits high levels of accuracy on dermatology specialty certificate exam questions. medRxiv. 2023:2023-07. medRxiv 2023.07.13.23292418; doi: https://doi.org/10.1101/2023.07.13.23292418

Evaluation of GPT-4 (GPT series) — Dermatology License questions, UK, CSE exam
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 85.00% (sample size: 89)
Comments: GPT-4, the most advanced large language model, exhibits remarkable accuracy - answering in excess of 85% of questions correctly, at a level that would likely be sufficient to pass the SCE exam
Dataset: 81 multiple-choice questions, excluding image-based queries curated by dermatologists from a series of the most recent CME articles from the Journal of American Academy of Dermatology ranging from July 2023 - September 2022 (simulated board certification examination)
Reference (2024): Sher, ArielKahan, ShaniRoster, KatiePeacock, AnjelicaPereira, Frederick et al. (2024) 54250 Assessing the Performance of ChatGPT in Answering Dermatology Practice Questions Journal of the American Academy of Dermatology, Volume 91, Issue 3, AB131

Evaluation of GPT-3.5 (GPT series) — Genetic, Biological, and Structural Differences in Skin of Color
Condition/subject: Skin of color (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 25.00% (sample size: 3)
Comments: subset of the 81 questions

Evaluation of GPT-3.5 (GPT series) — Ecchymosis due to Child Abuse and Neglect
Condition/subject: Ecchymosis (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 30.00% (sample size: 2)
Comments: subset of the 81 questions

Evaluation of GPT-3.5 (GPT series) — Physiology-related Dermatology questions 
Condition/subject: Physiology (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 51.00% (sample size: 5)
Comments: ChatGPT garnered a total score of 58.0% in the simulated board certification examination series. It exhibited a strong performance in pharmacology, with a 71% success rate, in contrast to a 51% score in physiology-related questions. Nonetheless, it demonstrated a pronounced weakness regarding questions concerning Genetic, Biological, and Structural Differences in Skin of Color, and Child Abuse and Neglect, with scores of 25% and 30%, respectively

Evaluation of GPT-3.5 (GPT series) — Continued Medical Education (CME) examination, simulated Board certification examination
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 58.00% (sample size: 81)
Comments: ChatGPT garnered a total score of 58.0% in the simulated board certification examination series. It exhibited a strong performance in pharmacology, with a 71% success rate, in contrast to a 51% score in physiology-related questions. Nonetheless, it demonstrated a pronounced weakness regarding questions concerning Genetic, Biological, and Structural Differences in Skin of Color, and Child Abuse and Neglect, with scores of 25% and 30%, respectively

Evaluation of GPT-3.5 (GPT series) — Pharmacology for Dermatology Certification
Condition/subject: Pharmacology , Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 71.00% (sample size: 5)
Comments: subset of the 81 questions
Dataset: Hundreds of post; publicly available Reddit data, pulling from the forums r/Dermatology, r/essentialoils, and r/tanning from January 2018 to August 2019.
Reference (2021): Sager MA, Kashyap AM, Tamminga M, Ravoori S, Callison-Burch C, Lipoff JB. Identifying and Responding to Health Misinformation on Reddit Dermatology Forums With Artificially Intelligent Bots Using Natural Language Processing: Design and Evaluation Study. JMIR Dermatol. 2021 Sep 30;4(2):e20975. doi: 10.2196/20975. PMID: 37632809; PMCID: PMC10334965.

Evaluation of Logistic regression (Bag-of-Words: Unigram, Bigram, Trigram) (NOT LLM/TRANSFORMER) — Dermatology Misinformation in Reddit's dermatology forums, Test Accuracy for "Tanning"
Condition/subject: Tanning (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 95.65% (sample size: )

Evaluation of Logistic regression (Bag-of-Words: Unigram, Bigram, Trigram) (NOT LLM/TRANSFORMER) — Dermatology Misinformation in Reddit's dermatology forums, Test Accuracy for "Essential Oils"
Condition/subject: Essential Oils (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 97.29% (sample size: )

Evaluation of XLNet, Pre-Trained and Fine-Tuned (XLNet) — Dermatology Misinformation in Reddit's dermatology forums, Test Accuracy for "Tanning"
Condition/subject: Tanning (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 98.61% (sample size: )

Evaluation of XLNet, Pre-Trained and Fine-Tuned (XLNet) — Dermatology Misinformation in Reddit's dermatology forums, Test Accuracy for "Essential Oils"
Condition/subject: Essential Oils (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 98.70% (sample size: )

Evaluation of BERT, Pre-Trained and Fine-Tuned (Google's Family of LLMs) — Dermatology Misinformation in Reddit's dermatology forums, Test Accuracy for "Essential Oils"
Condition/subject: Essential Oils (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 99.56% (sample size: )
Comments: Bidirectional Encoder Representations from Transformers (BERT) -

Evaluation of BERT, Pre-Trained and Fine-Tuned (Google's Family of LLMs) — Dermatology Misinformation in Reddit's dermatology forums, Test Accuracy for "Tanning"
Condition/subject: Tanning (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 100.00% (sample size: )
Dataset: The accuracy score represents the mean score of 3 dermatology residents who assessed the educational materials using a numeric scale: 1 (not accurate), 2 (somewhat accurate), and 3 (accurate).
Reference (2024): Roster K, Kann RB, Farabi B, Gronbeck C, Brownstone N, Lipner SR. Readability and Health Literacy Scores for ChatGPT-Generated Dermatology Public Education Materials: Cross-Sectional Analysis of Sunscreen and Melanoma Questions. JMIR Dermatol. 2024 Mar 6;7:e50163. doi: 10.2196/50163. PMID: 38446502; PMCID: PMC10955394.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Melanoma FAQs readability
Condition/subject: Melanoma (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 59.30% (sample size: 3)
Comments: Flesch Reading Ease score ranges from 0 to 100, with higher scores indicating easier readability. The AAD�s sunscreen FAQs and melanoma FAQs had Flesch Reading Ease scores of 60.9 (standard/average) and 56.2 (fairly difficult), respectively.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Sunscreen FAQs readability
Condition/subject: Mohs surgery (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 82.00% (sample size: 3)
Comments: Flesch Reading Ease score ranges from 0 to 100, with higher scores indicating easier readability. The AAD�s sunscreen FAQs and melanoma FAQs had Flesch Reading Ease scores of 60.9 (standard/average) and 56.2 (fairly difficult), respectively.
Dataset: 32 dermatologic clinical cases: Ten dermatology cases extracted from Clinical Advisor and 22 from McGraw Hill, reviewed by a dermatologist to ensure they could be reasonably answered without any images
Reference (2023): Ravipati A, Pradeep T, Elman SA. The role of artificial intelligence in dermatology: the promising but limited accuracy of ChatGPT in diagnosing clinical scenarios. Int J Dermatol. 2023 Oct;62(10):e547-e548. doi: 10.1111/ijd.16746. Epub 2023 Jun 12. PMID: 37306147.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Dermatology medical records: Top diagnoses
Condition/subject: Medical Records (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 37.50% (sample size: 32)
Comments: Top diagnoses of dermatology cases from educational materials scored for accuracy against top diagnoses created by dermatologists

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Dermatology medical records: Per-Case Inclusion/Differential diagnoses of dermatology cases from educational materials scored for accuracy against differential diagnoses created by dermatologists (the correct diagnosis is listed somewhere in its differential diagnosis )
Condition/subject: Medical Records (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 81.00% (sample size: 32)
Dataset: 29 cases in total, 14 from a general dermatology textbook assigned to patients in the non-skin of color group. 15 from a dermatology textbook which was aimed at skin of color and then assigned to this group during the study. Twelve of the skin of color cases involved histopathological reports. A correct diagnosis occurred when using GPT-4 in 100% of the skin of color cases involving histopathology, and this contrasted with an accuracy rate of 66.7% for GPT-3. However, this distinction was not reported by the research team to be statistically significant (P?=?.093). The histories and physical assessment details of each case were inputted by the investigators into the GPT-3.5 and GPT-4 models and used medical terminology, with questions posed to the AI to identify the top 3 differential diagnoses.
Reference (2024): Qureshi S, Alli SR, Ogunyemi B (2024). Accuracy of ChatGPT-3.5 and GPT-4 in diagnosing clinical scenarios in dermatology involving skin of color. Int J Dermatol. https://doi.org/10.1111/ijd.17425.

Evaluation of GPT-3.5 (GPT series) — Dx: Skin of color case reports involving histopathology
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 67.70% (sample size: 12)
Comments: ChatGPT could predict dermatoses in people with lighter and darker skin color with similar levels of accuracy. A notable finding was that GPT-3.5's accuracy was shown to decrease as additional clinical data was added, yet GPT-4's accuracy improved when additional data was included. A correct diagnosis occurred when using GPT-4 in 100% of the skin of color cases involving histopathology, and this contrasted with an accuracy rate of 66.7% for GPT-3.

Evaluation of GPT-4 (GPT series) — Dx: Skin of color case reports involving histopathology
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 12)
Comments: ChatGPT could predict dermatoses in people with lighter and darker skin color with similar levels of accuracy. A notable finding was that GPT-3.5's accuracy was shown to decrease as additional clinical data was added, yet GPT-4's accuracy improved when additional data was included. A correct diagnosis occurred when using GPT-4 in 100% of the skin of color cases involving histopathology, and this contrasted with an accuracy rate of 66.7% for GPT-3.
Dataset: Descriptive Data of 398 patients of whom 159 (39.9%) had fungal infections. Other conditions included eczema 36 (9%), alopecia 28 (7%), infestations 27 (6.8%), acne 25 (6.3%), psoriasis 19 (4.8%), benign tumors 7 (1.8%), bacterial infection 19 (4.8%), viral infection 15 (3.8%), and pigmentary disorders 20 (5%). 
Reference (2020): Patil S, Rao ND, Patil A, Basar F, Bate S. Assessment of Tibot(R) artificial intelligence application in prediction of diagnosis in dermatological conditions: results of a single centre study. Indian Dermatol Online J. 2020;11(6):910�4

Evaluation of Tibot AI (NOT an LLM ) (TibotAI) — Viral infection
Condition/subject: Viral infection (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 26.70% (sample size: 398)
Comments: The prediction accuracy (ability to get diagnosis in top three conditions) for alopecia, fungal infections, and eczema was 100%, 95.6%, and 91.7%, respectively. Mean prediction accuracy for correct diagnosis in the predicted top three diagnoses was 85.2%, and for correct diagnosis was 60.7%. Sensitivity and specificity of the application were approximately 86% and 98%, respectively. The sensitivity and positive predictive value of the application to diagnose alopecia was 100% and for fungal infections it was 96.85% and 90.05%, respectively. No skin types were recorded in this study, but it was generally between Fitzpatrick types four to six.

Evaluation of Tibot AI (NOT an LLM ) (TibotAI) — Infestation
Condition/subject: Infestation (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 59.30% (sample size: 398)
Comments: The prediction accuracy (ability to get diagnosis in top three conditions) for alopecia, fungal infections, and eczema was 100%, 95.6%, and 91.7%, respectively. Mean prediction accuracy for correct diagnosis in the predicted top three diagnoses was 85.2%, and for correct diagnosis was 60.7%. Sensitivity and specificity of the application were approximately 86% and 98%, respectively. The sensitivity and positive predictive value of the application to diagnose alopecia was 100% and for fungal infections it was 96.85% and 90.05%, respectively. No skin types were recorded in this study, but it was generally between Fitzpatrick types four to six.
Dataset: Descriptive Data of 398 patients of whom 159 (39.9%) had fungal infections. Other conditions included eczema 36 (9%), alopecia 28 (7%), infestations 27 (6.8%), acne 25 (6.3%), psoriasis 19 (4.8%), benign tumors 7 (1.8%), bacterial infection 19 (4.8%), viral infection 15 (3.8%), and pigmentary disorders 20 (5%). 
Reference (2020): Patil S, Rao ND, Patil A, Basar F, Bate S. Assessment of Tibot(R) artificial intelligence application in prediction of diagnosis in dermatological conditions: results of a single centre study. Indian Dermatol Online J. 2020;11(6):910�4

Evaluation of Tibot AI (NOT an LLM ) (TibotAI) — Benign tumors
Condition/subject: Benign tumors (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 71.40% (sample size: 398)
Comments: The prediction accuracy (ability to get diagnosis in top three conditions) for alopecia, fungal infections, and eczema was 100%, 95.6%, and 91.7%, respectively. Mean prediction accuracy for correct diagnosis in the predicted top three diagnoses was 85.2%, and for correct diagnosis was 60.7%. Sensitivity and specificity of the application were approximately 86% and 98%, respectively. The sensitivity and positive predictive value of the application to diagnose alopecia was 100% and for fungal infections it was 96.85% and 90.05%, respectively. No skin types were recorded in this study, but it was generally between Fitzpatrick types four to six.

Evaluation of Tibot AI (NOT an LLM ) (TibotAI) — Eczema
Condition/subject: Eczema (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 91.70% (sample size: 398)
Comments: The prediction accuracy (ability to get diagnosis in top three conditions) for alopecia, fungal infections, and eczema was 100%, 95.6%, and 91.7%, respectively. Mean prediction accuracy for correct diagnosis in the predicted top three diagnoses was 85.2%, and for correct diagnosis was 60.7%. Sensitivity and specificity of the application were approximately 86% and 98%, respectively. The sensitivity and positive predictive value of the application to diagnose alopecia was 100% and for fungal infections it was 96.85% and 90.05%, respectively. No skin types were recorded in this study, but it was generally between Fitzpatrick types four to six.
Dataset: Descriptive Data of 398 patients of whom 159 (39.9%) had fungal infections. Other conditions included eczema 36 (9%), alopecia 28 (7%), infestations 27 (6.8%), acne 25 (6.3%), psoriasis 19 (4.8%), benign tumors 7 (1.8%), bacterial infection 19 (4.8%), viral infection 15 (3.8%), and pigmentary disorders 20 (5%). 
Reference (2020): Patil S, Rao ND, Patil A, Basar F, Bate S. Assessment of Tibot(R) artificial intelligence application in prediction of diagnosis in dermatological conditions: results of a single centre study. Indian Dermatol Online J. 2020;11(6):910�4

Evaluation of Tibot AI (NOT an LLM ) (TibotAI) — Fungal infections
Condition/subject: Fungal infections (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 95.60% (sample size: 398)
Comments: The prediction accuracy (ability to get diagnosis in top three conditions) for alopecia, fungal infections, and eczema was 100%, 95.6%, and 91.7%, respectively. Mean prediction accuracy for correct diagnosis in the predicted top three diagnoses was 85.2%, and for correct diagnosis was 60.7%. Sensitivity and specificity of the application were approximately 86% and 98%, respectively. The sensitivity and positive predictive value of the application to diagnose alopecia was 100% and for fungal infections it was 96.85% and 90.05%, respectively. No skin types were recorded in this study, but it was generally between Fitzpatrick types four to six.

Evaluation of Tibot AI (NOT an LLM ) (TibotAI) — Alopecia
Condition/subject: Alopecia (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 398)
Comments: The prediction accuracy (ability to get diagnosis in top three conditions) for alopecia, fungal infections, and eczema was 100%, 95.6%, and 91.7%, respectively. Mean prediction accuracy for correct diagnosis in the predicted top three diagnoses was 85.2%, and for correct diagnosis was 60.7%. Sensitivity and specificity of the application were approximately 86% and 98%, respectively. The sensitivity and positive predictive value of the application to diagnose alopecia was 100% and for fungal infections it was 96.85% and 90.05%, respectively. No skin types were recorded in this study, but it was generally between Fitzpatrick types four to six.
Dataset: 84 multiple-choice sample questions from the sample SCE in Dermatology question bank
Reference (2024): Passby L, Jenko N, Wernham A. Performance of ChatGPT on Specialty Certificate Examination in Dermatology multiple-choice questions. Clin Exp Dermatol. 2024 Jun 25;49(7):722-727. doi: 10.1093/ced/llad197. PMID: 37264670.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Dermatology Specialty Certificate Examination (SCE)
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 63.00% (sample size: 84)
Comments: The typical pass mark for the dermatology SCE is 70-72%.

Evaluation of GPT-4 (GPT series) — Dermatology Specialty Certificate Examination (SCE)
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 90.00% (sample size: 84)
Comments: The typical pass mark for the dermatology SCE is 70-72%.