Aurametrix/dermatology_longitudinal huggingface dataset:
Dataset: 1000 images, 500 images each of melanoma and non-melanoma randomly selected.
Reference (2024): Nitin Chetla, Joseph Chang, Samantha Sattler, Matthew Chen, William Young Guo, Dia Shah, Jeremy Hugh (2024) What is the Capability and Accuracy of ChatGPT�s Newest Models in Diagnosing between Melanoma and Non-Melanoma lesions? JMIR Dermatology (Preprint) 14/10/2024:67551DOI: 10.2196/preprints.67551URL: https://preprints.jmir.org/preprint/67551

Evaluation of ChatGPT-4 Omni (GPT series) — Non-Melanoma Identification Accuracy  from Images of Skin Lesions
Condition/subject: Non-melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 6.56% (sample size: 899)

Evaluation of GPT-4 Omni, ChatGPT (GPT series) — Non-Melanoma Identification Accuracy from Images of Skin Lesions ��- Binary prompt
Condition/subject: Non-melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 25.25% (sample size: 998)

Evaluation of ChatGPT-4 Turbo (GPT series) — Melanoma Identification Accuracy from Images of Skin Lesions
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 54.60% (sample size: 1000)

Evaluation of ChatGPT-4 Omni (GPT series) — Melanoma Identification Accuracy from Images of Skin Lesions
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 57.70% (sample size: 1000)
Dataset: 135 patients' images, 92 (68.1%) were male and 43 (31.9%) were female. The median age was 71 years +/- 9 (Min: 56/Max: 91). Of those, 108 were malignant pathologies (54 basal cell carcinoma and 54 squamous cell carcinoma) and 27 benign pathologies (14 seborrheic keratoses, 2 actinic keratoses, and 11 actinic cheilitis)
Reference (2024): Medela A, Sabater A, Montilla IH, MacCarthy T, Aguilar A, Chiesa-Estomba CM. The utility and reliability of a deep learning algorithm as a diagnosis support tool in head & neck non-melanoma skin malignancies. Eur Arch Otorhinolaryngol. 2024 Sep 6. doi: 10.1007/s00405-024-08951-z. Epub ahead of print. PMID: 39242415.

Evaluation of Custom Deep Learning Algorithm, Vision Transformers (Transformer) — Basal cell carcinoma, image-based diagnosis, differential diagnosis (top 10)
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 90.00% (sample size: 54)
Comments: Malignant lesions (basal cell carcinoma and squamous cell carcinoma) were identified within the top-5 most likely diagnoses in above 90% of cases.

Evaluation of Custom Deep Learning Algorithm, Vision Transformers (Transformer) — Squamous cell carcinoma, image-based diagnosis, differential diagnosis (top 10)
Condition/subject: Squamous Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 90.00% (sample size: 54)
Comments: Malignant lesions (basal cell carcinoma and squamous cell carcinoma) were identified within the top-5 most likely diagnoses in above 90% of cases.
Dataset: generating immunohistochemical panels/immunophenotypes of 51 cutaneous diseases, including a diverse variety of epidermal, adnexal, hematolymphoid, and soft tissue entities. 
Reference (2024): McCrary MR, Galambus J, Chen WS. Evaluating the diagnostic performance of a large language model-powered chatbot for providing immunohistochemistry recommendations in dermatopathology. J Cutan Pathol. 2024 Sep;51(9):689-695. doi: 10.1111/cup.14631. Epub 2024 May 14. PMID: 38744501.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Not confabulated Immunohistochemistry references in dermatopathology
Condition/subject: Immunochemistry (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 72.80% (sample size: 51)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Clinical usefulness of Immunohistochemistry recommendations in dermatopathology
Condition/subject: Dermatopathology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 75.50% (sample size: 51)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Immunohistochemistry recommendations in dermatopathology: factual correctness
Condition/subject: Immunohistochemistry, dermatopathology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 86.10% (sample size: 51)
Dataset: Ten clinical case scenarios pertaining to different dermatological conditions from a publicly available website:  https://www.clinicaladvisor.com/home/dermatology-clinic/.
Reference (2024): Manoharan P, Surapaneni KM. Assessing the diagnostic capability of ChatGPT through clinical case scenarios in dermatology. Indian J Dermatol Venereol Leprol. 2024 May 25:1-3. doi: 10.25259/IJDVL_1267_2023. Epub ahead of print. PMID: 38841923.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Diagnostics from text-based clinical case scenarios pertaining to different dermatological conditions
Condition/subject: Diagnostics, clinical scenarios (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 90.00% (sample size: 10)

Evaluation of GPT-4.0, ChatGPT (GPT series) — Diagnostics from text-based clinical case scenarios pertaining to different dermatological conditions
Condition/subject: Diagnostics, clinical scenarios (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 90.00% (sample size: 10)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Firm Round Papule on the Foot   - Poroma  (clinical-based scenario)
Condition/subject: Poroma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: FAIL (sample size: 1)
Comments: "ChatGPT 3.5 has generated the wrong answer for the case scenario on Firm Round Papule on Foot. The answer generated by ChatGPT was �Verruca vulgaris� whereas the answer was �Poroma� in the key. But ChatGPT 4.0 has rightly diagnosed this case. In the case of a �swelling on the knee�, ChatGPT 3.5 rightly diagnosed the condition as a ganglionic cyst, whereas ChatGPT 4.0 diagnosed it to be a lipoma. The main discrepancy in the explanations generated was based on the typical features of lipoma and ganglionic cysts. While ChatGPT 3.5 has considered the location of joints to be the most common site for ganglionic cysts, ChatGPT 4.0 has given the diagnosis based on size and skin over the bump"

Evaluation of GPT-4.0, ChatGPT (GPT series) — Swelling on the knee - ganglionic cyst (clinical-based scenario)
Condition/subject: Ganglionic cyst (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: FAIL (sample size: 1)
Comments: "ChatGPT 3.5 has generated the wrong answer for the case scenario on Firm Round Papule on Foot. The answer generated by ChatGPT was �Verruca vulgaris� whereas the answer was �Poroma� in the key. But ChatGPT 4.0 has rightly diagnosed this case. In the case of a �swelling on the knee�, ChatGPT 3.5 rightly diagnosed the condition as a ganglionic cyst, whereas ChatGPT 4.0 diagnosed it to be a lipoma. The main discrepancy in the explanations generated was based on the typical features of lipoma and ganglionic cysts. While ChatGPT 3.5 has considered the location of joints to be the most common site for ganglionic cysts, ChatGPT 4.0 has given the diagnosis based on size and skin over the bump"
Dataset: Ten clinical case scenarios pertaining to different dermatological conditions from a publicly available website:  https://www.clinicaladvisor.com/home/dermatology-clinic/.
Reference (2024): Manoharan P, Surapaneni KM. Assessing the diagnostic capability of ChatGPT through clinical case scenarios in dermatology. Indian J Dermatol Venereol Leprol. 2024 May 25:1-3. doi: 10.25259/IJDVL_1267_2023. Epub ahead of print. PMID: 38841923.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Swelling on the knee - ganglionic cyst (clinical-based scenario)
Condition/subject: Ganglionic cyst (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: PASS (sample size: 1)
Comments: "ChatGPT 3.5 has generated the wrong answer for the case scenario on Firm Round Papule on Foot. The answer generated by ChatGPT was �Verruca vulgaris� whereas the answer was �Poroma� in the key. But ChatGPT 4.0 has rightly diagnosed this case. In the case of a �swelling on the knee�, ChatGPT 3.5 rightly diagnosed the condition as a ganglionic cyst, whereas ChatGPT 4.0 diagnosed it to be a lipoma. The main discrepancy in the explanations generated was based on the typical features of lipoma and ganglionic cysts. While ChatGPT 3.5 has considered the location of joints to be the most common site for ganglionic cysts, ChatGPT 4.0 has given the diagnosis based on size and skin over the bump"

Evaluation of GPT-4.0, ChatGPT (GPT series) — Firm Round Papule on the Foot   - Poroma  (clinical-based scenario)
Condition/subject: Poroma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: PASS (sample size: 1)
Comments: "ChatGPT 3.5 has generated the wrong answer for the case scenario on Firm Round Papule on Foot. The answer generated by ChatGPT was �Verruca vulgaris� whereas the answer was �Poroma� in the key. But ChatGPT 4.0 has rightly diagnosed this case. In the case of a �swelling on the knee�, ChatGPT 3.5 rightly diagnosed the condition as a ganglionic cyst, whereas ChatGPT 4.0 diagnosed it to be a lipoma. The main discrepancy in the explanations generated was based on the typical features of lipoma and ganglionic cysts. While ChatGPT 3.5 has considered the location of joints to be the most common site for ganglionic cysts, ChatGPT 4.0 has given the diagnosis based on size and skin over the bump"
Dataset: 100 histopathology-confirmed dermoscopic images (50 malignant, 50 benign) randomly selected from the International Skin Imaging Collaboration (ISIC) archive using a computer-generated randomization process. The ISIC archive was chosen due to its comprehensive and well-annotated collection of dermoscopic images, ensuring a diverse and representative sample. 
Reference (2024): Liu X, Duan C, Kim MK, Zhang L, Jee E, Maharjan B, Huang Y, Du D, Jiang X. Claude 3 Opus and ChatGPT With GPT-4 in Dermoscopic Image Analysis for Melanoma Diagnosis: Comparative Performance Analysis. JMIR Med Inform. 2024 Aug 6;12:e59273. doi: 10.2196/59273. PMID: 39106482; PMCID: PMC11336503.

Evaluation of GPT-4V (GPT series) — Melanoma malignancy discrimination, dermoscopic images
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 44.00% (sample size: )
Comments: In malignancy discrimination, Claude 3 Opus outperformed ChatGPT with 47.06% sensitivity, 81.63% specificity, and 64% accuracy, compared to 45.1%, 42.86%, and 44%, respectively. The McNemar test showed a significant difference (P<.001). Claude 3 Opus had an odds ratio of 3.951 (95% CI 1.685-9.263) in discriminating malignancy, while ChatGPT-4 had an odds ratio of 0.616 (95% CI 0.297-1.278).

Evaluation of GPT-4V (GPT series) — Melanoma: primary diagnosis, dermoscopic images
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 48.00% (sample size: )
Comments: In the primary diagnosis, Claude 3 Opus achieved 54.9% sensitivity (95% CI 44.08%-65.37%), 57.14% specificity (95% CI 46.31%-67.46%), and 56% accuracy (95% CI 46.22%-65.42%), while ChatGPT demonstrated 56.86% sensitivity (95% CI 45.99%-67.21%), 38.78% specificity (95% CI 28.77%-49.59%), and 48% accuracy (95% CI 38.37%-57.75%). The McNemar test showed no significant difference between the 2 models (P=.17).

Evaluation of Claude 3 Opus (Claude series) — Melanoma: primary diagnosis, dermoscopic images
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 56.00% (sample size: 100)
Comments: In the primary diagnosis, Claude 3 Opus achieved 54.9% sensitivity (95% CI 44.08%-65.37%), 57.14% specificity (95% CI 46.31%-67.46%), and 56% accuracy (95% CI 46.22%-65.42%), while ChatGPT demonstrated 56.86% sensitivity (95% CI 45.99%-67.21%), 38.78% specificity (95% CI 28.77%-49.59%), and 48% accuracy (95% CI 38.37%-57.75%). The McNemar test showed no significant difference between the 2 models (P=.17).
Dataset: 100 histopathology-confirmed dermoscopic images (50 malignant, 50 benign) randomly selected from the International Skin Imaging Collaboration (ISIC) archive using a computer-generated randomization process. The ISIC archive was chosen due to its comprehensive and well-annotated collection of dermoscopic images, ensuring a diverse and representative sample. 
Reference (2024): Liu X, Duan C, Kim MK, Zhang L, Jee E, Maharjan B, Huang Y, Du D, Jiang X. Claude 3 Opus and ChatGPT With GPT-4 in Dermoscopic Image Analysis for Melanoma Diagnosis: Comparative Performance Analysis. JMIR Med Inform. 2024 Aug 6;12:e59273. doi: 10.2196/59273. PMID: 39106482; PMCID: PMC11336503.

Evaluation of Claude 3 Opus (Claude series) — Melanoma malignancy discrimination, dermoscopic images
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 64.00% (sample size: 100)
Comments: In malignancy discrimination, Claude 3 Opus outperformed ChatGPT with 47.06% sensitivity, 81.63% specificity, and 64% accuracy, compared to 45.1%, 42.86%, and 44%, respectively. The McNemar test showed a significant difference (P<.001). Claude 3 Opus had an odds ratio of 3.951 (95% CI 1.685-9.263) in discriminating malignancy, while ChatGPT-4 had an odds ratio of 0.616 (95% CI 0.297-1.278).

Evaluation of Claude 3 Opus (Claude series) — Melanoma:differential diagnoses, dermoscopic images
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 76.00% (sample size: 100)
Comments: For the top 3 differential diagnoses, Claude 3 Opus and ChatGPT included the correct diagnosis in 76% (95% CI 66.33%-83.77%) and 78% (95% CI 68.46%-85.45%) of cases, respectively.

Evaluation of GPT-4V (GPT series) — Melanoma:differential diagnoses, dermoscopic images
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 78.00% (sample size: )
Comments: For the top 3 differential diagnoses, Claude 3 Opus and ChatGPT included the correct diagnosis in 76% (95% CI 66.33%-83.77%) and 78% (95% CI 68.46%-85.45%) of cases, respectively.
Dataset: 360 questions in Polish and English: Three Specialty Certificate Examination in Dermatology tests, in English and Polish, consisting of 120 single-best-answer, multiple-choice questions each (60% pass rate)
Reference (2024): Lewandowski M, ?ukowicz P, ?wietlik D, Bara?ska-Rybak W. ChatGPT-3.5 and ChatGPT-4 dermatological knowledge level based on the Specialty Certificate Examination in Dermatology. Clin Exp Dermatol. 2024 Jun 25;49(7):686-691. doi: 10.1093/ced/llad255. PMID: 37540015.

Evaluation of GPT-4 (GPT series) — Dermatology Practice Questions in Polish
Condition/subject: Certification (Polish)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 70.00% (sample size: 360)

Evaluation of GPT-4 (GPT series) — Dermatology Practice Questions in English
Condition/subject: Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.00% (sample size: 360)

Evaluation of GPT-4 (GPT series) — Clinical image-based Polish questions, Dermatology Practice
Condition/subject: Images, Certification (Polish)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 84.20% (sample size: 360)

Evaluation of GPT-4 (GPT series) — Clinical image-based English questions, Dermatology Practice
Condition/subject: Images, Certification (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 93.00% (sample size: 360)
Dataset: 38 clinical questions from professional dermatologists
Reference (2024): Lent HC, Ortner VK, Karmisholt KE, Wiegell SR, Nissen CV, Omland SH, Kamstrup MR, Togsverd?Bo K, Haedersdal M. A chat about actinic keratosis: examining capabilities and user experience of ChatGPT as a digital health technology in dermato?oncology. JEADV Clinical Practice. 2024 Mar;3(1):258-65. doi: 10.1002/jvc2.263

Evaluation of GPT-3.5, ChatGPT (GPT series) — Actinic Keratosis: treatment
Condition/subject: Actinic Keratosis (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 27.30% (sample size: 11)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Actinic Keratosis 
Condition/subject: Actinic Keratosis (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 31.60% (sample size: 38)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Actinic Keratosis: diagnosis
Condition/subject: Actinic Keratosis (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 37.50% (sample size: 8)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Actinic Keratosis: Patient education, including pathogenesis of AK and potential risk factors
Condition/subject: Actinic Keratosis (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 63.00% (sample size: 19)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Actinic Keratosis: What is the prevalence of actinic keratosis?
Condition/subject: Actinic Keratosis (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: FAIL (sample size: 1)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Actinic Keratosis: What are the signs and symptoms of actinic keratosis?
Condition/subject: Actinic Keratosis (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: FAIL (sample size: 1)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Actinic Keratosis: Which treatment for actinic keratosis gives the best cosmetic outcome?
Condition/subject: Actinic Keratosis (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: FAIL (sample size: 1)
Dataset: 10 most common patient MMS questions according to Google Search. Answers from ChatGPT and Bard were evaluated by 15 MMS surgeons, examining their appropriateness as a patient-facing informational platform, sufficiency of response in a clinical environment, and accuracy of content generated. Validated scales were employed to assess the comprehensibility of each response. Limitations include the significant degree of interrater variability observed.
Reference (2024): Lauck KC, Cho SW, DaCunha M, Wuennenberg J, Aasi S, Alam M, Arron ST, Bar A, Brodland DG, Cerci FB, Cohen JL, Coldiron B, Council ML, Harmon CB, Hruza G, L�uchli S, Moody BR, Wysong AS, Zitelli JA, Tolkachjov SN. The utility of artificial intelligence platforms for patient-generated questions in Mohs micrographic surgery: a multi-national, blinded expert panel evaluation. Int J Dermatol. 2024 Nov;63(11):1592-1598. doi: 10.1111/ijd.17382. Epub 2024 Aug 9. PMID: 39123288.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Mohs Micrographic Surgery:  Sufficiency of Answers to Patient Questions for Clinical Practice
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 33.00% (sample size: 150)
Comments: About 92% of all responses were deemed appropriate for a patient-facing platform outside of the clinic by the majority of reviewers. The majority of reviewers deemed all LLM responses appropriate. 75% of responses were rated as mostly accurate or higher. ChatGPT had the highest mean accuracy. The majority of the panel deemed 33% of responses sufficient for clinical practice. The mean comprehensibility scores for all platforms indicated a required 10th-grade reading level.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - Will I need plastic surgery if I have Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 33.33% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - How deep do they cut for basal cell carcinoma?
Condition/subject: Basal Cell Carcinoma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 40.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - Will I need plastic surgery if I have Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 46.67% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - What are disadvantages of Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 46.67% (sample size: 15)
Dataset: 10 most common patient MMS questions according to Google Search. Answers from ChatGPT and Bard were evaluated by 15 MMS surgeons, examining their appropriateness as a patient-facing informational platform, sufficiency of response in a clinical environment, and accuracy of content generated. Validated scales were employed to assess the comprehensibility of each response. Limitations include the significant degree of interrater variability observed.
Reference (2024): Lauck KC, Cho SW, DaCunha M, Wuennenberg J, Aasi S, Alam M, Arron ST, Bar A, Brodland DG, Cerci FB, Cohen JL, Coldiron B, Council ML, Harmon CB, Hruza G, L�uchli S, Moody BR, Wysong AS, Zitelli JA, Tolkachjov SN. The utility of artificial intelligence platforms for patient-generated questions in Mohs micrographic surgery: a multi-national, blinded expert panel evaluation. Int J Dermatol. 2024 Nov;63(11):1592-1598. doi: 10.1111/ijd.17382. Epub 2024 Aug 9. PMID: 39123288.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - How deep do they cut for basal cell carcinoma?
Condition/subject: Basal Cell Carcinoma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 53.33% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - Do you always need stitches after Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 53.33% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Cosmetic dermatology: Will I need plastic surgery if I have Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 56.00% (sample size: 15)

Evaluation of Bard (Google's Family of LLMs) — Mohs Micrographic Surgery:  Sufficiency of Answers to Patient Questions for Clinical Practice
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 58.00% (sample size: 150)
Comments: About 92% of all responses were deemed appropriate for a patient-facing platform outside of the clinic by the majority of reviewers. The majority of reviewers deemed all LLM responses appropriate. 75% of responses were rated as mostly accurate or higher. ChatGPT had the highest mean accuracy. The majority of the panel deemed 33% of responses sufficient for clinical practice. The mean comprehensibility scores for all platforms indicated a required 10th-grade reading level.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Risks and Complications: What are disadvantages of Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 60.00% (sample size: 15)
Dataset: 10 most common patient MMS questions according to Google Search. Answers from ChatGPT and Bard were evaluated by 15 MMS surgeons, examining their appropriateness as a patient-facing informational platform, sufficiency of response in a clinical environment, and accuracy of content generated. Validated scales were employed to assess the comprehensibility of each response. Limitations include the significant degree of interrater variability observed.
Reference (2024): Lauck KC, Cho SW, DaCunha M, Wuennenberg J, Aasi S, Alam M, Arron ST, Bar A, Brodland DG, Cerci FB, Cohen JL, Coldiron B, Council ML, Harmon CB, Hruza G, L�uchli S, Moody BR, Wysong AS, Zitelli JA, Tolkachjov SN. The utility of artificial intelligence platforms for patient-generated questions in Mohs micrographic surgery: a multi-national, blinded expert panel evaluation. Int J Dermatol. 2024 Nov;63(11):1592-1598. doi: 10.1111/ijd.17382. Epub 2024 Aug 9. PMID: 39123288.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - What are disadvantages of Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 60.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Surgical Technique: How deep do they cut for basal cell carcinoma?
Condition/subject: Basal Cell Carcinoma (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 64.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - What not to do after Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 66.67% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - What not to do after Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 66.67% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Procedure Details: Do you always need stitches after Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 70.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - Do you always need stitches after Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 73.33% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - How is Mohs different from regular surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 73.33% (sample size: 15)
Dataset: 10 most common patient MMS questions according to Google Search. Answers from ChatGPT and Bard were evaluated by 15 MMS surgeons, examining their appropriateness as a patient-facing informational platform, sufficiency of response in a clinical environment, and accuracy of content generated. Validated scales were employed to assess the comprehensibility of each response. Limitations include the significant degree of interrater variability observed.
Reference (2024): Lauck KC, Cho SW, DaCunha M, Wuennenberg J, Aasi S, Alam M, Arron ST, Bar A, Brodland DG, Cerci FB, Cohen JL, Coldiron B, Council ML, Harmon CB, Hruza G, L�uchli S, Moody BR, Wysong AS, Zitelli JA, Tolkachjov SN. The utility of artificial intelligence platforms for patient-generated questions in Mohs micrographic surgery: a multi-national, blinded expert panel evaluation. Int J Dermatol. 2024 Nov;63(11):1592-1598. doi: 10.1111/ijd.17382. Epub 2024 Aug 9. PMID: 39123288.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - How many hours does Mohs surgery take?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 73.33% (sample size: 15)

Evaluation of ChatGPT (GPT series) — Mohs Micrographic Surgery Patient Questions: Accuracy
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 75.00% (sample size: 150)
Comments: About 92% of all responses were deemed appropriate for a patient-facing platform outside of the clinic by the majority of reviewers. The majority of reviewers deemed all LLM responses appropriate. 75% of responses were rated as mostly accurate or higher. ChatGPT had the highest mean accuracy. The majority of the panel deemed 33% of responses sufficient for clinical practice. The mean comprehensibility scores for all platforms indicated a required 10th-grade reading level.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Post-operative Care: What not to do after Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 76.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Recovery: How long is recovery from Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 80.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - How is Mohs different from regular surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 80.00% (sample size: 15)
Dataset: 10 most common patient MMS questions according to Google Search. Answers from ChatGPT and Bard were evaluated by 15 MMS surgeons, examining their appropriateness as a patient-facing informational platform, sufficiency of response in a clinical environment, and accuracy of content generated. Validated scales were employed to assess the comprehensibility of each response. Limitations include the significant degree of interrater variability observed.
Reference (2024): Lauck KC, Cho SW, DaCunha M, Wuennenberg J, Aasi S, Alam M, Arron ST, Bar A, Brodland DG, Cerci FB, Cohen JL, Coldiron B, Council ML, Harmon CB, Hruza G, L�uchli S, Moody BR, Wysong AS, Zitelli JA, Tolkachjov SN. The utility of artificial intelligence platforms for patient-generated questions in Mohs micrographic surgery: a multi-national, blinded expert panel evaluation. Int J Dermatol. 2024 Nov;63(11):1592-1598. doi: 10.1111/ijd.17382. Epub 2024 Aug 9. PMID: 39123288.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - How many hours does Mohs surgery take?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 80.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - How serious is Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 80.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - How painful is Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 80.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Procedure Duration: How many hours does Mohs surgery take?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 82.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Comparison with Other Surgeries: How is Mohs different from regular surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 84.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Pain and Discomfort: How painful is Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 86.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - Will I need anesthesia if I have Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 86.67% (sample size: 15)
Dataset: 10 most common patient MMS questions according to Google Search. Answers from ChatGPT and Bard were evaluated by 15 MMS surgeons, examining their appropriateness as a patient-facing informational platform, sufficiency of response in a clinical environment, and accuracy of content generated. Validated scales were employed to assess the comprehensibility of each response. Limitations include the significant degree of interrater variability observed.
Reference (2024): Lauck KC, Cho SW, DaCunha M, Wuennenberg J, Aasi S, Alam M, Arron ST, Bar A, Brodland DG, Cerci FB, Cohen JL, Coldiron B, Council ML, Harmon CB, Hruza G, L�uchli S, Moody BR, Wysong AS, Zitelli JA, Tolkachjov SN. The utility of artificial intelligence platforms for patient-generated questions in Mohs micrographic surgery: a multi-national, blinded expert panel evaluation. Int J Dermatol. 2024 Nov;63(11):1592-1598. doi: 10.1111/ijd.17382. Epub 2024 Aug 9. PMID: 39123288.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - How serious is Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 86.67% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - How painful is Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 86.67% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - How long is recovery from Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 86.67% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Risks and Complications: How serious is Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 90.00% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Procedure details: Will I need anesthesia if I have Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 92.00% (sample size: 15)
Dataset: 10 most common patient MMS questions according to Google Search. Answers from ChatGPT and Bard were evaluated by 15 MMS surgeons, examining their appropriateness as a patient-facing informational platform, sufficiency of response in a clinical environment, and accuracy of content generated. Validated scales were employed to assess the comprehensibility of each response. Limitations include the significant degree of interrater variability observed.
Reference (2024): Lauck KC, Cho SW, DaCunha M, Wuennenberg J, Aasi S, Alam M, Arron ST, Bar A, Brodland DG, Cerci FB, Cohen JL, Coldiron B, Council ML, Harmon CB, Hruza G, L�uchli S, Moody BR, Wysong AS, Zitelli JA, Tolkachjov SN. The utility of artificial intelligence platforms for patient-generated questions in Mohs micrographic surgery: a multi-national, blinded expert panel evaluation. Int J Dermatol. 2024 Nov;63(11):1592-1598. doi: 10.1111/ijd.17382. Epub 2024 Aug 9. PMID: 39123288.

Evaluation of ChatGPT (GPT series) — Mohs Micrographic Surgery Patient Questions: Appropriateness for Patient-facing web platform
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 92.00% (sample size: 150)
Comments: About 92% of all responses were deemed appropriate for a patient-facing platform outside of the clinic by the majority of reviewers. The majority of reviewers deemed all LLM responses appropriate. 75% of responses were rated as mostly accurate or higher. ChatGPT had the highest mean accuracy. The majority of the panel deemed 33% of responses sufficient for clinical practice. The mean comprehensibility scores for all platforms indicated a required 10th-grade reading level.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - How long is recovery from Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 93.33% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - Will I need anesthesia if I have Mohs surgery?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 93.33% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Answer's sufficiency for Clinical Practice - What type of cancer is Mohs procedure for?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 93.33% (sample size: 15)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Applicability (Cancer Types): What type of cancer is Mohs procedure for?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 15)
Dataset: 10 most common patient MMS questions according to Google Search. Answers from ChatGPT and Bard were evaluated by 15 MMS surgeons, examining their appropriateness as a patient-facing informational platform, sufficiency of response in a clinical environment, and accuracy of content generated. Validated scales were employed to assess the comprehensibility of each response. Limitations include the significant degree of interrater variability observed.
Reference (2024): Lauck KC, Cho SW, DaCunha M, Wuennenberg J, Aasi S, Alam M, Arron ST, Bar A, Brodland DG, Cerci FB, Cohen JL, Coldiron B, Council ML, Harmon CB, Hruza G, L�uchli S, Moody BR, Wysong AS, Zitelli JA, Tolkachjov SN. The utility of artificial intelligence platforms for patient-generated questions in Mohs micrographic surgery: a multi-national, blinded expert panel evaluation. Int J Dermatol. 2024 Nov;63(11):1592-1598. doi: 10.1111/ijd.17382. Epub 2024 Aug 9. PMID: 39123288.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Moh's Surgery: Appropriateness of the answer for Website - What type of cancer is Mohs procedure for?
Condition/subject: Mohs surgery (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 100.00% (sample size: 15)
Dataset: 960 PEMs were generated across 4 LLMs and 8 dermatologic conditions. Handouts were generated by ChatGPT-3.5, GPT-4, DocsGPT, and DermGPT at two prompted reading levels. The Flesch-Kincaid reading level (FKRL) of current American Academy of Dermatology PEMs was evaluated for 4 common (atopic dermatitis, acne vulgaris, psoriasis, and herpes zoster) and 4 rare (epidermolysis bullosa, bullous pemphigoid, lamellar ichthyosis, and lichen planus) dermatologic conditions. 10 PEMs per condition at unspecified fifth- and seventh-grade FKRLs were evaluated with Microsoft Word readability statistics. The preservation of meaning across LLMs was assessed by 2 blinded dermatology resident trainees.
Reference (2024): Lambert R, Choo ZY, Gradwohl K, Schroedl L, Ruiz De Luzuriaga A. Assessing the Application of Large Language Models in Generating Dermatologic Patient Education Materials According to Reading Level: Qualitative Study. JMIR Dermatol. 2024 May 16;7:e55898. doi: 10.2196/55898. PMID: 38754096; PMCID: PMC11140271.

Evaluation of DermGPT (GPT series) — Acne vulgaris, fifth-grade reading level (common condition)
Condition/subject: Acne vulgaris (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DermGPT (GPT series) — Atopic dermatitis,  fifth-grade reading level (common condition)
Condition/subject: Atopic dermatitis (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DermGPT (GPT series) — Bullous pemphigoid, fifth-grade reading level (rare condition)
Condition/subject: Bullous pemphigoid (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DermGPT (GPT series) — Epidermolysis bullosa, fifth-grade reading level (rare condition)
Condition/subject: Epidermolysis bullosa (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DermGPT (GPT series) — Epidermolysis bullosa, seventh-grade reading level (rare condition)
Condition/subject: Epidermolysis bullosa (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DermGPT (GPT series) — Herpes zoster,  seventh-grade reading level (common condition)
Condition/subject: Herpes zoster (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DermGPT (GPT series) — Herpes zoster, fifth-grade reading level (common condition)
Condition/subject: Herpes zoster (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)
Dataset: 960 PEMs were generated across 4 LLMs and 8 dermatologic conditions. Handouts were generated by ChatGPT-3.5, GPT-4, DocsGPT, and DermGPT at two prompted reading levels. The Flesch-Kincaid reading level (FKRL) of current American Academy of Dermatology PEMs was evaluated for 4 common (atopic dermatitis, acne vulgaris, psoriasis, and herpes zoster) and 4 rare (epidermolysis bullosa, bullous pemphigoid, lamellar ichthyosis, and lichen planus) dermatologic conditions. 10 PEMs per condition at unspecified fifth- and seventh-grade FKRLs were evaluated with Microsoft Word readability statistics. The preservation of meaning across LLMs was assessed by 2 blinded dermatology resident trainees.
Reference (2024): Lambert R, Choo ZY, Gradwohl K, Schroedl L, Ruiz De Luzuriaga A. Assessing the Application of Large Language Models in Generating Dermatologic Patient Education Materials According to Reading Level: Qualitative Study. JMIR Dermatol. 2024 May 16;7:e55898. doi: 10.2196/55898. PMID: 38754096; PMCID: PMC11140271.

Evaluation of DermGPT (GPT series) — Lamellar ichthyosis, fifth-grade reading level (rare condition)
Condition/subject: Lamellar ichthyosis (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DermGPT (GPT series) — Psoriasis,  fifth-grade reading level (common condition)
Condition/subject: Psoriasis (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DermGPT (GPT series) — Dermatologic patient education material handouts meeting prompted fifth-grade reading level (common conditions)
Condition/subject: Patient Education (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 40)

Evaluation of DocsGPT (GPT series) — Bullous pemphigoid, fifth-grade reading level (rare condition)
Condition/subject: Bullous pemphigoid (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of DocsGPT (GPT series) — Psoriasis,  fifth-grade reading level (common condition)
Condition/subject: Psoriasis (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of GPT-3.5 (GPT series) — Atopic dermatitis,  seventh-grade reading level (common condition)
Condition/subject: Atopic dermatitis (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)

Evaluation of GPT-4 (GPT series) — Epidermolysis bullosa, seventh-grade reading level (rare condition)
Condition/subject: Epidermolysis bullosa (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 0.00% (sample size: 10)