Aurametrix/dermatology_longitudinal huggingface dataset:
Dataset: Diverse Dermatology Images (DDI) Dataset representative of diverse skin tones. DDI contains 656 clinical images obtained from the Stanford Clinic and includes some rare dermatological conditions that have previously been described in literature. The Fitzpatrick skin tone (FST) was carefully labeled based on in-person visit documentation, demographic photo, and image of the lesion. They are represented as groups of two i.e. FST I-II (light skin tone), FST III-IV, and FST V-VI (dark skin tone). There are 208, 241, and 207 clinical images across the three groups respectively. USMLE set included 612 questions
Reference (2024): Jiang Y, Omiye JA, Zakka C, Moor M, Gui H, Alipour S, Mousavi SS, Chen JH, Rajpurkar P, Daneshjou R. Evaluating General Vision-Language Models for Clinical Medicine. medRxiv. 2024 Apr 14:2024-04. doi: https://doi.org/10.1101/2024.04.12.24305744

Evaluation of GPT-4V (GPT series) — Image-based FST V-VI, top-1 diagnosis
Condition/subject: Fitzpatrick V-VI skin type (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 4.35% (sample size: 207)

Evaluation of GPT-4V (GPT series) — Image-based FST I-II, top-1 diagnosis
Condition/subject: Fitzpatrick I-II skin type (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 5.28% (sample size: 208)
Comments: For Fitzpatrick skin tone (FST) prediction, GPT-4V only provided skin tones for 603 images and reported there was not enough information for the remaining 53 images. Diverse Dermatology Images (DDI) Dataset representative of diverse skin tones. DDI contains 656 clinical images obtained from the Stanford Clinic and includes some rare dermatological conditions that have previously been described in literature. The Fitzpatrick skin tone (FST) was carefully labeled based on in-person visit documentation, demographic photo, and image of the lesion. They are represented as groups of two i.e. FST I-II (light skin tone), FST III-IV, and FST V-VI (dark skin tone). There are 208, 241, and 207 clinical images across the three groups respectively.

Evaluation of GPT-4V (GPT series) — Image-based diagnostics of dermatology conditions: Top diagnosis
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 6.20% (sample size: 656)
Comments: In dermatology, GPT-4V had an overall top-1 and top-3 diagnostic accuracy of 6.2% and 21% respectively.. There was a significant accuracy drop when predicting on images of darker skin tones (p<0.001). GPT-4V accurately identified Fitzpatrick skin tones for 56.5% of images. For the multiple-choice styled USMLE image-based test questions, GPT-4V had an accuracy of 59%.
Dataset: Diverse Dermatology Images (DDI) Dataset representative of diverse skin tones. DDI contains 656 clinical images obtained from the Stanford Clinic and includes some rare dermatological conditions that have previously been described in literature. The Fitzpatrick skin tone (FST) was carefully labeled based on in-person visit documentation, demographic photo, and image of the lesion. They are represented as groups of two i.e. FST I-II (light skin tone), FST III-IV, and FST V-VI (dark skin tone). There are 208, 241, and 207 clinical images across the three groups respectively. USMLE set included 612 questions
Reference (2024): Jiang Y, Omiye JA, Zakka C, Moor M, Gui H, Alipour S, Mousavi SS, Chen JH, Rajpurkar P, Daneshjou R. Evaluating General Vision-Language Models for Clinical Medicine. medRxiv. 2024 Apr 14:2024-04. doi: https://doi.org/10.1101/2024.04.12.24305744

Evaluation of GPT-4V (GPT series) — Image-based FST III-IV, top-1 diagnosis
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 8.71% (sample size: 241)
Comments: For Fitzpatrick skin tone (FST) prediction, GPT-4V only provided skin tones for 603 images and reported there was not enough information for the remaining 53 images. Diverse Dermatology Images (DDI) Dataset representative of diverse skin tones. DDI contains 656 clinical images obtained from the Stanford Clinic and includes some rare dermatological conditions that have previously been described in literature. The Fitzpatrick skin tone (FST) was carefully labeled based on in-person visit documentation, demographic photo, and image of the lesion. They are represented as groups of two i.e. FST I-II (light skin tone), FST III-IV, and FST V-VI (dark skin tone). There are 208, 241, and 207 clinical images across the three groups respectively.

Evaluation of GPT-4V (GPT series) — Image-based FST V-VI top-3 diagnosis
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 15.94% (sample size: 207)

Evaluation of GPT-4V (GPT series) — Image-based FST I-II, top-3 diagnosis
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 16.82% (sample size: 208)

Evaluation of GPT-4V, ChatGPT (GPT series) — Image-based diagnostics of dermatology conditions: Differential diagnosis (top 3)
Condition/subject: Differential diagnosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 21.00% (sample size: 656)

Evaluation of GPT-4V (GPT series) — Image-based FST III-IV, top-3 diagnosis
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 29.46% (sample size: 241)
Dataset: Diverse Dermatology Images (DDI) Dataset representative of diverse skin tones. DDI contains 656 clinical images obtained from the Stanford Clinic and includes some rare dermatological conditions that have previously been described in literature. The Fitzpatrick skin tone (FST) was carefully labeled based on in-person visit documentation, demographic photo, and image of the lesion. They are represented as groups of two i.e. FST I-II (light skin tone), FST III-IV, and FST V-VI (dark skin tone). There are 208, 241, and 207 clinical images across the three groups respectively. USMLE set included 612 questions
Reference (2024): Jiang Y, Omiye JA, Zakka C, Moor M, Gui H, Alipour S, Mousavi SS, Chen JH, Rajpurkar P, Daneshjou R. Evaluating General Vision-Language Models for Clinical Medicine. medRxiv. 2024 Apr 14:2024-04. doi: https://doi.org/10.1101/2024.04.12.24305744

Evaluation of GPT-4V (GPT series) — Melanoma in situ; Image-based diagnostics
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 37.80% (sample size: 656)
Comments: GPT-4V predicts melanoma in situ at a frequency of 37.8% compared to the true frequency of 0.91%. This overdiagnosis could imply that the model is overly sensitive to features associated with melanoma in situ, leading it to misclassify other conditions as this type of melanoma.

Evaluation of GPT-4V (GPT series) — Melanocytic nevi; Image-based diagnostics
Condition/subject: Moles (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 39.00% (sample size: 656)
Comments: Top predictions for GPT-4V are mostly malignant dermatological conditions

Evaluation of GPT-4V (GPT series) — Malignancy predictions from dermatological images: accuracy
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 39.62% (sample size: 656)
Comments: In malignancy predictions, Dermatologists had an accuracy of 67.99% (95% CI: 64.27% - 71.55%) compared to GPT-4V�s accuracy of 39.62% (95% CI: 35.41% - 43.95%). This difference was statistically significant with a p-value of 3.93�10?21. However, GPT-4V outperformed in other metrics including a sensitivity of 0.86 compared to dermatologists� 0.71
Dataset: Diverse Dermatology Images (DDI) Dataset representative of diverse skin tones. DDI contains 656 clinical images obtained from the Stanford Clinic and includes some rare dermatological conditions that have previously been described in literature. The Fitzpatrick skin tone (FST) was carefully labeled based on in-person visit documentation, demographic photo, and image of the lesion. They are represented as groups of two i.e. FST I-II (light skin tone), FST III-IV, and FST V-VI (dark skin tone). There are 208, 241, and 207 clinical images across the three groups respectively. USMLE set included 612 questions
Reference (2024): Jiang Y, Omiye JA, Zakka C, Moor M, Gui H, Alipour S, Mousavi SS, Chen JH, Rajpurkar P, Daneshjou R. Evaluating General Vision-Language Models for Clinical Medicine. medRxiv. 2024 Apr 14:2024-04. doi: https://doi.org/10.1101/2024.04.12.24305744

Evaluation of GPT-4V (GPT series) — Image-based diagnostics of dermatology conditions: Fitzpatrick skin tone (FST) labeling
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 56.50% (sample size: 656)
Comments: For Fitzpatrick skin tone (FST) prediction, GPT-4V only provided skin tones for 603 images and reported there was not enough information for the remaining 53 images. Diverse Dermatology Images (DDI) Dataset representative of diverse skin tones. DDI contains 656 clinical images obtained from the Stanford Clinic and includes some rare dermatological conditions that have previously been described in literature. The Fitzpatrick skin tone (FST) was carefully labeled based on in-person visit documentation, demographic photo, and image of the lesion. They are represented as groups of two i.e. FST I-II (light skin tone), FST III-IV, and FST V-VI (dark skin tone). There are 208, 241, and 207 clinical images across the three groups respectively.

Evaluation of GPT-4V (GPT series) — Image-based diagnostics of dermatology conditions: USMLE questions
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 59.00% (sample size: 612)

Evaluation of GPT-4V (GPT series) — Squamous cell carcinoma; Image-based diagnostics
Condition/subject: Squamous Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 59.00% (sample size: 656)

Evaluation of GPT-4V (GPT series) — Squamous cell carcinoma in-situ (SCCIS); Image-based diagnostics
Condition/subject: Squamous Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 68.00% (sample size: 656)

Evaluation of GPT-4V (GPT series) — Basal cell carcinoma; Image-based diagnostics
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 83.00% (sample size: 656)
Dataset: Diverse Dermatology Images (DDI) Dataset representative of diverse skin tones. DDI contains 656 clinical images obtained from the Stanford Clinic and includes some rare dermatological conditions that have previously been described in literature. The Fitzpatrick skin tone (FST) was carefully labeled based on in-person visit documentation, demographic photo, and image of the lesion. They are represented as groups of two i.e. FST I-II (light skin tone), FST III-IV, and FST V-VI (dark skin tone). There are 208, 241, and 207 clinical images across the three groups respectively. USMLE set included 612 questions
Reference (2024): Jiang Y, Omiye JA, Zakka C, Moor M, Gui H, Alipour S, Mousavi SS, Chen JH, Rajpurkar P, Daneshjou R. Evaluating General Vision-Language Models for Clinical Medicine. medRxiv. 2024 Apr 14:2024-04. doi: https://doi.org/10.1101/2024.04.12.24305744

Evaluation of GPT-4V (GPT series) — Malignancy predictions from dermatological images: sensitivity
Condition/subject: Images, Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 86.00% (sample size: 656)
Comments: When comparing dermatologists to GPT-4V�s malignancy predictions,  dermatologists had an accuracy of 67.99% (95% CI: 64.27% - 71.55%) compared to GPT-4V�s accuracy of 39.62% (95% CI: 35.41% - 43.95%). This difference was statistically significant with a p-value of 3.93�10?21. However, GPT-4V outperformed in other metrics including a sensitivity of 0.86 compared to dermatologists� 0.71.
Dataset: survey questions in April 2023 for drug recommendations generated by ChatGPT with data from secondary databases, that is, Taiwan's National Health Insurance Research Database and an US medical center database, and validated by dermatologists. 
Reference (2024): Iqbal U, Lee LT, Rahmanti AR, Celi LA, Li YJ. Can large language models provide secondary reliable opinion on treatment options for dermatological diseases? J Am Med Inform Assoc. 2024 May 20;31(6):1341-1347. doi: 10.1093/jamia/ocae067. PMID: 38578616; PMCID: PMC11105123.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Medication Recommendations for Common Dermatological Conditions, Q-value cutoff of 10 (least confident disease-medication associations)
Condition/subject: Dermatology medications (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 72.92% (sample size: N/A)
Comments: Drug suggestions were also evaluated using the Q-value, measure used to assess the strength or reliability of disease-medication associations in the drug suggestion model. Varying cutoff values for disease-medication associations, a cutoff of 3 achieved 95.14% accurate prescriptions, 5 yielded 85.42%, and 10 resulted in 72.92%. Human expert validation agreement surpassed Q-value cutoff-based agreement.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Medication Recommendations for Common Dermatological Conditions, Q-value cutoff of 5 (moderate level confidence in disease-medication associations)
Condition/subject: Dermatology medications (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 85.42% (sample size: N/A)
Comments: Drug suggestions were also evaluated using the Q-value, measure used to assess the strength or reliability of disease-medication associations in the drug suggestion model. Varying cutoff values for disease-medication associations, a cutoff of 3 achieved 95.14% accurate prescriptions, 5 yielded 85.42%, and 10 resulted in 72.92%. Human expert validation agreement surpassed Q-value cutoff-based agreement.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Medication Recommendations for Common Dermatological Conditions, Q-value cutoff of 3 (most confident disease-medication associations)
Condition/subject: Dermatology medications (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 95.14% (sample size: N/A)
Comments: Drug suggestions were also evaluated using the Q-value, measure used to assess the strength or reliability of disease-medication associations in the drug suggestion model. Varying cutoff values for disease-medication associations, a cutoff of 3 achieved 95.14% accurate prescriptions, 5 yielded 85.42%, and 10 resulted in 72.92%. Human expert validation agreement surpassed Q-value cutoff-based agreement.
Dataset: survey questions in April 2023 for drug recommendations generated by ChatGPT with data from secondary databases, that is, Taiwan's National Health Insurance Research Database and an US medical center database, and validated by dermatologists. 
Reference (2024): Iqbal U, Lee LT, Rahmanti AR, Celi LA, Li YJ. Can large language models provide secondary reliable opinion on treatment options for dermatological diseases? J Am Med Inform Assoc. 2024 May 20;31(6):1341-1347. doi: 10.1093/jamia/ocae067. PMID: 38578616; PMCID: PMC11105123.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Medication Recommendations for Common Dermatological Conditions
Condition/subject: Dermatology medications (English)
Category: Medication Recommendations and Treatment Efficacy — Clinical Practice
Validity/accuracy: 98.87% (sample size: N/A)
Comments: ChatGPT achieved a high 98.87% dermatologist approval rate for common dermatological medication recommendations. . While ChatGPT offered accurate drug advice, it occasionally included incorrect ATC codes, leading to issues like incorrect drug use and type, nonexistent codes, repeated errors, and incomplete medication codes.
Dataset: 24 text-based questions completed by five pediatric dermatologists. These included 16 single-answer, multiple-choice questions  (MCQs), two multiple-answer questions (MAQs), and six free-response, case-based questions (CBQs). MCQs and MAQs were selected from the American Board of Dermatology (ABD) 2021 Certification Sample Test and used with permission from the ABD.7 CBQs were extracted from the �Photoquiz� section of the journal Pediatric Dermatology from issues published between July 2022 and July 2023, which lie beyond ChatGPT-3.5 and 4.0's training period.
Reference (2024): Huang CY, Zhang E, Caussade MC, Brown T, Stockton Hogrogian G, Yan AC. Pediatric dermatologists versus AI bots: Evaluating the medical knowledge and diagnostic capabilities of ChatGPT. Pediatr Dermatol. 2024 Sep-Oct;41(5):831-834. doi: 10.1111/pde.15649. Epub 2024 May 9. PMID: 38721744.

Evaluation of GPT-3.5 (GPT series) — Surgery, Pediatric Dermatology
Condition/subject: Mohs surgery (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 40.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Surgery, Pediatric Dermatology
Condition/subject: Mohs surgery (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 21)

Evaluation of GPT-3.5 (GPT series) — Pediatric Dermatology (multiple choice and mutiple answer questions)
Condition/subject: Mohs surgery (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 76.20% (sample size: 21)

Evaluation of GPT-3.5 (GPT series) — Pharmacology, Pediatric Dermatology
Condition/subject: Pharmacology, Pediatric Dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 80.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Pediatric Dermatology (multiple choice and mutiple answer questions)
Condition/subject: Pediatric Dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 90.50% (sample size: 21)

Evaluation of GPT-3.5 (GPT series) — Basic Sciences, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-3.5 (GPT series) — Research Methods, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-3.5 (GPT series) — Genetics, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-3.5 (GPT series) — Connetvie Tissue Diseases, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)
Dataset: 24 text-based questions completed by five pediatric dermatologists. These included 16 single-answer, multiple-choice questions  (MCQs), two multiple-answer questions (MAQs), and six free-response, case-based questions (CBQs). MCQs and MAQs were selected from the American Board of Dermatology (ABD) 2021 Certification Sample Test and used with permission from the ABD.7 CBQs were extracted from the �Photoquiz� section of the journal Pediatric Dermatology from issues published between July 2022 and July 2023, which lie beyond ChatGPT-3.5 and 4.0's training period.
Reference (2024): Huang CY, Zhang E, Caussade MC, Brown T, Stockton Hogrogian G, Yan AC. Pediatric dermatologists versus AI bots: Evaluating the medical knowledge and diagnostic capabilities of ChatGPT. Pediatr Dermatol. 2024 Sep-Oct;41(5):831-834. doi: 10.1111/pde.15649. Epub 2024 May 9. PMID: 38721744.

Evaluation of GPT-3.5 (GPT series) — Dermopathology, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-3.5 (GPT series) — Pigmentary Lesions, Pediatric Dermatology
Condition/subject: Pigmentary disorders (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Basic Sciences, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Research Methods, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Genetics, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Connetvie Tissue Diseases, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Pharmacology, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Dermopathology, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)

Evaluation of GPT-4 (GPT series) — Pigmentary Lesions, Pediatric Dermatology
Condition/subject: Pigmentary disorders (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)
Dataset: 24 text-based questions completed by five pediatric dermatologists. These included 16 single-answer, multiple-choice questions  (MCQs), two multiple-answer questions (MAQs), and six free-response, case-based questions (CBQs). MCQs and MAQs were selected from the American Board of Dermatology (ABD) 2021 Certification Sample Test and used with permission from the ABD.7 CBQs were extracted from the �Photoquiz� section of the journal Pediatric Dermatology from issues published between July 2022 and July 2023, which lie beyond ChatGPT-3.5 and 4.0's training period.
Reference (2024): Huang CY, Zhang E, Caussade MC, Brown T, Stockton Hogrogian G, Yan AC. Pediatric dermatologists versus AI bots: Evaluating the medical knowledge and diagnostic capabilities of ChatGPT. Pediatr Dermatol. 2024 Sep-Oct;41(5):831-834. doi: 10.1111/pde.15649. Epub 2024 May 9. PMID: 38721744.

Evaluation of GPT-4 (GPT series) — Medical Billing, Pediatric Dermatology
Condition/subject: Pediatric dermatology (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 21)
Dataset: 134 responses to 18 questions about LLMs. Of 134 respondents, 87 respondents (64.9%) have used LLMs, with 45 respondents (51.7%) using the technology daily or weekly and 77 respondents (88.5%) reporting feeling extremely likely or somewhat likely to continue to use LLMs in the future. ChatGPT was the most popular LLM used (85.1%), followed by Google Bard (25.3%). 107 were post-residency level dermatology clinicials and 27 were current residents.
Reference (2024): Gui H, Rezaei SJ, Schlessinger D, Weed J, Lester J, Wongvibulsin S, Mitchell D, Ko J, Rotemberg V, Lee I, Daneshjou R. Dermatologists' Perspectives and Usage of Large Language Models in Practice: An Exploratory Survey. J Invest Dermatol. 2024 Oct;144(10):2298-2301. doi: 10.1016/j.jid.2024.03.028. Epub 2024 Apr 4. PMID: 38582369.

Evaluation of ChatGPT-3.5, ChatGPT-4, Bard, Bing Chat, BioGPT (GPT series) — Practicing Dermatologists who find LLMs extremely accurate 
Condition/subject: LLMs (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 7.00% (sample size: 86)

Evaluation of ChatGPT-3.5, ChatGPT-4, Bard, Bing Chat, BioGPT (GPT-series) — Practicing Dermatologists requiring minimal editing (<5 changes) to LLM responses
Condition/subject: LLMs (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 41.40% (sample size: 87)

Evaluation of ChatGPT-3.5, ChatGPT-4, Bard, Bing Chat, BioGPT (GPT-series) — Practicing Dermatologists extremely likely to continue using LLMs
Condition/subject: LLMs (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 51.70% (sample size: 87)

Evaluation of ChatGPT-3.5, ChatGPT-4, Bard, Bing Chat, BioGPT (GPT-series) — Practicing Dermatologists who find LLMs somewhat accurate
Condition/subject: LLMs (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 58.10% (sample size: 86)

Evaluation of ChatGPT-3.5, ChatGPT-4, Bard, Bing Chat, BioGPT (GPT series) — Practicing Dermatologists concerned about accuracy of LLMs
Condition/subject: LLMs (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 78.00% (sample size: 132)

Evaluation of ChatGPT-3.5, ChatGPT-4 (GPT series) — ChatGPT's popularity among dermatologists using LLMs
Condition/subject: ChatGPT (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 85.10% (sample size: 87)
Comments: ChatGPT was the most popular LLM used (85.1%), followed by Google Bard (25.3%). Most popular uses of LLM among respondents included asking questions about diagnoses and medical management, creating patient handouts, and writing insurance authorization letters.
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of GPT-4, ChatGPT (GPT series) — Atopic Dermatitis/Top Diagnosis Matched
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 5)
Comments: Accuracy Rate: 100%; Differential Diagnosis: 0% CC, 82% MC, 18% PC, 0% MI

Evaluation of GPT-4, ChatGPT (GPT series) — MCC, Merkel cell carcinoma/Top Diagnosis Matched
Condition/subject: Merkel Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 5)
Comments: Accuracy Rate: 100%; Differential Diagnosis: 0% CC, 27% MC, 45% PC, 27% MI

Evaluation of GPT-4, ChatGPT (GPT series) — Ecthyma/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Ecthyma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 5)
Comments: Accuracy Rate: 100%; Differential Diagnosis: 0% CC, 27% MC, 55% PC, 18% MI

Evaluation of GPT-4, ChatGPT (GPT series) — CTCL, cutaneous T-cell lymphoma/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: cutaneous T-cell lymphoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 5)
Comments: Accuracy Rate: 100%; Differential Diagnosis: 0% CC, 64% MC, 18% PC, 18% MI

Evaluation of GPT-4, ChatGPT (GPT series) — MCC, Merkel cell carcinoma/Differential Diagnosis Completely Correct or Mostly Correct
Condition/subject: Merkel Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 27.00% (sample size: 5)
Comments: 0% CC, 27% MC, 45% PC, 27% MI

Evaluation of GPT-4, ChatGPT (GPT series) — Ecthyma/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Ecthyma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 27.00% (sample size: 5)
Comments: 0% CC, 27% MC, 55% PC, 18% MI
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of ChatGPT-4 (GPT series) — Top Diagnosis/Diagnosis and Workup of Dermatologic Conditions
Condition/subject: Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 37.50% (sample size: 110)

Evaluation of ChatGPT-4 (GPT series) — BCC, basal cell carcinoma/Per-Case Inclusion
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 40.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Ecthyma/Per-Case Inclusion
Condition/subject: Ecthyma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 40.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Syphilis/Per-Case Inclusion
Condition/subject: Syphilis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 40.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — BCC, basal cell carcinoma - Correct diagnosis (top Dx)
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 40.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Ecthyma - Correct diagnosis (top Dx)
Condition/subject: Ecthyma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 40.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Syphilis - Correct diagnosis (top Dx)
Condition/subject: Syphilis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 40.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Tinea Corporis/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Tinea corporis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 45.00% (sample size: 5)
Comments: 0% CC, 45% MC, 55% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — CTCL, cutaneous T-cell lymphoma/Per-Case Inclusion
Condition/subject: cutaneous T-cell lymphoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 4)
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of ChatGPT-4 (GPT series) — CTCL, cutaneous T-cell lymphoma - Correct diagnosis (top Dx)
Condition/subject: cutaneous T-cell lymphoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 4)

Evaluation of GPT-4, ChatGPT (GPT series) — Most likely diagnosis (differential diagnoses of  dermatologic conditions described from the perspective of a non-dermatologist physician)
Condition/subject: Diagnosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 10)
Comments: The percentage of cases where ChatGPT-4's most likely diagnosis matched the top diagnosis on the dermatologists' list. This occurred in 50% of the vignettes: basal cell carcinoma, viral exanthem, syphilis, psoriasis, and melanoma

Evaluation of GPT-4, ChatGPT (GPT series) — Overall Accuracy of individual diagnosis suggested by ChatGPT-4 across all cases.
Condition/subject: Diagnostics (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 52.00% (sample size: 10)
Comments: Proxy for Accuracy: rate of accurate diagnoses and the percentage of matching diagnoses.  Overall, 52% of ChatGPT4�s diagnoses were accurate and 62% of its recommended workup suggestions were deemed completely correct by board-certified dermatologists.  The percentage of ChatGPT-4's diagnoses that were also found on the dermatologists' differential diagnosis lists: 52% across all vignettes. The percentage of correct diagnoses provided by ChatGPT-4 for each individual vignette ranged from 20% to 100%.

Evaluation of ChatGPT-4 (GPT series) — Viral exanthem/Per-Case Inclusion
Condition/subject: Viral exanthem (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — MCC, Merkel cell carcinoma/Per-Case Inclusion
Condition/subject: Merkel Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 5)
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of ChatGPT-4 (GPT series) — Tinea Corporis/Per-Case Inclusion
Condition/subject: Tinea corporis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Melanoma/Per-Case Inclusion
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Viral exanthem - Correct diagnosis (top Dx)
Condition/subject: Viral exanthem (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — MCC, Merkel cell carcinoma - Correct diagnosis (top Dx)
Condition/subject: Merkel Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Tinea Corporis - Correct diagnosis (top Dx)
Condition/subject: Tinea corporis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of ChatGPT-4 (GPT series) — Melanoma   - Correct diagnosis (top Dx)
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 60.00% (sample size: 5)

Evaluation of GPT-4, ChatGPT (GPT series) — Workup suggested (diagnostic steps a physician would take to confirm a suspected diagnosis) deemed completely correct by dermatologists
Condition/subject: Viral exanthem (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 62.00% (sample size: 10)
Comments: ChatGPT-4 was rated as providing a completely correct workup an average of 62% of the time for any given clinical vignette
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of GPT-4, ChatGPT (GPT series) — Completely correct workup of dermatological conditions
Condition/subject: Workup (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 62.00% (sample size: 110)
Comments: Out of 10 vignettes, ChatGPT-4 was rated as providing a completely correct, or mostly correct workup in all vignettes except for those describing atopic dermatitis, Merkel cell carcinoma, and ecthyma (rated by 11 dermatologists)

Evaluation of ChatGPT-4 (GPT series) — CTCL, cutaneous T-cell lymphoma/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: cutaneous T-cell lymphoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 64.00% (sample size: 5)
Comments: 0% CC, 64% MC, 18% PC, 18% MI

Evaluation of ChatGPT-4 (GPT series) — MCC, Merkel cell carcinoma - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Merkel Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 73.00% (sample size: 11)

Evaluation of ChatGPT-4 (GPT series) — Per-Case Inclusion/Differential Diagnosis (the correct diagnosis is listed somewhere in its differential diagnosis )
Condition/subject: Differential diagnosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 80.00% (sample size: 10)
Comments: Correct in Differential Diagnosis: 80%

Evaluation of ChatGPT-4 (GPT series) — Atopic Dermatitis/Differential Diagnosis Completely Correct or Mostly Correct
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 82.00% (sample size: 5)
Comments: 0% CC, 82% MC, 18% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — PsO, psoriasis/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Psoriasis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 82.00% (sample size: 5)
Comments: 0% CC, 82% MC, 18% PC, 0% MI
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of ChatGPT-4 (GPT series) — Syphilis/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Syphilis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 82.00% (sample size: 5)
Comments: 27% CC, 55% MC, 18% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — Melanoma/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 82.00% (sample size: 5)
Comments: 27% CC, 55% MC, 18% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — Atopic Dermatitis - Workup Completely or Mostly Correct
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 82.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — MCC, Merkel cell carcinoma - Workup Completely or Mostly Correct
Condition/subject: Merkel Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 82.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — Ecthyma - Differential Diagnosis Completely or Mostly Correct
Condition/subject: Ecthyma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 82.00% (sample size: 11)

Evaluation of ChatGPT-4 (GPT series) — CTCL, cutaneous T-cell lymphoma - Differential Diagnosis Completely or Mostly Correct
Condition/subject: cutaneous T-cell lymphoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 82.00% (sample size: 11)

Evaluation of ChatGPT-4 (GPT series) — Viral exanthem/Differential Diagnosis Completely Correct or Mostly Correct
Condition/subject: Viral exanthem (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 91.00% (sample size: 5)
Comments: 18% CC, 73% MC, 9% PC, 0% MI
Dataset: Ten clinical vignettes describing common dermatologic conditions from the perspective of a non-dermatologist physician, generated by students and an intern. Three board-certified dermatology faculty members independently created differential diagnosis lists for each vignette, listing three to five potential diagnoses ordered from most to least likely. These lists were validated and consolidated by a third dermatologist to form the final expert differential diagnosis list for each vignette. Eleven board-certified dermatologists rated the accuracy of ChatGPT-4's diagnoses and the appropriateness of its recommended workups using predefined Likert scales
Reference (2024): Greif C, Mpunga N, Koopman IV, Pye A, Hivnor CM, Owen JL. Evaluating the effectiveness of ChatGPT4 in the diagnosis and workup of dermatologic conditions. Dermatology Online Journal. 2024;30(4).

Evaluation of ChatGPT-4 (GPT series) — BCC, basal cell carcinoma/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 91.00% (sample size: 5)
Comments: 55% CC, 36% MC, 9% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — Ecthyma - Workup Completely or Mostly Correct
Condition/subject: Ecthyma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 91.00% (sample size: 11)
Comments: Likert scale evaluation of appropriateness of ChatGPT4�s recommended clinical workup by vignette (N=11)

Evaluation of ChatGPT-4 (GPT series) — PsO, psoriasis/Per-Case Inclusion
Condition/subject: Psoriasis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of ChatGPT-4 (GPT series) — PsO, psoriasis - Correct diagnosis (top Dx)
Condition/subject: Psoriasis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of ChatGPT-4 (GPT series) — Viral exanthem/Top Diagnosis Matched
Condition/subject: Viral exanthem (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 5)
Comments: Accuracy Rate: 60%; Differential Diagnosis: 18% CC, 73% MC, 9% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — PsO, psoriasis/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Psoriasis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 5)
Comments: Accuracy Rate: 100%; Differential Diagnosis: 0% CC, 82% MC, 18% PC, 0% MI

Evaluation of ChatGPT-4 (GPT series) — BCC, basal cell carcinoma/Differential Diagnosis Completely Completely Correct or Mostly Correct
Condition/subject: Basal Cell Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 5)
Comments: Accuracy Rate: 40% 2/5; Differential Diagnosis: 55% CC, 36% MC, 9% PC, 0% MI