Aurametrix/dermatology_longitudinal huggingface dataset:
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Custom GPT: O�Malley, A., Representative Skin Tones. 2024. (GPT series) — Skin tone 6 (NIS scale) images -  approximately corresponds to FST IV (olive or medium-dark skin tones).
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 46.01% (sample size: 100)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Custom GPT: O�Malley, A., Representative Skin Tones. 2024. (GPT series) — Skin tone 4 (NIS scale) images -  approximately corresponds to FST III (medium skin tones).
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 53.69% (sample size: 100)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Midjourney (Midjourney) — Skin tone 1 (NIS scale) images -  approximately corresponds to FST I-II (lighter skin tones).
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 56.97% (sample size: 99)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Custom GPT: O�Malley, A., Representative Skin Tones. 2024. (GPT series) — Skin tone 1 (NIS scale) images -  approximately corresponds to FST I-II (lighter skin tones).
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 59.09% (sample size: 100)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Custom GPT: O�Malley, A., Representative Skin Tones. 2024. (GPT series) — Skin tone 7 (NIS scale) images -  approximately corresponds to  FST V-VI
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 61.82% (sample size: 100)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Custom GPT: O�Malley, A., Representative Skin Tones. 2024. (GPT series) — Skin tone 5 (NIS scale) images -  approximately corresponds to FST IV (olive or medium-dark skin tones).
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 64.77% (sample size: 100)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Custom GPT: O�Malley, A., Representative Skin Tones. 2024. (GPT series) — Skin tone 3 (NIS scale) images -  approximately corresponds to FST III (medium skin tones).
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 70.75% (sample size: 100)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Dall-E (GPT series) — Skin tone 3 (NIS scale) images -  approximately corresponds to FST III (medium skin tones).
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 71.01% (sample size: 96)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 100 images of people with psoriasis generated by two standard AI models (Dall-E and Midjourney). Additionally, a custom model was developed which incorporated a prompt injection aimed at �forcing� the AI (Dall-E 3) to reflect the skin tone distribution of the US population according to the 2012 American National Election Survey. This custom model generated another set of 100 images. The skin tones in these images were assessed by three researchers using the New Immigrant Survey skin tone scale, with the median value representing each image. 
Reference (2024): O'Malley A, Veenhuizen M, Ahmed A. Ensuring Appropriate Representation in Artificial Intelligence-Generated Medical Imagery: Protocol for a Methodological Approach to Address Skin Tone Bias. JMIR AI. 2024 Nov 27;3:e58275. doi: 10.2196/58275. PMID: 39602221; PMCID: PMC11635324. // was   Ensuring appropriate representation in AI-generated medical imagery: A Methodological Approach to Address Skin Tone Bias (Preprint) DOI: 10.2196/58275 URL: https://preprints.jmir.org/preprint/58275

Evaluation of Custom GPT: O�Malley, A., Representative Skin Tones. 2024. (GPT series) — Skin tone 2 (NIS scale) images -  approximately corresponds to FST I-II (lighter skin tones).
Condition/subject: Skin of color (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 81.36% (sample size: 100)
Comments: The range of skin tones present in the images generated by both Dall-E 3 and Midjourney are significantly different to what would be expected in the US population.  CustomGPT shows a better result. Custom GPT Model with prompt injection closely aligns with the expected demographic distribution, evidenced by a lower Chi-Square statistic (17.35) and a non-significant p-value (P = .0435), indicating minimal deviation from the target distribution. The New Immigrant Survey (NIS) skin tone scale is a standardized scale used to classify and assess skin tone, primarily for research on social and demographic factors. The scale ranges from 1 to 10, where each score represents a gradation of skin tone, with 1 being the lightest skin tone and 10 being the darkest. Validity is computed as Goodness of Fit Percentage estimated based on Standardized Residuals
Dataset: 45 questions asked, the responses generated by the chatbots were then compared against established guidelines from the European Society of Cardiology, American Academy of Dermatology, and American Society of Clinical Oncology. In addition to the content, the readability of the responses was evaluated using four different readability scales: the Flesch Reading Scale, Gunning Fog Scale Level, Flesch-Kincaid Grade Level, and Dale-Chall Score.
Reference (2024): Olszewski R, Brzezinski J, Watros K, Manczak M, Owoc J, Jeziorski K. Exploring the role of AI-driven chatbots in patient care: a critical evaluation amidst healthcare staff shortages. European Heart Journal. 2024 Oct;45(Supplement_1):ehae666-3495.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Dermatology questions included into set of 45 general questions; relative readability, Flesch Reading Ease (FRES)
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 17.67% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).

Evaluation of Bard (Google's Family of LLMs) — Dermatology questions included into set of 45 general questions; relative readability, Flesch Reading Ease (FRES)
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 39.34% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).
Dataset: 45 questions asked, the responses generated by the chatbots were then compared against established guidelines from the European Society of Cardiology, American Academy of Dermatology, and American Society of Clinical Oncology. In addition to the content, the readability of the responses was evaluated using four different readability scales: the Flesch Reading Scale, Gunning Fog Scale Level, Flesch-Kincaid Grade Level, and Dale-Chall Score.
Reference (2024): Olszewski R, Brzezinski J, Watros K, Manczak M, Owoc J, Jeziorski K. Exploring the role of AI-driven chatbots in patient care: a critical evaluation amidst healthcare staff shortages. European Heart Journal. 2024 Oct;45(Supplement_1):ehae666-3495.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Dermatology questions included into set of 45 general questions; relative readability, Gunning Fog Scale (GFI) as percentage vs easy to read (10)
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 50.68% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).

Evaluation of GPT-3.5, ChatGPT (GPT series) — Dermatology questions included into set of 45 general questions; relative readability, Flesch-Kincaid Grade Level (FKGL) as percentage vs appropriatness for high school (10)
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 62.62% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).
Dataset: 45 questions asked, the responses generated by the chatbots were then compared against established guidelines from the European Society of Cardiology, American Academy of Dermatology, and American Society of Clinical Oncology. In addition to the content, the readability of the responses was evaluated using four different readability scales: the Flesch Reading Scale, Gunning Fog Scale Level, Flesch-Kincaid Grade Level, and Dale-Chall Score.
Reference (2024): Olszewski R, Brzezinski J, Watros K, Manczak M, Owoc J, Jeziorski K. Exploring the role of AI-driven chatbots in patient care: a critical evaluation amidst healthcare staff shortages. European Heart Journal. 2024 Oct;45(Supplement_1):ehae666-3495.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Dermatology questions included into set of 45 general questions; relative readability, Dale-Chall Score (DCRF) as percentage vs easy to read (7.5)
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 63.18% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).

Evaluation of Bard (Google's Family of LLMs) — Dermatology questions included into set of 45 general questions; relative readability, Gunning Fog Scale (GFI) as percentage vs easy to read (10)
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 63.40% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).
Dataset: 45 questions asked, the responses generated by the chatbots were then compared against established guidelines from the European Society of Cardiology, American Academy of Dermatology, and American Society of Clinical Oncology. In addition to the content, the readability of the responses was evaluated using four different readability scales: the Flesch Reading Scale, Gunning Fog Scale Level, Flesch-Kincaid Grade Level, and Dale-Chall Score.
Reference (2024): Olszewski R, Brzezinski J, Watros K, Manczak M, Owoc J, Jeziorski K. Exploring the role of AI-driven chatbots in patient care: a critical evaluation amidst healthcare staff shortages. European Heart Journal. 2024 Oct;45(Supplement_1):ehae666-3495.

Evaluation of PiAI (PiAI) — Dermatology questions included into set of 45 general questions; relative readability, Flesch-Kincaid Grade Level (FKGL) as percentage vs appropriatness for high school (10)
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 71.32% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).

Evaluation of Bard (Google's Family of LLMs) — Dermatology questions included into set of 45 general questions; relative readability, Dale-Chall Score (DCRF) as percentage vs easy to read (7.5)
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 73.24% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).
Dataset: 45 questions asked, the responses generated by the chatbots were then compared against established guidelines from the European Society of Cardiology, American Academy of Dermatology, and American Society of Clinical Oncology. In addition to the content, the readability of the responses was evaluated using four different readability scales: the Flesch Reading Scale, Gunning Fog Scale Level, Flesch-Kincaid Grade Level, and Dale-Chall Score.
Reference (2024): Olszewski R, Brzezinski J, Watros K, Manczak M, Owoc J, Jeziorski K. Exploring the role of AI-driven chatbots in patient care: a critical evaluation amidst healthcare staff shortages. European Heart Journal. 2024 Oct;45(Supplement_1):ehae666-3495.

Evaluation of ChatGPT (GPT series) — Dermatology questions included into set of 45 general questions; percent valid response
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 100.00% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).

Evaluation of Copilot, Microsoft (GPT series) — Dermatology questions included into set of 45 general questions; percent valid response
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 100.00% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).
Dataset: 45 questions asked, the responses generated by the chatbots were then compared against established guidelines from the European Society of Cardiology, American Academy of Dermatology, and American Society of Clinical Oncology. In addition to the content, the readability of the responses was evaluated using four different readability scales: the Flesch Reading Scale, Gunning Fog Scale Level, Flesch-Kincaid Grade Level, and Dale-Chall Score.
Reference (2024): Olszewski R, Brzezinski J, Watros K, Manczak M, Owoc J, Jeziorski K. Exploring the role of AI-driven chatbots in patient care: a critical evaluation amidst healthcare staff shortages. European Heart Journal. 2024 Oct;45(Supplement_1):ehae666-3495.

Evaluation of PiAI (PiAI) — Dermatology questions included into set of 45 general questions; percent valid response
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: 100.00% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).

Evaluation of Bard (Google's Family of LLMs) — Dermatology questions included into set of 45 general questions; percent valid response
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: Less than 100% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).
Dataset: 45 questions asked, the responses generated by the chatbots were then compared against established guidelines from the European Society of Cardiology, American Academy of Dermatology, and American Society of Clinical Oncology. In addition to the content, the readability of the responses was evaluated using four different readability scales: the Flesch Reading Scale, Gunning Fog Scale Level, Flesch-Kincaid Grade Level, and Dale-Chall Score.
Reference (2024): Olszewski R, Brzezinski J, Watros K, Manczak M, Owoc J, Jeziorski K. Exploring the role of AI-driven chatbots in patient care: a critical evaluation amidst healthcare staff shortages. European Heart Journal. 2024 Oct;45(Supplement_1):ehae666-3495.

Evaluation of GPT-3.5, ChatSpot (GPT series) — Dermatology questions included into set of 45 general questions; percent valid response
Condition/subject: Readability (English)
Category: Patient Education Materials and Readability Studies — Patient Education
Validity/accuracy: Less than 100% (sample size: 45)
Comments: Only three chatbots (Microsoft Copilot, PiAI, and ChatGPT) responded to all 45 questions. In terms of readability, the Flesch Reading Ease Scale scores ranged from 17.67 (ChatGPT) to 39.34 (Bard), indicating the relative complexity of the responses. The Flesch-Kincaid Grade Level, which reflects the academic grade level required to comprehend the text, ranged from 14.02 (PiAI) to 15.97 (ChatGPT). The Gunning Fog Scale Level, another measure of readability, varied from 15.77 (Bard) to 19.73 (ChatGPT). Lastly, the Dale-Chall Score, which assesses the understandability of the text, ranged from 10.24 (Bard) to 11.87 (ChatGPT).
Dataset: 30 clinical scenarios involving common and rare cutaneous tumors across anatomic sites presented to ChatGPT to determine whether wide local excision (WLE) or Mohs surgery (MS) were appropriate treatment options, correlating recommendations with MS appropriate use criteria (AUC)
Reference (2023): O'Hern K, Yang E, Vidal NY. ChatGPT underperforms in triaging appropriate use of Mohs surgery for cutaneous neoplasms. JAAD Int. 2023 Jun 9;12:168-170. doi: 10.1016/j.jdin.2023.06.002. PMID: 37404248; PMCID: PMC10316650.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — DPTE; Location: Buttock
Condition/subject: DPTE (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: AUC: 3; Decision: Incorrect

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — EMPD; Location: Perineum
Condition/subject: EMPD (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: AUC: 8; Decision: Incorrect

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Leiomyosarcoma; Location: Hand
Condition/subject: Leiomyosarcoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: AUC: 8; Decision: Incorrect

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Melanoma; Location: Chin
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: AUC: 7; Decision: Incorrect

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — MCC; Location: Cheek
Condition/subject: MCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: AUC: 7; Decision: Incorrect

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — SCC; Location: Neck
Condition/subject: SCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: AUC: 8; Decision: Incorrect

Evaluation of GPT-3.5, ChatGPT (GPT series) — SCC; Location: Eyebrow
Condition/subject: SCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: AUC: 3; Decision: Incorrect

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Angiosarcoma; Location: Lip
Condition/subject: Angiosarcoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 1)
Comments: AUC: 5; Decision: Not Assessed

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — BCC; Location: Forearm
Condition/subject: BCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 1)
Comments: AUC: 6; Decision: Not Assessed

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — LMM; Location: Trunk
Condition/subject: LMM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 1)
Comments: AUC: 4; Decision: Not Assessed
Dataset: 30 clinical scenarios involving common and rare cutaneous tumors across anatomic sites presented to ChatGPT to determine whether wide local excision (WLE) or Mohs surgery (MS) were appropriate treatment options, correlating recommendations with MS appropriate use criteria (AUC)
Reference (2023): O'Hern K, Yang E, Vidal NY. ChatGPT underperforms in triaging appropriate use of Mohs surgery for cutaneous neoplasms. JAAD Int. 2023 Jun 9;12:168-170. doi: 10.1016/j.jdin.2023.06.002. PMID: 37404248; PMCID: PMC10316650.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — LMM; Location: Helix
Condition/subject: LMM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 1)
Comments: AUC: NA; Decision: Not Assessed

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — LMM; Location: Foot
Condition/subject: LMM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 1)
Comments: AUC: NA; Decision: Not Assessed

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Melanoma; Location: Shoulder
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 1)
Comments: AUC: 5; Decision: Not Assessed

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Melanoma; Location: Cheek
Condition/subject: Melanoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 1)
Comments: AUC: NA; Decision: Not Assessed

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — SCC; Location: Thigh
Condition/subject: SCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 50.00% (sample size: 1)
Comments: AUC: 6; Decision: Not Assessed

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Cutaneous tumors, treatment, surgery, appropriatness of wide local excision (WLE) vs Mohs surgery (MS) recommendation
Condition/subject: Cutaneous tumors (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 68.00% (sample size: 30)
Comments: ChatGPT demonstrated 68% (n = 15) congruence with the MS AUC when triaging surgical management of 22 clinical scenarios characterized as clearly appropriate or inappropriate by the MS AUC. For all 5 cases characterized as indeterminate by the MS AUC, ChatGPT recommended against MS. For 3 cases of invasive melanoma, ChatGPT recommended MS for lentigo maligna melanoma of the helix, while recommending WLE for superficial spreading melanoma of the cheek and lentigo maligna melanoma of the dorsal foot.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — AFX; Location: Neck
Condition/subject: AFX (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 8; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — BCC; Location: Eyelid
Condition/subject: BCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 9; Decision: Correct
Dataset: 30 clinical scenarios involving common and rare cutaneous tumors across anatomic sites presented to ChatGPT to determine whether wide local excision (WLE) or Mohs surgery (MS) were appropriate treatment options, correlating recommendations with MS appropriate use criteria (AUC)
Reference (2023): O'Hern K, Yang E, Vidal NY. ChatGPT underperforms in triaging appropriate use of Mohs surgery for cutaneous neoplasms. JAAD Int. 2023 Jun 9;12:168-170. doi: 10.1016/j.jdin.2023.06.002. PMID: 37404248; PMCID: PMC10316650.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — BCC; Location: Scalp
Condition/subject: BCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 9; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — BCC; Location: Nose
Condition/subject: BCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 7; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — BCC; Location: Back
Condition/subject: BCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 1; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Bowenoid Papules; Location: Penis
Condition/subject: Bowenoid Papules (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 3; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — DFSP; Location: Upper Arm
Condition/subject: DFSP (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 9; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — DFSP; Location: Upper Arm
Condition/subject: DFSP (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 9; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — LM; Location: Trunk
Condition/subject: LMM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 7; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — LM; Location: Helix
Condition/subject: LMM (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 8; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — SCC; Location: Shoulder
Condition/subject: SCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 8; Decision: Correct

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — SCC; Location: Neck
Condition/subject: SCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 7; Decision: Correct
Dataset: 30 clinical scenarios involving common and rare cutaneous tumors across anatomic sites presented to ChatGPT to determine whether wide local excision (WLE) or Mohs surgery (MS) were appropriate treatment options, correlating recommendations with MS appropriate use criteria (AUC)
Reference (2023): O'Hern K, Yang E, Vidal NY. ChatGPT underperforms in triaging appropriate use of Mohs surgery for cutaneous neoplasms. JAAD Int. 2023 Jun 9;12:168-170. doi: 10.1016/j.jdin.2023.06.002. PMID: 37404248; PMCID: PMC10316650.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — SCC; Location: Shoulder
Condition/subject: SCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 7; Decision: Correct

Evaluation of GPT-3.5, ChatGPT (GPT series) — SCC; Location: Abdomen
Condition/subject: SCC (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 3; Decision: Correct

Evaluation of GPT-3.5, ChatGPT (GPT series) — Sebaceous Carcinoma; Location: Eyebrow
Condition/subject: Sebaceous Carcinoma (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: AUC: 9; Decision: Correct