Aurametrix/dermatology_longitudinal huggingface dataset:
Dataset: Questions from Chinese NMLE, NPLE and NNLE from year 2017 to 2021. In NMLE and NPLE, each exam consists of 4 units, while in NNLE, each exam consists of 2 units. The questions with figures, tables or chemical structure were manually identified and excluded by clinician. 600 multiple-choice questions (MCQs) Available at https://github.com/zonghui0228/LLM-Chinese-NMLE
Reference (2024): Zong H, Li J, Wu E, Wu R, Lu J, Shen B. Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses. BMC Med Educ. 2024 Feb 14;24(1):143. doi: 10.1186/s12909-024-05125-7. PMID: 38355517; PMCID: PMC10868058.

Evaluation of GPT-3.5 (GPT series) — Essential knowledge, Accuracy
Condition/subject: Dermatology questions in the 2021 NMLE exam, National Medical Licensing Examination, Certification (Chinese)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 15)
Comments: ChatGPT excelled in clinical epidemiology, human parasitology, and dermatology, with all questions answered correctly. However, the model faltered in subcjects such as pathology, pathophysiology, public health regulations, physiology, and anatomy, with the proportion of correct answers was less than 0.5 . Overall, ChatGPT failed to pass the accuracy threshold of 0.6 in any of the three types of examinations over the five years. Specifically, in the NMLE, the highest recorded accuracy was 0.5467, which was attained in both 2018 and 2021. In the NPLE, the highest accuracy was 0.5599 in 2017. In the NNLE, the most impressive result was shown in 2017, with an accuracy of 0.5897, which is also the highest accuracy in our entire evaluation. ChatGPT�s performance showed no significant difference in different units, but significant difference in different question types. ChatGPT performed well in a range of subject areas, including clinical epidemiology, human parasitology, and dermatology, as well as in various medical topics such as molecules, health management and prevention, diagnosis and screening.
Dataset: Skin Cancer ISIC: 2357 images; Skin Cancer MNIST: HAM10000: 10015 images; Skin Cancer Malignant vs. Benign: 1800 images
Reference (2024): Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang, Potential of multimodal large language models for data mining of medical images and free-text reports, Meta-Radiology, Volume 2, Issue 4, 2024, 100103, ISSN 2950-1628, https://doi.org/10.1016/j.metrad.2024.100103. (https://www.sciencedirect.com/science/article/pii/S2950162824000572)

Evaluation of Claude-3-Opus (Claude series) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix

Evaluation of Gemini-1.5-Flash (Google's Family of LLMs) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix

Evaluation of GPT-4o (GPT series) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix

Evaluation of GPT-4-Turbo (GPT series) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix

Evaluation of LLAMA3 (LLaMA Series by Meta) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix

Evaluation of Yi-Large (Yi series) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix
Dataset: Skin Cancer ISIC: 2357 images; Skin Cancer MNIST: HAM10000: 10015 images; Skin Cancer Malignant vs. Benign: 1800 images
Reference (2024): Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang, Potential of multimodal large language models for data mining of medical images and free-text reports, Meta-Radiology, Volume 2, Issue 4, 2024, 100103, ISSN 2950-1628, https://doi.org/10.1016/j.metrad.2024.100103. (https://www.sciencedirect.com/science/article/pii/S2950162824000572)

Evaluation of Yi-Large-Turbo (Yi series) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix

Evaluation of Gemini-1.0-Pro-Vision (Google's Family of LLMs) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.33% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix

Evaluation of Gemini-1.5-Pro (Google's Family of LLMs) — Image-based recognition of skin disease malignancy, Accuracy
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.33% (sample size: 3)
Comments: Accuracy/Validity is based on three cases presented in the paper and its appendix

Evaluation of Claude-3-Opus (Claude series) — Benign Lesion: dermoscopic image-based diagnosis
Condition/subject: Actinic Keratosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Actinic Keratosis was given.  Claude-3-Opus' Dx: Based on the image provided, there appears to be a lesion present on the skin. The lesion has the following characteristics:    Irregular, asymmetrical shape    Mottled, patchy coloration ranging from light to dark brown    Slightly raised and rough texture compared to surrounding skin. Given these visual characteristics, this lesion is most consistent with a pigmented nevus, commonly known as a mole. Moles are benign growths formed by clusters of melanocytes, the pigment-producing cells in the skin. While the majority of moles are harmless, it's important to monitor any changes in size, shape, color, or texture. Sudden or progressive changes could be potential signs of malignant transformation, such as melanoma. I recommend performing regular self-examinations and having any concerning moles evaluated by a dermatologist for a more definitive diagnosis and appropriate management.
Dataset: Skin Cancer ISIC: 2357 images; Skin Cancer MNIST: HAM10000: 10015 images; Skin Cancer Malignant vs. Benign: 1800 images
Reference (2024): Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang, Potential of multimodal large language models for data mining of medical images and free-text reports, Meta-Radiology, Volume 2, Issue 4, 2024, 100103, ISSN 2950-1628, https://doi.org/10.1016/j.metrad.2024.100103. (https://www.sciencedirect.com/science/article/pii/S2950162824000572)

Evaluation of Claude-3-Opus (Claude series) — Benign Lesion: dermoscopic image-based diagnosis
Condition/subject: Benign Lesion (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Benign Lesion was given. Claude-3-Opus' Dx: Based on the provided image, I cannot definitively determine if the lesion is benign or malignant. The image shows a small, dark irregular lesion on the skin, which has some concerning features that could potentially indicate a malignant melanoma, such as asymmetry and color variation. However, additional information and tests would be needed to make a definitive diagnosis, such as the size of the lesion, any recent changes, and the results of a biopsy. My recommendation would be to have this lesion evaluated in-person by a dermatologist as soon as possible for a thorough assessment and potential biopsy, as early detection is crucial if this does turn out to be a melanoma. Let me know if you have any other questions!

Evaluation of Gemini-1.0-Pro-Vision (Google's Family of LLMs) — Actinic Keratosis: dermoscopic image-based diagnosis
Condition/subject: Actinic Keratosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Actinic Keratosis was given. Gemini-1.0-Pro-Vision' Dx: There is a pigmented lesion on the patient's back. It is round, with a diameter of about 6 mm. The color is light brown, and the borders are irregular. The surface is smooth, and there is no scaling or crusting. This lesion is most likely a seborrheic keratosis. Seborrheic keratoses are common, benign growths that are found in older adults. They are caused by an overgrowth of the cells in the epidermis, the outer layer of the skin. Seborrheic keratoses are typically harmless, and they do not require treatment. However, they can sometimes be mistaken for skin cancer, so it is important to have them checked by a dermatologist if you are concerned.
Dataset: Skin Cancer ISIC: 2357 images; Skin Cancer MNIST: HAM10000: 10015 images; Skin Cancer Malignant vs. Benign: 1800 images
Reference (2024): Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang, Potential of multimodal large language models for data mining of medical images and free-text reports, Meta-Radiology, Volume 2, Issue 4, 2024, 100103, ISSN 2950-1628, https://doi.org/10.1016/j.metrad.2024.100103. (https://www.sciencedirect.com/science/article/pii/S2950162824000572)

Evaluation of Gemini-1.5-Flash (Google's Family of LLMs) — Actinic Keratosis: dermoscopic image-based diagnosis
Condition/subject: Actinic Keratosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Actinic Keratosis was given. Gemini-1.5-Flash' Dx: There is a lesion in the image; it appears to be a seborrheic keratosis, a common, noncancerous skin growth. It is characterized by its brown, waxy, and slightly raised appearance.

Evaluation of Gemini-1.5-Flash (Google's Family of LLMs) — Benign Lesion: dermoscopic image-based diagnosis
Condition/subject: Benign Lesion (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Benign Lesion was given. Gemini-1.5-Flash' Dx: Malignant. The lesion has irregular borders and a variety of colors, suggesting melanoma. A biopsy is needed for definitive diagnosis.

Evaluation of Gemini-1.5-Pro (Google's Family of LLMs) — Actinic Keratosis: dermoscopic image-based diagnosis
Condition/subject: Actinic Keratosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Actinic Keratosis was given. Gemini-1.5-Pro' Dx: Yes, there is a lesion in the picture. This is a pigmented lesion with irregular borders and color variation, suspicious for melanoma. It needs to be biopsied.

Evaluation of GPT-4o (GPT series) — Actinic Keratosis: dermoscopic image-based diagnosis
Condition/subject: Actinic Keratosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Actinic Keratosis was given. GPT-4o' Dx: Based on the dermoscopic image provided:    Findings: There is an irregular, pigmented lesion with varying shades of brown. The lesion exhibits an asymmetric pattern with irregular borders. There are areas of pigmentation that vary in color and density.    Assessment: The appearance is concerning for a potential melanoma, given the asymmetry, irregular borders, and heterogeneous pigmentation. Further evaluation, including a biopsy, is recommended to confirm the diagnosis and rule out malignancy.    Conclusion: The lesion shows features suggestive of melanoma, warranting prompt further investigation.
Dataset: Skin Cancer ISIC: 2357 images; Skin Cancer MNIST: HAM10000: 10015 images; Skin Cancer Malignant vs. Benign: 1800 images
Reference (2024): Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang, Potential of multimodal large language models for data mining of medical images and free-text reports, Meta-Radiology, Volume 2, Issue 4, 2024, 100103, ISSN 2950-1628, https://doi.org/10.1016/j.metrad.2024.100103. (https://www.sciencedirect.com/science/article/pii/S2950162824000572)

Evaluation of GPT-4o (GPT series) — Benign Lesion: dermoscopic image-based diagnosis
Condition/subject: Benign Lesion (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Benign Lesion was given. GPT-4o' Dx: �	Assessment: Malignant.�	Reasons: The lesion is asymmetrical with irregular, uneven borders. The pigmentation is uneven with multiple shades of brown. The overall appearance is suggestive of melanoma due to these atypical features.�	Immediate further investigation, including a biopsy, is recommended to confirm the diagnosis and plan appropriate treatment.

Evaluation of GPT-4-Turbo (GPT series) — Actinic Keratosis: dermoscopic image-based diagnosis
Condition/subject: Actinic Keratosis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Actinic Keratosis was given. GPT-4-Turbo' Dx: The image provided appears to be a dermoscopy photo, a technique used in dermatology to examine skin lesions with a dermatoscope. Here�s a professional evaluation based on the image: Observation: The lesion depicted in the image shows a complex pattern with multiple colors, including shades of brown and black, and a pinkish hue in the background. There are irregular pigment networks and possible structural changes within the lesion; Assessment: The presence of multiple colors and irregular structures suggests this could be a melanocytic lesion. The asymmetry and color variation may raise a concern for melanoma, a type of skin cancer that arises from melanocytes.;     Diagnosis: This lesion has characteristics that could be indicative of melanoma, including asymmetry, border irregularity, color variation, and a diameter that seems larger than 6 mm (based on the scale shown).;     Recommendations: A clinical evaluation is necessary to assess the lesion further. I recommend an excisional biopsy to ensure accurate diagnosis. Histopathological examination of the biopsy will confirm whether this is benign, like a dysplastic nevus, or malignant, such as melanoma.;     Conclusion: Due to the suspicious nature of the lesion seen in the dermoscopy image, prompt dermatological consultation and possible biopsy are advised to rule out melanoma and to guide appropriate management.
Dataset: Skin Cancer ISIC: 2357 images; Skin Cancer MNIST: HAM10000: 10015 images; Skin Cancer Malignant vs. Benign: 1800 images
Reference (2024): Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang, Potential of multimodal large language models for data mining of medical images and free-text reports, Meta-Radiology, Volume 2, Issue 4, 2024, 100103, ISSN 2950-1628, https://doi.org/10.1016/j.metrad.2024.100103. (https://www.sciencedirect.com/science/article/pii/S2950162824000572)

Evaluation of GPT-4-Turbo (GPT series) — Benign Lesion: dermoscopic image-based diagnosis
Condition/subject: Benign Lesion (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1)
Comments: Image of Benign Lesion was given. GPT-4-Turbo' Dx: Based on the dermoscopy image provided, here is a professional evaluation of the lesion:    Observation:    The lesion exhibits a dense cluster of dark brown to black pigmentation, which is irregularly shaped and unevenly distributed. The background skin shows a uniform coloration with no visible structures.    Assessment:    The lesion shows characteristics concerning for a melanocytic nevus (mole) or melanoma, particularly due to its irregular shape and uneven color distribution.    Diagnosis:    Given the visual complexity and irregularity, the lesion could potentially be malignant. It exhibits features suggestive of melanoma, such as asymmetry, border irregularity, multiple colors, and a diameter that appears to be less than 6 mm but irregular.    Recommendations:        An excisional biopsy is highly recommended for accurate histopathological diagnosis. This will determine whether the lesion is benign (like a dysplastic nevus) or malignant (melanoma)      Comprehensive skin examination and possibly total body skin photography for monitoring are advised, especially if the biopsy confirms a melanoma.   Conclusion:    The lesion is suspicious for melanoma based on dermoscopy characteristics. Immediate dermatological action in the form of a biopsy is imperative to confirm the diagnosis and guide further treatment.

Evaluation of Gemini-1.0-Pro-Vision (Google's Family of LLMs) — Benign Lesion: dermoscopic image-based diagnosis
Condition/subject: Benign Lesion (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: Image of Benign Lesion was given. Gemini-1.0-Pro-Vision' Dx: The tumor is benign. The tumor is a compound nevus. It is a benign tumor that is made up of both melanocytes and keratinocytes. Melanocytes are the cells that produce melanin, which is the pigment that gives skin its color. Keratinocytes are the cells that make up the outer layer of the skin. Compound nevi are usually round or oval and are brown or black in color. They are often found on the face, neck, and arms. Compound nevi are not cancerous and do not need to be treated.
Dataset: Skin Cancer ISIC: 2357 images; Skin Cancer MNIST: HAM10000: 10015 images; Skin Cancer Malignant vs. Benign: 1800 images
Reference (2024): Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang, Potential of multimodal large language models for data mining of medical images and free-text reports, Meta-Radiology, Volume 2, Issue 4, 2024, 100103, ISSN 2950-1628, https://doi.org/10.1016/j.metrad.2024.100103. (https://www.sciencedirect.com/science/article/pii/S2950162824000572)

Evaluation of Gemini-1.5-Pro (Google's Family of LLMs) — Benign Lesion: dermoscopic image-based diagnosis
Condition/subject: Benign Lesion (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 100.00% (sample size: 1)
Comments: Image of Benign Lesion was given. Gemini-1.5-Pro' Dx: Benign. This lesion demonstrates symmetry, a well-defined border, and uniform color, all features of benign lesions.

Evaluation of Gemini-pro-series, Google (Google's Family of LLMs) — Image-based recognition of the exact type of skin disease, lesion segmentation and anatomical localization
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: Positive direction of the effect (sample size: 14172)
Comments: GPT-Series models were most proficient in lesion segmentation and anatomical localization, while Gemini-Series models excelled in report generation and lesion detection in dermascopic images.

Evaluation of GPT-4-series;OpenAI (GPT series) — Image-based recognition of the exact type of skin disease, lesion detection and report generation
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: Positive direction of the effect (sample size: 14172)
Comments: GPT-Series models were most proficient in lesion segmentation and anatomical localization, while Gemini-Series models excelled in report generation and lesion detection in dermascopic images.

Evaluation of Multimodal Large Language Models (MLLM) (Multimodal Large Language Models) — Image-based recognition of the exact type of skin disease
Condition/subject: Atopic dermatitis (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 1800)
Comments: None of the models accurately identified the type of skin disease afflicting the patients. This deficiency may stem from the sensitivity of the skin disease images or the insufficient training of the large language model on this particular category of diseases. Numerical accuracy metrics for the medical image tasks are not explicitly mentioned. Instead, the paper offers a qualitative assessment of the performance of various multimodal large language models (MLLMs), including the Gemini series and GPT series, across different medical imaging categories and tasks.
Dataset: Skin Cancer ISIC: 2357 images; Skin Cancer MNIST: HAM10000: 10015 images; Skin Cancer Malignant vs. Benign: 1800 images
Reference (2024): Yutong Zhang, Yi Pan, Tianyang Zhong, Peixin Dong, Kangni Xie, Yuxiao Liu, Hanqi Jiang, Zihao Wu, Zhengliang Liu, Wei Zhao, Wei Zhang, Shijie Zhao, Tuo Zhang, Xi Jiang, Dinggang Shen, Tianming Liu, Xin Zhang, Potential of multimodal large language models for data mining of medical images and free-text reports, Meta-Radiology, Volume 2, Issue 4, 2024, 100103, ISSN 2950-1628, https://doi.org/10.1016/j.metrad.2024.100103. (https://www.sciencedirect.com/science/article/pii/S2950162824000572)

Evaluation of Multimodal Large Language Models (MLLM) (Multimodal Large Language Models) — Image-based recognition of the exact type of skin disease
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 2357)
Comments: None of the models accurately identified the type of skin disease afflicting the patients. This deficiency may stem from the sensitivity of the skin disease images or the insufficient training of the large language model on this particular category of diseases. Numerical accuracy metrics for the medical image tasks are not explicitly mentioned. Instead, the paper offers a qualitative assessment of the performance of various multimodal large language models (MLLMs), including the Gemini series and GPT series, across different medical imaging categories and tasks.

Evaluation of Multimodal Large Language Models (MLLM) (Multimodal Large Language Models) — Image-based recognition of the exact type of skin disease
Condition/subject: Skin cancer, skin disease (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 0.00% (sample size: 10015)
Comments: None of the models accurately identified the type of skin disease afflicting the patients. This deficiency may stem from the sensitivity of the skin disease images or the insufficient training of the large language model on this particular category of diseases. Numerical accuracy metrics for the medical image tasks are not explicitly mentioned. Instead, the paper offers a qualitative assessment of the performance of various multimodal large language models (MLLMs), including the Gemini series and GPT series, across different medical imaging categories and tasks.
Dataset: 20 questions of patients� greatest concerns (from published literature), covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice.
Reference (2024): Yan S, Du D, Liu X, Dai Y, Kim MK, Zhou X, Wang L, Zhang L, Jiang X. Assessment of the Reliability and Clinical Applicability of ChatGPT's Responses to Patients' Common Queries About Rosacea. Patient Prefer Adherence. 2024 Jan 31;18:249-253. doi: 10.2147/PPA.S444928. PMID: 38313827; PMCID: PMC10838492.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Special Manifestations of Rosacea: Are the medications used to treat skin rosacea effective for eye symptoms?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 80.00% (sample size: 3)
Comments: worst score in the Rosacea dataset

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Skin Care: Does using skincare products worsen the skin symptoms of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 86.67% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Skin Care: How to choose a moisturizer for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 86.67% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Treatment: How to manage adverse reactions during rosacea treatment?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 88.89% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Special Manifestations of Rosacea: What is the correct approach to treating ocular rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 88.89% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Treatment: What kind of results can be expected from intense pulsed light therapy for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 91.11% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Triggers and Diet: Triggers of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 91.11% (sample size: 3)
Dataset: 20 questions of patients� greatest concerns (from published literature), covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice.
Reference (2024): Yan S, Du D, Liu X, Dai Y, Kim MK, Zhou X, Wang L, Zhang L, Jiang X. Assessment of the Reliability and Clinical Applicability of ChatGPT's Responses to Patients' Common Queries About Rosacea. Patient Prefer Adherence. 2024 Jan 31;18:249-253. doi: 10.2147/PPA.S444928. PMID: 38313827; PMCID: PMC10838492.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Rosacea Special manifestations of rosacea, Reliability of Recommendations
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 92.22% (sample size: 12)
Comments: Reliability

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Treatment: How to manage adverse reactions during rosacea treatment?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 92.59% (sample size: 3)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Rosacea Skin care, Reliability of Recommendations
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 92.78% (sample size: 12)
Comments: Reliability

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Treatment: How to avoid the side effects of long-term oral antibiotic (such as doxycycline) treatment for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 93.33% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea Comon Patient Queries
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 94.60% (sample size: 60)
Comments: 94.6% reliability in general and range from 92.22% to 97.78% among four categories

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Rosacea Treatment, Reliability of Recommendations
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 95.28% (sample size: 24)
Comments: Reliability

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Treatment: How to alleviate the persistent erythema of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 95.56% (sample size: 3)
Dataset: 20 questions of patients� greatest concerns (from published literature), covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice.
Reference (2024): Yan S, Du D, Liu X, Dai Y, Kim MK, Zhou X, Wang L, Zhang L, Jiang X. Assessment of the Reliability and Clinical Applicability of ChatGPT's Responses to Patients' Common Queries About Rosacea. Patient Prefer Adherence. 2024 Jan 31;18:249-253. doi: 10.2147/PPA.S444928. PMID: 38313827; PMCID: PMC10838492.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Treatment: How effective are the treatments for different subtypes of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 95.56% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Treatment: What level of efficacy can be achieved with current rosacea treatments?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 96.30% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Skin Care: Does using skincare products worsen the skin symptoms of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 96.30% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Special Manifestations of Rosacea: Are the medications used to treat skin rosacea effective for eye symptoms?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 96.30% (sample size: 3)

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Rosacea Triggers and diet, Reliability of Recommendations
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 97.78% (sample size: 24)
Comments: Reliability

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Treatment: What level of efficacy can be achieved with current rosacea treatments?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 97.78% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Skin Care: Which brand of cosmetics can be used for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 97.78% (sample size: 3)
Dataset: 20 questions of patients� greatest concerns (from published literature), covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice.
Reference (2024): Yan S, Du D, Liu X, Dai Y, Kim MK, Zhou X, Wang L, Zhang L, Jiang X. Assessment of the Reliability and Clinical Applicability of ChatGPT's Responses to Patients' Common Queries About Rosacea. Patient Prefer Adherence. 2024 Jan 31;18:249-253. doi: 10.2147/PPA.S444928. PMID: 38313827; PMCID: PMC10838492.

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Rosacea Treatment, Clinical Applicability of Recommendations
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 98.61% (sample size: 12)
Comments: Clinical Applicability

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Rosacea Skin care, Clinical Applicability of Recommendations
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 99.07% (sample size: 12)
Comments: Clinical Applicability

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Rosacea: Special manifestations, Clinical Applicability of Recommendations
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 99.07% (sample size: 12)
Comments: Clinical Applicability

Evaluation of GPT-3.5 (ChatGPT) (GPT series) — Rosacea Triggers and Diet, Clinical Applicability of Recommendations
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 12)
Comments: Clinical Applicability

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Treatment: How can one stay updated on the clinical progress of rosacea in a timely manner?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Treatment: How can one determine the current stage of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Triggers and Diet: What dietary considerations should be taken into account for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Dataset: 20 questions of patients� greatest concerns (from published literature), covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice.
Reference (2024): Yan S, Du D, Liu X, Dai Y, Kim MK, Zhou X, Wang L, Zhang L, Jiang X. Assessment of the Reliability and Clinical Applicability of ChatGPT's Responses to Patients' Common Queries About Rosacea. Patient Prefer Adherence. 2024 Jan 31;18:249-253. doi: 10.2147/PPA.S444928. PMID: 38313827; PMCID: PMC10838492.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Triggers and Diet: What is the impact of alkaline and acidic foods on rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Triggers and Diet: Are vitamins or other nutritional supplements beneficial for treating rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Skin Care: Are the currently popular skincare products helpful for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Special Manifestations of Rosacea: What should be done if someone is socially isolated due to rhinophyma rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Content Reliability: Special Manifestations of Rosacea: Does rosacea have any comorbidities with other systems, such as digestive system diseases?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Treatment: How to alleviate the persistent erythema of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Treatment: How to avoid the side effects of long-term oral antibiotic (such as doxycycline) treatment for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Dataset: 20 questions of patients� greatest concerns (from published literature), covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice.
Reference (2024): Yan S, Du D, Liu X, Dai Y, Kim MK, Zhou X, Wang L, Zhang L, Jiang X. Assessment of the Reliability and Clinical Applicability of ChatGPT's Responses to Patients' Common Queries About Rosacea. Patient Prefer Adherence. 2024 Jan 31;18:249-253. doi: 10.2147/PPA.S444928. PMID: 38313827; PMCID: PMC10838492.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Treatment: What kind of results can be expected from intense pulsed light therapy for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Treatment: How effective are the treatments for different subtypes of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Treatment: How can one stay updated on the clinical progress of rosacea in a timely manner?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Treatment: How can one determine the current stage of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Triggers and Diet: Triggers of rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Triggers and Diet: What dietary considerations should be taken into account for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Triggers and Diet: What is the impact of alkaline and acidic foods on rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)
Dataset: 20 questions of patients� greatest concerns (from published literature), covering four main categories: treatment, triggers and diet, skincare, and special manifestations of rosacea. Each question was inputted into ChatGPT separately for three rounds of question-and-answer conversations. The generated answers will be evaluated by three experienced dermatologists with postgraduate degrees and over five years of clinical experience in dermatology, to assess their reliability and applicability for clinical practice.
Reference (2024): Yan S, Du D, Liu X, Dai Y, Kim MK, Zhou X, Wang L, Zhang L, Jiang X. Assessment of the Reliability and Clinical Applicability of ChatGPT's Responses to Patients' Common Queries About Rosacea. Patient Prefer Adherence. 2024 Jan 31;18:249-253. doi: 10.2147/PPA.S444928. PMID: 38313827; PMCID: PMC10838492.

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Triggers and Diet: Are vitamins or other nutritional supplements beneficial for treating rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Skin Care: How to choose a moisturizer for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Skin Care: Which brand of cosmetics can be used for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Skin Care: Are the currently popular skincare products helpful for rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Special Manifestations of Rosacea: What is the correct approach to treating ocular rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Special Manifestations of Rosacea: What should be done if someone is socially isolated due to rhinophyma rosacea?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)

Evaluation of GPT-3.5, ChatGPT (GPT series) — Rosacea: Clinical Applicability: Special Manifestations of Rosacea: Does rosacea have any comorbidities with other systems, such as digestive system diseases?
Condition/subject: Rosacea (English)
Category: Dermatological Conditions and Management — Clinical Practice
Validity/accuracy: 100.00% (sample size: 3)