Aurametrix/dermatology_longitudinal huggingface dataset:
Dataset: 542 Medical domain-specific questions (MCQs) including 144 dermatology questions
Reference (2025): Panagoulias, D.P. et al. (2025). COGnitive Network Evaluation Toolkit for Medical Domains: Evaluating Large Language Model Performance in Medical Diagnostics�An Assessment Framework and Dataset for Healthcare AI. In: Nagar, A., Jat, D.S., Mishra, D., Joshi, A. (eds) Intelligent Sustainable Systems. Worlds4 2024. Lecture Notes in Networks and Systems, vol 1178. Springer, Singapore. https://doi.org/10.1007/978-981-97-9559-8_38  https://huggingface.co/datasets/DimitriosPanagoulias/COGNET-MD/viewer/default/train?q=dermatology&views%5B%5D=train

Evaluation of ChatGPT o4-mini (GPT series) — What are some prominent causes of erythema multiforme? A. Drugs B. Herpes zoster C. Streptococcal infections D. Primary tuberculosis E. Orf
Condition/subject:  (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)
Comments: A-E

Evaluation of command-a-03-2025 (Cohere series) — Handling what substance could possibly lead to contact dermatitis? A. Chromate B. Cobalt C. Mercury D. Silver E. Aluminium
Condition/subject: A-B-C (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)
Comments: A,B,C: Handling the following substances can potentially lead to **contact dermatitis**: **A. Chromate**   **B. Cobalt**   **C. Mercury**  - **Chromate (A):** A well-known allergen and irritant, commonly found in cement, leather, and industrial materials.  - **Cobalt (B):** Often found in metal alloys, jewelry, and medical implants, and is a common cause of allergic contact dermatitis.  - **Mercury (C):** Used in some industrial processes, cosmetics (e.g., skin-lightening creams), and dental amalgam, and can cause both allergic and irritant contact dermatitis.  **D. Silver (D)** and **E. Aluminium (E)** are less commonly associated with contact dermatitis:  - Silver is generally inert and rarely causes dermatitis, though rare cases exist.  - Aluminium is more likely to cause irritant dermatitis in high concentrations but is not a common allergen.  Thus, the correct answers are **A, B,** and **C**.

Evaluation of ChatGPT o3-2025-04-16 (GPT series) — Handling what substance could possibly lead to contact dermatitis? A. Chromate B. Cobalt C. Mercury D. Silver E. Aluminium
Condition/subject:  (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)
Comments: A,B and C

Evaluation of gemini-2.0-flash-001 (Google's family of models) — Handling what substance could possibly lead to contact dermatitis? A. Chromate B. Cobalt C. Mercury D. Silver E. Aluminium
Condition/subject:  (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)
Comments: A,B and C
Dataset: 542 Medical domain-specific questions (MCQs) including 144 dermatology questions
Reference (2025): Panagoulias, D.P. et al. (2025). COGnitive Network Evaluation Toolkit for Medical Domains: Evaluating Large Language Model Performance in Medical Diagnostics�An Assessment Framework and Dataset for Healthcare AI. In: Nagar, A., Jat, D.S., Mishra, D., Joshi, A. (eds) Intelligent Sustainable Systems. Worlds4 2024. Lecture Notes in Networks and Systems, vol 1178. Springer, Singapore. https://doi.org/10.1007/978-981-97-9559-8_38  https://huggingface.co/datasets/DimitriosPanagoulias/COGNET-MD/viewer/default/train?q=dermatology&views%5B%5D=train

Evaluation of stephen (unknown) — Handling what substance could possibly lead to contact dermatitis? A. Chromate B. Cobalt C. Mercury D. Silver E. Aluminium
Condition/subject:  (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: B and C

Evaluation of ChatGPT 4o (GPT series) — Which feature is least common in basal cell carcinomata? A. Pigmentation B. Scarring C. Cystic degeneration D. Hyperkeratosis E. Telangiectasia
Condition/subject: A-B-C (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: C (Cystic degeneration) < D (Hyperkeratosis) < B (Scarring)

Evaluation of goldmane (Gemini 2.5 June 2025) (Google's family of models) — Which feature is least common in basal cell carcinomata? A. Pigmentation B. Scarring C. Cystic degeneration D. Hyperkeratosis E. Telangiectasia
Condition/subject: A-B-C (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: D Atypical for BCC; a classic feature of SCC and actinic keratosis.; C Defines the uncommon nodulocystic BCC variant.' B Defines morpheaform BCC (~5-10% of cases); also seen with ulceration; A Defines pigmented BCC (~6-10% of cases); mimics melanoma; E. Telangiectasia A hallmark sign of nodular BCC, the most common subtype.

Evaluation of claude-sonnet-4-20250514 (Claude series) — Which feature is least common in basal cell carcinomata? A. Pigmentation B. Scarring C. Cystic degeneration D. Hyperkeratosis E. Telangiectasia
Condition/subject: A-B-C (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: Hyperkeratosis (D) - Least common < Cystic degeneration C < Scarring (B)
Dataset: 542 Medical domain-specific questions (MCQs) including 144 dermatology questions
Reference (2025): Panagoulias, D.P. et al. (2025). COGnitive Network Evaluation Toolkit for Medical Domains: Evaluating Large Language Model Performance in Medical Diagnostics�An Assessment Framework and Dataset for Healthcare AI. In: Nagar, A., Jat, D.S., Mishra, D., Joshi, A. (eds) Intelligent Sustainable Systems. Worlds4 2024. Lecture Notes in Networks and Systems, vol 1178. Springer, Singapore. https://doi.org/10.1007/978-981-97-9559-8_38  https://huggingface.co/datasets/DimitriosPanagoulias/COGNET-MD/viewer/default/train?q=dermatology&views%5B%5D=train

Evaluation of goldmane (Google's family of models) — The Kobner phenomenon is associated with which of the following skin conditions? A. herpes zoster B. pityriasis rosea C. viral warts D. parapsoriasis E. molluscum contagiosum
Condition/subject: C (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: Both C. Viral Warts and E. Molluscum Contagiosum are textbook examples of conditions that exhibit the Koebner phenomenon. The mechanism for both is autoinoculation of the causative virus into the skin via trauma (like scratching). In a multiple-choice question format, this represents a flawed question with two correct answers.

Evaluation of claude-sonnet-4-20250514 (Claude series) — The Kobner phenomenon is associated with which of the following skin conditions? A. herpes zoster B. pityriasis rosea C. viral warts D. parapsoriasis E. molluscum contagiosum
Condition/subject: C (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)
Comments: C. viral warts Viral warts commonly exhibit the Kobner phenomenon. Trauma to the skin can lead to inoculation of human papillomavirus (HPV) into the injured area, resulting in new wart formation at the site of trauma. This is why warts often appear along scratch marks or areas of repeated irritation.

Evaluation of goldmane (Google's family of models) — Which of the following are considered as pre-malignant skin lesions? A. Histiocytoma B. Seborrhoeic keratoses C. Parapsoriasis D. Solar keratoses E. Epidermal naevi
Condition/subject: D-E (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: D: In a typical exam setting asking for a single answer, D. Solar Keratoses would be the expected choice as it is the most prevalent and quintessential pre-malignant skin lesion. C. Parapsoriasis (Large Plaque) Pre-malignant (pre-lymphomatous) Mycosis Fungoides (CTCL). E. Epidermal Naevus Benign (with very low risk of secondary malignancy)
Dataset: 542 Medical domain-specific questions (MCQs) including 144 dermatology questions
Reference (2025): Panagoulias, D.P. et al. (2025). COGnitive Network Evaluation Toolkit for Medical Domains: Evaluating Large Language Model Performance in Medical Diagnostics�An Assessment Framework and Dataset for Healthcare AI. In: Nagar, A., Jat, D.S., Mishra, D., Joshi, A. (eds) Intelligent Sustainable Systems. Worlds4 2024. Lecture Notes in Networks and Systems, vol 1178. Springer, Singapore. https://doi.org/10.1007/978-981-97-9559-8_38  https://huggingface.co/datasets/DimitriosPanagoulias/COGNET-MD/viewer/default/train?q=dermatology&views%5B%5D=train

Evaluation of claude-sonnet-4-20250514 (Claude series) — Which of the following are considered as pre-malignant skin lesions? A. Histiocytoma B. Seborrhoeic keratoses C. Parapsoriasis D. Solar keratoses E. Epidermal naevi
Condition/subject: D-E (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: D. Solar keratoses (also called actinic keratoses) are considered premalignant lesions with the potential to progress to squamous cell carcinoma. They develop from chronic sun exposure and represent dysplastic changes in keratinocytes that can evolve into invasive carcinoma over time. // A. Histiocytoma (dermatofibroma) - Benign fibrous lesion, not premalignant B. Seborrhoeic keratoses - Benign epidermal tumors with virtually no malignant potential (though very rarely basal cell carcinoma can arise within them) C. Parapsoriasis - This is more complex. Large plaque parapsoriasis can progress to mycosis fungoides (cutaneous T-cell lymphoma), so it does have some premalignant potential, but it's less commonly recognized as a classic premalignant lesion compared to actinic keratoses E. Epidermal naevi - Generally benign developmental lesions, though very rarely malignant transformation can occur. Solar keratoses are the most clearly established premalignant skin lesions among these options, with a well-documented risk of progression to squamous cell carcinoma if left untreated

Evaluation of Gemini 2.5 Flash (Google's family of models) — What are the five systemic diseases that can cause generalized pruritus? They include: A. carcinoid syndrome B. primary biliary cirrhosis C. chronic adrenal failure. D. systemic lymphoma E. obstructive jaundice
Condition/subject: B-D-E (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: All of the listed conditions (A. Hodgkin's disease, B. Pityriasis rosea, C. Diabetes mellitus, D. Haemolytic jaundice, E. Polycythaemia rubra vera) can include generalized pruritus as a symptom.) Answeer given: B,D,E
Dataset: 542 Medical domain-specific questions (MCQs) including 144 dermatology questions
Reference (2025): Panagoulias, D.P. et al. (2025). COGnitive Network Evaluation Toolkit for Medical Domains: Evaluating Large Language Model Performance in Medical Diagnostics�An Assessment Framework and Dataset for Healthcare AI. In: Nagar, A., Jat, D.S., Mishra, D., Joshi, A. (eds) Intelligent Sustainable Systems. Worlds4 2024. Lecture Notes in Networks and Systems, vol 1178. Springer, Singapore. https://doi.org/10.1007/978-981-97-9559-8_38  https://huggingface.co/datasets/DimitriosPanagoulias/COGNET-MD/viewer/default/train?q=dermatology&views%5B%5D=train

Evaluation of ChatGPT 4o (GPT series) — What are the five systemic diseases that can cause generalized pruritus? They include: A. carcinoid syndrome B. primary biliary cirrhosis C. chronic adrenal failure. D. systemic lymphoma E. obstructive jaundice
Condition/subject:  (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 100.00% (sample size: 1)
Comments: B-D-E; A Carcinoid syndrome is less common.  Consider replacing C chronic adrenal failure with another known pruritus-associated condition such as: Chronic kidney disease (uremia), Polycythemia vera

Evaluation of X-preview (unknown (LLM arena)) — Which among the following options might prove beneficial for an old man suffering from a gravitational ulcer with adjacent eczema? A. The usage of a supportive elastic bandage B. Daily application of Betnovate-C ointment C. Povidine-iodine ointment on the ulcer D. Diuretic therapy for oedema reduction E. Lassar's paste application around the ulcer
Condition/subject: A-C-E (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: A,C,B,E: A. The usage of a supportive elastic bandage. Compression therapy is the cornerstone of managing venous ulcers, improving venous return and reducing edema. While E. Lassar's paste application around the ulcer protects periwound skin, and B. Betnovate-C ointment may address eczema, compression (A) is primary for ulcer healing. Avoiding potent steroids (B) directly on the ulcer and long-term antiseptics (C) aligns with best practices. Diuretics (D) are less relevant unless systemic edema is present.

Evaluation of gemini-2.0-flash-001 (Google's family of models) — Which among the following options might prove beneficial for an old man suffering from a gravitational ulcer with adjacent eczema? A. The usage of a supportive elastic bandage B. Daily application of Betnovate-C ointment C. Povidine-iodine ointment on the ulcer D. Diuretic therapy for oedema reduction E. Lassar's paste application around the ulcer
Condition/subject: A-C-E (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 75.00% (sample size: 1)
Comments: A,E: The most beneficial option, addressing the underlying cause of the ulcer, is A. The usage of a supportive elastic bandage. E is also useful as an adjunct.
Dataset: 542 Medical domain-specific questions (MCQs) including 144 dermatology questions
Reference (2025): Panagoulias, D.P. et al. (2025). COGnitive Network Evaluation Toolkit for Medical Domains: Evaluating Large Language Model Performance in Medical Diagnostics�An Assessment Framework and Dataset for Healthcare AI. In: Nagar, A., Jat, D.S., Mishra, D., Joshi, A. (eds) Intelligent Sustainable Systems. Worlds4 2024. Lecture Notes in Networks and Systems, vol 1178. Springer, Singapore. https://doi.org/10.1007/978-981-97-9559-8_38  https://huggingface.co/datasets/DimitriosPanagoulias/COGNET-MD/viewer/default/train?q=dermatology&views%5B%5D=train

Evaluation of gemini-2.5-flash-preview-05-20 (Google's family of models) — Which among the following options might prove beneficial for an old man suffering from a gravitational ulcer with adjacent eczema? A. The usage of a supportive elastic bandage B. Daily application of Betnovate-C ointment C. Povidine-iodine ointment on the ulcer D. Diuretic therapy for oedema reduction E. Lassar's paste application around the ulcer
Condition/subject: A-C-E (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 50.00% (sample size: 1)
Comments: A vs A-C-E indicated by th author of ths benchmarks

Evaluation of Chatgpt-4.1-mini-2025-04-14 (GPT series) — Which among the following options might prove beneficial for an old man suffering from a gravitational ulcer with adjacent eczema? A. The usage of a supportive elastic bandage B. Daily application of Betnovate-C ointment C. Povidine-iodine ointment on the ulcer D. Diuretic therapy for oedema reduction E. Lassar's paste application around the ulcer
Condition/subject:  (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 50.00% (sample size: 1)
Comments: A vs A-C-E indicated by th author of ths benchmarks
Dataset: 60 ratings by three attending physicians of the twenty clinical vignettes based on alignment with scientific consensus, possibility of clinical harm, comprehensiveness, possibility of demographic bias.
Reference (2025): Rao AS, Kim J, Mu A, Young CC, Kalmowitz E, Senter-Zapata M, Whitehead DC, Garibyan L, Landman AB, Succi MD. Synthetic medical education in dermatology leveraging generative artificial intelligence. NPJ Digit Med. 2025 May 4;8(1):247. doi: 10.1038/s41746-025-01650-x. PMID: 40320492; PMCID: PMC12050279.

Evaluation of ChatGPT 4 (GPT series) — Scientific accuracy of Clinical Vignette (used in teaching to test clinical reasoning, diagnostic skills, and decision-making) generated by LLM / also assessed for alignment with scientific consensus, possibility of clinical harm, comprehensiveness, possibility of demographic bias, and overall quality.
Condition/subject: Scarlet Fever, Local Subcutaneous Reaction, Hyperhidrosis, Stomatitis, Acne Vulgaris, Staphylococcal Scalded Skin Syndrome, Ichthyosis, Impetigo, Stevens-Johnson Syndrome, Tinea Corporis, Cauliflower Ear, Dermatoses Caused by Plants, Lentigo, Frostbite, Melanoma, Herpes Zoster, Cellulitis, Folliculitis, Pigmented Nevi, and Keloids (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 89.00% (sample size: 60)
Comments: Physician experts gave the vignettes high average scores on a Likert scale in scientific accuracy (4.45/5), comprehensiveness (4.3/5), and overall quality (4.28/5) and low scores for potential clinical harm (1.6/5) and demographic bias (1.52/5). A strong correlation (r?=?0.83) was observed between comprehensiveness and overall quality. Vignettes did not incorporate significant demographic diversity.

Evaluation of ChatGPT 4 (GPT series) — Comprehensiveness of Clinical Vignette (used in teaching to test clinical reasoning, diagnostic skills, and decision-making) generated by LLM / also assessed for alignment with scientific consensus, possibility of clinical harm, comprehensiveness, possibility of demographic bias, and overall quality.
Condition/subject: Scarlet Fever, Local Subcutaneous Reaction, Hyperhidrosis, Stomatitis, Acne Vulgaris, Staphylococcal Scalded Skin Syndrome, Ichthyosis, Impetigo, Stevens-Johnson Syndrome, Tinea Corporis, Cauliflower Ear, Dermatoses Caused by Plants, Lentigo, Frostbite, Melanoma, Herpes Zoster, Cellulitis, Folliculitis, Pigmented Nevi, and Keloids (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 80.00% (sample size: 60)
Comments: Physician experts gave the vignettes high average scores on a Likert scale in scientific accuracy (4.45/5), c (4.3/5), and overall quality (4.28/5) and low scores for potential clinical harm (1.6/5) and demographic bias (1.52/5). A strong correlation (r?=?0.83) was observed between comprehensiveness and overall quality. Vignettes did not incorporate significant demographic diversity.
Dataset: 60 ratings by three attending physicians of the twenty clinical vignettes based on alignment with scientific consensus, possibility of clinical harm, comprehensiveness, possibility of demographic bias.
Reference (2025): Rao AS, Kim J, Mu A, Young CC, Kalmowitz E, Senter-Zapata M, Whitehead DC, Garibyan L, Landman AB, Succi MD. Synthetic medical education in dermatology leveraging generative artificial intelligence. NPJ Digit Med. 2025 May 4;8(1):247. doi: 10.1038/s41746-025-01650-x. PMID: 40320492; PMCID: PMC12050279.

Evaluation of ChatGPT 4 (GPT series) — Overall quality of Clinical Vignette (used in teaching to test clinical reasoning, diagnostic skills, and decision-making) generated by LLM / also assessed for alignment with scientific consensus, possibility of clinical harm, comprehensiveness, possibility of demographic bias.
Condition/subject: Scarlet Fever, Local Subcutaneous Reaction, Hyperhidrosis, Stomatitis, Acne Vulgaris, Staphylococcal Scalded Skin Syndrome, Ichthyosis, Impetigo, Stevens-Johnson Syndrome, Tinea Corporis, Cauliflower Ear, Dermatoses Caused by Plants, Lentigo, Frostbite, Melanoma, Herpes Zoster, Cellulitis, Folliculitis, Pigmented Nevi, and Keloids (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 85.60% (sample size: 60)
Comments: Physician experts gave the vignettes high average scores on a Likert scale in scientific accuracy (4.45/5), comprehensiveness (4.3/5), and overall quality (4.28/5) and low scores for potential clinical harm (1.6/5) and demographic bias (1.52/5). A strong correlation (r?=?0.83) was observed between comprehensiveness and overall quality. Vignettes did not incorporate significant demographic diversity.

Evaluation of ChatGPT 4 (GPT series) — Potential clinical harm of Clinical Vignette (used in teaching to test clinical reasoning, diagnostic skills, and decision-making) generated by LLM / also assessed for alignment with scientific consensus, possibility of clinical harm, comprehensiveness, possibility of demographic bias, and overall quality.
Condition/subject: Scarlet Fever, Local Subcutaneous Reaction, Hyperhidrosis, Stomatitis, Acne Vulgaris, Staphylococcal Scalded Skin Syndrome, Ichthyosis, Impetigo, Stevens-Johnson Syndrome, Tinea Corporis, Cauliflower Ear, Dermatoses Caused by Plants, Lentigo, Frostbite, Melanoma, Herpes Zoster, Cellulitis, Folliculitis, Pigmented Nevi, and Keloids (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 32.00% (sample size: 60)
Comments: Physician experts gave the vignettes high average scores on a Likert scale in scientific accuracy (4.45/5), comprehensiveness (4.3/5), and overall quality (4.28/5) and low scores for potential clinical harm (1.6/5) and demographic bias (1.52/5). A strong correlation (r?=?0.83) was observed between comprehensiveness and overall quality. Vignettes did not incorporate significant demographic diversity.
Dataset: 60 ratings by three attending physicians of the twenty clinical vignettes based on alignment with scientific consensus, possibility of clinical harm, comprehensiveness, possibility of demographic bias.
Reference (2025): Rao AS, Kim J, Mu A, Young CC, Kalmowitz E, Senter-Zapata M, Whitehead DC, Garibyan L, Landman AB, Succi MD. Synthetic medical education in dermatology leveraging generative artificial intelligence. NPJ Digit Med. 2025 May 4;8(1):247. doi: 10.1038/s41746-025-01650-x. PMID: 40320492; PMCID: PMC12050279.

Evaluation of ChatGPT 4 (GPT series) — Demographic bias of Clinical Vignette (used in teaching to test clinical reasoning, diagnostic skills, and decision-making) generated by LLM / also assessed for alignment with scientific consensus, possibility of clinical harm, comprehensiveness,  and overall quality.
Condition/subject: Scarlet Fever, Local Subcutaneous Reaction, Hyperhidrosis, Stomatitis, Acne Vulgaris, Staphylococcal Scalded Skin Syndrome, Ichthyosis, Impetigo, Stevens-Johnson Syndrome, Tinea Corporis, Cauliflower Ear, Dermatoses Caused by Plants, Lentigo, Frostbite, Melanoma, Herpes Zoster, Cellulitis, Folliculitis, Pigmented Nevi, and Keloids (English)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: 30.40% (sample size: 60)
Comments: Physician experts gave the vignettes high average scores on a Likert scale in scientific accuracy (4.45/5), comprehensiveness (4.3/5), and overall quality (4.28/5) and low scores for potential clinical harm (1.6/5) and demographic bias (1.52/5). A strong correlation (r?=?0.83) was observed between comprehensiveness and overall quality. Vignettes did not incorporate significant demographic diversity.
Dataset: 100 questions were created in Turkish, divided into four groups of 25 questions each, based on the seniority of dermatology resident physicians
Reference (2025): G��er G�rok N, �zt�rk S. The Performance of AI in Dermatology Exams: The Exam Success and Limits of ChatGPT. J Cosmet Dermatol. 2025 May;24(5):e70244. doi: 10.1111/jocd.70244. PMID: 40387311; PMCID: PMC12087418.

Evaluation of ChatGPT 3.5 (GPT series) — first?year Dermatology Exam
Condition/subject:  (Turkish)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: PASS (sample size: 25)
Comments:  ChatGPT 3.5 passed the first and second?year exams but failed the third and fourth?year exams. ChatGPT 4.0 passed the first, second, and third?year exams but failed the fourth?year exam.

Evaluation of ChatGPT 3.5 (GPT series) — second?year Dermatology Exam
Condition/subject:  (Turkish)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: PASS (sample size: 25)
Comments:  ChatGPT 3.5 passed the first and second?year exams but failed the third and fourth?year exams. ChatGPT 4.0 passed the first, second, and third?year exams but failed the fourth?year exam.

Evaluation of ChatGPT 3.5 (GPT series) — third?year Dermatology Exam
Condition/subject:  (Turkish)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: FAIL (sample size: 25)
Comments:  ChatGPT 3.5 passed the first and second?year exams but failed the third and fourth?year exams. ChatGPT 4.0 passed the first, second, and third?year exams but failed the fourth?year exam.

Evaluation of ChatGPT 3.5 (GPT series) — fourth?yearDermatology Exam
Condition/subject:  (Turkish)
Category: Dermatology Examinations and Practice Questions — Professional Education
Validity/accuracy: FAIL (sample size: 25)
Comments:  ChatGPT 3.5 passed the first and second?year exams but failed the third and fourth?year exams. ChatGPT 4.0 passed the first, second, and third?year exams but failed the fourth?year exam.
Dataset: 64 dermatology questitons from HealthBench's 5,000 realistic health conversations, each with a custom physician-created rubric to grade model responses. (15 criteria for validity score used in this dataset)
Reference (2025): Arora RK, Wei J, Hicks RS, Bowman P, Qui�onero-Candela J, Tsimpourlas F, Sharman M, Shah M, Vallone A, Beutel A, Heidecke J. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint arXiv:2505.08775. 2025 May 13. https://doi.org/10.48550/arXiv.2505.08775

Evaluation of gemini-2.5-flash-preview-04-17 (Google's Family of LLMs) — Adult acne; mild breakouts, gradually worse over 6 months. 
Condition/subject:  (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 76.19% (sample size: 15)
Comments: mild breakouts, gradually worse over 6 months.  - described prescription options and appropriately framed as things a doctor would prescribe but  was noticeably longer and more technical thus penalized for communication

Evaluation of glm-4-air-250414 (Zhipu.AI) — Adult acne
Condition/subject:  (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 87.30% (sample size: 15)
Comments: mild breakouts, gradually worse over 6 months.  - described prescription options and appropriately framed as things a doctor would prescribe.

Evaluation of claude-opus-4-20250514 (Claude series) — Hair shedding
Condition/subject:  (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 88.06% (sample size: 15)
Comments: I just noticed a bunch of hair coming out on my brush this morning It’s normal to shed some hair throughout the day, but if you’re seeing more than usual, it can be unsettling. Sometimes a sudden increase in shedding can be related to stress or changes in diet, but other factors could be at play as well. Are there any other changes you’ve noticed? It just feels so sudden, and my scalp is itchy too. I’m scared it means something is seriously wrong. I can’t stop thinking about it. An itchy scalp can have many causes, from mild irritation to certain skin conditions, and stress itself can sometimes exacerbate hair shedding. It’s understandable to feel worried if you see more hair than usual. Have you been under more stress than usual, or changed your hair care routine? Could it be a serious illness? I’m so anxious I can’t even sleep, I feel like I’m going to lose all my hair if this continues.
Dataset: 64 dermatology questitons from HealthBench's 5,000 realistic health conversations, each with a custom physician-created rubric to grade model responses. (15 criteria for validity score used in this dataset)
Reference (2025): Arora RK, Wei J, Hicks RS, Bowman P, Qui�onero-Candela J, Tsimpourlas F, Sharman M, Shah M, Vallone A, Beutel A, Heidecke J. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint arXiv:2505.08775. 2025 May 13. https://doi.org/10.48550/arXiv.2505.08775

Evaluation of redsword (unknown (LLM arena)) — Hair shedding
Condition/subject:  (English)
Category: Medical Records and Diagnostic Processes — Clinical Practice
Validity/accuracy: 83.58% (sample size: 15)
Comments: I just noticed a bunch of hair coming out on my brush this morning It’s normal to shed some hair throughout the day, but if you’re seeing more than usual, it can be unsettling. Sometimes a sudden increase in shedding can be related to stress or changes in diet, but other factors could be at play as well. Are there any other changes you’ve noticed? It just feels so sudden, and my scalp is itchy too. I’m scared it means something is seriously wrong. I can’t stop thinking about it. An itchy scalp can have many causes, from mild irritation to certain skin conditions, and stress itself can sometimes exacerbate hair shedding. It’s understandable to feel worried if you see more hair than usual. Have you been under more stress than usual, or changed your hair care routine? Could it be a serious illness? I’m so anxious I can’t even sleep, I feel like I’m going to lose all my hair if this continues.