Research task (AI & ML):
Compose a concise technical report sketching an experimental plan for deciding whether a modern vision-language model could stand in for human specialists when labeling medical images of skin lesions.

Evaluation rubric:
criterion: The response discusses the primary objective, the chosen study design (e.g. model selection, multi-rater, paired non-inferiority trial, inter-rater agreements), the exact size of the human expert panel and test set, and the primary clinical success criterion (e.g. the lower bound of the 95% confidence interval for the accuracy difference exceeding -5%, sensitivity of at least 95% for melanoma detection, negative predictive value of 99% for ruling out malignant lesions).
weight: 3
axis: Explicit Criteria

criterion: The response's narrative structure shows a sophisticated flow, where the ethical considerations in the final sections are directly linked back to specific methodological choices made earlier (e.g. explaining how the multi-round Delphi consensus process from the Methods section mitigates the ethical risk of relying on a single-rater baseline, referencing how the test set can address ethical concern regarding bias against underrepresented skin, discussing model transparency and its impact on bias/fairness).
weight: 3
axis: Synthesis of Information

criterion: The response includes a glossary defining at least 5 key technical terms specific to the intersection of AI and medical diagnostics (e.g. non-inferiority margin, adjudication, Parameter-Efficient Fine-Tuning (PEFT), Fleiss Kappa, and Predetermined Change Control Plan (PCCP)).
weight: 2
axis: Communication Quality

criterion: The experimental objective is explicitly defined not only in terms of diagnostic substitution but also in terms of a secondary objective to evaluate the vision-language model's ability to generate descriptive text that identifies at least three specific, predefined morphological features (e.g. irregular pigment network, arborizing vessels, pearly border).
weight: 3
axis: Instruction Following

criterion: The response articulates a cost-effectiveness analysis which details the specific economic model to be used (e.g. cost-utility analysis, cost-benefit analysis, decision tree model) and the variables to be measured (e.g. cost per correct diagnosis, time saved by specialists, quality-adjusted life year).
weight: 2
axis: Implicit Criteria

criterion: The response discusses a dataset strategy that includes a plan for both external and temporal validation, specifying that the model will be tested on a held-out dataset from an institution not involved in training and on a set of images collected after the model's development was finalized.
weight: 5
axis: Explicit Criteria

criterion: The response specifies that ground-truth labels will be provided by board-certified dermatologists.
weight: 4
axis: Implicit Criteria

criterion: The response indicates inclusion of at least three lesion classes for classification (e.g. melanoma, nevus, basal cell carcinoma).
weight: 2
axis: Implicit Criteria

criterion: The response lists at least two evaluation metrics from the following: accuracy, sensitivity/recall, specificity, F1-score, AUROC.
weight: 3
axis: Implicit Criteria

criterion: The response proposes a statistical significance or non-inferiority test comparing the model and specialists (e.g. McNemar test, confidence intervals, paired t-test).
weight: 4
axis: Implicit Criteria

criterion: The response indicates the model and humans will annotate the same set of images for accurate comparison of model v.s. human performance.
weight: 1
axis: Implicit Criteria

criterion: The response provides a fail-safe or fallback procedure for model's low-confidence outputs (e.g. human-in-the-loop validation, defaulting to rule-based system, requesting additional input or clarification).
weight: 1
axis: Miscellaneous

criterion: The response mentions conducting a sample-size calculation or power analysis to determine proper dataset size.
weight: 2
axis: Implicit Criteria

criterion: The response proposes at least two distinct interpretability methods and justifies their selection by tying them to the specific model architecture and the clinical need for transparency (e.g. proposing Gradient-weighted Class Activation Mapping (Grad-CAM) to visualize lesion features for dermatologists, SHapley Additive exPlanations (SHAP) to quantify feature importance for model debugging, proposing Local Interpretable Model-agnostic Explanations (LIME) to understand specific cases, using Integrated Gradients to attribute outputs to specific image pixels and words).
weight: 3
axis: Implicit Criteria

criterion: The response explicitly mentions need for Institutional Review Board (IRB) or ethics board approval before the commencement of the study.
weight: 4
axis: Implicit Criteria

criterion: The response includes a patient privacy plan which specifies that a formal Business Associate Agreement (BAA) must be executed with any third-party cloud or AI service provider used for data processing, ensuring their contractual obligation to uphold Health Insurance Portability and Accountability Act (HIPAA) standards.
weight: 2
axis: Implicit Criteria

criterion: The response discusses at least one limitation of the vision language model in labeling medical images in reality (e.g. dataset bias, domain shift, small sample size, privacy issues).
weight: 3
axis: Implicit Criteria

criterion: The response includes any clinical treatment recommendations or diagnoses for patients.
weight: -4
axis: Implicit Criteria

criterion: The response cites nonexistent work, incorrect datasets, or made-up metrics (e.g. fabricated research papers, non-existent dataset, misrepresenting data distribution, inverted performance scores).
weight: -5
axis: References & Citation Quality

criterion: The response uses statistical methods incorrectly (e.g. applies unpaired test to paired setup, chi-squared test of independence for paired, dependent data, simple logistic regression for paired data structure).
weight: -3
axis: Synthesis of Information

criterion: The response discusses an adjudication process for establishing ground truth that details how qualitative disagreements in free text descriptions will be resolved, specifying a method for synthesizing a final consensus text that incorporates key features identified by the majority of agreeing experts.
weight: 3
axis: Implicit Criteria