Doctor Approved: Generating Medically Accurate Skin Disease Images through AI-Expert Feedback
Authors: Janet Wang
Yunbei Zhang
Zhengming Ding
Jihun Hamm
Venue: NeurIPS 2025 poster ()
Keywords: Synthetic Clinical Data
Machine Learning for Healthcare
Skin Imaging Analysis
Primary area: applications

Abstract: Paucity of medical data severely limits the generalizability of diagnostic ML models, as the full spectrum of disease variability can not be represented by a small clinical dataset. To address this, diffusion models (DMs) have been considered as a promising avenue for synthetic image generation and augmentation. However, they frequently produce _medically inaccurate_ images, deteriorating the model performance. Expert domain knowledge is critical for synthesizing images that correctly encode clinical information, especially when data is scarce and quality outweighs quantity. Existing approaches for incorporating human feedback, such as reinforcement learning (RL) and Direct Preference Optimization (DPO), rely on robust reward functions or demand labor-intensive expert evaluations. Recent progress in Multimodal Large Language Models (MLLMs) reveals their strong visual reasoning capabilities, making them adept candidates as evaluators. In this work, we propose a novel framework, coined MAGIC (**M**edically **A**ccurate **G**eneration of **I**mages through AI-Expert **C**ollaboration), that synthesizes clinically accurate skin disease images for data augmentation. Our method creatively translates expert-defined criteria into actionable feedback for image synthesis of DMs, significantly improving clinical accuracy while reducing the direct human workload. Experiments demonstrate that our method greatly improves the clinical quality of synthesized skin disease images, with outputs aligning with dermatologist assessments. Additionally, augmenting training data with these synthesized images improves diagnostic accuracy by +9.02% on a challenging 20-condition skin disease classification task, and by +13.89% in the few-shot setting. Beyond image synthesis, MAGIC illustrates a task-centric alignment paradigm: instead of adapting MLLMs to niche medical tasks, it adapts tasks to the evaluative strengths of general-purpose MLLMs by decomposing domain knowledge into attribute-level checklists. This design offers a scalable and reliable path for leveraging foundation models in specialized domains.

Doctor Approved: Generating Medically Accurate Skin Disease Images through Al-Expert Feedback
Janet Wang Yunbei Zhang
Zhengming Ding Jihun Hamm
Tulane University
(swang47, yzhang111, zdingl, jhamm3) e @ullane.... edu
Abstract
Paucity of medical data severely limits the generalizability of diagnostic ML mod- els, the full spectrum of disease variability can not be represented by a small clinical dataset. To address this, diffusion models (DMs) have been considered as a promising avenue for synthetic image generation and augmentation. However, they frequently produce medically inaccurate images, deteriorating the model performance. Expert domain knowledge critical for synthesizing images that correctly encode clinical information, especially when data is scarce and quality outweighs quantity. Existing approaches for incorporating human feedhack, such as reinforcement learning (RL) and Direct Preference Optimization (DPO), rely on robust reward functions or demand labor-intensive expert evaluations. Recent progress in Multimodal Large Language Models (MLLMs) reveals their strong visual reasoning capabilities, making them adept candidates as evaluators. this work, we propose novel framework, coined MAGIC (Medically Accurate Generation of Images through AI-Expert Collaboration). that synthesizes clini- cally accurate skin disease images for data augmentation. Our method creatively translates expert-defined criteria into actionable feedback for image synthesis DMs, significantly improving clinical accuracy while reducing the direct human workload. Experiments demonstrate that our method greatly improves the clinical quality of synthesized skin disease images, with outputs aligning with dermatolo- gist assessments. Additionally, augmenting training data with these synthesized images improves diagnostic accuracy by +9.02% on challenging 20-condition skin disease classification task, and by +13.89% in the few-shot setting. Beyond image synthesis. MAGIC illustrates task-centric alignment paradigm: instead of adapting MLLMs to niche medical tasks, it adapts tasks to the evaluative strengths of purpose MLLMs by decomposing domain knowledge into attribute-lvvel checklists. This design offers scalable and reliable path for leveraging foundation models in specialized domains. Our implementation detail and code available https: httptb...uuu..gg/////// com/ jane/janet---e sw /MAGIC. git.
Introduction
Recent advances in deep learning have made dermatological diagnosis increasingly accessible, offering significant potential for teledermatology in rural regions [6, 15, 37, 52]. However, privacy constraints and proprietary rights over skin images often lead to data scarcity, especially for rare conditions, making it difficult capture the full complexity and variability of skin diseases for training robust diagnostic models. In response, various data augmentation strategies have been proposed-most straightforwardly, by aggregating open-source dermatological images [1, 60]. Yet,
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

this approach does not guarantee access to high-quality samples of the precise clinical presentations needed, such as specific combinations of skin tones, body sites, and other lesion characteristics.
Image synthesis by Text-to-Image (T2I) Diffusion Models (DMs) [12] has emerged as a promis- ing solution enrich datasets under the guidance of prompts. Such controlled generation helps mitigate long-tail distributions, reduce biases against underrepresented groups, and improve model aspects of building reliable diagnostic systems [31, 48, 58, 35J. While the effectiveness of diffusion-basee synthetic augmentation for common objects is debatable compared to retrieval- -bssed methods, their value in the medical domain remains significant due to the proprietary nature of medical data and the general infeasibility of retrieval [22]. T2I DMs have been employed to augment medical datasets across various imaging modalities [3, 28, 30, 42, 63, 13]. Previous works have also attempted to fine-tune DMs on skin disease images to enhance subsequent diagnostic model performance. However, these approaches typically involved end-to-end generation without expert participation during the training process, relegating expert assessment or filtering to post-generation stage, rather than actively guiding the model to create clinically accurate images. [2, 45, 46, 58].
Aligning DMs via Reinforcement Learning from Human Feedback (RLHF) has been explored to adapt these models and generate images that meet human preferences. In particular, [33] proposes reward-weighted likelihood maximization to achieve alignment. Building on this, [53] engages expert pathologists to assess sampled bone marrow images against a clinical plausibility checklist and train a reward function on binary feedback to emulate clinician assessments when fine-tuning a conditionn DM. More recently. [4, 17] considers the denoising process as multi-step Markov Decision Process (MDP) and adopts policy gradient optimization to fine-tune DMs based on human feedback. However, such methods still require reliable reward functions, whose training demands substantial computational resources and vast amounts of human-labeled feedback. To address these limitations, [66] proposes using Direct Preference Optimization (DPO) [43], which enables DM fine-tuning directly on preference data, bypassing the need for explicit reward model and allowing iterative parameter updates based on human feedback at each timestep of the denoising process.
Inspired by recent advances in Reinforcement Learning from AI Feedback (RLAIF) [32] and the strong visual reasoning capabilities MLLMs, we propose MAGIC (Medically Accurate Generation of Images through Al-Expert Collaboration), a semi-autornatdd framework that utilizes MLLMs for visual evaluation. In this framework, human experts are primarily required to: (1) craft, from credible sources, checklists that are easily verifiable by a MLLM, and (2) oversee the MLLM's feedback on synthetic images during the training of T2I DMs. By iteratively learning from the feedback enhanced with expert knowledge, MAGIC steers the T21 DMs toward more medically consistent generations. This approach highlights the potential of Al-expert collaboration, as MAGIC effectively leverages existing domain knowledge without labor-intensive annotation. Moreover, MAGIC incorporates an Image-to-ly Image (121) module within its training pipeline to initiate denoising from intermediate timesteps rather than pure Gaussian noise. This accelerates the sampling stage while ensuring factorized lcsion transformations that do not deviate excessively from the real data distribution.
Through rigorous experiments, we demonstrate that our MAGIC framework performs effectively with both reward- based fine-tuning (RFT) and DPO, exhibiting particular strength with DPO. The MAGIC-DPO pipeline optimizes DMs to generate synthetic data that accurately represent each condition's 's unique visual features, with improvements observed as training progresses and more mmage-feedback pairs are used (Fig. 2). This is also validated by increasing dermatologist evaluation scores (Fig. 4d) and decreasing Fréchet Inception Distance (FID) scores (Fig. 4c), indicating improved clinical accuracy and fidelity. As a result, we also observe significant improvements in classification performance over baseline, highlighting MAGIC's potential to advance AI dermatology. Overall, main contributions are: (i) We propose MAGIC, novel
framework that integrates expert knowledge into DMs, enabling their subsequent line-tuning with both DPO and RFT. The framework incorporates an 121 module to efficiently align the model for producing medically accurate images. (ii) Our framework employs an Al-Expert collaboration paradigm that offloads the work of visual evaluation to powerful MILMM under minimal expert supervision, significantly reducing time and labor required from medical experts. (iii) MAGIC, particularly when combined with DPO (MAGIC-DPO), generates high-quality, clinically accurate images, achieving notable improvements in FID scores and classification performance. It yields a +9.02% boost in accuracy on a challenging 20-condition classification task and + 13.89% improvement in few-shot scenarios.
2

---

(a). Clinical Feedback Curation
Lupus:
ing acaty acered
"An image
lupus erythematosus"
5/5 Score
Diflusion Model
Al Expext
Collsboration
2/5 Score
(b). tine-tune DM with Feedback
(c). Train Classifier with Augmented Data
Scores,
Ground truit sarcokdoals
Feedback
Enhancad
Medically
Medical
guided
Accurate Data
Fine-tuning
Figure 1: Illustration of our proposed MAGIC: (a) A preliminary fine-tuned diffusion model (DM) transforms a source image (c.g.. sarcoidosis) to target condition (c.g., lupus erythematosus); an MILMM then provides expert checklist-band based feedback scores on the generated image pair. (b) This feedback guides the subsequent fine-tuning (e.g.. RFT or DPO) of the DM. (c) The feedback-enhanced DM synthesizes medically accurate dermatological images for robust classifier training.
Related Works
DM-hased Augmentation for Skin Disease Classification. Existing studies have explored diffusion models (DMs) to generate synthetic dermatological images for augmenting the training data of diagnostic models. Along this line, [46] implemented seed-based approach, sampling small set of real images from the Fitzpatrick 17k dataset [23] and generating synthetic data using the inpainting feature of OpenAl's DALL-E 2. Subsequently, [45] leveraged Stable Diffusion's ''s T2I pipeline, fine-tuned with Dreambooth, to produce images of specific disease conditions. Other related works [2, 31] have similarly employed DM-based augmentation enhance diagnostic accuracy and generalization on their internal skin disease datasets. Building on these advances, [58] proposed a diffusion augmentation framework specifically targeting minority skin types. Their approach involved Textual Inversion [19] and Low-Rank Adaptation (LoRA) [26] for fine-tuning, coupled with image-to-image generation for inference. This method enabled the creation of images depicting novel lesion concepts previously unseen by the DM. Their study revealed that images synthesized using this -uuidaee strategy improved the diagnostic performance of subsequent classifiers for minority skin types, even when reference data from these groups was absent from the training set. However, expert involvement these previously proposed methods, if any, is typically confined to post-generation assessment filtering, rather than actively guiding the image creation process.
Fine-tune Diffusion Models (DMs) with Feedback. Approaches tine-tuning DMs with human feedback broadly fall into two categories: reward-based and preference-basee Reward-based methods [5, 16, 18, 34, 65] depend on robust reward models, the training which typically requires substantial datasets and extensive human evaluations. In the medical domain, for instance, [53] leveraged weighted maximization to synthesize plausible bone marrow images, by fine-tuning class-conditional DM with pathologist's feedback synthetic images. In contrast, preference- based approaches aim to derive policies directly from preference data, thereby bypassing the need for explicit reward functions [9, 14, 32]. A key development in this area is Direct Preference Optimization (DPO) [43], originally proposed for fine-tuning language models directly using preferences. While DPO adaptations for diffusion models have primarily been tested for feege-ke alignment (57, 66], their application to medical image generation remains largely unexplored, especially for clinical images of skin diseases, which exhibit high complexity and variations.
as..-M--as-a-uudge Collecting high-quality feedback has traditionally relied on human labelers, approach that is both costly and difficult to scale. Recent research demonstrates that powerful

---

SAR
ERY
ALL
LUP
GRA I VIT
Iteration
Figure 2: Evolution of synthetic skin conditions generated by MAGIC-DPO, illustrating its ability to learn unique visual features from feedback across training iterations. The Top Row demonstrates the model transforming Sarcoidosis (SAR) into Erythema Multiforme (ERY). learning features like target "oull's-eye lesions with concentric rings. The Middle Row demonstrates the model transforming Allergic Contact Dermatitis (ALL) into Lupus Erythematosus (LUP), progressively developing butterfly rash covering the cheeks. The Bottom Row demonstrates the model transforming Granuloma Annulare (GRA) into Vitiligo (VTT), evolving show characteristic depigmented patches.
proprictary MLLMs, such as GPT-4V and GPT-4o [40], can serve as effective generalist evaluators for vision-language tasks [8, 21, 69]. These models have proven particularly valuable in complex tasks requiring human-like judgment, including visual conversations and detailed image captioning, where MLLMs are often incorporated into evaluation benchmarks to assess model responses [54, 68, 72]. More recently, these models have shown capabilities in encoding clinical knowledge and acting evaluators in medical reasoning [51]. Although employing MLLMs as collaborators in Al dermatology holds great potential to enhance the reliability of diagnostic models, the optimal paradigm for their collaboration with medical experts still remains underexplored.
Method
3.1 Preliminaries
Diffusion Models (DMs). DMs are designed to learn the probability distribution p(x) by reversing Markovian forward process, denoted as q( xz-1), which incrementally introduces noise into the images. The reversal, a denoising process, is implemented through a neural network tasked with predicting either the mean of æt-1 the noise E,-1 from the forward process. In our approach, utilize a network p ( t) to predict the mean æ;-1, rather than the added noise. We employ the Mean Squared Error (MSE) performance metric, defining the objective function of network as follows:
LDM B (aap:/")))) (aa (g [|ã xx) - (o 0)1²]
(I)
where (x, 00) represents the posterior of the forward process.
In conditional generative modeling, diffusion models are adapted to learn the conditional distribution p(z|c), where represents conditioning information, such as image categories captions. This adap- tation involves augmenting the denoising network with additional input, e, resulting in μ t; c). To generate sample from the learned distribution pa(z)c) we initiate the process by drawing sample Ør ~ (0, I), which then progressively denoised through iterative application of cg. based specific samplers adopted [25]. The reverse process is modeled
pe (0 It, N (2111 c,t),
(2)

---

In our skin disease image generation framework, we leverage the 12I pipeline of Stable Diffusion [44] to transform lesion features while preserving body part information the image. This strategy effectively reduces semantic distortion during generation and ensures factorized translation of lesions, thereby enhancing medical plausibility. Specifically, we start with a real input dermatological image Zo (e.g., sarcoidosis), add partial noise to it, and transform it into a different target skin condition (e.g., lupus erythematosus), by denoising this partily noised images. And the denoising process is governed by μs and denoise strength parameter 7.
Multi-Step MDP Formulation. We formulate the diffusion model's denoising process multi-step Markov Decision Process (MDP), following [5, 55]. In our model, the state s E S includes the current denoising time step, denoised image data and prompt. The action space A includes possible image transformations at each time step. The state transition function P(s'ls, (s, a) describes the image evolution, and the reward function r r(s,a) a) assigns values based on the image quality at each time step,
aiming maximize cumulative returns J(T) = E, IE T-1 (Sc, a.)]. The MDP is formulated as
e (c. (c)))))- a, - 1- Po (5o) 6o.
P ((8++1 s.) & (6e, 5e4s, (1455 bmy-1-1) i a (ar (s.) po (2r-1-e leal-ee_,,,, (,
(3)
r(s.. ax) e r((c,,, 1y-
where 6, represents the Dirac delta distribution, and T denotes the maximize denoising timesteps.
3.2 Preliminary Diffusion Models Fine-tuning
Previous studies have shown that off-the-shelf diffusion models struggle to represent skin lesion concepts, making preliminary fine-tuning necessary before aligning with expert feedback [58]. Following [58], we employ Latent Diffusion Models (L.MMs) [44], which operate in autoencoder latent space to reduce computational demands while maintaining generation quality. For simplicity, we abuse notation and use æ to represent the latent input to the diffusion process rather than the original image. Our framework utilizes Textual Inversion [20] derive unique embeddings that capture the semantics of cach condition extracted from training data. Each image is paired with a descriptive string containing placeholders (e.g.. 'an image (S.) } as input. The optimal embedding u., encapsulating the lesion concept S., is then obtained by minimizing reconstruction loss while keeping the LDM fixed. To enhance the efficiency of the LDM fine-tuming process, we employ LoRA [26]. adapting the model with the discovered tokens from Textual Inversion. This approach maintains the pre-trained model weights while introducing only two compact matrices A and B (where A e R B E R**"), These matrices are embedded within the attention layers, enabling the detailed capture of skin lesion characteristics previously unrepresented in the initial model, aligned with the learned target embedding c..
3.3 Expert Feedback Curation
While diffusion models can synthesize visually realistic medical images, their clinical validity often remains questionable [53]. Incorporating medical expertise is therefore crucial for guiding these models generate medically accurate images. To provide this clinical guidance, our framework leverages structured feedback derived from checklists that are designed by an experienced dermatologist. These checklists evaluates five distinct aspects of cach condition: [Locat ion, Lesion Type, Shape/ / siize, Color, Texture] (see Appendix B for complete details). Assessment against these aspects yiclds binary outcome (c.g., satisfied/not satisfied) for each criterion. To automate this evaluation, we instructed MLLM analyze each synthesized image based on the target condition's checklist and return a 5-dimensional binary score list, where each dimension corresponds to a criterion's 's satisfaction (see Appendix C for instruction details). To accommodate both reward- based and preference- based alignment strategies, we generate pair of images from each text prompt and submit each single image the MLLM for this assessment. Thus, the MLLM's score list for each image in a pair individually stands a sample for RFT, while the pair of score lists can used for DPO. Examples of this MLI.M assessment using OpenAl's GPT-4o are illustrated in Fig. 3, showing yielded score lists such as [1, 0, 0, 1, 0] and [1, 1, 1, 1] for a given pair. Ultimately, each 5-dimensional MI MLL M-gencrated score list is aggregated into overall binary score (e.g., 0 for negative example, for positive example) using predefined algorithm (detailed in Appendix A.2). This semi-automated pipeline allows to significantly accelerate the curation expert feedback. Notably, only synthetic images are sent to GPT-4o API services and no real patient images are processed by the MLLM, to preserve privacy.
3.4 Finetuning with Expert Feedback
After collecting pairwise preferences, we explore two complementary ways to integrate them into optimizing the diffusion model parameters 6.
5

---

pair images: (0, 1}
Evaluation Chec klists
Image
Image 1
corpetionditon: erythematozus
Location:
fealure: rashes
hapet'szer symmetric buteriyy cheeks
X
Calon pink-ttoved skin
Texture: scaly scarre
X
vi win
v win
Target consitio: granuloma annulare
Image
Image
Location: trunk limbs
Les foature: non scaly bumps or papules
X
S shapetsier anraler ring shaped)
X
Calon skon-cloree, prikk reddish
Texture: cenerally smooth, litl to flaking or crust
x
x
Target condition villige
Image
Image 1
Location: fate
Lesion festure: depigmented petches
X
Shapeniine: inegular symmetic
X
Colon white orpale, loss of sisin color
X
Texture: smooth, only colorie lest
X
Figure 3: Illustration of the image assessment process by OpenAI's GPT-4o using condition sppecifi checklists for target skin conditions such as lupus erythematosus, granuloma annulare, and vitiligo. Each generated image in pair evaluated against five clinical criteria. The image with more satisfied criteria is considered the preferred sample in a comparison. Additional examples are in Appendix 7.
Reward-model guided fine-tuning (RFT) R HXX33 x C C->R2o learned scalar that predicts the likelihood an image x conditioned on class satisfies every checklist item. We follow [33, 53] and mix real and synthetic images when training RR with MSE loss. Formally, with feedhack labels y E (0. 1} we minimize Luv(e) E ((reey (v R2(2, c))² After fitting ø. refine 0 by maximising the expected reward-weigen weighted
log-probability action sequence generated along each denoising trajectory a {(+,, a=)
Luri(0) = B) [ - (zt, e)>T=> log To (a: st] + S. B(x.))
- log (e ss]].
(4)
where and denote synthetic and real image pools, respectively, and B, balances fidelity expert feedback against faithfulness to the original data distribution.
Direct Preference Optimization (DPO) Given a pair of trajectories (a",, ,a") that yield a winner image x" and loser image æ' under expert comparison, DPO increases the likelihood of every action of the winning branch while decreasing the likelihood the corresponding af on the losing branch. Similar to reinforcement learning methods [7, 49, 50], rewards are assigned by Vs,, a a, 78,,, a₂) = for winning the game and vt E a, rst, at) = -1 for losing the game. Following [66], we also assume that if the final image is preferred, then any state-a0e action pair in its generation path is superior to the corresponding pair in the non- preferrfedeed path. To maximize learning from each generation process under this assumption, construct t' >I sub-segments that allow model to from intermediate states
(P) Fw #2) [log p(8 log
mafa (a"") Veat (1F
9 log ma(afle)) meer(a (x)
(5)
where E [0 1], effectively increasing data utilization by factor
3.5 Synthetic Augmentation for Classifier Training
After fine-tuning diffusion model with expert-enhanced feedback, we leverage the model to synthesize images for dataset augmentation, primarily through image-to-image translation approach. For any given sample x with label y, we first randomly select different target label y' from the label sct. We then use the text prompt "an image of {v') innooorrrrrriiiin the specific text embedding for y' leamned via Textual Inversion-to guide image z'. This process is designed that x' preserves most of the anatomical context of the original sample x while primarily displaying the lesion semantics of the target label y'. thereby achieving factorized transformation. This generation strategy offers key benefit: helps mitigate the risk of the classifier learning spurious correlations by preventing it from associating lesions with specific body locations, encouraging focus on the intrinsic characteristics of the skin lesions. During the subsequent classifier training phase, we intentionally control the influence of synthetic data using ratio parameter p E (0, 1).
the DM in generating

---

which determines the percentage of synthetic images added to each training batch. While our method aims to generate medically accurate images, potential domain shifts between real and synthetic data remain an important consideration, Indeed, our experiments indicate that varying the proportion of synthetic data can significantly affect classifier performance on real test data (see Fig. 4a).
4 Experiments
Dataset. Following prior work [58], we use the Fitzpatrick 17k dataset to evaluate our synthetic augmentation
pipeline [23]. Fitzpatrick17k contains clinical photos of 114 skin conditions, each annotated with a condition label and Fitzpatrick Skin Type (FST). Although there are other datasets of clinical photos (e.g.. SCIN [62] and DDI[II]. they are primarily collected within the United States and feature lighter skin tones. Fitzpatrick I encompasses a wider range of skin types, making it particularly suitable for evaluating generalizable diagnostic approaches. For our experiments, we focus on a subset of the Fitzpatrick 17k 17 dataset consisting of 20 skin conditions. We chose these based on two criteria: (1) they present the largest class sizes in the datasct, and (2) they have well-established descriptions available from reputable clinical sources (e.g., Mayo Clinic, Cleveland Clinic), which allowed dermatologists to craft reliable diagnostic checklists of key visual features for these diseases. These checklists, verified by clinicians, distill essential visual cues for each condition, detailed in the Appendix B. The distribution of the selected classes is provided in the Appendix A.
Models and Baselines, We utilize Stable Diffusion v2-1 1 [44] for image generation. For classification tasks, we employ ResNet18 [24] and DINOv2 [41] as backbone architectures, For medical image generation, we evaluate four different methods: (1) diffusion model fine-tuned with Textual Inversion and LoRA, generating images via text-to-image T21;; (2) the same fine-tuned model but generating via image-to-image (+ 12I); and (3/4) our proposed MAGIC (RFT/DPO) with expert feedback. We assess synthetic image quality using both FID score and human evaluation. For classification experiments, we first establish a baseline by training a classifier solely real data. We then generate an equivalent number of synthetic images using each generation method (excluding the off-the-shelf DM due to its lack of domain-specific knowledge [58]), and train classifiers on combined real and synthetic datasets. Implementation details are provided in Appendix A.
Implementation Details. To adapt the model to skin lesion concepts, our preliminary fine-tuning process proceeds in two stages: (i) We learn unique discase-related tokens by updating the text encoder via Textual Inversion [19), thereby introducing new vocabulary specific to cach condition; and (ii) we tic the newly learned tokens to fine-grained visual cues within the images by updating the UNet parameters via LoRA [26]. Further details on prompts and hyperparameters can be found in the Appendix A.
For training with expert feedback, all experiments share unified sampling-feedbutt pipeline. For each mini- batch image-prompt pairs drawn from the real set, the current diffusion model generates two synthetic variants image-to-image path, intentionally targeting skin-disease classes that differ from the
via the originals to maximise diversity. Each synthetic image is then scored with the cundition-speciff checklists (Appendix B). which submit to GPT-40 [40]. The API returns binary vectors indicating whether each criterion met; if the lesion is deemed invalid, an all-zero vector is assigned. From every pair vectors we derive a winner-loser label and store the associated latents, timesteps, and prompt emheddings. We subsequently branch into two finetuning regimes: (i) in the reward-model route we fit a scalar network R2 to these binary outcomes and update by the -eighhee likelihood of Eq. (4); (ii) in the DPO route treat each preferenec tuple in [66] and optimize the multi-segment loss of Eq. (5). Both routes draw from the same pool of feedback pairs, subsequent comparisons isolate the effect the finetuning algorithm itself. Examples are visualised in Fig. 3.
For classifier training, we randomly split the dataset into training and hold-out sets at 50/50 ratio, resulting in 3,100 training and 3,100 test images. The baseline classifier is trained exclusively on this 3, 3.100-image training set. During inference, we apply the same hyperparameters used in the DPO sampling stage when generating synthetic images with the DPO fine-tuned model. We generate one synthetic image for each real image, intentionally assigning a target label that differs from the real image's original label while corresponding to the same body region. Following established practices, we combine synthetic and real images to optimize performance, maintaining fixed ratio of synthetic to real examples in each training batch. All experiments are conducted five rounds on RTX 6000 Ada GPUs. Our experimental evaluation encompasses both CNN-based and meatffomer-bbaeed classifier architectures. fine-tuned according to protocols outlined in previous work [58].
5
Analysis
5.1
Experimental Results
Classification results. We comprehensively evaluate synthetic image quality by its impact on downstream classification using ResNet18 and DINOv2 architectures (Tables and 2). Our MAGIC framework markedly enhances performance across both models compared to baselines. Standard fine-tuned Text- toxt-to-Inne Image (T21)
generation degrades R8 accuracy by -3.74% and DINOv2 by -2. 16%,, while the fine-tuned Image-to-
7

---

Table 1:
Performance of 8 8--aeet8-based classi- fiers trained on real and synthetic data.
Table 2: Performance of DINOv2-based classi- fiers trained on real and synthetic data.
Method
Acc
F1
Prec
Rec
Method
Acc
F1
Prec
Rec
Real
29.31
28.73
28.61
29.13
Real
49.89
49.43
50.03
49.31
+ -22
25.57
24.63
24.44
25.16
+T2
47.73
47.26
47.51
47.43
-3.74
-4.11
-4.17
-3.97
-2.16
-2.17
-2.52
-1.88
+ 121
31.45
31.09
31.03
31.49
121
50.71
50.17
51.04
49.89
+2.14
+2.35
+2.42
+2.36
+0.82
+0.74
+1.01
+0.58
+ MAGIC
33.49
30.40
29.12
29.67
MAGIC
51.16
52.66
52.17
52.69
(RFT)
44.18
+1.67
+0.51
+0.54
(RFT)
+1.27
+3.23
+2.14
43.38
+ MAGIC
38.33
37.01
38.41
36.06
MAGIC
55.01
54.05
54.96
53.70
(DPO)
+9.02
+8.28
+9.80
+6.94
(DPO)
+5.12
+4.62
+4.93
+4.33
51 Soure
24
5m
Oum
Fae
3054
17-2 QUITETT OUe
1110
Gre Oun
(a) Ratio P
(b) #Feedback
(c) FID Score
(d) Synthetic Data Eval
Figure 4: Experimental results showing (a) the impact of ratio p. (b) feedback volume accuracy, (c) FID score comparison across different methods, and (d) evaluation results on synthetic data showing the percentage of criteria met. Our method consistently outperforms baseline methods in most metrics, achieving lower FID scores and higher criteria satisfaction rates.
Image (121) approach offers modest gains, increasing ResNet18 accuracy by +2.14% and DINOv2 by +0.82%. The feedback integrated via our MAGIC framework proves beneficial for both Reward model guided Fine-Tuning (RFT) and Direct Preference Optimization (DPO) strategies. Specifically, MAGIC- CF-FFT improved accuracy over the real data baseline by +4.18% 9% for ResNet18 and +2221% for DINOv2. MAGIC-DPO demonstrated even more substantial gains, boosting accuracy by +9.02% for ResNet18 (from 29.31 % 38.33%) and by +5.12% for DINOv2 (from 99.89% to 55.01%), 01),, with similar improvements FI, precision, and recall. We further validate the MAGIC framework on additional datasets, SCIN and PAD-UFES-20, with results in Appendix D.3.
The DPO approach within the MAGIC framework (MAGIC-DPO) shows particular strength. Its advantage may stem from directly optimizing for preference alignment without intermediate reward model. This can be more robust and generalize better, proving especially advantagcous when the number of feedback pairs is limited, as common specialized medical domains, thus sidestepping potential instabilities quality of expert guidance remains crucial for generating synthetic images that are not only visually plausible but also encode clinically relevant diagnostic features. This enhanced alignment is reflected across our evaluations, including improved qualitative outputs (Fig. 2), FID scores (Fig. 4c), and expert preference measures (Fig. 4d). Appendix D.5 for details).
reward modeling. The
We also evaluate MAGIC-DPO with other classifier hackhones
Expert evaluation generated images. To further assess the quality and medical plausihility of images generated by our methods, we engaged medical experts to evaluate the synthetic data based on our specific checklist criteria. For each method, we sampled 10 images per skin condition, resulting in 200 images per method. Each image was evaluated against 5 criteria, with binary outcomes (satisfied/not satisfied). Fig. 4d summarizes these evaluation results, displaying the pereentage of images meeting different numbers of criteria (with details Appendix B). The results show that images from the pretrained diffusion model rarely satisfied more than one criterion, and none met more than three. Standard Text-to- Text-to-Image (T21) generation showed minimal improvement, with only 2.0% of images meeting 3 or more criteria and only single image meeting 4 criteria overall. Fine-tuned Image-to- Image (12I) generation yielded better outputs, with 18.5% of its images meeting
criteria, underscoring 12T's greater suitability for medical tasks. Our MAGIC framework significantly builds on this; MAGIC-RFT RFT (Ours RFT) further increased the proportion of high-quality images, with 38.9% meeting 3 or more criteria. Notably, MAGIC-DR -PO (Ours DPO) demonstrated the best performance. with 55.5% of its images satisfying or more criteria. This substantial improvement over both tunetuned I21 and MAGIC-RE RET correlates directly with the observed enhancements in classifier performance.

---

Table Performance of DINOv2-based clas-
Table 4: Performance of classifiers across different
sifiers in few-shot setting.
backbones and Coarse/Structured checklists.
Method
Acc
F1
Prec
Rec
Model
Method
Acc
F1
Prec
Rec
Real (all)
49.89
49.43
50,03
49.37
Real
29.31
28.73
28.61
29.13
Real (310)
26.45
19.50
21.86
20.19
MAGIC
32.83
30.58
29.75
31.18
T21
25.58
19.58
20.87
19.27
RN18
Coarse
+3.52
+1.85
+1.14
+2,05
-2.17
+0.08
-0,99
-0.92
MAGIC
38.33
37.01
38.41
36.06
121
30.10
27.26
28.07
27.00
Structured
+9.02
+8.28
+9.80
+6.94
+3.65
+7.76
+6.21
+6.81
Real
49.89
49.43
50.03
49.31
MAGIC
37,39
36,90
37.95
36,94
+ MAGIC
51.16
52.66
52.17
52.69
(DPO)
+10.94
+17.40
+16.09
+16.75
DINO
Coarse
+1.27
+3.23
+2,14
+3.38
MAGIC-A
40.34
39.43
42.20
38.77
MAGIC
55.01
54.05
54.96
53,70
(DPO)
+13.89
+19.93
+20.34
+18.58
Structured
+5.12
+4.62
+4.93
+4.39
Few-shot Setting. We further evaluate our framework in a few-shot setting where only a small number of labeled data are available. This scenario better reflects real-world conditions. as collecting and labeling medical data is both challenging and expensive. We simulate this setting by randomly selecting 10% of the DINOv2 training set (310 images) while keeping the test set fixed. We fine-tuned the diffusion model on these 310 real images using our DPO-based approach (MAGIC-DR) DPO) and other baselines. As shown in Table 3, MAGIC-DPP improves classifier accuracy by +10.94% (from 26.45 26.45% to 37 39%% compared to training with only the limited real data, significantly outperforming standard T21 and 121 augmentation basclines in this data-scarce context. Moreover, practical scenarios, unlabeled medical data from the same distribution may be available even when expert labeling is cost- cost-prohibitive Our MAGIC framework can effectively utilize such unlabeled data; specifically, during the DPO fine-tuning stage, unlabeled data is processed by the diffusion model with randomly selected skin conditions, and feedback is evaluated solely based on the target condition. This makes our framework well-suited for leveraging unlabeled data. This augmented approach, termed MAGIC-A (also DPO (PO-based), demonstrates that by incorporating an equal number of unlabeled samples (310), we can further improve accuracy by an additional 2.95% over MAGIC- HAGIC-DPO, reaching 40.34% accuracy.
by Design. The MAGIC framework is explicitly designed to minimize the risk of MILMM hallucination through our Al-Expert collaboration paradigm. The MLLM is not asked to perform open-ended reasoning. Instead, its role constrained to evaluating an image against predefined clinical check- list. These checklists, designed by dermatologists, decompose complex medical concepts into simple, visually verifiable features. For instance, while an MILM may not intrinsically understand "Lupus Erythematosus." it can effectively verify "symmetric butterfly rash across the checks". This approach transforms a complex diagnostic reasoning task into series of closeu-question evaluations, which are far less susceptible to unconstrained hallucination. Even so, an analysis of GPT-4o's 's hallucination is important. To that end, we have GPT-4o evaluate 100 real images from internal dataset with clinical records confirmed by in-person visits. The MILM is tasked with describing each image based on the five criteria and dermatolo- gist assess these descriptions. As shown in Fig. 5, the dermatologist assigned an alignment score of 3 or greater (on 5 5-point scale) to approximately 86% of the image-descriptin pairs. Notably, many of the lower-scoring examples were also identified by the dermatologist as being visually ambiguous and challenging for human to assess from an image alone.
Hallucination-Resien
(6N)
score 5 (40%)
soore 4 (21%)
Figure 5: Distribution of align- ment scores indicating the number of checklist criteria met in the description.
5.2 Abaltion Study
Effeet of Checklist Quality. We investigate the impact of checklist detail level on the MAGIC framework's efficacy in DPO training. by comparing two types of expert-designed checklists: a "Coarse" version using single-sentenee descriptions for each condition, and a more detailed, "Structured" version (as used throughout the main paper and detailed in Appendix B). Table shows that the quality of the checklist is crucial feedback quality. For the ResNet18 (RN18), augmenting with MAGIC-DPO using Coarse checklists improved accuracy by +3.52% over the real data baseline (from 29.3 31% to 32.83%), whereas Structured checklists led to much larger gain of +9.02% (to 38.33%). A similar trend was observed with the DINOv2 (DINO): Coarse checklists yielded +1.27% accuracy improvement (from 49,89%, to 51.16%). while Structured checklists achieved +5.12% boost (to 55.01%). These results underscore that more detailed and well--stuee structured expert guidance in the checklists significantly enhances the quality of synthetic images and subsequent classitier performance. To explore the upper bounds of this effect, we have dermatologist craft even more fine-graine checklists with nine criteria (Location, Distribution, Lesion Type, Shape/Size, Border, Elevation, Texture, Color, and

---

Translucency//onnlet.. We then use these 9-criteria checklists in our MAGIC framework. We observed small, additional improvement our original S-criteria structured checklist, detailed in the Appendix D.6.
Effect of feedback volume. In addition to feedback quality, also investigate how the quantity of feedback influences image quality and classifier performance. As DPO training progresses, more image pairs are used, providing additional feedback to guide the diffusion model. We visually demonstrate the evolution of generated images across epochs in Fig. 2. Additionally, we also train classifiers using synthetic data that is generated from different training stages. Results in Fig. 4b show that accuracy consistently improves DPO training accumulates more feedback, with performance stabilizing after receiving feedback from approximately 512 image pairs. Based on these findings, we fix the feedhack volume at 1024 image pairs for all our experiments.
Effeet of the ratio p of synthetic data. We investigate how the ratio p of synthetic data affects classifier performance. Initial experiments with purely synthetic data failed to achieve performance comparable to real data-trained classifiers. It's expected that, without the guidance of real data, classifiers tend to overfit to the synthetic data distribution. We therefore systematically controlled the percentage of synthetic data used in each training batch across different values of p. while keeping the total volume of synthetic data constant. As shown Fig. 4a, performance improves when P is less than 0.5 (when synthetic data constitutes less than half of the training data). The performance remains stable when E [0.1, 0.3]. We adopt =
0.2 for all our experiments.
Choice of MILM. The MAGIC framework is designed to be model-agnostic, al- lowing for the flexibility to use Therefore, we also test our MAGIC frame- work with an open-source MLLM, Google DeepMind's 's MedGemma-4B [47], a foundation model for medical text and image comprehen- sion. We used MedGemma as evaluator to assess the same set of 1,024 pairs of synthe- sized images that GPT-4o had assessed. As shown in Tahle 5, we observed that MedGemma is comparable to GPT-44 as an evaluator in aligning the DM using the MAGIC framework. This demonstrates the flexibility of our MAGIC framework, which is adaptive to both large, closed-source, generalist models and smaller, open-source, comain-specifee alternatives. Notably, we observe small performance gap between the pipelines integrated with the two MLLMs. We believe this gap stems from the specific nature of evaluation lask, which is not open-ended medical reasoning but rather a constrained, visual instruction-follont
the most suitable MLLM for given task.
Table 5: Performance of linear classifiers trained on synthetic data from the MAGIC pipeline, aligned using feedback from different MLLMs.
MLLM
Model
Acc
F1
Prec
Rec
ModGemma-4B
ResNet-18 DINOv2
36.97
35.55
37.12
36.32
54.19
53.08
54.78
53,53
GPT-40
ResNet-18
38.33
37.01
36.06
DINOv2
55.01
54.05
$4.96
53.70
evaluation. While MedGemma possesses specialized medical knowledge, GPT-4o's 's massive scale and training on vastly diverse datasets have endowed it with powerful general visual reasoning and instruction mnstuuctio-follooinng capabilities.
Specifically, we hypothesize that GPT-4o's edge comes from its superior ability to parse the descriptive, often non-clinical language of the checklists (e.g.. "bull's-eye lesions." " "butterfly rash") and precisely map these concepts to visual features. In contrast, while MedGemma is fine-tuned on medical data, smaller scale may slightly limit its raw visual-language alignment and nuanced insructtionfollowing abilities. Most importantly, we see the comparable performance of MedGemma as a validation of our framework's 's flcxibility, demonstrating that MAGIC is not dependent single, closed-source model and can he adapted using accessible alternatives.
Conclusion
In this work, we introduced MAGIC, a novel cemi-automated framework designed to refine Diffusion Models hy effectively integrating expert-enhanced clinical knowledge. Our approach uniquely leverages the visual reasoning capabilities of MI..Ms to interpret and apply expert-defined checklists, therehy guiding DMs to produce images with high clinical fidelity while significantly reducing the burden human experts. Our experiments demonstrate that MAGIC, substantially improves the clinical quality of synthesized skin disease images, as validated by both quantitative metrics like FID scores and qualitative assessments by dermatologists. Furthermore, augmenting training data with images generated by MAGIC led to significant enhancements in downstream classification accuracy for skin discases, even in few-shot scenarios. These results underscore the efficacy our Al-Expert collaboration paradigm in translating nuanced clinical criteria into actionable feedback for generative models. We acknowledge that our framework's 's performance is linked to the capabilities of the MILM used. However, MAGIC is designed to be model-agnostic and flexible. This flexibility is also an advantage, as the framework's 's performance will naturally improve with the continually advancing interpretive capabilities of MILMs. Beyond image synthesis, MAGIC demonstrates a task-centric alignment paradigm: instead of adapting MLLMs to niche medical tasks, adapts tasks to the strengths of general purpose MLLMs by decomposing domain knowledge into attribute- checklists. This task-centric alignment is particularly valuable given that the most powerful MLLMs are often proprietary, and training domain- domain-specific MLLMs is costly. This design offers scalable and reliable path for leveraging foundation models in specialized domains.
10

---

Acknowledgment
We thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the NSF EPSCoR-Louisiana Materials Design Alliance (LAMDA) program #01A-1946231.
References
[1] 1st Pushkar Aggarwal. Data augmentation dermatology image recognition using machine learning. Skin Research and Technology, 25(6):115-82,, 2019.
[2] Mohamed Akrout, Bálint Gyepesi, Péter Holló, Adrienn Poór, Blága Kineső, Stephen Solis, Katrina Cirone, Jeremy Kawahara, Dekker Slade, Latif Abid, al. Diffusion-based data augmentation for skin disease classification: Impact across original medical datasets fully synthetic images. In International Conference on Medical Image Computing and Cumputer-Assisted Intervention, pages 99-109. Springer, 2023.
[3] Hazrat Ali, Shafaq Murad, and Zubair Shah. Spot the fake lungs: Generating synthetic medical images using neural diffusion models. In Irish Conference Artificial Intelligence and Cognitive Science, pages 32-39. Springer, 2022.
[4] Kevin Black, Michacl Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. 2023.
[5] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.1338 13301, 2023.
[6] Titus Brinker, Achim Hekler, Alexander Enk, Joachim Klode. Axel Hauschild, Carola Berking, Bastian Schilling. Sebastian Haferkamp, Jochen Utikal. Christof Kalle, Stefan Fröhling. and Michael Weichenthal. convolutional neural network trained with dermoscopic images performed on par with 145 dermatologists in a clinical melanoma image classification task. European Journal of Cancer, 111:148-154 154, 03 2019.
[7] Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 36856456) 2019.
[8] Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-a-judge: Assessing multimodal Ilm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning, 2024.
[9] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Logg, and Dario Amodei. Deep rein- forcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[10] Yunsung Chung. Yunbei Zhang, Nassir Marrouche, and Jihun Hamm. Sok: Can synthetic images replace real data? survey of utility and privacy of synthetic image generation. arXiv preprint arXiv:2506. 19360, 2025.
[11] Roxana Daneshjou, Kailas Vodrahalli, Roberto A Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey, Olivier Gevaert, et al. Disparities in dermatology ai performance a diverse, curated clinical image set. Science advances, rcabb614,, 2022.
[12] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[13] Dexin Duan, Peilin Liu, Bingwei Hui, and Fei Wen. Brain-inspired online adaptation for remote sensing with spiking neural network. IEEE Transactions on Geoscience and Remote Sensing, 2025.
[14] Miroslav Dudik, Katja Hofmann, Robert E. Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory, pages 563-587. PMLR, 2015.
[15] Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin M. Ko, Susan Swetter, Helen M. Blau, and Sebastian Thrun. Dermatologist-leww classification of skin cancer with deep neural networks. Nature, 542:115-118, 2017.
[16] Ying Fan and Kangwook Lee. Optimizing ddpm sampling with shortcut fine- fine-tuning. arXiv preprint arXiv:2301.13362 2023,

---

[17] Ying Fan, Olivia Watkins, Yuqing Du Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mo- hammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 79858 01588-99855. Curran Associates, Inc., 2023.
[18] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mo- hammad Ghavamzadch, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. arXiv preprint arXiv:2305. 16381, 2023.
[19] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
[20] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen Or. An image is worth one word: Personalizing text-to- to-image generation using textual inversion. arXiv preprint 22Xi:22088006188 2022.
[21] Wentao Ge, Shumian Chen, Guiming Chen, Junying Chen, Zhihong Chen, Shuo Yan, Chenghao Zhu, Ziyue Lin, Wenya Xic, Xidong Wang, et al. Milm-bench, evaluating multi-modal Ilms using gpt-4v. arXiv preprint or Xiv:: 1. 13951, 2023.
[22] Scott Geng, Cheng-Yu Yu Hsieh, Vivek Ramanujan, Matthew Wallingford, Chun-Liang Li, Pang Wei W Koh, and Ranjay Krishna. The unmet promise of synthetic training images: Using retrieved real images performs better. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 29227929. Curran Associates, Inc.. 2024.
[23] Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Acrin Kim, Arash Konchek, and Omar Badri. Evaluating deep neural networks trained clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1820-1828, 2021.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[25] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
[26] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[27] Xin Hu, Janet Wang, Jihun Hamm, Ric Yotsu, and Zhengming Ding. Enhancing skin discase diagnosis: Interpretable visual concept discovery with sam. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 172-181. IEFF, 2025.
[28] Peng Huang. Xue Gao, Lihong Huang, Jing Jiao, Xiaokang Li, Yuanyuan Wang, and Yi Guo. Chest- diffusion: a light- text-to-image model for report-to-cxr generation. In 2024 IEEE International Symposium on Biomedical Imaging (ISBI), pages 1-5. IEEE, 2024.
[29] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. Visual prompt tuning. In European conference on computer vision, pages 109-727.35 Springer, 2022.
[30] Firas Khader, Gustav Müller-Franzee Soroosh Tayehi Arasteh, Tianyu Han, Christoph Haarburger, Maxim- ilian Schulze-Hagen, Philipp Schad, Sandy Engelhardt, Bettina Baeßler, Sehastian Foersch, et al. Denoising diffusion probabilistic models for 3d medical image generation. Scientific Reports, 13( 3())73333, 2023.
[31] Ira Klena, Olivia Wiles, Isabela Albuquerque, Sylvestre-Alvise Rebufli, Ryutaro Tanno, Abhijit Guha Roy, Shekoofeh Azizi, Danielle Belgrave, Pushmeet Kohli, Taylan Cemgil, al. Generative models improve fairness of medical classifiers under distribution shifts. Nature Medicine, pages 1-8, 2024.
[32] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop. Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.0022 00267, 2023.
12

---

[33] Kimin Lee. Hao Liu. Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzzdeh,, and Shixiang Shane Gu. Aligning text-to-image models using human feedback, 2023.
[34] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadch, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2330..1 1292, 2023.
[35] Ming Liu, Hao Chen, Jindong Wang, Liwen Wang, Bhiksha Raj, and Wensheng Zhang. On fairness of unified multimodal large language model for image generation. The Thirty-ninth Annual Conference on Newral Information Processing Systems, 2025.
[36] Yiyang Liu, James Chenhao Liang, Ruixiang Tang, Yugyung Lee, Majid Rabbani, Sohail A. Dianat, Raghuveer Rao, Lifu Huang, Dongfang Liu, Qifan Wang, and Cheng Han. Re-imagining multimodal instruction tuning: A representation view. In ICLR, 2025.
[37] Yuan Liu, Ayush Jain, Clara Eng. David H Way. Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, et al. A deep learning system for differential diagnosis of skin diseases. Nature medicine, 26(6):900-99) 900000998,,00000 2020.
[38] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2537-2546, 2019.
[39] Zheda Mai, Ruiwen Li, Jihwan Jeong. David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28 51, 2022.
[40] OpenAL. Gpt- system card, 2024.
[41] Maxime Oquah, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Syanaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Learning robust visual features without supervision. 2023.
Dinov2:
[42] Walter HL Pinaya, Petru- Daniel Tudosiu, Jessica Dafflon, Pedro F Da Costa, Virginia Fernandez. Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso. Brain imaging generation with latent diffusion models. In MICCA/ Workshop on Deep Generative Models, pages 117-126. Springer, 2022.
[43] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. Advances in Neural Information Processing Systems, 36:53728 53741, 2023.
[44] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684- 10695, June 2022.
[45] Luke W Sagers, James A Diao, Luke Melas-Kyriazi, Matthew Groh, Pranav Rajpurkar, Adewole S Adamson, Veronica Rotemberg, Roxana Daneshjou, and Arjun K Manrai. Augmenting medical image classifiers with synthetic data from latent diffusion models. arXiv preprint S..Xi::238..1455: 2023.
[46] Luke William Sagers, James Diao, Matthew Groh, Pranav Rajpurkar, Adewole Adamson, and Ar- jun Kumar Manrai. Improving dermatology classifiers across populations using images generated by large diffusion models. In NewrlPS 2022 Workshop on Synthetic Data for Empowering MI. Research, 2022.
[47] Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly. Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint S.Xi::2507.05201 2025.
[48] Joonghyuk Shin, Minguk Kang, and Jacsik Park. Fill-up: Balancing long-tailed data with generative models. arviv preprint arXiv:2306.07200 2023.
[49] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwisser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484449) 2016.
[50] David Silver. Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
13

---

[51] Karan Singhal, Shekoofeh Azizi, Tao Tu. S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172-188, 2023.
[52] Luis R Spenksen, Timothy Kassis, Susan T Conover, Berta Marti-Fuster, Judith S Birkenfeld, Jason Tucker-Schwartz, Asif Naseem, Rohert R Stavert, Caroline Kim, Maryanne M Senna, et al. Using deep learning for dermatologist-leve detection of suspicious pigmented skin lesions from wide-field images. Science Translational Medicine, 13(581 )ceabb3652, 2021.
[53] Shenghuan Sun. Greg Goldgof, Atul Butte, and Ahmed M. Alaa, Aligning synthetic medical images with clinical knowledge using human feedhack. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 13408-13428. Curran Associates, Inc., 2023.
[54] Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li. Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang. Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2.309. 14525, 2023.
[55] Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning. 1998.
[56] Cheng-Hao Tu, Zheda Mai, and Wei-Lum Chao. Visual query tuning: Towards effective usage of intermedi- ate representations for parameter and memory efficient transfer learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 7725-7735, 2023.
[57] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 8228 8238, 2024.
[58] Janet Wang, Yunsung Chung, Zhengming Ding, and Jihun Hamm. From majority to minority: A diffusion- based augmentation for underrepresented groups in skin lesion analysis. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 14-23. Springer, 2024.
[59] Janet Wang, Xin Hu, Yunbei Zhang, Diabate A Almamy, Vagamon Bamba, Konan Amos Sébastien Koffi, Yao Koffi Aubin, Zhengming Ding, Jihun Hamm, and Rie R Yotsu. eSkinHealth: A multimodal dataset for neglected tropical skin diseases. In ACM Multimedia 2025, 2025.
[60] Janet Wang. Yunbei Zhang, Zhengming Ding, and Jihun Hamm. Achieving reliable and fair skin lesion diagnosis via unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 5157-5166, 2024.
[61] Taowen Wang, Yiyang Liu, James Chenhao Liang, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng. Zenglin Xu, Cheng Han, et al. Mºpt: Multimodal prompt tuning for zero-shot instruction learning. In EMNLP, 2024.
[62] Abbi Ward, Jimmy Li, Julie Wang, Sriram I Lakshminarasimhan, Ashley Carrick, Bilson Campana, Jay Hartford, Pradeep K. Sreenivasaiah, Tiya Tiyasirisokchai, Sunny Virmani, Rence Wong, Yossi Matias, Greg S. Corrado, Dale R. Webster, Margaret Ann Smith, Dawn Siegel, Steven Lin, Justin Ko, Alan Karthikesalingam, Christopher Semturs, and Pooja Rao. Creating an empirical dermatology dataset through crowdsourcing with web search advertisements. JAMA Network Open, 7(1 022246615--2446661 11 2024.
[63] Yuxiang Wei, Yanteng Zhang, Xi Xiao, Tianyang Wang, Xiao Wang, and Vince D. Calhoun. More-brain: Routed mixture of experts for interpretable and generalizable cross-subject fmri visual decoding, 2025.
[64] Xi Xiao, Yunbei Zhang, Xingjian Li, Tianyang Wang, Xiao Wang. Yuxiang Wei, Jihun Hamm, and Min Xu. Visual instance-waa prompt tuning. In ACM Multimedia 2025, 2025.
[65] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.055 05977, 2023.
[66] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai L.,, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. arXiv preprint arXiv:2311. 132331, 2023.
[67] Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huangg. Ying Nian Wu, and Dongtang Liu. Visual fourier prompt tuning. Advances in Neural Information Processing Systems. 2024.
14

---

[68] Kaichen Zhang, Bo Li, Peiyuan Zhang. Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, al. Lums-eval: Reality check the evaluation of large multimodal models. arXiv preprint urXiv:2407. 12772, 2024.
[69] Xinlu Zhang. Yujie Lu, Weizhi Wang. An Yan, Jun Yan, Lianke Qin, Heng Wang. Xifeng Yan, William Yang Wang. and Linda Ruth Petzold. Gpt-4v (ision) as generalist evaluator saioo-language tasks. arXiv preprint xxXi:2311.01300 2023.
[70] Yunbei Zhang, Akshay Mehra, and Jihun Hamm. Ot-vp: Optimal transport-guicn visual prompting for cest-time adaptation. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pages 1122 1132, February 2025.
[71] Yunbei Zhang, Akshay Mehra, Shuaicheng Niu, and Jihun Hamm. DPCore: Dynamic prompt coresel for continual test-time adaptation. In Forty-second International Conference on Machine Learning, 2025.
[72] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li. Dacheng Li. Eric Xing, et al. Judging Ilm an-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-4666 2023.
15

---

Appendix
A Additional Implementation Details
this section, we present additional implementation details of our proposed method.
A.1 Pre-Feedback Fine-tuning
For textual inversion, learn the embedding for each skin condition through various prompts. These prompls are used to ensure robust learning the text embedding across different phrasings and contexts:
skin weinndi daiiinniiinn sease_prromps
"a photo of a {)",
"a rendering of a ()",
"a crepped photo of the ()",
"the photo of 11", "a close-up photo of ()",
"a cropped photo of a (}",
"a photo of the ()",
"a photo of one ()",
"a close-up photo of the (,",
"a rendition of the ()",
rendition of ()"
text embeddings are learned
entire training set. The Adam W optimizer used with learning rate
of 5 x 10
For LoRA, the rank is set to 32, and the learning rate 5 10 for AdamW optimizer.
A.2 MILM Score Processing for Preference Pairs
To translate 5-dimensional binary MLLM scores into preference signals for DPO, pairs of generated images are processed. For each image in pair, its 5 binary scores are summed to get Sj and S2. max(S1, S2) S 2, both images are deemed low quality (outcome e.g., [0, 0j). If min(S1, S2) = 5. or if S S2 > 2, the pair is marked "both win" (e.g.. [1, 1]). Otherwise, if Sj 82, the first image the "winner" (e.g., [1, OD: if S2 S1, preeferrr-ddd samples DPO loss computation. The
the second wins (e.g.. j0. 1j). This determines distribution of these outcomes is in Table 6.
A.3 DPO fine-tuning
We conduct DPO fine-tuning for 128 iterations and each iteration, pairs (16 images) will sampled. The denoise strength is set to 0.3. The DPO loss will be computed with the feedback. We utilize AdamW optimizer with learning rate of 0.0001.
A.4 Classifier Training
We utilize the Adam optimizer with a learning rate 0.01 and step learning rate scheduler that reduces the learning rate 0.1 of its previous value every 50 epochs. The classifier is trained for 200 epochs to ensure stable results. Each result reported in the table represents the average of five with different random seeds.
B Expert Designed Checklist
We enclose the checklist used in the experiment in this section. For each skin condition, we design checklist evaluations from perspective of [Location, Lesion Type, Shape/Size Color, Texture] to capture the visual concept from the synthetic data. The details are shown in Table 14.
C Automate Evaluation via MLLMs
For each pair of data, we use the following prompt to collect feedback from ChatGPT-44:
prompt Evvluutte images against the
16

---

following checklist:
Icondition (enddttion_chekklist] checklist]
Return 1ist indicating whether it satisfics each checklist
item (1 for satisfied, 0 otherwise),
Only the 1ist of results should be returned. Expected format:
[1, 0, 1, 0, 0)"*
D Addtional Results
D.1 Distribution of Feedback
For each pair of data, our approach categorizes feedback into three types: both win ((w- 0, IJ.. both lose (l = 0, = 1)), and better than the other ([w = 0,7 = 1] or [l 0, 1]). We present the distribution of feedback received during DPO training Table
D.2 More examples of image pairs
We provide two more image pairs in Fig.
D.3 Results on SCIN and PAD-UFES-20
The SCIN dataset [62], collected via voluntary image donation platform from Google Search users in the United States, typically includes up to three images per case, each evaluated by up three dermatologists. This diagnostic process yields a weighted skin condition label for each case. To ensure label accuracy for our study, we selected the condition with the highest weight as the definitive label. discarding ambiguous cases where multiple conditions had equal probabilities. Our analysis concentrated on the 10 most prevalent classes in the real world. Given that the SCIN dataset exhibits an imbalanced class distribution, we first sampled uniformly distributed test set, following methodologies similar to ImageNet-LTT LT [38]. Furthermore, guided by approaches like that of [48], we employed our MAGIC-DE IPO framework to generate additional synthetic images for each condition, aiming to augment the test set towards more uniform distribution. Further details the dataset distribution are provided in Table 8. However, experiments conducted with this augmented SCIN dataset yielded suboptimal results, potentially attributable to inherent noise within the dataset, a challenge noted works such as [27].
Our MAGIC framework's 's effectiveness is further validated on the SCIN dataset, with detailed performance for both ResNet18 and DINOv2 classifiers presented in Table 9 and Table 10. For the ResNet18 classifier on SCIN, models trained on real data achieved an accuracy of 13... Standard T2I augmentation slightly decreased this 22. 22.60% (-0.5%), while 12T augmentation offered modest improvement 24.13% (+1.0%) In contrast, our MAGIC framework demonstrated more substantial gains: MAGIC-RFT increased accuracy to 26.58% ( (+3.5%), and MAGIC-DD DPO further improved to 29.43% (+6.3%). A similar trend was observed with the DINOv2 classifier, which had baseline accuracy of 30.61% on real SCIN data. T21 augmentation reduced accuracy 28%18% 2.44%) and 12I provided small increase to 15% (+1.5%). Both MAGIC strategies again outperformed these: MAGIC-RFT achieved 33.82% accuracy (+3.2%), while MAGIC-DPO led with 35.65% (-5.0%). These results on the SCIN dataset consistently show the advantages leveraging MAGIC, with both RFT and DPO components enhancing performance over standard augmentation techniques, and DPO often yielding the highest accuracy.
To quickly evaluate the cross-datasel generalizability of our method, we identified four overlapping classes between the hospital-grade PAD-UFES-20 dataset and Fitzpatrick 17k subset (ACK, BCC, MEL, and SCC), and ran MAGIC-DPO pipeline. The results, presented in the Table 11, demonstrate that the MAGIC framework is generalizable to hospital-gran -rade datasets.
D.4 Score change during training
Figure 6 illustrates how the clinical quality of generated images, assessed by the number of satisfied expert- defined criteria, evolves throughout the feedback-guided training phase of our MAGIC framework. Initially, images from the Pre-trained model and the fine-tuned Text-to-Image (T2I) model satisfy very few criteria, with average scores of 0.3 and 0.5, respectively. Even the fine-tuned Image-to-Image (I2I) model, at the beginning of feedback training (Iteration 0), achieves an average of only criteria met. As the model receives more feedback and training progresses (Iterations 32 through 128), a significant improvement is observed. The distribution of scores progressively shifts towards satisfying a higher number of clinical criteria, with the average number of criteria met increasing steadily from 1.4 to 3.0 by Iteration 128. This trend clearly demonstrates the diffusion
17

---

model's ability to learn from and adapt to the expert-derived feedback over time, resulting in generated images that are increasingly more aligned with clinical requirements for medical accuracy.
D.5 Evaluation with Other Backbones
stronger backhone such as DINOv2 starts with a clear advantage: it has a complex transformer architecture and is pre-trained on massive dataset, enabling it to extract robust image features from the start. In contrast, a smaller model like ResNet-18 struggles with the limited and challenging real data. The differing gains (+9.02% for ResNet18 vs. +5.12% for DINOv2) therefore highlight a key finding: our augmentation framework provides the most significant benefit data-scarce or model-constrained scenarios.
However, the fact that MAGIC still substantially boosts the performance of a powerful model like DINOv2 is a strong testament to the quality of our synthetic data. It demonstrates that MAGIC generates images with clinically accurate features that even a strong classifier cannot extract from the limited real datasct alone. This conclusion is supported by few-shot experiments in Table 3. When the training set reduced to just 10% (310 images), MAGIC-DPO provided sizable +10.94% accuracy improvement for the DINOv2 classifier. This shows that as data becomes more scarce, the value of our synthetic augmentation becomes even more pronounced. To further explore this interesting effect, we have run additional experiments with other classifier backbones of various sizes. The results, presented in the Table 12, are consistent with our observations.
D.6 Effect of Checklist Granularity
As shown in Table 13, this analysis leads to two key insights: (1) detailed, structured checklist critical for the framework's 's success, as shown by the significant performance jump from the "Coarse" to the "Structured" checklist. (2) There may be diminishing returns after certain level of detail is achieved, as shown by the smaller gain when moving from the 5-criteria to the 9-criteria checklist. This suggests our original 5-criteria checklist was already capturing the most essential features for high-quality generation.
D.7 Discussion about Diversity
There are two distinct and important aspects of diversity:
Inter-Site Diversity: Can the model generate the same condition with clinically appropriate, site- specific features (c.g., does Lupus on the scalp look different from Lupus on the face)?
Intra-Site Diversity: Can the model generate multiple, varied appearances of the same condition at the same site (c.g., 100 lifferent-looking examples of "Lupus on the facc")?
The MAGIC framework is designed to address both of these challenges, as claborated below.
The model's ' ability to render a condition with site-specific features is driven by the synergy between our 121 pipeline and the structured checklists. The 121 pipeline grounds the generation process in specific anatomical context by starting with real source image (e.g.. scalp with hair). Guided by the feedback loop. the model is then tasked with generating features that satisfy the clinical checklist within the visual and anatomical constraints that source image. While many of the selected conditions do not exhibit strong site specifiiif, our expert-designed checklists are nuanced enough to include these manifestations wherever applicable. For example, the checklist for "Lupus Erythematosus" explicitly guides the model toward a "symmetric butterfly rash across the cheeks" when the target is the face, and "discoid coin-shaped lesion" otherwise. Over time, the MI. M.-rrvvnn feedback rewards the model for plausibly blending the target lesion features with the source anatomical context.
Inter-Site Diversity:
2. Intra-Site Diversity: As noted in our initial rebuttal, our framework enhances diversity primarily through variation in the source images. To generate 100 diverse images of "Lupus Erythematosus on the face." we begin with 100 different real source images of faces, which naturally contain diversity in skin tone, age, gender, and so on. Our 121 process transforms the lesion on each unique face into lupus while preserving the source's individual characteristics. The resulting synthetic images are therefore as diverse as the original source images. Furthermore, this potential is not limited to the labeled training set, as our framework can effectively use unlabeled images source for generation, dramatically expanding the pool of available image contexts. Additionally, the model generates diverse outputs even when starting from the exact same source image. The denoising diffusion process is inherently stochastic, beginning with a randomly noised vector and denoising it into a different final image. Our DPÓ fine-tuning ensures that these random variations remain within the manifold of what is clinically plausible, resulting in meaningful variations in lesion presentations.
Here, we qualitatively confirm that images generated by our MAGIC framework exhibit both types of diversity, shown by Fig. 8
18

---

E
Discussions
While our MAGIC framework demonstrates significant promise, several exciting avenues for future work could enhance its efficiency and adaptability in clinical settings. The current approach relies LoRA for efficient fine-tuning, but exploring alternative Parameter- Efficient Fine-Tuning (PEFT) methods, such as Visual Prompt Tuning (VPT) [29, 64, 56, 67, 61, 36), could offer different tradc-offs in performance and computational cost, especially when adapting to new visual concepts. Furthermore, our few-shot experiments highlight the framework's ' potential to leverage unlabeled data, which could be formalized into robust semi-supervised learning paradigm to further mitigate data scarcity. In real-world scenario, diagnostic models must evolve as new disease data becomes available. Integrating principles from continual learning and online adaptation [71, 39, 70, 13J would enable the generative model to learn new skin conditions incrementally without suffering from catastrophic forgetting of previously learned ones. Additionally, we will investigate our pipeline's potential in real-world dermatological applications, such synthesizing data for skin neglected tropical diseases (skin NTDs). This group of skin conditions impacts millions of people but suffers from serious data scarcity [59]. Future work could also explore the broader utility and privacy implications such synthetic data replacement strategies [10].
F
Limitations
The efficacy of our MAGIC framework, like similar (codback-drive approaches, is naturally guided by the detail within the expert-crafted checklists and the continually advancing interpretive capabilities of Multimodal Large Language Models (MLLMs). The scope of conditions and populations within the dermatology datasets utilized (Fitzpatrick1 17k and SCIN) provides the foundation for the current findings, and extending this work even broader and more varied datasets presents an exciting avenue for future research. While MAGIC demonstrates considerable potential in dermatology. its promising AI- Al-Expert collaboration paradigm also invites future exploration and adaptation to enhance synthetic data generation in other medical imaging fields, each with its unique visual characteristies and clinical requirements.
G Failure Analysis of GPT-40
Based qualitative review of the MLLM's 's evaluations, we identify several recurring challenges and potential failure modes, which are detailed below. A systematic analysis of these failure modes is valuable direction for future research:
Subtle Textural and Morphological Details: The MLLM can struggle with very fine surface textures or lesions that are not well-defined. For example, small papules or pustules can be difficult for the model to assess, especially when their color is indistinguishable from the surrounding skin.
Complex Color Nuances: Diffcrentiating between similar shades (c.g., pink vs. reddish) or accurately interpreting colors on darker skin tones, such as the "purple/dark brown with grayish scales" described for Psoriasis, can be difficult from a 2D image alone.
Inferring 3D Characteristics: Features that imply three-dimenssonality, such as the "firm nodules" of Prurigo Nodularis or the "pearly hump" of Basal Cell Carcinoma, are inherently challenging to assess from a single 2D photograph.
Ambiguity and Confounding Factors: For example, one case, GPT-4o incorrectly identified medicine powder on a patient's 's skin as a white, hypopigmented scale. Without the accompanying clinical record, a human dermatologist would likely find it difficult to distinguish this from the image alone. This highlights that for both humans and AI, visual data can be ambiguous, and other factors like suboptimal lighting or the lack of clinical metadata can impede a purely visual assessment.
H Ethical Considerations and Safeguards
Our MAGIC framework is designed with privacy in mind. This "factorized transformation" preserves only the high-level anatomical context while overwriting the Ene-grained lesion details. This dissociates the original identity from the new condition, which both enhances privacy and reduces the risk of the classifier learning spurious correlations. While this approach is designed to be privacy-conscioss, we acknowledge that it does not offer the formal guarantecs of methods like Differential Privacy. Additionally, privacy risks can be further minimized by running open-source HIPAA-compliant MLLMs locally as evaluators. While formal, quantitative analysis guaranteeing the complete removal of all identifying features was beyond the scope this work, we agree that aligning diffusion models while removing identifying cues is an important future direction for medical and other privacy- critical domains.
19

---

Average: 0.5
Average 0.5
Avarage: 1.4
Average: 2.0
Aswcage: 2.7
Average: 2.9
Asacage: 5.0
Figure 6: Feedback distribution as training progresses.
A pair of images: 1}
Evaluation Chec klists
Image
Image
anywhore
plocucs papulcs
imggstr
Texture: ocotee that peotecioff
x
Torger cond ton, prurigo nodiuaaris
Image
Image1 1
Lucation: me lage
Lesion footure muttiplc fimm nedules
Shapahace: round
Color pinkhhe//brow buso sonde- typerpigmented
tick, rough, craned sratbeed
Figure 7: Two image pairs with the corresponding checklist.
SES
SPS
Figure 8: First row: diverse images for "Lupus Erythematosus on the face". Second row: diverse images of "Lupus Erythematosus" generated on different anatomical sites.
20

---

Table 6: Distribution of feedback
both win only one win
both lose
count
295
397
332
Table 7: Skin Condition Distribution for Fitzpatrick 17kk 17k
Skin Condition
Real Training
Real Test
Syuthetic
Acne
92
91
93
Actinic Keratosis
88
87
164
Allergic Contact Dermatitis
215
215
Basal Cell Carcinoma
234
234
154
Eczema
102
102
166
Erythema Multiforme
118
118
155
Folliculitis
171
171
114
Granuloma Annulare
106
105
148
Keloid
78
78
135
Lichen Planus
246
245
151
Lupus Erythematosus
205
205
172
Melanoma
130
155
Mycosis Fungoides
91
91
165
Pityriasis Rosea
96
97
156
Prurigo Nodularis
85
85
152
Psoriasis
326
327
165
Sarcoidosis
174
175
162
Scabies
170
169
176
Squamous Cell Carcinoma
290
291
175
Vitiligo
83
83
161
Total
3100
3100
3100
Table Skin Condition Distribution for SCIN
Skin Condition
Real Training
Real Test
Synthetic
Eczema
409
36
Urticaria
178
Folliculitis
33
Tinea
72
58
Psoriasis
57
39
70
Herpes Simplex
49
36
76
Acne
44
80
Herpes Zoster
41
82
Pityriasis rosca
41
32
82
Tinea Versicolor
27
34
93
Total
1022
340
574
Table Performance of R classi- fiers trained on real and synthetic data for SCIN.
Table 10: Performance DINOv2-based classi- fiers trained on real and synthetic data for SCIN.
Training data
Prec
Rec
Training data
Acc
F1
Prec
Rec
Real
23.13
10.94
12.20
10.70
Real
30.61
18.37
21.23
17.45
+ T21
22.60
10.44
12.43
10.96
+ T21
28.18
17.48
20.23
16.15
-0.5
-0.5
+0.2
+0.3
-2.4
-0.9
-1.0
-1.3
121
24.13
10.90
12.03
11.06
121
32.15
20.10
23.80
19.06
+1.0
-0.2
+0.4
+1.5
+1.7
+2.6
+1.6
MAGIC
26.58
11.69
15.79
11.89
MAGIC
33.82
20.08
24.16
18.70
RFT
+3.5
+0.7
+3.6
RFT
+3.2
+1.7
+2.9
+1.2
MAGIC
29.43
12.16
18.18
11.47
MAGIC
35.65
21.39
24.00
19.40
DPO
+1.2
+6.0
DPO
+5.0
+3.0
+2.8
21

---

Table 11: Performance (%) of MAGIC-DPO on PAD-UFES-20.
Model
Setting
Accuracy (%)
F1 (%)
Precision (%)
Recall (%)
Real
63.02
51.97
61.07
48.34
ResNet-18
MAGIC
70.65
58.48
62.40
51.65
Real
67.88
60.50
66.29
57.10
DINOv2
MAGIC
73.85
63,81
67.17
59.41
Table 12: Classifier accuracy (%) real data MAGIC-augmented training. Gain denotes absolute
improvement.
Model
#Params (M)
Pre-train Dataset
Real Ace (%)
MAGIC Ace (%)
Gain (%)
ResNet-18
12
ImageNet-Ik -k
29.31
38.33
+9.02
ResNet-50
26
ImageNet-1K
35.56
46.24
+10.68
ViT-B/16
86
ImageNet-21k
45.22
51.44
+6.22
DINOv2
87
LVD-142M
49.89
55.01
+5.12
Table 13: Effect of checklist granularity on performance (%).
Model
Setting
Acc (%)
F1 (%)
Pree (%)
Rec (%)
w/ Coarse Checklist (1 sentence)
32.83
30.58
29.75
31.18
ResNet-18
w/ Fine-grained Checklist (7 criteria)
38.33
37.01
38.41
36.06
Highly Fine- sineigrained Checklist (9 criteria)
39.39
37.86
40.64
39.75
Coarse Checklist (1 sentence)
$1.16
$2.66
$2.17
52.69
DINO-v2
Fine-grained Checklist (7 criteria)
55.01
54.05
54.96
53.70
w/ Highly Fine-grained Checklist (9 criteria)
55.88
55.60
56.56
55.51
22

---

Table 14: Skin Conditions and Their Checklist Properties
Skin Condition
Checklist Details
Acne
1. Location: Face, forehead, chest, shoulders, upper back areas with many oil glands)
2. Lesion Type: Bumps including comedones (whiteheads, blackheads) and inflamed pimples (papules, pustules, nodules)
3. Shape/Size: Small clogged-pore bumрs; larger tender nodules/ cysts in severe cases
4. Color: Red or skin-colored bumps tmay appear purple/brown on dark skin); blackheads have dark plug, whiteheads have white tip
5. Texture: Oily or shiny skin with multiple bumps; some lesions with pus or crust if ruptured
Actinic keratosis
1. Location: Sun-exposed areas (face, scalp, ears, neck,
forearma, backs of hands)
2. Lesion Type: Rough, scaly patch or small crusty bump
3. Shape/Size: Flat slightly raised lesion, usually under
2.5 cm
4. Color: Pink, red, or brownist sh, pessibly with a yellowish
crust; on darker skin can appear gray or dark
5. Texture: Dry, coarse, sandpaper-like surface; may have hard wart-like feel
Allergic contact dermatitis
1. Location: Where allergen contacts skin (hands, face, eyelids neck, etc.)
2. Lesion Type: Red patches often with small blisters (vesicles swelling
3. Shape/Size: Irregular shape fellewing exposure pattern; size depends on contact area
4. Color: Pink to red on light skin; can be darker, purple, or brownish dark skin
5. Texture: May be weepy, crusty, scaly; inflamed and swollen in acute cases
Basal carcinoma
1. Location: Sun-exposed areas (face, nose, ears, neck, scalp, shoulders)
2. Lesion Type: Pearly waxy bump/nodule, flat scaly patch with raised edge
3. Shape/Size: Small, round/oval; can ulcerate or develop a central depression
4. Color:
Translucent pearly en fair skin; brown/black glossy dark on darker skin
5. Texture:
Smooth, shiny surface; can crust or scab with contral ulcoration
Continued on next page
23

---

Table 14 continued from previous page
Skin Condition
Checklist Details
Eczema
1. Location: Flexural areas (inner elbows, behind knees), hands
ankles, neck, eyelids, cheeks
2. Lesion Type; Patches or plaques, sometimes with small blisters bumps
3. Shape/Size: Ill-defined patches varying in size; often bilateral or symmetric
4. Color:
Red or pink on lighter skin; purple, gray, or dark
brown darker skin
5. Texture: Dry, flaky, scaly; become thick and leathery (Lichenification)
Erythema multiforme
1. Location: Hands, feet, arms, legs, can involve mucous
membranes (lips, mouth, eyes)
2. Lesion Type: Target (bull's-eye) lesions with concentric rings
3. Shape/Size: Round lesions (1-3 cm) with a dark center, pale
ring, and outer red ring
4. Color:
Center is dark red/purple, ring is lighter pink, outer zone is red; dark skin, may be grayish hyperpigmented center
5. Texture: Mostly flat but can have a blistered or raised center
Folliculitis
1. Location: Hair-bearing areas prone friction shaving beard, scalp, underarms, legs, buttocks)
2. Lesion Type: Small pustules or red papules centered around hair follicles
3. Shape/Size: Clusters of 2-5 mm bumps; each with central hair
4. Color: Red or pink on light skin; darker or hyperpigmented on dark skin; pus may appear white/yellow
5. Texture: Dome-shaped, often with a fluid-filled top; can crust if ruptured
Granuloma annulare
1. Location: Hands, feet, wrists, ankles (localized) ; can appear on trunk/limbs if generalized
2. Lesion Type: Smooth, firm bumps (papules) forming rings; typically non-scaly
3. Shape/Size: Annular (ring-shaped) up to few wide; papules are a few mm each
4. Color: skin-colored, pink, or reddish; can appear purple on darker skin
5. Texture: Generally smooth; little to no flaking crust
Continued on next page
24

---

Table 14 continued from previous page
Skin Condition
Checklist Details
Keloid
1. Location: Scars on chest, shoulders, earlobes, jawline, or
any site of skin injury
2. Lesion Type; Overgrown scar tissue extending beyond the original wound
3. Shape/Size: Raised, irregularly shaped scar; can be small or grow Large over time
4. Color: Pink or red on lighter skin; darker, purple or brown on darker skin
5. Texture: Smooth, hairless, firm/rubbery; shiny surface
Lichen planus
1. Location: Wrists, forearms, ankles, scalp, nails, mouth, genitals
2. Lesion Type; Flat-topped papules; can form plaques or lines from scratching
3. Shape/Size: Polygonal, 2-10 mm papules
4. Color: Viclaceous (purple) on light skin; gray-brow or hyperpigmented on dark skin
5. Texture: Shiny surface with fine white lines (Wickham's striae); be scaly if scratched
Lupus erythematosus
1. Location: Face (butterfly rash across cheeks/nose); can appear on scalp/ears; photosensitive areas
2. Lesion Type: Flat or slightly raised rash (malar/butterflyy discoid lesions
be scaly and scarred 3. Shape/Size: Butterfly rash covers the bridge of nose and
both cheeks; discoid lesions are coin-shaped (1-3 cm)
4. Color: Pink-red on light skin; can be darker red or hyperpigmented on darker skin
5. Texture: Malar rash smooth slightly raised; discoid be rouga/scaly with scarring
Melanoma
1. Location: Can appear anywhere (trunk, limbs, face, nails); in darker skin, often on palms/soles or under nails
2. Lesion Type: Atypical mole patch; irregular shape and color
3. Shape/Size: edges
Asymetrric, often >6 mn, with notched/bumpy
4. Color:
Multiple shades (orown, black, red, white, blue); dark skin, often very dark with variation
5. Texture: Smooth early; may become raised, crusted, or ulcerated if advanced
Continued on next page
25

---

Table 14 continued from previous page
Skin Condition
Checklist Details
Mycosis fungoides
1. Location: Usually sun-sunn areas (buttocks, lower
abdomen, thighs); can spread more widely later
2. Lesion Type; Patches (like eczema), plaques (thickened), or tumor nodules (advanced)
3. Shape/Size: Irregular shapes, patches often a few om wide; plaques larger/thicker; nodules can be several cm Pink-red to reddish-brown, darker or hyperpigmented on darker skin
4. Color:
5. Texture: Dry, scaly for patches; plaques thicker/scaly; nodules can smooth or ulcerated
Pityriasis rosea
1. Location: Trunk (back, chest, abdomen) primarily;
occasienally upper arms, thighs
2. Lesion Type: Herald patch (large oval) followed by multiple smaller oval patches/papules
3. Shape/Size: Herald patch ~2-6 cm; daughter lesions ~1-2 cm
often align in 'Christnas tree' pattern
4. Color: Pink/salmon on light skin; gray, brown, parplish dark skin
5. Texture: Fine collarette scale inner edge; not typically thick or crusty
Prurigo nodularis
1. Location: Arms, legs, upper back, shoulders, scalp, areas casily reached for scratching
2. Lesion Type: Firm, itchy nodules, often with crusted scabbed top
3. Shape/Size: Round nodules 1-3 cm; multiple lesions often present
4. Color: May be pink, red, brown, black, skin-toned; older
lesions can be hyperpigmented
5. Texture: Thick, rough; scabs from scratching; firm to touch
Psoriasis
1. Location: Elbows, knees, scalp, lewer back; can affect nails
r palms, scles, be widespread
2. Lesion Type: well-demarcated plaques with thick, scaly surface; can also be snaller papules
3. Shape/Size: Round/oval or irregular plaques; can range from small patches to large areas
4. Color: On light skin, pink/red with silvery scales; dark skin, purple/dark brown with grayish scales
5. Texture: pry, flaky scales that can be peeled off; underlying skin may bleed (Auspitz sign)
Continued on next page
26

---

Table 14 continued from previous page
Skin Condition
Checklist Details
Sarcoidosis
1. Location: Face (nose, cheeks lupus pernio), shins
erythema nodosum), scars/tattoos, can be widespread
2. Lesion Type; Firm plaques, nodules, or discolored patches; red burps shins in erythema nodosum
3. Shape/Size: Plaques are broad and raised; nodules can cm: patchy discolorations vary
4. Color:
Purplish or red-brown lumps; can be lighter/darker patches on dark skin; scars turn red
5. Texture: Smooth, firm rubbery;
lesions (erythera nodosum) are tender lumps under the skin
Scabies
1. Location: Finger webs, wrists, waist, buttocks, genitals, armpits; in infants: pains, soles, scalp
2. Lesion Type: Tiny burrows (thin, wavy lines) plus small itchy bumps or vesicles
3. Shape/Size: Burrows ~5-15 mm leng; bumps -1-2 mm in clusters
4. Color: skin-toned to pink/red; en darker skin, may appear
darker or hyperpigmented
5. Texture: Scratch marks, crusted spots from itching; burrows feel like slight ridges
Squamous cell carcinoma
1. Location: sun-exposed areas (face, ears, lips, hands),
chronic scars, or wounds; can appear on mucosal surfaces
2. Lesion Type: Crusty scaly bump, ulcer, plaque; central depressior
have raised borders
3. Shape/Size: Firm nodule or patch, grow rapidly
am untreated; may
4. Color: Pink/red lighter skin; brown darker on brown/
Black skin; show white/yellow keratin
5. Texture: Rough, thick, crusted surface; may bleed or ulcerate; firm on palpation
Vitiligo
1. Location: Face (around cyes, mouth), hands, feet, arms, legs genitals; can occur anywhere on body
2. Lesion Type: Depigmented patches with well-defined borders; hair may turn white affected area
3. Shape/Size: Irregular shapes; can start small and enlarge over time, often symmetrical
4. Color: Completely white pale cempared to surrounding skin
; high contrast on darker skin
5. Texture: Normal skin texture (nc scaling or thickening), only color is lost
27

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction summarize the paper's contributions.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are discussed in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specificen specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
28

---

model The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The information needed to reproduce the main experimental results of the paper can be found in the "Experiments" section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. . a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closcd-soure source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer: [No]
Justification: Code will be released upon acceptance.
Guidelines:
The answer means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: //tp:///ipppp// http://ipps.ccppllll/pp cc/pllic/////ccccccccccccccccccc
guides/ des /Code Cubme asi onPol 1 cy) for more details.
While we encourage release of code and data, we understand that might not possible, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nnpss.p.....//// ce/ for more details.
29

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommeneed, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper specifies all the training and test details necessary to understand the results in the "Experiment" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
All experiments have been repeated for rounds.
Guidelines:
The answer means that paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful symmetric error bars would yield results that are out of range (e.g. negative error rates). error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
show in tables figures
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The information needed to reproduce the main experimental results of the paper can be found in the "Experiments" section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
30

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, every respect, of Ethics https: : htt///neurrppp ips.. ca/ httpp/////etcc/ctieeeeeo
NeurIPS Code
Answer:
[Yes]
Justification:
No ethical concerns.
Guidelines:
The answer means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should he released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
31

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The creators or original owners assets used the paper are properly cited and credited.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. popular datasets, papersu thoode. com/ cat asetss for some datasets. Their licensing guide can help determine the license a dataset.
curated licenses
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
answer means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
32

---

Justification:
for The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important. original. or non non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The usage LLM is descrihed "Experiments" section.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: what should or should not be described.
33


Reviews:
[{"review_id": "3yBVA7hRYZ", "reviewer": "Reviewer_d8Vt", "summary": "This paper introduces a framework called MAGIC. They propose to fine-tune diffusion models to produce clinically accurate synthetic skin-disease images. The main contribution is to translate dermatologist-crafted check-lists into machine-readable feedback: a multimodal LLM (GPT-4o) scores each generated image on five visual criteria, and the diffusion model is aligned with either reward-weighted likelihood or DPO. Experiments on 2 datasets show that MAGIC-DPO raises ResNet-18 accuracy from 29 → 38 % and DINOv2 from 50 → 55 %, while more than half of the synthetic images satisfy ≥3 clinical criteria.", "questions": "Please discuss the weaknesses and here are some further questions for your paper:\n\nHow does MAGIC perform on unseen hospital-grade datasets e.g., PAD-UFES-20 or in a blinded dermatologist Turing test?\n\nThe I2I pipeline denoises real patient images conditioned on a new disease label. Could residual patient identity or background cues leak into the synthetic set, violating privacy regulations? You should discuss this problem \n\nFine-tuning with 1 024 feedback pairs took 128 iterations; please report wall-clock time and GPU hours to guide practitioners.", "limitations": "No limitation.\n\nHowever, as you are a medical ML research, please add a candid section on misuse (e.g., fake dermatology imagery for fraud), bias across skin tones, and safeguards for releasing tuned diffusion weights.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- Compared to the other medical imaging paper I reviewed in NeurISP this year, this paper is well structured and algorithmic steps are enumerated, which is good.     \n\n- It highlights an inexpensive path to reduce expert burden by pairing MLLM judges with concise clinical check-lists.    \n\n- The workflow in experiment is clear, dermatology synthesis -> prove it in the downstream tasks.     \n\n**Weaknesses**\n\n- This is an application-only paper. It builds directly on prior DPO-diffusion work [54] and MLLM-judge paradigms; we can't say low novelty but the innovation is largely domain transfer.       \n\n- No analysis of hallucinations for the MLLM-as-a-judge process. Why not use some open-source model to replace GPT-4o and explore the hallucinations and potential bias?     \n\n- When the model scale of Classifier is larger, the performance increment is also lower. You should explore this marginal effect.", "was_revised": true, "final_justification": "My main concerns on this paper is resolved. I will increase my final rating to Accept.", "rebuttal": "We thank the reviewer for recognizing the quality of our work and our success in translating the MAGIC framework into tangible downstream performance improvements. We also sincerely appreciate the constructive feedback on our submission. \n## W1: Applicaiton-oriented & Novelty\n---\nWhile our framework leverages established methods like DPO-diffusion and MLLM-as-a-judge, this is the first work as far as we know that proposes a framework of combining expert domain knowledge and MLLM-based evaluation into actionable feedback for synthesizing domain-accurate images using DMs. We show significantly improved clinical accuracy while reducing the direct human workload in the dermatology domain, where generating clinically accurate images from limited data is non-trivial due to high intra-class variation and inter-class similarity of conditions. However, the core contribution is not merely a domain transfer but a new, semi-automated, and effective paradigm for image synthesis, whose potential could be extended to real-world applications beyond dermatology and even medical domains.\n## W2: Analysis of hallucination & Open-source MLLM\n---\nWe address this in two parts: by detailing our framework's built-in safeguards and by presenting two new empirical analyses.\n\n1. **Framework Design to Mitigate Hallucination:** The MAGIC framework is explicitly designed to minimize the risk of MLLM hallucination through our AI-Expert collaboration paradigm. The MLLM is not asked to perform open-ended reasoning. Instead, its role is constrained to evaluating an image against a predefined clinical checklist. These checklists, designed by dermatologists, decompose complex medical concepts into simple, visually verifiable features. For instance, while an MLLM may not intrinsically understand \"Lupus Erythematosus,\" it can effectively verify \"symmetric butterfly rash across the cheeks\". This approach transforms a complex diagnostic reasoning task into a series of closed-question evaluations, which are far less susceptible to unconstrained hallucination. \n\n    Even so, we agree that an analysis of GPT-4o's performance is important. To that end, we conducted a new study where we had GPT-4o evaluate 100 real images from an internal dataset with clinical records confirmed by in-person visits. The MLLM was tasked with describing each image based on the five criteria (location, lesion type, shape/size, color, and texture). We then invited a board-certified dermatologist to assess these descriptions. As shown in the table below, the dermatologist assigned an **alignment score** of 3 or greater (on a 5-point scale) to approximately 86% of the image-description pairs. Notably, many of the lower-scoring examples were also identified by the dermatologist as being visually ambiguous and challenging for a human to assess from an image alone.\n    \n|Score(↑)|1|2|3|4|5|\n|-|-|-|-|-|-|\n|# Examples|8|6|16|21|49|\n\n\n2. **Choice of MLLM and Comparative Analysis:** The MAGIC framework is designed to be model-agnostic, allowing for the flexibility to use the most suitable MLLM for a given task. For this foundational study, we selected GPT-4o for its exceptional visual reasoning capabilities. However, we agree with the reviewer that a comparative analysis using open-source models is a valuable addition. To that end, we have conducted new experiments testing our MAGIC framework with an open-source MLLM, Google DeepMind's **MedGemma-4B**[1], an open model for medical text and image comprehension. We used MedGemma as an evaluator to assess the same set of 1024 pairs of synthesized images that GPT-4o had assessed. We observed that MedGemma is comparable to GPT-4o as an evaluator in aligning the DM using the MAGIC framework. This demonstrates the flexibility of our MAGIC framework, which is adaptive to both large, closed-source, genric models and smaller, open-source, domain-specific alternatives. The results are shown in the table below.\n\n|MLLM|Classifier|Accuracy(%)|F1(%)|Precision(%)|Recall(%)|\n|-|-|-|-|-|-|\n|GPT-4o|ResNet-18|38.33|37.01|38.41|36.06|\n|GPT-4o|DINOv2|55.01|54.05|54.96|53.70|\n|MedGemma|ResNet-18|36.97|35.55|37.12|36.32|\n|MedGemma|DINOv2|54.19|53.08|54.78|53.53|\n\n[1]. MedGemma Technical Report. (2025)\n\nWe'll include the results and analyses in the future version of the work.\n\n## W3: Analysis of Classifier Backbones\n---\nA stronger backbone such as DINOv2  starts with a clear advantage: it has a complex transformer architecture and is pre-trained on a massive dataset, enabling it to extract robust image features from the start. In contrast, a smaller model like ResNet-18  struggles with the limited and challenging real data. The differing gains (**+9.02%** for ResNet18 vs. **+5.12%** for DINOv2) therefore highlight a key finding: our augmentation framework provides the most significant benefit in data-scarce or model-constrained scenarios.\n\nHowever, the fact that MAGIC still substantially boosts the performance of a powerful model like DINOv2 is a strong testament to the quality of our synthetic data. It demonstrates that MAGIC generates images with clinically accurate features that even a strong classifier cannot extract from the limited real dataset alone. This conclusion is supported by our few-shot experiments in **Table 3**. When the training set was reduced to just 10% (310 images), MAGIC-DPO provided a sizable **+10.94%** accuracy improvement for the DINOv2 classifier. This shows that as data becomes more scarce, the value of our synthetic augmentation becomes even more pronounced. To further explore this interesting effect, we have run additional experiments with other classifier backbones of various sizes. The results, presented in the table below, are consistent with our observations. We'll include the analysis in our work.\n\n|Model|#Params(M)|Pre-train Dataset|Real Acc(%)|MAGIC Acc(%)|Gain(%)|\n|-|-|-|-|-|-|\n|ResNet-18|12|ImageNet-1k|29.31|38.33|+9.02|\n|ResNet-50|26|ImageNet-1k|35.56|46.24|+10.68|\n|ViT-B/16|86|ImageNet-21k|45.22|51.44|+6.22|\n|DINOv2|87|LVD-142M|49.89|55.01|+5.12|\n\n## Q1: Generalizability and Clinical Realism\n---\nTo directly address these questions, we have conducted two new experiments to evaluate the generalizability and clinical realism of our MAGIC framework. \n\n1. **Generalizability to Unseen Dataset:** To quickly evaluate the cross-dataset generalizability of our method, we identified four overlapping classes between the PAD-UFES-20 dataset and our Fitzpatrick17k subset (ACK, BCC, MEL, and SCC), and ran the MAGIC-DPO pipeline. The results, presented in the tables below, demonstrate that the MAGIC framework is generalizable to hospital-grade datasets.\n\n|ResNet-18|Accuracy(%)|F1(%)|Precision(%)|Recall(%)|\n|-|-|-|-|-|\n|Real|63.02|51.97|61.07|48.34|\n|MAGIC|70.65|58.48|62.40|51.65|\n\n|DINOv2|Accuracy(%)|F1(%)|Precision(%)|Recall(%)|\n|-|-|-|-|-|\n|Real|67.88|60.50|66.29|57.10|\n|MAGIC|73.85|63.81|67.17|59.41|\n\n2. **Blinded Dermatologist Evaluation:** We conducted a blinded Turing test to assess the clinical realism of our generated images. We randomly sampled a mixed set of 500 real images and 500 synthetic images generated by MAGIC-DPO. We also resized the images to the same dimensions. An expert dermatologist, who was not involved in the project, was asked to rate the likelihood of each of the 1,000 images being synthetic on a 5-point scale given the condition label. The expert's ratings for the real and synthetic images followed a remarkably similar distribution, and the expert tended not to make absolute predictions. This indicates that the expert could not confidently distinguish the synthetic images from the real clinical photos. This experiment complements the quality evaluation in Fig. 4(d) by directly assessing realism.\n\n||1-very likely|2-likely|3-neutral|4-unlikely|5-very unlikely|\n|-|-|-|-|-|-|\n|Real|9|151|90|249|1|\n|Syn|11|131|83|267|8|\n\nWe'll include the results and analyses in the future version of the work.\n\n## Q2: Privacy Concern about I2I Pipeline\n---\nOur MAGIC framework is designed with privacy in mind. This \"factorized transformation\" preserves only the high-level anatomical context while overwriting the fine-grained lesion details. This dissociates the original identity from the new condition, which both enhances privacy and reduces the risk of the classifier learning spurious correlations.\nWhile this approach is designed to be privacy-conscious, we acknowledge that it does not offer the formal guarantees of methods like Differential Privacy. Additionally, privacy risks can be further minimized by running open-source or HIPAA-compliant MLLMs locally as evaluators. While a formal, quantitative analysis guaranteeing the complete removal of all identifying features was beyond the scope of this work, we agree that aligning diffusion models while removing identifying cues is an important future direction for medical and other privacy‑critical domains.\n\n## Q3: Computation Analysis\n---\nWe have included the table for wall-clock time on a single NVIDIA RTX 6000 Ada here. We will add the table to the Appendix.\n|Method|Config|Wall-Clock Time(h)|\n|-|-|-|\n|Textual Inversion|500 steps/token, 20tokens|4.05|\n|LoRA|3000 steps|0.44|\n|RFT|2024 feedback|2.01|\n|DPO|2024 feedback|3.77|\n\n## Limitation\n---\nWe agree completely that a candid discussion on the broader impacts of this method is essential. Following this feedback, we will add a new \"Ethical Considerations and Safeguards\" section to our paper to discuss these concerns.\n\n---\nWe once again thank the reviewer for their valuable time and constructive feedback. We hope our detailed responses and new experimental results have fully addressed the concerns raised and have better highlighted the significance of our contributions. We would be happy to answer any further questions you may have during the discussion period."}, {"review_id": "GbkFsxDa3L", "reviewer": "Reviewer_Jzjk", "summary": "This paper introduces MAGIC (Medically Accurate Generation of Images through AI–Expert Collaboration), a novel semi-automated framework to generate clinically accurate dermatological images using diffusion models (DMs), and multimodal large language models (MLLMs) like GPT-4o. MAGIC fine-tunes image generation via expert-designed criteria translated into machine-readable feedback. It can leverage two learning paradigms: Reward-model Fine-Tuning (RFT) or Direct Preference Optimization (DPO), with DPO yielding superior results.", "questions": "1. I wonder how did you make sure, GPT4o is reasonable to be used as an expert?\n2. For expert evaluation in your experiment part, did you invite real medical doctors to evaluate the generated images?\n3. Section 3.2 first sentence need citations to make sure its correctness.\n4. Also you didn't start from a random noise, I wonder what the benefit of your method is. Do you have any experiments to show it?", "limitations": "1. Technical dependence: Relies on proprietary MLLM capabilities that may change over time\n2. Domain specificity: Requires adaptation of checklists and evaluation criteria for each new medical domain", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghts: 1. The method is sound to me. This paper introduces a practical and semi-automated way to incorporate human feedback using MLLMs, reducing costly manual evaluations.\n2. The problems they proposed are critical, especially that the generated images look pretty fake in the medical domain.\n3. The experiment results demonstrate the method's superior performance. The ablation studies are sufficient, and experimental details are great.\n\nWeaknesses: 1. Only two datasets are used for the experiment.\n2. The authors simply use GPT4-o for the assessment. I doubt that if GPT4-o is 100% percent correct to serve as an expert.\n3. Missing comparisons with other medical-specific synthetic data generation approaches.", "was_revised": true, "final_justification": "The reviewer has tried their best to solve my problems, and the follow-up experiments successfully showed the benefits of their proposed methods. Based on the re-justification, i raise my score to 5 accordingly.", "rebuttal": "We thank the reviewer for the thoughtful assessment and valuable feedback. We are very pleased that they found our method sound, the problem critical, and our experiments comprehensive and well-supported.\n\n## W1: Choice of Datasets\n---\nFrom the limited number of available clinical datasets, we strategically chose Fitzpatrick17k and SCIN as they represent real-world challenges, such as complex variations and data scarcity. To further evaluate the generalizability of our method, we also performed a new evaluation on a new dataset: PAD-UFES-20. To quickly evaluate our method, we identified four overlapping classes between the PAD-UFES-20 dataset and our Fitzpatrick17k subset (ACK, BCC, MEL, and SCC), and ran the MAGIC-DPO pipeline. The results, presented in the tables below, demonstrate that the MAGIC framework is also generalizable to hospital-grade datasets.\n\n|ResNet-18|Accuracy(%)|F1(%)|Precision(%)|Recall(%)|\n|-|-|-|-|-|\n|Real|63.02|51.97|61.07|48.34|\n|MAGIC|70.65|58.48|62.40|51.65|\n\n|DINOv2|Accuracy(%)|F1(%)|Precision(%)|Recall(%)|\n|-|-|-|-|-|\n|Real|67.88|60.50|66.29|57.10|\n|MAGIC|73.85|63.81|67.17|59.41|\n\n## W2 & Q1: Reliability of GPT-4o\n---\nWe address this in two parts:\n1. **Framework Design to Ensure Reasonableness:** We do not use the MLLM as an autonomous expert. Instead, its role is carefully constrained within our AI-Expert collaboration paradigm. The MLLM is not asked to perform open-ended reasoning; its role is limited to evaluating an image against a predefined clinical checklist. These checklists, designed by dermatologists, decompose complex medical concepts into simple, visually verifiable features. For instance, while an MLLM may not intrinsically understand \"Lupus Erythematosus,\" it can effectively verify a feature like a \"symmetric butterfly rash across the cheeks\". This approach transforms a complex diagnostic reasoning task into a series of closed-question evaluations, making the MLLM's role both reasonable and verifiable. \n\n    Even so, we agree that an analysis of GPT-4o's performance is important. To that end, we conducted a new study where we had GPT-4o evaluate 100 real images from an internal dataset with clinical records confirmed by in-person visits. The MLLM was tasked with describing each image based on the five criteria (location, lesion type, shape/size, color, and texture). We then invited a board-certified dermatologist to assess these descriptions. As shown in the table below, the dermatologist assigned an **alignment score** of 3 or greater (on a 5-point scale) to approximately **86%** of the image-description pairs. Notably, many of the lower-scoring examples were also identified by the dermatologist as being visually ambiguous and challenging for a human to assess from an image alone.\n    \n|Score(↑)|1|2|3|4|5|\n|-|-|-|-|-|-|\n|# Examples|8|6|16|21|49|\n\n2. **Choice of MLLM and Comparative Analysis:** The MAGIC framework is designed to be model-agnostic, allowing for the flexibility to use the most suitable MLLM for a given task. For this foundational study, we selected GPT-4o for its exceptional visual reasoning capabilities. However, we agree that a comparative analysis using open-source models is a valuable addition. To that end, we have conducted new experiments testing our MAGIC framework with an open-source MLLM, Google DeepMind's **MedGemma-4B**[1], an open model for medical text and image comprehension. We used MedGemma as an evaluator to assess the same set of 1024 pairs of synthesized images that GPT-4o had assessed. We observed that MedGemma is comparable to GPT-4o as an evaluator in aligning the DM using the MAGIC framework. This demonstrates the flexibility of our MAGIC framework, which is adaptive to both large, closed-source, genric models and smaller, open-source, domain-specific alternatives. The results are shown in the table below.\n\n|MLLM|Classifier|Accuracy(%)|F1(%)|Precision(%)|Recall(%)|\n|-|-|-|-|-|-|\n|GPT-4o|ResNet-18|38.33|37.01|38.41|36.06|\n|GPT-4o|DINOv2|55.01|54.05|54.96|53.70|\n|MedGemma|ResNet-18|36.97|35.55|37.12|36.32|\n|MedGemma|DINOv2|54.19|53.08|54.78|53.53|\n\n[1]. MedGemma Technical Report. (2025)\n\nWe'll include the results and analyses in the future version of the work.\n\n## W3: Comparision with Other Medical Synthesis Method\n---\nIn our 'Related Works' section, we discuss several relevant studies in dermatological data synthesis. Our experimental comparisons were designed to be relevant to our core contribution, which is the expert-feedback alignment loop. Our work is directly compared against two SOTA methods: (i) **The method from [50]:** This represents an effective approach for skin disease image synthesis *without* an active expert-feedback loop. Our \"+I2I\" baseline is based on this work. (2) **The method from [46]:** This is a key attempt at aligning a medical image generator with expert feedback by training a reward function. Our MAGIC-RFT model is our adaptation and comparison to this methodology, tailored from a class-conditional DM to our T2I DM. Furthermore, by comparing with [50], we are also indirectly assessed against a method that has been systematically evaluated against other recognized approaches in the field, including [2, 39, 40]. This allows us to situate our performance within the broader context of existing literature. ([2, 39, 40, 46, 50] in our paper).\n\n## Q2: Expert Evaluation on the Generated Images\n---\nFor the final assessment of the generated images, we have engaged **real medical experts** to evaluate the synthetic data based on our specific checklist criteria, and the results are present in **Fig.4 (d)**. Additionally, we also conducted a blinded Turing test to assess the clinical realism of our generated images. We randomly sampled a mixed set of 500 real images and 500 synthetic images generated by MAGIC-DPO. We also resized the images to the same dimensions. An expert dermatologist, who was not involved in the project, was asked to rate the likelihood of each of the 1,000 images being synthetic on a 5-point scale given the condition label. The expert's ratings for the real and synthetic images followed a remarkably similar distribution, and the expert tended not to make absolute predictions. This indicates that the expert could not confidently distinguish the synthetic images from the real clinical photos. This experiment complements the quality evaluation in Fig. 4(d) by directly assessing realism. We'll include this analysis in our work.\n\n||1-very likely|2-likely|3-neutral|4-unlikely|5-very unlikely|\n|-|-|-|-|-|-|\n|Real|9|151|90|249|1|\n|Syn|11|131|83|267|8|\n\n## Q3: Claim in the First Sentence of Section 3.2\n---\nThe claim is well-supported by prior work **[50]**, which demonstrated that an off-the-shelf diffusion model lacks the necessary \"domain-specific knowledge\" for this task. They attribute this to the model being unfamiliar with specific skin lesion concepts and semantics. Furthermore, fine-tuning diffusion models has become a common and necessary practice for synthetic augmentation in this field, as exemplified by recent studies, including [2, 28, 39, 40, 46, 50]. Following the reviewer's suggestion, we will add references, including [50], to the end of this sentence to make the citation explicit and properly credit these foundational observations. ([2, 28, 39, 40, 46, 50] in our paper).\n\n## Q4: Benefits of Denoising from Partially Noised Images\n---\nThe benefit of starting from a partially noised image instead of pure Gaussian noise is that it enables a \"factored transformation,\" where we can change the skin condition while preserving the underlying anatomy. This approach has several key advantages:\n- It helps to ensure the model develops a disentangled understanding of skin lesion concepts separate from other irrelevant visual cues.\n- It reduces semantic distortion during generation.\n- It helps prevent the downstream classifier from learning spurious correlations (e.g., associating a specific disease with a specific body part or background cue).\n- It accelerates the image generation process.\n\nIn our experiments, the **\"T2I\"** method generates images from pure Gaussian noise, while the **\"I2I\"** method starts from a partially noised real image. The results in **Table 1** and **Table 2** clearly show the superiority of the I2I approach, where the T2I augmentation is outperformed by its I2I counterpart. Given this strong experimental evidence, we chose the more effective I2I pipeline for our MAGIC framework.\n\n## L1: Technical Dependence on MLLMs\n---\nWe acknowledge that our framework's performance is linked to the capabilities of the MLLM used. However, MAGIC is designed to be model-agnostic and flexible. This flexibility is also an advantage, as the framework's performance will naturally improve with the continually advancing interpretive capabilities of MLLMs. As demonstrated in our response to **W2**, our framework can incorporate open-source models (e.g., MedGemma).\n\n## L2: Domain Specificity\n---\nThe reviewer is correct that adapting MAGIC to new medical domains requires new, expert-designed checklists. This is an intended feature of our design. The core purpose of MAGIC is to translate specific, nuanced clinical knowledge into actionable feedback for a generative model. Therefore, achieving accuracy in any new domain fundamentally requires integrating targeted expert knowledge. We see this adaptability as a strength, allowing for rigorous, domain-aware data synthesis in other areas with their own unique visual characteristics and clinical requirements.\n\n---\n\nWe once again thank the reviewer for their valuable time and constructive feedback. We hope our detailed responses and new experimental results have fully addressed the concerns raised and have better highlighted the significance of our contributions. We would be grateful if you would consider these clarifications in your final evaluation of our work and are happy to answer any further questions you may have during the discussion period."}, {"review_id": "2ISEe1iEdv", "reviewer": "Reviewer_Embq", "summary": "This paper proposes MAGIC, a novel framework integrating AI-generated expert feedback via MLLMs into diffusion models for the generation of clinically accurate synthetic dermatological images. \n\nThe method notably reduces human annotation workloads and demonstrates significant performance improvements in downstream classification tasks.", "questions": "x. How sensitive is MAGIC's performance to inaccuracies or omissions in expert-provided checklists? Could the authors elaborate on robustness tests regarding potential checklist variability?\n\nx. What insights can the authors share about the performance drop when exceeding a certain ratio of synthetic data (p > 0.5)? Could there be practical strategies to maximize benefits from synthetic images while avoiding potential domain shifts?\n\nx. Can the authors provide insights on the failure modes or common inaccuracies observed when GPT-4o evaluates dermatological images? Are there specific lesion characteristics that are particularly challenging for the MLLM to assess correctly?", "limitations": "The paper accurately identifies reliance on high-quality expert-generated checklists as a potential limitation. \n\nFurther exploration into automating or systematically validating checklist creation could mitigate human bias.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths: \n\n+ Effectively leverages Multimodal Large Language Models for image evaluation, significantly reducing human annotation labor.\n\n+ Demonstrates tangible performance improvements: a notable increase in diagnostic accuracy (+9.02% in general settings and +13.89% in few-shot scenarios). \n\n+ Robust baseline comparisons (T2I, I2I) and well-justified use of Direct Preference Optimization (DPO) and Reward-based Fine-Tuning (RFT).\n\n\nMaojr weakness: \n\n-  The effectiveness of MAGIC heavily relies on checklist quality and clarity (highlighted in ablation studies on page 9). If dermatologists overlook key characteristics or checklist detail is insufficient, the method's performance may degrade significantly.\n\n- While leveraging GPT-4o is innovative, the practical implications and reliability of AI-based medical judgments in clinical workflows are not discussed in depth.", "was_revised": true, "final_justification": "A timely, well-formatted paper for the healthcare applications. Recommend acceptance.", "rebuttal": "We thank the reviewer for their positive feedback and for recognizing our method's tangible impact on diagnostic accuracy, its innovative use of MLLMs, and the robustness of our experimental design. We also appreciate the constructive feedback.\n## W1: Reliance on Checklist Quality\n---\nThis is an insightful question. Our ablation study was designed to demonstrate this very point: the quality of the expert-provided checklist is pivotal for aligning diffusion models with clinical reality. While the goal is to significantly reduce the expert labor burden by automating the repetitive evaluation of thousands of images, we believe that the initial, high-leverage task of ensuring checklist quality is an essential and manageable role for human experts. \n\nTo ensure this quality in our work, we employed a robust, multi-step process. \n**First**: the checklists were crafted using multiple credible clinical sources for comprehensiveness. \n**Second**: these checklists were then cross-validated by several dermatologists to ensure clinical accuracy and minimize the risk of overlooking key diagnostic features. \n\nThe results of our ablation study in **Table 4** confirm that this expert effort is critical; the detailed, well-structured checklist enabled a much larger accuracy boost compared to a coarse one. This underscores a core message of our paper: generating high-quality synthetic medical data requires successfully integrating granular expert knowledge into the generation process. While this represents a dependency on expert input, it also highlights the framework's primary strength: its ability to effectively translate nuanced clinical criteria into actionable feedback, guiding the diffusion models toward clinical reality.\n\n## W2: Reliability of GPT-4o\n---\nWe address this in two parts: by detailing our framework's built-in safeguards and by presenting two new empirical analyses.\n1. **Framework Design to Mitigate Hallucination**: The MAGIC framework is explicitly designed to minimize the risk of MLLM hallucination through our AI-Expert collaboration paradigm. The MLLM is not asked to perform open-ended reasoning. Instead, its role is constrained to evaluating an image against a predefined clinical checklist. These checklists, designed by dermatologists, decompose complex medical concepts into simple, visually verifiable features. For instance, while an MLLM may not intrinsically understand \"Lupus Erythematosus,\" it can effectively verify \"symmetric butterfly rash across the cheeks\". This approach transforms a complex diagnostic reasoning task into a series of closed-question evaluations, which are far less susceptible to unconstrained hallucination. \n\n    Even so, we agree that an analysis of GPT-4o's performance is important. To that end, we conducted a new study where we had GPT-4o evaluate 100 real images from an internal dataset with clinical records confirmed by in-person visits. The MLLM was tasked with describing each image based on the five criteria (location, lesion type, shape/size, color, and texture). We then invited a board-certified dermatologist to assess these descriptions. As shown in the table below, the dermatologist assigned an **alignment score of 3** or greater (on a 5-point scale) to approximately **86%** of the image-description pairs. Notably, many of the lower-scoring examples were also identified by the dermatologist as being visually ambiguous and challenging for a human to assess from an image alone.\n    \n|Score(↑)|1|2|3|4|5|\n|-|-|-|-|-|-|\n|# Examples|8|6|16|21|49|\n\n2. **Choice of MLLM and Comparative Analysis:** The MAGIC framework is designed to be model-agnostic, allowing for the flexibility to use the most suitable MLLM for a given task. For this foundational study, we selected GPT-4o for its exceptional visual reasoning capabilities. However, we agree that a comparative analysis using open-source models is a valuable addition. To that end, we have conducted new experiments testing our MAGIC framework with an open-source MLLM, Google DeepMind's **MedGemma-4B**[1], an open model for medical text and image comprehension. We used MedGemma as an evaluator to assess the same set of 1024 pairs of synthesized images that GPT-4o had assessed. We observed that MedGemma is comparable to GPT-4o as an evaluator in aligning the DM using the MAGIC framework. This demonstrates the flexibility of our MAGIC framework, which is adaptive to both large, closed-source, genric models and smaller, open-source, domain-specific alternatives. The results are shown in the table below.\n\n|MLLM|Classifier|Accuracy(%)|F1(%)|Precision(%)|Recall(%)|\n|-|-|-|-|-|-|\n|GPT-4o|ResNet-18|38.33|37.01|38.41|36.06|\n|GPT-4o|DINOv2|55.01|54.05|54.96|53.70|\n|MedGemma|ResNet-18|36.97|35.55|37.12|36.32|\n|MedGemma|DINOv2|54.19|53.08|54.78|53.53|\n\n\n\nWe'll include the results and analyses in the future version of the work. Additionally, we agree that the direct use of AI for medical judgments is a complex topic involving crucial discussions on ethics, regulation, and reliability. A full exploration of these issues is beyond the scope of this foundational technical paper, but our work contributes by improving the underlying data quality, which is a prerequisite for any reliable clinical AI system.\n\n[1]. MedGemma Technical Report. (2025)\n\n## Q1: MAGIC's Sensitivity/Robustness to Checklist Quality\n---\nOur work presents a direct robustness test for this, and we have conducted an additional experiment to explore this sensitivity further.\n\nFirst, the ablation study in **Section 5.2, \"Effect of Checklist Quality**,\" serves as our primary robustness test. By comparing the performance of models trained with data generated using \"Coarse\" vs. \"Structured\" checklists, we explicitly tested the system's sensitivity to checklist detail. The results in **Table 4** show that performance is indeed highly sensitive to checklist quality, confirming that granular expert knowledge is a critical driver for generating high-quality synthetic data.\n\nTo explore the upper bounds of this effect, we conducted a new experiment. We had a dermatologist craft even more fine-grained checklists with nine criteria (Location, Distribution, Lesion Type, Shape/Size, Border, Elevation, Texture, Color, and Translucency/Content). We then used these 9-criteria checklists in our MAGIC framework. We observed a small, additional improvement over our original 5-criteria structured checklist, as shown in the table below.\n\n|ResNet-18|Accuracy(%)|F1(%)|Precision(%)|Reall(%)|\n|-|-|-|-|-|\n|w/CoarseChecklist (1sentence)|32.83|30.58|29.75|31.18|\n|w/Fine-grainedChecklist (7-criteria)|38.33|37.01|38.41|36.06|\n|w/HighlyFine-grainedChecklist (9-criteria)|39.39|37.86|40.64|39.75|\n\n|DINO-v2|Accuracy(%)|F1(%)|Precision(%)|Reall(%)|\n|-|-|-|-|-|\n|w/CoarseChecklist (1sentence)|51.16|52.66|52.17|52.69|\n|w/Fine-grainedChecklist (7-criteria)|55.01|54.05|54.96|53.70|\n|w/HighlyFine-grainedChecklist (9-criteria)|55.88|55.60|56.56|55.51|\n\nThis combined analysis leads to two key insights: **(1)** A detailed, structured checklist is critical for the framework's success, as shown by the significant performance jump from the \"Coarse\" to the \"Structured\" checklist. **(2)** There may be diminishing returns after a certain level of detail is achieved, as shown by the smaller gain when moving from the 5-criteria to the 9-criteria checklist. This suggests our original 5-criteria checklist was already capturing the most essential features for high-quality generation.\n\n## Q2: Maximize Benefits from Synthetic Images\n---\nOur insight is that when the proportion of synthetic data becomes too high (ρ > 0.5), the classifier begins to overfit to the specific nuances and distribution of the generated images, thereby losing some of its ability to generalize to the real-world test data. Even high-quality synthetic data has a slightly different distribution from real clinical photos. The most effective practical strategy to mitigate this, and the one employed in our work, is to use a mix of real and synthetic data in each training batch. \n\n## Q3: Failure Analysis of GPT-4o\n---\nBased on a qualitative review of the MLLM's evaluations, we identified several recurring challenges and potential failure modes, which are detailed below. A systematic analysis of these failure modes is a valuable direction for future research.\n\n- Subtle Textural and Morphological Details: The MLLM can struggle with very fine surface textures or lesions that are not well-defined. For example, small papules or pustules can be difficult for the model to assess, especially when their color is indistinguishable from the surrounding skin.\n- Complex Color Nuances: Differentiating between similar shades (e.g., pink vs. reddish) or accurately interpreting colors on darker skin tones, such as the \"purple/dark brown with grayish scales\" described for Psoriasis, can be difficult from a 2D image alone.\n- Inferring 3D Characteristics: Features that imply three-dimensionality, such as the \"firm nodules\" of Prurigo Nodularis or the \"pearly bump\" of Basal Cell Carcinoma, are inherently challenging to assess from a single 2D photograph.\n- Ambiguity and Confounding Factors: For example, in one case, GPT-4o incorrectly identified medicine powder on a patient's skin as a white, hypopigmented scale. Without the accompanying clinical record, a human dermatologist would likely find it difficult to distinguish this from the image alone. This highlights that for both humans and AI, visual data can be ambiguous, and other factors like suboptimal lighting or the lack of clinical metadata can impede a purely visual assessment.\n\n### Limitations\n---\nWe thank the reviewer for this excellent suggestion. We agree that exploring methods to automate or more systematically validate the checklist creation process is critical. This is a valuable avenue for future research that builds directly on our framework and we will add this to our discussion of future work."}, {"review_id": "AcJZnWvU7L", "reviewer": "Reviewer_WddE", "summary": "The authors introduce a novel diffusion-based framework for generating skin images, titled \"Medically Accurate Generation of Images through AI-Expert Collaboration.\" This method effectively converts criteria set by experts into practical feedback for synthesizing images of Diffusion Models, markedly enhancing clinical precision and decreasing the workload on human operators.  Within this framework, human experts are mainly needed to: (1) create checklists based on credible sources that are easily verifiable by an MLLM, and (2) monitor the MLLM’s feedback on synthetic images throughout the training of T2I DMs. Experiments show that their approach significantly enhances the clinical quality of synthesized images of skin diseases, aligning closely with dermatologist evaluations. Furthermore, incorporating these synthesized images into the training data boosts diagnostic accuracy by 9.02% on a complex skin disease classification task involving 20 conditions, and by 13.89% in a few-shot scenario.", "questions": "Results are limited to one single modality. Do the authors think the method can be adapted to the modalities with ease? Can the authors comment on the key parts to be changed for adoption?\n \nCan the authors illustrate whether the model is able to generate enough diverse data for pictures of similar anatomic sites?\n\nAt line 185, the authors mentioned that \"only synthetic images are sent to GPT-4o API services and no real patient images are processed by the MLLM, to preserve privacy.\" However, the clinical images that are the subject of this paper are large field-of-view images, which can contain PHI. As the proposed framework only alters the image with respect to the target disease category, the privacy concerning certain portions of the image will most likely remain intact. In this case, even the synthetic image may still contain private content, and the sharing of data remains a concern. Nonetheless, this issue does not affect the validity of the paper's results, as it can be addressed by using a HIPAA-compliant, local model in the future.\n\nIn Figure 2: the direction of the arrow between the classes on the left side of the image does not align with the text description in the caption.", "limitations": "Results are only limited to one single modality; further testing involving different modalities may be required\nDiversity in the generation is a question. The authors did not illustrate if the model is able to generate enough diverse data for pictures of similar sites.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n-  The need for augmented data in training medical models is a significant problem. More importantly, the development of methods that are grounded in concepts and are related to the lesion appearance and diagnostic properties is an essential need in the field. This work touches this issue and provides a solution.\n- The claims presented in the article are backed with sufficient techinical justification. In this respect the authors used a combination of image generation quality metrics (e.g. FID) and also some diagnostically related metrics (the expert checklist) to gauge the quality of the generated images. \n- The use of MLLM during the evaluation of the generated images increases the efficiency of the approach as it decreases the burden on the clinician/expert. \n- The evaluation results from the AI-Human collaborative system are fed back to the diffusion model to fintune the model. The authors used 2 alternative ways to do this (Reward-model guided fine tuning and Direct Preference Optimization) and validated their performance through a classification experiment. \n- The paper is well organized and clearly written. It provides adequate information to the reader to understand the approach \n- Numerous diffusion-based data generation models have been discussed in the literature. The authors build on the literature by incorporating human feedback through \"skin condition checklists\" to enhance the grounding of generation models.\n\n**Weakness**\n- The authors used a generic AI model to check the generated images against the skin condition checklist generated by experts. It is not clear how capable this model is of assessing the images in terms of the queried conditions. \n- It is not clear if the authors did any qualitative evaluation conducted by an expert. In the conclusions section they mentioned about qualitative evaluations but it is not clear if they are refering to \"skin condition checklist\" evaluation by the MLLM or some manual evaluation.", "was_revised": true, "final_justification": "The authors replied all my comments and offered to update the camera ready paper with addition details provided in their comments", "rebuttal": "We thank the reviwer for the recognition of our method's technical justification and clear presentation, and we appreciate the constructive feedback provided. \n\n## W1: Reliability of GPT-4o\n---\nWe address this in two parts: by detailing our framework's built-in safeguards and by presenting two new empirical analyses.\n\n1. **Framework Design to Mitigate Hallucination:** The MAGIC framework is explicitly designed to minimize the risk of MLLM hallucination through our AI-Expert collaboration paradigm. The MLLM is not asked to perform open-ended reasoning. Instead, its role is constrained to evaluating an image against a predefined clinical checklist. These checklists, designed by dermatologists, decompose complex medical concepts into simple, visually verifiable features. For instance, while an MLLM may not intrinsically understand \"Lupus Erythematosus,\" it can effectively verify \"symmetric butterfly rash across the cheeks\". This approach transforms a complex diagnostic reasoning task into a series of closed-question evaluations, which are far less susceptible to unconstrained hallucination. \n\n    Even so, we agree that an analysis of GPT-4o's performance is important. To that end, we conducted a new study where we had GPT-4o evaluate 100 real images from an internal dataset with clinical records confirmed by in-person visits. The MLLM was tasked with describing each image based on the five criteria (location, lesion type, shape/size, color, and texture). We then invited a board-certified dermatologist to assess these descriptions. As shown in the table below, the dermatologist assigned an **alignment score** of 3 or greater (on a 5-point scale) to approximately **86%** of the image-description pairs. Notably, many of the lower-scoring examples were also identified by the dermatologist as being visually ambiguous and challenging for a human to assess from an image alone.\n    \n|Score(↑)|1|2|3|4|5|\n|-|-|-|-|-|-|\n|# Examples|8|6|16|21|49|\n\n2. **Choice of MLLM and Comparative Analysis:** The MAGIC framework is designed to be model-agnostic, allowing for the flexibility to use the most suitable MLLM for a given task. For this foundational study, we selected GPT-4o for its exceptional visual reasoning capabilities. However, we agree that a comparative analysis using open-source models is a valuable addition. To that end, we have conducted new experiments testing our MAGIC framework with an open-source MLLM, Google DeepMind's **MedGemma-4B**[1], an open model for medical text and image comprehension. We used MedGemma as an evaluator to assess the same set of 1024 pairs of synthesized images that GPT-4o had assessed. We observed that MedGemma is comparable to GPT-4o as an evaluator in aligning the DM using the MAGIC framework. This demonstrates the flexibility of our MAGIC framework, which is adaptive to both large, closed-source, genric models and smaller, open-source, domain-specific alternatives. The results are shown in the table below.\n\n|MLLM|Classifier|Accuracy(%)|F1(%)|Precision(%)|Recall(%)|\n|-|-|-|-|-|-|\n|GPT-4o|ResNet-18|38.33|37.01|38.41|36.06|\n|GPT-4o|DINOv2|55.01|54.05|54.96|53.70|\n|MedGemma|ResNet-18|36.97|35.55|37.12|36.32|\n|MedGemma|DINOv2|54.19|53.08|54.78|53.53|\n\n[1]. MedGemma Technical Report. (2025)\n\nWe'll include the results and analyses in the future version of the work.\n\n## W2: Expert Evaluation\n---\nA separate qualitative evaluation was conducted entirely by human medical experts. This is detailed in **Section 5.1** under \"Expert evaluation on generated images\". In this study, we engaged medical experts to evaluate the synthetic data against the five clinical criteria. The results of this manual evaluation are summarized in **Fig.4(d)**. \n\nAdditionally, we also conducted a blinded Turing test to assess the clinical realism of our generated images. We created a mixed set of 500 real images and 500 synthetic images generated by MAGIC-DPO. An expert dermatologist, who was not involved in the project, was asked to rate the likelihood of each of the 1,000 images being synthetic on a 5-point scale given the condition label. The expert's ratings for the real and synthetic images followed a remarkably similar distribution, and the expert tended not to make absolute predictions. This indicates that the expert could not confidently distinguish the synthetic images from the real clinical photos. This experiment complements the quality evaluation in Fig. 4(d) by directly assessing realism.\n\n||1-very likely|2-likely|3-neutral|4-unlikely|5-very unlikely|\n|-|-|-|-|-|-|\n|Real|9|151|90|249|1|\n|Syn|11|131|83|267|8|\n\n## Q1, L1: Adapting MAGIC to other Modalities\n---\nThis is an excellent question. While our current work focuses on dermatological photos, the core MAGIC framework is designed to be modality-agnostic. Its fundamental principle is to integrate structured, domain-specific knowledge into the generative process via expert checklists that are executable by MLLMs. As we note in our conclusion, the framework is adaptable to other medical fields, each with its \"unique visual characteristics and clinical requirements\". For example, adapting the method to modalities like radiology or digital pathology would be feasible and would primarily require changes to the following three components:\n\n- **The Expert Checklists**: This is the most critical and domain-specific component. For a new modality, relevant experts (e.g., radiologists) would need to create new checklists that capture the essential visual features of pathologies in that domain. For a chest X-ray, this could include criteria for lesion location, opacity, borders, and size.\n- **The Base Diffusion Model**: Before applying the MAGIC feedback loop, the base diffusion model would need to be fine-tuned on a representative dataset from the new modality. This preliminary step is crucial for teaching the model the fundamental anatomy and characteristics of the new image type.\n- **The MLLM Evaluator**: While powerful MLLMs are increasingly generalist, optimal performance might be achieved by selecting a model best suited for the target modality. For instance, one might choose an MLLM specifically fine-tuned on radiological images and reports to ensure the most accurate interpretation of the expert checklist.\n\n## Q2, L2: Generate Diverse Data\n---\nThe ability to generate diverse data is a key strength of our framework, which we achieve through its novel I2I mechanism and can verify both qualitatively and quantitatively.\n1. **Leveraging Source Image Diversity via the I2I Pipeline**: The primary driver of diversity is our use of the I2I pipeline. Our process transforms a real source image to a new target condition while preserving the original anatomical context. This means the diversity of the generated images—in terms of skin tone, age, gender, lighting, and viewing angle—is directly inherited from the diversity of the real source images. For example, to generate 100 diverse images of \"lupus on the face,\" we can start with 100 diverse real-world images of faces (with any other condition), and the resulting synthetic images will retain this rich variation. And this is where MAGIC shines. As demonstrated in our few-shot learning experiments, the framework is not limited to using labeled images from the training set as a source. We can use any relevant image, including unlabeled data, to generate new examples. This dramatically expands the potential for diversity, as the framework's capacity to generate varied data grows with the number of external source images one can access.\n2. **Qualitative and Quantitative Evidence**: We can see the model's ability to generate diverse conditions on the same anatomy in **Fig.2**. By comparing the first and last columns, it is clear that the framework can successfully transform a single image into a new target condition with distinct features, while preserving the underlying anatomical information. This increases dataset diversity by allowing different conditions to be realistically rendered in various contexts. We will add more qualitative comparisons between real and synthesized images in our work. Quantitatively, the improvement in classification performance also indicates the improved generalizability resulting from enhanced diversity. However, we agree that rigorous quantification of diversity to assess the quality of synthesized images is an important question, and we will investigate this in our future work.\n\n## Q3: Privacy Concern about GPT-4o\n---\nThe MAGIC framework transforms the lesion and dissociates the condition from any particular person, reducing the risk of privacy leakage. Additionally, privacy risks can be further minimized by running open-source, HIPAA-compliant (as suggested by the reviewer) MLLMs locally as evaluators. Moreover, synthesizing images that preserve a higher level of differential privacy would require dedicated DP strategies. While a formal, quantitative analysis guaranteeing the complete removal of all identifying features was beyond the scope of this work, we agree that aligning diffusion models while removing identifying cues is an important future direction for medical and other privacy-critical domains. We will add a note on this to our discussion in the \"**Ethical Considerations and Safeguards**\" section of the final paper.\n\n## Q4: Image-caption Mismatch in Fig.2\n---\nWe thank the reviewer for their careful reading and for catching this error. The caption text is accurate, and we will correct the arrows in the figure for the final version of the paper."}]

Author rebuttal:

Meta-review: This work presents MAGIC, a novel framework that effectively generates clinically accurate synthetic dermatological images by fine-tuning diffusion models with AI-generated expert feedback from a multimodal LLM (GPT-4o). The paper's principal strength is its pragmatic and well-validated solution to a critical problem: reducing the immense burden of manual expert annotation in medical imaging. The proposed semi-automated workflow, which translates dermatologist checklists into machine-readable rewards for Direct Preference Optimization (DPO) or reward-based fine-tuning, is sound and demonstrates tangible, significant improvements in downstream diagnostic accuracy (e.g., +9.02% on a 20-class classification task). While reviewers noted weaknesses concerning the domain-transfer novelty, the inherent reliance on checklist quality, and the potential hallucinations of the MLLM judge, these are considerably outweighed by the method's practical utility, rigorous experimental design, and strong empirical results. The authors' rebuttal successfully addressed these concerns, as all reviewers upgraded their scores. The rebuttal provided sufficient justification and promised clarifications, thereby solidifying the paper's contribution to the field.
Decision: Accept (poster)