Title: Teaching Humans Subtle Differences with DIFFusion
Venue: NeurIPS 2025
Accepted: no
Keywords: counterfactual

machine teaching

image generation
Abstract: Scientific expertise often requires recognizing subtle visual differences that remain challenging to articulate even for domain experts. We present a system that leverages generative models to automatically discover and visualize minimal discriminative features between categories while preserving instance identity. Our method generates counterfactual visualizations with subtle, targeted transformations between classes, performing well even in domains where data is sparse, examples are unpaired, and category boundaries resist verbal description. Experiments across six domains, including black hole simulations, butterfly taxonomy, and medical imaging, demonstrate accurate transitions with limited training data, highlighting both established discriminative features and novel subtle distinctions that measurably improved category differentiation. User studies confirm our generated counterfactuals significantly outperform traditional approaches in teaching people to correctly differentiate between fine-grained classes, showing the potential of generative models to advance human visual learning and scientific research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explore piecing together various popular methods and models such as the usage of CLIP embeddings and Diffusion models in a novel way to propose an algorithm to detect subtle discriminative differences and synthetically introduce them to images while keeping individual examples recognizable. The methodology boasts the ability to train on unpaired small datasets and also claims to pick up differences missed by domain experts. This work could have meaningful impact in scientific education and training, particularly in domains where visual expertise is crucial but difficult to teach through traditional methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The proposed methodology can work on unpaired datasets as well as on limited training data which makes it useful for niche subjects with less available data.\n* Using machine learning to provide a method to teach humans subtle differences is a novel idea and provides a significant improvement in human classification ability.\n* Comprehensive quantitative testing done across multiple domains and appropriate baseline comparisons including recent methods like TIME and Concept Sliders.\n* The paper includes thoughtful analysis of dataset bias visualization and the relationship between training set size and performance.\n\nWeaknesses: \n* While the authors have provided the basic details behind the sub-components further details behind the reasoning of choosing the exact methodology is missing. Furthermore the experiments focuses on paired classifications, it would be good to have experiments that capture performance for multi-class problems.\n* More commentary providing an analysis of the trade-off between identity preservation and class change would have provided a stronger backbone to the proposed algorithm.\n* While there is an experiment provided showcasing the teaching capability, the paper is lacking an expert verification to ensure the changes generated align with scientific facts. It would be good to have a benchmark against current knowledge before expanding further.\n* There needs to be heavy adversarial tests done to make sure this methodology does not get influenced by inherent biases in machine learning models (especially the CLIP model used for embedding) if it is to be used for teaching.\n\nQUESTIONS:\n1. How do you ensure the difference of means approach doesn't amplify false correlations in training data? \n2. What happens when class boundaries are not linearly separable in CLIP embedding space?\n3. How would you validate that discovered visual patterns represent genuine scientific phenomena rather than dataset artifacts?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DIFFusion, a counter-factual image editing framework to teach humans how to notice sub-pixel differences between visually similar classes. DIFFusion starts from a real image, and then invert it into diffusion-model noise maps, followed by adding a class-diff vector in CLIP-embedding space and then finally re-decode. \n\nBy controlling the strength of this vector and the number of denoising steps skipped, DIFFusion outputs identity-preserving images in which only the minimal, discriminative features can change. Experiments are run on six binary class datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Proposes an interesting idea of diffusion-based image editing not as an explainability only but as an explicit machine-teaching mechanism, which is very interesting. The approach synthesizes counterfactual variations of one single example, then trains humans to recognise class-distinguishing micro-features that experts may find challenging to verbalise.  The core idea of subtracting CLIP embedding mean of the source class from that of the target class to create a class-difference vector Δc, then adding a scaled version of Δc to the conditioning signal of a diffusion model may come across as simple but could have implications if many domains.  The overall concept is to fill a gap where labelers cannot articulate decision boundaries but still need to learn them.\n2. The approach is methodological Sound and eq 1-8 provide a clean mathematical derivation. Also, principled ablations on ω and Tskip.\n3. The experimental suite spans six markedly different datasets three of which almost never appear in main CV benchmarking research papers.\n4. The authors have open-sourced code, pre-trained checkpoints, and scripts for all experiments.\n\nWeaknesses:\n1. DIFFusion is intrinsically framed around a single source:target, and arithmetic vector Δc is computed as the difference between two CLIP class means. Here success is if an oracle flips from one class label to the other, and the user study tests a before/after recognition of exactly two categories.  Extending this to multi-class involve non-trivial re-design of the sampling policy, likely new evaluation metrics.\n2. The other challenge is that the entire pipeline hinges on CLIP’s semantic space being both linear and meaningful for a given domain.  In OODs, CLIP embeddings can collapse, mis-align, or over-compress small cues, potentially steering Δc in arbitrary directions.  There is no mention of quantifying how sensitive performance is to embedding quality.\n3. The user evaluation comprises 30 participants split into three groups of ten—a sample that yields wide confidence intervals and low statistical power, especially when broken down by dataset and expertise level.  Although the appendix displays error bars, the NeurIPS checklist marks “statistical significance” as NA, and no formal power analysis is provided. \n4. Each image requires DDPM inversion+generation of ten candidate edits. Out of these only the first LPIPS-minimal, label-flipping image is kept.  On an A100 GPU, it is tens of seconds per image and ~40 GPU-hours for the six-datasets.  This kind of overhead may be acceptable for expert teaching tools but is a limitation in real-time or mobile deployment.\n5. The paper is missing an analysis of adversarial or privacy risks.  An attacker could craft deceptive counterfactuals or exploit the inversion step to recover information about proprietary training images.\n\nQUESTIONS:\n1. Please quantify how the visual-algebra automatically isolates discriminative features, meaning are the ∆c directions regularized? \n2. Please provide a legend for ω and Tskip.\n3. Since selecting the first edit that flips prediction could bias toward minimal LPIPS, how would you verify no adversarial artefacts emerge with this apptoach?\n4. Please report standard deviations over the 5 ω/Tskip samples to gauge stability.\n5. Can you provide the exact video frame rate and if participants could replay. Also clarify IRB approval ID.\n6. Please clarify if DIFFusion can chain two vectors, for example Monarch->Viceroy->Queen without any re-inversion?\n7. What is the computational cost of DIFFusion?\n\nLIMITATIONS:\nN/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDIFFusion is a diffusion-based method presented to help human experts understand subtle discriminative features in images difficult to articulate by experts.\nThe presented method is said to perform well in challenging scenarios, which is shown in six experimental settings and evaluated with multiple metrics.\n\nDisclaimer: \nI do want to highlight my lack of expertise in this particular field. I can assess the soundness of this paper, however I am less confident in assessing whether the chosen baselines are the correct ones to compare to, or if the evaluation is standardized. For these matters I'd refer to the other reviewers.\n\nSTRENGTHS AND WEAKNESSES:\n* Strengths\n1. The experiments are designed to rigorously evaluate the effectiveness of the DIFFusion system in generating counterfactual visualizations and its utility in teaching humans to discern subtle visual difference.\n\n2. The user study seems convincing, showing solid improvements by using DIFFusion.\n\n3. Table 2 paired with Fig. 3 show superior performance across the chosen datasets, which is convincing and compelling.\n\n4. Theory seems sound.\n\n* Weaknesses\n\n1. In the user study, the best baselines often perform worse than a user analyzing unpaired data. This makes me think that it is possible that either the experiment is not useful to evaluate the claims, or else the baselines are not correctly chosen. I'd like the authors to clarify this point.\n\n2. Not clear where the technical novelty lies. Please clarify\n\n3. It is said that othere teaching methods \"typically require aligned, abundant data and focus on single modalities\" and that \"our work extends these efforts\". I think it would strengthen the paper to clarify what has been done to mitigate the problems with data and whether DIFFusion would perform equally than the baselines in the presence of abundant data.\n\nQUESTIONS:\n1. Why do the best baselines sometimes perform worse than a user analyzing unpaired data in the user study, raising concerns about the experiment's utility or the baseline selection?\n\n2. Could you please elaborate on the core technical novelty of DIFFusion, distinguishing its contributions from existing state-of-the-art image editing and counterfactual generation methods?\n\n3. Could the authors clarify the specific mechanisms within DIFFusion that mitigate these data requirements, especially regarding \"aligned, abundant data\"? Furthermore, how would DIFFusion's performance, particularly its LPIPS, compare to baselines when abundant data is available, and would it still offer a significant advantage?\n\nLIMITATIONS:\nSensitivity to Dataset Bias and Limited Precise Control: A primary limitation is that DIFFusion edits images based on differences between class mean embeddings. This makes the method sensitive to dataset bias\n\nPerformance on \"Common Objects\" Datasets: While DIFFusion excels on scientific datasets and those where visual details are hard to describe textually (like Black-Holes and KikiBouba), its LPIPS (perceptual distance) performance on \"common objects\" datasets like AFHQ and CelebA-HQ is not always the best compared to baselines", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method that leverages diffusion models to highlight differences between visual categories in images, with the goal of teaching humans to recognize differences. The method is tailored for domains where text descriptions are not easy, such as scientific imaging. The model consists of four main steps, inversion, space arithmetic, generation, and fine-tuning (optional). The results indicate that the method works well and produces counterfactual images that preserve the identity of the original image while making sufficient changes to change the decision of a pre-trained reference classifier.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Simple method that leverages existing components, including pretrained diffusion models and CLIP encoders.\n* Almost perfect scores in terms of class flipping.\n* Realistic image generation.\n\nWeaknesses:\n* The selected baselines are not necessarily well aligned with the task. Machine teaching has been used to approach this problem in the past, and there are solutions that are even multi-class (e.g., [1]). The selected baselines are recent models that could be hacked for this task, but are not specific. Can the authors compare to a baseline that aims to teach humans too?\n* The technical innovation is limited, and it is unclear what is the novelty. For instance, the solution is built in terms of binary classifiers, when previous works have approached the problem with multi-class problems.\n* The methodology makes sense, but it seems to be a straightforward application of existing methods and uses image manipulation strategies that are well known in the community, including space arithmetic and latent inversion.\n\n[1] Becoming the expert - interactive multi-class machine teaching. CVPR 2015.\n\nQUESTIONS:\n* What makes the proposed methodology different with respect to previous works in machine teaching for vision problems?\n* What other directly related baseline could be used to compare the significance of the proposed contributions, which is directly designed for human teaching?\n* How can be the model extended to multi-class settings?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-02"}