L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancement

Authors: Morgan Bruce Talbot, Gabriel Kreiman, James J. DiCarlo, Guy Gaziv
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Human-aligned models, robust neural networks, visual perception, perceptual learning, medical machine learning

Abstract:
The currently leading artificial neural network models of the visual ventral stream - which are derived from a combination of performance optimization and robustification methods - have demonstrated a remarkable degree of behavioral alignment with humans on visual categorization tasks. We show that image perturbations generated by these models can enhance the ability of humans to accurately report the ground truth class. Furthermore, we find that the same models can also be used out-of-the-box to predict the proportion of correct human responses to individual images, providing a simple, human-aligned estimator of the relative difficulty of each image. Motivated by these observations, we propose to augment visual learning in humans in a way that improves human categorization accuracy at test time. Our learning augmentation approach consists of (i) selecting images based on their model-estimated recognition difficulty, and (ii) applying image perturbations that aid recognition for novice learners. We find that combining these model-based strategies leads to categorization accuracy gains of 33-72% relative to control subjects without these interventions, on unmodified, randomly selected held-out test images. Beyond the accuracy gain, the training time for the augmented learning group was also shortened by 20-23%, despite both groups completing the same number of training trials. We demonstrate the efficacy of our approach in a fine-grained categorization task with natural images, as well as two tasks in clinically relevant image domains - histology and dermoscopy - where visual learning is notoriously challenging. To the best of our knowledge, our work is the first application of artificial neural networks to increase visual learning performance in humans by enhancing category-specific image features.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730535452096

thread_type:
official_review

<thread_object>
creator: reviewer_njdg

<content>
summary:
This paper proposes a method to enhance human visual learning by designing a model-based selection and enhancement algorithm to improve classification accuracy during testing. First, the authors select images to present to novice learners based on a model’s estimated recognition difficulty for each image. Next, they apply image perturbations intended to aid recognition for novice learners. The authors conduct experiment on the three benchmark datasets, including the natural image and the clinical image to verify the effectiveness of their proposal.

strengths:
1. This paper is well-motivated, and a decent amount of technical details are given.
2. The idea of improving the categorization performance of the novice learner by leveraging the capacity of the robustified artificial neural network is both interesting and practical.
3. The reported improvement in novice learners' performance is notable, with gains in both test accuracy and reduced training time.

weaknesses:
1. The establishment of the empirical observations is somewhat unconvincing. Do these observations hold in more complex classification tasks or when applied to medical imaging?
2. The related work section lacks discussion of both the machine teaching and human-machine vision alignment methods.
3. The size of the particants is somewhat small. 
4. The perception of enhanced images may be altered due to perturbations.

questions:
1. The empirical observations are derived from a 16-way animal categorization task on natural images, which seems somewhat simplistic. It would be valuable to examine how these observations hold up in more complex categorization tasks or with different types of images, especially medical images. Given the typically limited availability of medical images, the proposed method could have promising applications in the medical imaging field.

2. Beyond the empirical observations, is there any physiological insight or analysis on why the proposed model-based selection and enhancement method could improve novice learners’ performance in categorization tasks?

3. The authors do not discuss the related machine teaching literature. An in-depth comparison with machine teaching methods, particularly with "Teaching Categories to Human Learners with Visual Explanations" (CVPR 2018), would be valuable. This work similarly considers image difficulty; an introduction and comparison with it are beneficial.

4. The authors should also discuss the connection to human-machine vision alignment methods, such as "Harmonizing the Object Recognition Strategies of Deep Neural Networks with Humans" (NeurIPS 2022).

5. The sample size of participants is relatively small, and expanding the participant pool is recommended to enhance the reliability of the results; also, recruiting participants from diverse backgrounds would improve the generalizability of the findings. If expanding the participant pool is impractical due to time or budget constraints, performing a power analysis or discussing effect sizes could help strengthen the reliability of the analysis.

6. When using perturbations to enhance images, how does the method ensure that essential image details remain unchanged, particularly when using a large ϵ (e.g., 20)? This concern is especially pertinent for medical images, where even slight pixel changes may alter critical information. I recommend involving medical experts to review the enhanced images or using quantitative similarity measures (such as SSIM or FSIM) to verify that essential details are preserved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681039849

thread_type:
official_review

<thread_object>
creator: reviewer_5u3k

<content>
summary:
This paper presents L-WISE, a framework leveraging adversarially robust ANNs to estimate image difficulty and apply nuanced perturbations that facilitate human learning in visual categorization tasks. By selecting challenging images and amplifying category-specific features, L-WISE improves human categorization accuracy by 33-72% and reduces training time by 20-23% across both general and clinical domains (e.g., dermoscopy and histology). The authors also discuss ethical implications, emphasizing the benefits of enhanced medical training and cautioning against potential biases that may arise from reliance on model-derived guidance.

strengths:
1. The application of Robustified ANNs for improving human visual performance on image categorization seems like an interesting avenue. 
2. L-Wise empirically demonstrate gains in categorization accuracy and training efficiency. 
3. The paper addresses ethics concerns. Since the work mentions use of clinical data ethics discussion is of critical importance.

weaknesses:
1. The paper focused on the performance of ventral stream. But we know that the human visual stream has a dorsal stream (where) that locates an object and the ventral stream (what) stream. And the interplay of these two streams forms the basis of human visual system. In this work the authors mainly focused on the ventral stream. From only quantified data, we can see the gains but it is very hard to trace this back to the nuanced perturbations the ANN produces. Hence, the suggestion is to use human gaze. The human gaze will precisely pin-point the "where" aspect and then will truly help us understand if at all the model perturbations are helping improve human performance. Can the authors please explain this? 
2. A robust DNN actually has worse performance on nominal data points. Data points that have not been corrupted adversarially. What was the motivation of the authors to select such a model for their experiments? 
3. The perturbations -  The perturbations if I am not mistaken are very subtle ones. For fine grained classifications, humans do follow curriculum learning but learning structures gradually from simpler to harder concepts. No experiments have shown this. It would be great if the authors can provide some empirical results/ explanation that can explain how will their method occur when you focus on structural cues rather than model perturbations that will benefit fine grained categorization. 
4. I am providing some citations related to Gaze and dual stream hypothesis that can help authors clarify my concerns 

a. A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains, NeurIPS 2023.
b. Literature related to papers accepted in NeurIPS Gaze Meets ML workshop. That workshop accepted papers will provide intuition of how human gaze can be used in coherence with DL models.

questions:
1. For model perturbations, can the authors please provide heat maps or any qualitative results that will help us track the ANN perturbations to human visual learning? 
2. Is there a way to show if the study scales/generalizes to other ANNs as well other than Adversarially trained ones? Since human subjects have been used here, I am not sure how feasible experiments will be. This can be a general neural network or a network trained by the CutMix [a] loss that provides robustness benefits as well. 
3. I feel generating perturbations based of dual stream networks and then using human gaze to track these will be a much stronger claim to the work. Can the authors please address this question? 
4. What about adversarially trained transformers? The attention maps are different from CNN feature maps. How will the study be applicable for perturbations based of a transformer backbone? 
5. Please also address concerns raised in the weakness sections. 

a. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019,
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710214546

thread_type:
official_review

<thread_object>
creator: reviewer_wuum

<content>
summary:
The paper proposes a novel approach to augment human learning in image categorization tasks. By leveraging robustified ANNs, the study introduces model-guided image selection and enhancement strategies that increase human test-time categorization accuracy by up to 72% and reduce training duration by around 20-23%. L-WISE employs selecting images based on predicted difficulty levels and enhancing images with pixel perturbations.The proposed approach is tested on natural images, dermoscopy, and histology images. The results demonstrates efficacy of L-WISE in aiding novice learners in fine-grained categorization tasks. This research represents one of the first applications of ANNs in optimizing human visual learning in clinically relevant domains.

strengths:
- Presents an innovative use of robustified ANNs to predict task difficulty and enhance images, aiding human perception and learning.
- Shows broad applicability by successfully testing across diverse domains, such as natural image classification, dermoscopy, and histology.
- Achieves practical efficiency by reducing training time and improving test-time accuracy, beneficial for fields requiring rapid, accurate human image categorization training.

weaknesses:
- Lacks a dedicated related work section, which would help contextualize the research.
- Both low and high logits from ANNs show significant variation in human accuracy, making predictions less reliable in certain logit intervals.
- Uses only the ResNet-50 architecture, limiting generalization; further testing with models like vision transformers (ViT) is needed to support broader conclusions.
- Image enhancement may introduce biases, potentially improving accuracy only for certain major classes; additional metrics like precision and recall per class, rather than just mean accuracy, should be reported to provide a clearer assessment.

questions:
1. How to choose $\epsilon$ for different tasks and image domains?
2. What criteria determine if a model is "robustified" enough for use? Have you considered specific metrics to evaluate the robustness of guide models, and how do these metrics correlate with human learning outcomes?
3. Did you collect qualitative feedbacks from participants? Did the new curriculum and enhanced images increase mental stress of human learners? Additional learning costs beyond training time should be considered, such as cognitive load and emotional well-being.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732727747223

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of changes in rebuttal revision

comment:
We thank the reviewers for all of their thoughtful feedback and insights, and for their encouraging words expressing interest in our work. We agree that the constructive feedback provided would improve the manuscript and clarify its contributions. Below, we provide a summary of the most important changes implemented in this revision:

**Related Work:**

In response to reviewer feedback we have added a substantial dedicated Related Work section, with discussion on prior approaches to image difficulty prediction, image enhancement, alignment of ANN models with humans, and using ANN models to assist human learning (machine teaching).

**Figures & Presentation:**

**Fig. 1:**
- Added data points from additional human participants to logit predictions (panel A) to reduce measurement uncertainty.
- Revised example image selection.
- New, cleaner figure design (for example, example images in panel B2 are now lined up directly underneath the corresponding data in panel B1).
- Fixed/added text labels.
- Added example images to provide intuition for the proposed model-based recognition difficulty score.

**Fig. 2:**
- Cleaner and clearer explanatory text labels.
- Fixed typos.
- Changed focus from robustified models to ANN-based classification models more broadly.

**Fig. 3:**
- Added interpretability images: heatmaps and ``difference’’ images indicating which image regions are affected by enhancement.
- Cleaner/clearer figure design and text labels.

**Fig. 4:**
- New figure design.
- Improved visualization of the joint test-accuracy and training speed gains as a scatter plot.
- New analysis for per–class precision and recall metrics across the three considered image domains.
- Added interpretability images for dermoscopy and histology domains: heatmaps and ``difference’’ images between original and enhanced image versions.
- Cleaner and clearer explanatory text labels.

**Added/Extended Supplementary Material:**

**Fig. S1** - Additional examples of ImageNet image enhancement, with interpretability images similar to the new Figures 3 and 4.

**Fig. S4** - Expanded analysis that more comprehensively compares the proposed image difficulty score with the prior state of the art.

**Fig. S5** - Extension of image difficulty prediction to other model types and robustification levels (robustified vision transformer, CutMix, ResNet-50 non-robustified and robustified using $\epsilon=1,3,10$).

**Fig. S6** - Quantitative human psychophysics analysis of image category enhancement effectiveness by various models (robustified vision transformer, CutMix, ResNet50 non-robustified and robustified using $\epsilon=1,3,10$).

**Fig. S7** - Qualitative visual analysis of enhancement effects by various models (robustified vision transformer, CutMix, ResNet50 non-robustified and robustified using $\epsilon=1,3,10$).

**Fig. S11** - Support for empirical observations presented (for the 16-way ImageNet task) in Fig 1 for the three different image domains focused on in the learning tasks.

**Table S1** - Precision/Recall per class in every learning task considered, including confidence intervals.

**Table S2** - Demographic characteristics of study participants, aggregated across all experiments.

**Section S6** - Comparing different guide models (robustified vision transformer, CutMix, ResNet-50 non-robustified and robustified using $\epsilon=1,3,10$).

**Section S8** - Paragraphs 2-4 are new, outlining extension of image difficulty prediction and image enhancement observations to learning domains and class-wise breakdown of learning gains attributed to L-WISE.

**Section S9** - New analysis showing that participants who receive L-WISE assistance are more likely than control group participants to persist through the entire learning task.

**Section S11** - Details on participant recruitment and demographics.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents an advancement in using ANNs to enhance human visual learning of image categories. The research makes several good contributions to both theoretical understanding and practical applications.
The primary contribution is the development of L-WISE (Logit-Weighted Image Selection and Enhancement) which uses robustified ANNs to improve how humans learn to categorize images. The system works in two ways: it predicts how difficult images will be for humans to categorise, and it can enhance images to make their key features more recognisable. When combined, these capabilities allow for creating optimised learning sequences that significantly improve both learning speed and accuracy.
A key strength of the research is its comprehensive empirical validation. The authors demonstrated L-WISE's effectiveness across multiple domains, including natural images (moth species), medical dermoscopy (skin lesions), and histology (tissue samples). The results were good, showing improvements in accuracy and training time.

The research also makes a theoretical contribution by establishing that robustified ANNs can serve as accurate predictors of image recognition difficulty for humans. This finding extends our understanding of human-AI alignment and suggests new ways that AI models can be used to understand and enhance human perception.

However, the study does have some limitations. The authors acknowledge they didn't exhaustively explore optimal curriculum design strategies or enhancement parameters. There's also a key concern about the homogenising effect of their enhancement technique - for example, enhanced images of benign moles tend to look very similar, potentially obscuring important real-world variations that medical practitioners need to recognise.

Despite these limitations, the research represents a good step forward in using AI to enhance human learning. The fact that the improvements were consistent across different domains and tasks suggests the approach has broad applicability. Furthermore, the authors' transparent discussion of limitations and potential risks demonstrates a thoughtful consideration of the wider implications of their work.