L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancement
Authors: Morgan Bruce Talbot
Gabriel Kreiman
James J. DiCarlo
Guy Gaziv
Venue: ICLR 2025 Poster (2025)
Keywords: Human-aligned models
robust neural networks
visual perception
perceptual learning
medical machine learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Robust artificial neural networks can be used to design effective training strategies for enhancing visual category learning in human subjects, resulting in significant improvements in human accuracy and efficiency across multiple image domains.

Abstract: The currently leading artificial neural network models of the visual ventral stream - which are derived from a combination of performance optimization and robustification methods - have demonstrated a remarkable degree of behavioral alignment with humans on visual categorization tasks. We show that image perturbations generated by these models can enhance the ability of humans to accurately report the ground truth class. Furthermore, we find that the same models can also be used out-of-the-box to predict the proportion of correct human responses to individual images, providing a simple, human-aligned estimator of the relative difficulty of each image. Motivated by these observations, we propose to augment visual learning in humans in a way that improves human categorization accuracy at test time. Our learning augmentation approach consists of (i) selecting images based on their model-estimated recognition difficulty, and (ii) applying image perturbations that aid recognition for novice learners. We find that combining these model-based strategies leads to categorization accuracy gains of 33-72% relative to control subjects without these interventions, on unmodified, randomly selected held-out test images. Beyond the accuracy gain, the training time for the augmented learning group was also shortened by 20-23%, despite both groups completing the same number of training trials. We demonstrate the efficacy of our approach in a fine-grained categorization task with natural images, as well as two tasks in clinically relevant image domains - histology and dermoscopy - where visual learning is notoriously challenging. To the best of our knowledge, our work is the first application of artificial neural networks to increase visual learning performance in humans by enhancing category-specific image features.

Published as a conference paper at ICLR 2025
L- L-WISE: BOOSTING HUMAN VISUAL CATEGORY LEARNING THROUGH MODEL-BASED IMAGE SELEC-
TION AND ENHANCEMENT
Morgan B. Talbot- 2,3 Gabriel Kreiman -22 James J. DiCarlo and Guy Gaziv A.4
"Boston Children's Hospital, Harvard Medical School
Center for Brains, Minds, and Machines, MIT
& Deett of Health Sciences and Technology. MIT
McGovern Institute for Brain Research, Dept. of Brain and Cognitive Sciences, MIT MIIT Quest for Intelligence
ABSTRACT
The currently leading artificial neural network models of the visual ventral stream which are derived from combination of performance optimization and robus- tification methods have demonstrated a remarkable degree of behavioral align- ment with humans on visual categorization tasks. We show that image perturba- tions generated by these models can enhance the ability of humans to accurately report the ground truth class. Furthermore, we find that the same models can also be used out-of-the-bov to predict the proportion of correct human responses to individual images, providing a simple, suman-aligned estimator of the relative difficulty of each image. Motivated by these observations, we propose to augment visual learning in humans in way that improves human categorization accuracy at test time. Our learning augmentation approach consists of (i) selecting images based on their model -estimestt recognition difficulty, and (ii) applying image per- turbations that aid recognition for novice learners. We find that combining these model-based strategies leads to categorization accuracy gains of 33-72% relative to control subjects without these interventions, on unmodified, randomly selected held-out test images. Beyond the accuracy gain, the training time for the aug- mented learning group was also shortened by 20-23%, despite both groups com- pleting the same number of training trials. We demonstrate the efficacy of our approach in a fine-gra grained categorization task with natural images, as well as two tasks in clinically relevant image domains histology and dermoscopy- where visual learning is notoriously challenging. To the best of our knowledge, our work is the first application of artificial neural networks increase visual learning per- formance in humans by enhancing category-specific image features.
Code
Webpage
INTRODUCTION
Over the last decade, artificial neural network (ANN) models have demonstrated superior perfor- mance as image-computable emulators of neural processing along the human and monkey ventral visual stream. Iterative efforts have developed models that are increasingly aligned with primate vi- sion, as measured by their ability to predict both neural activity and behavioral responses (Schrimpl) et al., 2018). Beyond prediction, promising class of these models "robustified" deep ANNS (Mądry al.] 2018) has been shown enable the generation of image perturbations that pre- dictably control both ventral stream neural activity (Guo et al., 2022) and human object categoriza- tion reports (Gaziv et al.) 2023] Croce & Hein 2020). our work, we ask whether the prediction and control capabilities of robustified ANNs be used to enhance human performance visual categorization tasks.
Beyond categorization of familiar visual categories, one practically important task is learning to rec- ognize new, unfamiliar categories. Although humans can readily learn many new categories even from single example (Lake et al., 2011). some consequential tasks require extensive training to reach high levels of performance. For example, medical specialists such as pathologists and radiol- ogists devote numerous hours to mastering the diagnosis of various diseases from medical images.

---

Published as a conference paper at ICLR 2025
A1.
100
B1.
Rate humans choose
Contrasti imited Adaptive Hislogram Equaltaation
ground truth [%]]
85
MuttiSccll Retinox
with Color Reslonsioo
Adobe Lightroom
Ulmoddieed images
'hard* baay""
70
20
0
10
20
c=0
Perturbation pixel budget
C=
Other off-the-shelf Image enhancement methods +
Ground truth logit value [a.u.]
A2.
B2.
"bird"
'turtl''
Ground truth
"izaird'
'Insact'
"haan"
"blrd"
Figure 1: Robustified ANNs can be used out-of-the-box as image recognition difficulty estima- tors and ground truth percept enhancers. We consider a 16-way basic animal classification task. Panel Al shows the correspondence between human categorization accuracy and model-computed ground truth logit activation values. The curve denotes a logistic regression model predicting the probability of a correct response using only the logit value (p 0.001 from the Wald statistic, AUC 0.72 under 10-fold cross validation). A2 shows example images with varying ground truth logit values (predicted difficulty). BI shows how perturb- ing images via ground truth logit maximization increases human recognition accuracy progressively with the fz-norm perturbation pixel hudget Other off-the-shelf image enhancement methods do not increase catego- rization accuracy, despite inducing larger perturbations of e 43, e 106, and e 26 on average from left to right. B2 shows example images: unmodified (left), enhanced by ground truth logit maximization with pixel budgets e = 10 and = 20 (middle), and enhanced by baseline off the-shelf methods (to the right of the dotted line). All vertical error bars are 95% confidence intervals by bootstrap. Horizontal error bars in panel Al show the standard deviation among images within each logit value bin.
We identify a potential strategy to accelerate the visual learning process by extrapolating from the perceptual learning literature. Simple visual tasks, such as line orientation discrimination, reveal a curriculum effect whereby providing easy examples to novice human learner, before gradually increasing the difficulty, promotes faster perceptual learning (Lu & Dosher, 2022). Motivated by these findings, we ask whether the human-aligned nature of robustified ANNs allows them to aug- ment human learning in complex image domains, chiefly by reducing the initial task difficulty and then increasing the difficulty as learning progresses. A demonstration that these models can be used enhance learning serves as additional scientific test of the models' alignment with human perception, while also pioneering a potentially beneficial application of ANNS in education.
To test the viability of enhancing image category learning in humans, first establish two key empirical observations, summarized in Fig. (i) we find that the human error rate in an image categorization task is strongly predicted by the ground truth logit activation value of a robustified ANN, making it a valid image recognition difficulty score for humans; (ii) we find that this relation- ship also holds reverse -ixel-level perturbations can be generated by the model to maximize the ground truth logit activation, producing an "enhanced" version of the image that is casier for humans to recognize as the ground truth category. While previous findings demonstrate that model-guided perturbations can modulate human perception away from the ground truth category (Gaziv et al.) 2023), we conversely seek to amplify category-relevant features to facilitate correct classification. We observe that our model-based image enhancements yield significant increases in human accu- racy on a classification task with basic animal categories, unlike conventional image enhancement techniques that adjust lowelevel image properties such lighting and contrast. We thus propose an algorithm that combines model-based image enhancement and image difficulty prediction to gener- ate optimized curricula for novice humans learning challenging image categorization tasks.
2

---

Published as a conference paper at ICLR 2025
A.
Labeled dataset of naturally unfamiliar categories
B.
sequence
Learn to categorize images using feedback
C.
Test accuracy unmodified, randomly sampled held-out images
Category
Participant chooses category
Category
Category N
Partiipent chocoees tadegary
Catopery!
Calogory
Canageyy2
Category
feedback)
Category
Category 2
Random sampling
"Category N"
Testing sequence
"Naive"
Correct/Incorret feedback (+correct category)
visual learning
D. Model-based enhanced
Logit representation
E.
visual learning
Use model in two ways
Sample images by recognition difficulty schedule
Enhance images by perturbation budget schedule
Allowable image recognitton difficulty score
ANN
Gr edadualy match tasting Saquened
Enhancement perturbation
pixel budget
Gradually AEquercc
Image
classification model
Prob. Prob.(class i) softmax(())
Training sequence
Training sequence
Figure? 2: Robustified ANNS can be used to boost image category learning in humans. A novice human learner undertakes challenging image categorization task, which consists of training phase (B) and a test phase (C). Images for both phases are randomly drawn from labeled image dataset of unfamiliar fine- grained categories (A). Feedback (correct/ncooreet,, with indication of the correct category) delivered after each trial during the training phase only. Our proposed "Logit- Weighted Image Selection and Enhancement" difficulty score to sample images based on predefined increasing schedule of maximal difficulty per trial (E), and by enhancing images for eaxier recognition with an enhancement magnitude that decreases along a predefined schedule (F).
(I. WISE) approach uses an ANN model (D) to augment the visual curriculun by using
Our proposed method, Logit-Weightt Image Selection and Enhancement" (L- (L-WISE), is illustrated Fig. 2 In our primary experimental setting, a human participant learns an unfamiliar image cat- egorization task by viewing a series of examples, providing category judgment for each image before receiving feedback indicating the correct category (Fig. B). Upon completion of the train- ing phase, test accuracy is measured on held-out images in similar trials without feedback (Fig (C). L-WISE intervenes on this naive visual learning baseline by using a robustified ANN model in two ways: (i) it uses the ground truth logit difficulty score to sample training images based prede- fined, increasing schedule of maximal difficulty per trial (Fig. 2D.)); (ii) enhances training images with perturbation magnitude that decreases along similarly predefined schedule (Fig. PD.F).
Despite the human visual system being well-adapted for rapidly learning new visual categories, we find that L-WE gives rise substantial accuracy gains of 33-72% in test-time accuracy margins above chance relative to control participants. In addition to improved accuracy, L-WISE also signif- icantly reduces the training phase duration by 20-23% with constant number of trials. We demon- strate these effects across three varied image domains and category spaces: moth species in natural photographs, skin lesions in dermoscopy images, and pathologie findings in histology images.
Our main contributions are:
state-of-the-art in predicting image recognition difficulty for humans, using the robustified ANN ground truth logit activation value as a simple but effective difficulty metric.
We establish
We show that robustified ANNs can guide image perturbations that enhance the ability of humans to accurately report the associated ground truth category label.
We propose novel model-based visual learning augmentation approach for humans that substan- tially increases test-time categorization accuracy and also reduces training time. To the best of our knowledge, this the first application image enhancement augment human visual learning.
We demonstrate the broad applicability of our proposed method in variety of image domains, including clinically relevant dermoscopy and histology categorization tasks.
3

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
We develop two important capabilities that form the foundation of our approach to assisting human learners: (I) -fate-of-the-art predictions of the recognition difficulty of images for humans, and (2) image perturbations that increase human categorization accuracy. Many works have ranked the difficulty of images to design curricula for training ANNs (Wang et al. 2021b). Leading approaches include the c-score learning speed proxy (Jiang et al. 2 and the prediction depth (Baldock et al.) 202T] calculated for each image. Mayo al. (2023) applied both of these techniques prediet the recognition difficulty of natural images for humans, defined as either the minimum viewing time required to classify a given image correctly, or (as in our work) the proportion of humans who correctly classify it. Here, we show that a robustified ANN model's logit score associated with the ground truth class is a more accurate predictor of image difficulty for humans than prior methods.
Enhancing image quality has been the focus of many previous studies (Qi ct al. 20211. ranging from correction of low-level properties such as lighting and contrast (e.g., Zuiderveld (1994); Jobson et al. (197)) to ANN models that "upsample" images to higher resolutions (Anwar et al. 2020).
However, very little research has focused on enhancing images to more strongly represent a specific category. Previous works in this vein focused on making images easier for ANN models to correctly classify (Kim et al. 2023] (Tusspov et al. (023) or less vulnerable to subsequent adversarial attacks (Salman et al.) 202 Frosio & Kautz (023). However, such methods do not strongly affect human responses due to perceptual misalignment between humans and naive ANNs (aazi et al.) 2023).
Many studies have focused on model-humen alignment. Brain-Scort benchmarks models in terms of neural representations and behavior (Schrimpf et al. 2018). "Harmonization" " methods drive align- ment using auxiliary objective on ANN- feature importance maps and crowd-sourced human maps (Fel et al. 2022). Other works introduce architecture components to account for addi- tional aspects of human vision, such as the dorsal-stream pathway (Choi et al., (023), recurrent connections for contextual reasoning (Bomatter et al. 202T) and visual search (Gupta et al. 202T).
A key property that enables ANNs generate human-interpretable image perturbations is that of perceptually aligned gradients, which is closely related to ad adversarial robustness and can be induced through adversarial training (Ganz et al., 2023] Gaziv et al., (023). In our present work, we apply dddersarialll-trainee -trained ANNs to enhance images such that they are more strongly associated with their ground truth label by the guiding model and by humans. To the best of our knowledge, we are the first to demonstrate improved human performance on image classification tasks through category- specific image enhancement.
Our primary goal is to apply difficulty prediction and image enhancement to augment human learn- ing. The emerging field of machine teaching (Zhul 2015) employs machine learning to find or generate optimal "teaching sets" that can be used to train models or humans. While many such ap-
proaches have been successfully applied to training machine learning models (c.g.. Liu et al. (2017); Qiu et al. (023)), ) few studies have successfully enhanced image category learning in humans and most of these focus on teaching set selection. [Singla et al. (2014) propose STRICT, which optimizes the expected decrease in learner error based on how the selected images and their labels constrain a linear hypothesis class in a feature space. [Johns et al. (2015) extend a similar approach to select images in an online fashion by modeling the learner's progress. MaxGrad (Wang al. 202 la) uses bi-level optimization to iteratively refine a teaching set by modeling learners as optimal empirical risk minimizers. Most similar to our work are approaches like EXPLAIN (Mac Aodha et al. 2018), which uses ANN class activation maps (CAMs) to highlight relevant image regions while providing feedback to the learner. EXPLAIN also selects a curriculum of images based on (i) multi-class adaptation of STRICT, (ii) representativennss (mean feature-space distance to other images of the same class), and (iii) the estimated difficulty (entropy) of the CAM explanations. [Chang et al. (2023) use bounding boxes to highlight image regions attended to by experts and not novices, allowing hu- mans to more accurately match bird or flower images to one species among five shown in a gallery.
(c.
Our approach to learning augmentation departs from previous studies in several ways. We make explicit estimates of image difficulty with unprecedented accuracy to select easier images for carly- stage learners. Our work is unique in employing category- novel technique in itself, to improve the teaching efficacy of given set of images. While| Aodha cet al. (2018) and (Cang et al. (2023) help learners by explicitly highlighting where learners should attend to in each image, we take a distinct and complementary approach by implicitly highlighting what learners should attend to in order to classify images correctly.
image enhancement, which is
4

---

Published as a conference paper at ICLR 2025
3 OVERVIEW OF APPROACH AND EXPERIMENTS
Our approach to improving visual learning in humans is based on two key observations regarding human-aligned ANN models of the ventral visual stream: (i) they can accurately predict the recog- nition difficulty of specific images for humans (Fig. [T],, and (ii) they can be used to perturb images in way that enhances the ability of humans to accurately report the original ground truth category (Fig. D3). In other words, these models can be used out-of-the-box calegory-recogniionn difficulty estimators and category-percepl enhancers. As such, we propose using them to design sequences of images that humans can use to more efficiently learn recognize unfamiliar image categories. We test our approach in a variety of challenging image domains: natural photographs (moth species classification), dermoscopy images (skin lesion classification), and histology images (benign vs. pre-cancerous colon tissue classification).
3.1 TRAINING TASK-SPECIHE ROBUSTIFIED MODELS
To obtain robustified models for task-specific category spaces, we adversarially trained ResNet-50
ANNs (He et al. 2016 on the ImageNet- 1K (Deng et al., 2009) and iNaturalist 2021 (Van Horn
ct al. 2021) datasets (separately) using the same techniques as Madry et al. (2018). To adapt the resulting model the three categorization tasks of interest, we conducted additional adversarial fine-tuning on the corresponding smaller datasets: a subset of moth species images from iNaturalist (after iNaturalist pretraining). the HAM00000
dataset (Tschandl et al. (2018) (ImageNet pretraining), and the MHIST histology dataset (Wei et al. (222)) (ImageNet pretraining).
3.2 PREDICTING CATEGORY RECOGNITION DIFFICULTY
We propose a simple approach to predicting the human categorization error rate on specific images, in the form of a new image recognition difficulty score: the logit activation (pre-softmax) at the ground truth category output unit of robustified ANN. The higher this logit value is, the lower the human categorization eπrоr rate. We establish this relationship through human participant responses during a natural image categorization task with 16 basic animal categories (Fig. [T].. We found this logit score to be the new state-of ofetth-rrrt in predicting human error rates (see Appendix Fig. (S6).
3.3 GENERATING IMAGE PERTURBATIONS TO ENHANCE CATEGORY PERCEPTS
While the ground truth logit score predicts image difficulty at baseline, we show that we can also generate perturbations that maximize this value by backpropagating from the logit score pixel space and running projected gradient ascent to update the pixel values, limiting the "pixel budget" l2 norm of the perturbation to value e. These perturbations make images casier for humans to recognize with respect to their ground truth labels (Fig. [BB.. This approach is analogous to Gaziv et al. 2023), but is designed to enhance the ground truth percept rather than guiding away from it.
3.4 BOOSTING IMAGE CATEGORY LEARNING IN HUMANS
Our approach to augmenting visual learning is summarized in Fig. In the naive baseline scenario (Fig. ---C), the novice human learner is presented with a sequence of training trials through an online platform. In each training trial, randomly selected image from one of N categories is presented, and the learner attempts to choose the correct category among N possible labels. The learner receives feedback after each training trial indicating the correct category label and whether or not their response was correct. In most of our experiments, to ensure that participants begin at the chance level and avoid possible priors induced by the category names, we assign each category to a random Greek name unrelated to the task (with random assignments for each participant). After completion of the training phase, the experiment transitions into a test phase, in which the same task continues over a held-out set of images and no feedback is provided (Fig. (C). During the test phase we measure the main visual learning outcome of interest, the test accuracy.
Hamesssing key observations of robustified ANNs, our "L-WISE" approach uses one such model to optimize the training phase in a way that improves the test accuracy via two mechanisms: (i) sam- pling training images based on their predicted recognition difficulty, and (ii) enhancing the training images. We adjust the "strength" of both mechanisms in a time-dependent manner: for the former, we set the maximum allowable difficulty score for an image to be shown in given trial, and for the latter, we limit the enhancement perturbation magnitude to an & norm pixel-budget e. The user of approach can flexibly define arbitrary time-dependent profiles for image selection and enhance- ment (Fig. (D-F). In this study, we used a step-wise linear ramp schedule for the allowable image difficulty given time, and exponential tapering of the enhancement e. Intuitively, this should
5

---

Published as a conference paper at ICLR 2025
A.
Rate humans choose di
80
L-WISE (Ours)
Train [lkodback) Test loa foodback)
Moth clazs 2 Moth class
ground truth [%)
Control
+58%
Chance
20
Enhancement Diffieulty score
Image-difficult schedule
C.
Moth class 3
80
percentlee
40
0
Moth class 4
pixel budget
8-
Pixet-budget schedule
D.
Original images
Enhanccdd with Difference
Difference heat map
pixel budget (magnified 5x) c=8
20
40
Trial number
120
140
160
Figure 3: Novice learners who had their curriculum augmented by our method showed im- proved test-time categorization accuracy for previously unfamiliar categories. This figure shows empirical results from 4-way fine- -raingd moth species classification task. Panel A shows examples of the moth classes, side-by-side with their model-enhanced versions at the highest pixel budget used in our exper- iments (c 8). While subtle, one notable difference is the distinctive wing spots of moth class 2, which are enlarged the enhanced version of the image. Also included are difference images showing the (5x magni- fied) difference between original and enhanced images, and heat maps with more red coloration in regions of larger changes from enhancement. compares the averuge smoothed accuracy of participants in L-WISE group and a comtrol group. Shaded areas denote the standard error of the mean. The test accuracy gain of the I-WISS group relative to the control group is stotistically significant (x2 (1) test, p < 0.001). C, D show the rrial-dependent empirical profiles of the average image difficulty percentile of selected images, which (noisily) increases step-wise, wise, and the perturbation pixel budget for enhancement (c), which decreases step-wise. These profiles uniform in the control group (black doned lines), denoting mundomly-choseer non non-enhanced images.
correspond to an easy-to-challenging trend during the training phase. Extensively optimizing these schedules was not goal of our study, which serves primarily as proof-of-concept. Applying image selection and enhancement led to significant gains in the accuracy of human participants on unmod- ified, mundomly-selected test images, while also reducing the time needed to complete the training phase (which consists constant number of trials). This result was robustly observable across the varied image domains and category spaces we tested.
RESULTS
We used robustified ANNs to both enhance images and predict the difficulty of images across multi- ple domains. We applied both of these techniques improve the final test performance (on unmod- ified, randomly-chosen images) of novice humans learning challenging image classification tasks.
4.1 ROBUST MODELS CAN BOTH PREDICT IMAGE RECOGNITION DIFFICULTY AND REDUCE IT
We tested the effects of novel model-based image enhancement algorithm on human image cate- gorization accuracy. We demonstrate that we can enhance images by maximizing the ground truth logit from a robustified ANN (ResNet-50) using gradient ascent image pixel space. As the magni- tude of the enhancement perturbations grows (E2 norm pixel budget e), human participants become increasingly accurate on 16-way animal photograph classification task derived from ImageNet (Fig. DB chance 1/16). While mean accuracy on the original, unmodified (c = 0) images was 0.75, mean accuracy on enhanced images was as high as 0.84 (at = 20). The accuracy gains from enhancement appear to approach saturation point as the perturbations grow larger. The improve- ments in accuracy are also somewhat dependent on the starting ground truth logit score, as shown in Appendix Fig. S accuracy gains are larger for "difficult" images than for "easy" images. Bascline enhancement algorithms Contrast-Limiter Adaptive Histogram Equalization (CLAHE, Zuiderveld
(194]). ) Multi-Scale Retinex with Color Restoration (MSRCR, Jobson al. 19977;
Petro et al.

---

Published as a conference paper at ICLR 2025
L-WISE (Ours) "Naive" visual learning (control)
70
Natural images (Moths) Dermoscopy images
0.8
Class-level
accuracy 1% correct]
65
Histology images
0.7
Precision
Recall
LWWISE
+
+
0.3
35
0.2
Training duration [min]
14
0.2
0.3
Control 0.7 0.8
Dermoscopy images
Histology images
4-way classification
2-way classification
Original images
Enhanced with pixel budget EmB
Difference (magnified 5x)
Difference map
benign male
melanome
benign keratosis
besal opl carcinoma
hyperplastic
sessilo serrafed
palyp
andenoma
Figure 4: Our approach can boost time efficiency and final accuracy image category learn- ing for humans across varied image domains, including in clinically relevant tasks. Panel A compares the mean test-phase accuracy and training phase duration of human participants who were random- ized to L-WISE control groups and learned moth photo, dermoscopy, histology classification task. All differences between L-WISE and the control group are statistically significant (x" (1) test, < 0.05). Panel B shows precision and recall L-WISE and control groups, with each point representing specific class in of the three tasks. All error bars show 95% bootstrap confidence intervals. Each class from the dermoscopy and histology tasks illustrated in panels C and respectively, similarly the moth classes Fig. 3A.
2014)) ) and Adobe Photoshop Lightroom's "Auto" enhancement feature (LR,, Adobe Inc. (024)) had significant effect on human accuracy, despite inducing image perturbations of considerably larger le norm on average than the le pixel budget values we used for model-based enhancement.
We also demonstrate that the robustified model's ground truth logit Lg((r) is strongly correlated with the rate at which humans choose the ground truth category associated with image x in a 16- way basic animal classification task (Fig. [TA). We used robustified ResNet-50 to calculate Lg for each of the 2,400 distinct natural images used in the task, and applied logistic regression to pre- diet binary correct vs. incorrect responses individual image trials. We pooled responses to original images with those to modified control-group images that were not enhanced by robust models, recomputing Lgt for each image version (see also Appendix Fig. SS). The logistic re- gression model (Fig. [ used Lg to predict the binary correctness of the trial responses with Area Under the Receiver Operating Characteristic Curve (AUC) 0.72 (10-fold cross-validation, < 0.001 via Wald statistic). Notably, we find that this simple approach predicts the difficulty of individual images for humans more accurately than existing state-of-the-an metrics (Mayo et al. 2023), see Appendix Fig. SS6). In addition to ResNet-50, we demonstrate both difficulty prediction and image enhancement with XCTT vision transformers (Ali et al.) (202T), Appendix Figs. S7.
4.2 WISE IMPROVES BOTH TEST ACCURACY AND LEARNING SPEED FOR HUMANS
We applied both image difficulty prediction and image enhancement as part of novel method that designs image sequence currieula for novices learning challenging image classification tasks.

---

Published as a conference paper at ICLR 2025
Idaea moth photos
Skin lesion dermoscopy images
Mean acc.
Training duration
Mean acc.
Training duration
Chance level
0.25
0.25
Control
0.47 (0.45, 0.50)
14.0 (13.8, 14.2)
0.38 (0.36, 0.40)
13.5 (13.4, 13.7)
ET
0,58* (0.55, 0,61)
11.8 (11.7, 11.9)
0.45* (0.43, 0.47)
13.1 (12.9, 13.3)
ET (shuffled)
0.53* (0.50, 0.56)
15.1 (14.8, 15.4)
0.39 (0.36, 0.42)
13.3 (13.2, 13.4)
DS
0.49 (0.47, 0.52)
13.9 (13.7, 14.1)
0.44* (0.42, 0,48)
11.5 (11.4, 11.6)
DS (shuffled)
0.58* (0.55, 0.60)
12.6 (12.5, 12.8)
0.45* (0.42, 0.48)
11.0 (10.9, 11.1)
L-WISE
0.60* (0.58, 0.64)
11.1 (11.0, 11.2)
0.47* (0.44, 0.50)
10.5 (10.4, 10.5)
Table 1: Both image enhancement tapering (ET) and image difficulty selection (DS) contribute to the ability of L-WISE to assist learners. The benefits of image enhancement are dependent on easy-to-hard sequencing ("ET" outperforms "ET (shuffled))), but the benefits of difficulty- selection appear to stem from simply showing an easier distribution of images during training ("DS (shuffled)" performs as well as or better than "DS"). Training durations are in minutes. Values in parentheses show 95% confidence intervals from 10,000 bootstrap replicates. denotes significant differences in accuracy from the control group (p < 0.01, (1) test).
Our proposed algorithm, Segit--eighted Image Selection and Enhancement" (L-WISE), operates on image trial sequences used to train human participants on image classification tasks through trial-by-trial feedback. The performance of each participant evaluated in subsequent testing phase without feedback, which includes only randomly-selected. unmodified images (unaffected by L-WISE). During the early portion of the training phase, L-WISE randomly selects images from below a certain difficulty percentile that stepwise-linent Selected images are enhanced at each trial during this period, via perturbations within an l2 pixel budget that decreases in a repwise-exponentia manner (Figs.) 2 R and 3Cd
increases as the training phase progresses.
We tested L-WIEE's efficacy at improving test-time accuracy and training duration of human learn- ers on three challenging image category learning tasks (Figs.
Participants were randomly assigned to a control group with randomly-selected, non-enhanced images throughout the task, or to an L-WISE group. L-WISE increased the average test-time accuracy margin above chance levels by 57.6% on a 4-way moth species classification task (p < 0.001 on x1(1) test), by 72.3% on a 4-way skin lesion dermoscopy task (p < 0.001), and by 33.1 1% (p- = 0.023) (Fig. [T) In all three tasks, participant accuracy in the L-WISH group increased ini- tially and then declined to varying degrees more and more difficult images were selected and the degree of enhancement was simultancously reduced. In addition to improving test-time accuracy, L-WISE decreased the mean time learn the task (with fixed number of training trials) by 20% for the moth task, 23% for the dermoscopy task, and 22% for the histology task (Fig. [A).
binary colon histology task
4.3 IMAGE ENHANCEMENT AND SELECTION BOTH CONTRIBUTE TO EFFICACY OF L-WISE
We tested several ablated versions of L-WISE (in the moth and dermoscopy tasks) to determine the relative contributions of its components: (A) image enhancement based on logit maximization, (B) selection of images according logit-estimated difficulty, and (C) easy-to-hard curriculum trends enabled by A and B (T. In "Fnhancement Tapering" (ET), only the enhancement component of L-WISE is active, with random image selection as in the control group. Conversely, "Diffi- culty Selection" (DS), images are selected based on difficulty but not enhanced. In "ET (shuffled)" and "DS (shuffled)," after applying ET or DS, the ordering of affected training trials is randomly permuted. Shuffling flattens easy-to-hane hard trends, isolating effects from (i) the mere presence of en- hanced images in ET, and (ii) seeing easier images on average in DS (DS limits max. difficulty of early images, so DS/DS (shuffled) have easier training images than Control on average; see Fig.
The results show that both ET and DS have significant benefits in isolation. ET increased the test- phase accuracy margin above chance by 46.8% for the moth task and 56.5% for the dermoscopy task, while DS increased the same margin by 8. (not significant) and 53.2% respectively. ET (shuffled) was less effective, increasing the margin above chance by 23.0% for the moth task and 11.2% (not significant) for the dermoscopy task. Surprisingly. DS (shuffled) outperformed DS without shuffling, increasing the margin above chance by 45.2% for the moth task and 58.2% for the dermoscopy task (the increase of DS (shuffled) relative to DS is statistically significant for the moth task only). Unab- lated L--ISE numerically outperformed all ablated conditions, increasing the margin above chance
8

---

Published as a conference paper at ICLR 2025
by 57.6% for the moth task and 72.3% in the dermoscopy task. However, additional paired com- parisons indicated that the differences between these increases and those from ET or DS (shuffled) not statistically significant for either task. ET and DS did demonstrate a statistically significant additive benefit in terms of learning speed, however. On the moth task, training duration was 6% shorter for L--ISE than for the next fastest group, which was ET. Similarly, on the dermoscopy task, training duration was 5% shorter for L-WISE than for the next fastest group, DS (shuffled).
5 DISCUSSION
In this study, we demonstrate that robustified ANNs can be used to both predict the empirical recog- nition difficulty of individual images for humans, and also generate enhanced versions of images that are easier for humans to correctly categorize. We harness these capabilities to develop a model- based curriculum design algorithm to augment human image category learning. We show that a combination of selecting images within a certain difficulty range, and perturbing those images to enhance the perception of the ground truth category, leads substantial improvements in human training speed and test-time classification accuracy on randomly-selected, unmodified images.
The results of our ablation study show that at least portion these improvements can be achieved with image enhancement alone: humans can learn from perturbed images and subsequently achieve superior generalization to unseen examples (Table [T. There are several possible explanations for this effect. Image enhancements might draw the learner's 's attention to relevant features, such as the distinctive dot in the middle of each wing of the Idaea biselata moth in Fig. 3A (Hufnagel 1767). or the irregular border and multiple colors that appear to be enhanced in the me image in Fig. 4C
(Tsa et al. 2015). Enhancements might also diminish features that distract from or contradict the ground truth: for example, in Appendix Fig. [STI (second image from the left), buffalo standing behind the ground truth "antelope" are variously blurred or nearly erased. Analogously, images with high ground truth logits (low predicted difficulty) might tend to have clearer class-relevant features and fewer distracting or contradictory features. These parallel explanations for the effects of image enhancement and image selection could help clarify why the "enhancement tapering" and "difficulty selection" strategies in isolation provide comparable accuracy gains to each other, and why combining both strategies did not lead to large additive improvements in accuracy (although additive increases in learning speed were observed).
5.1 LIMITATIONS
This work is a proof-of-concept demonstration that robustified ANNs can be applied to augment image category learning in humans. We did not exhaustively search for optimal curriculum design strategies or image enhancement hyperparameters, nor did we study the "dose-dependency" of im- age enhancement or selection. L-WISE applics a fixed schedule of maximal image difficulty and image enhancement magnitude for all learners: human learning could plausibly be augmented more effectively by adapting the degree of image enhancement and the difficulty of selected images to the leamer's 's progress in real time (Lu & Dosher 2022) Mettler & Kellman 2014).
One caveat to our approach is that logit maximization can sometimes appear to have a homoge- nizing effect on image distributions. For example, "benign mole" dermoscopy images enhanced with high- budgets tend to all resemble smooth and uniform blobs (see Appendix Fig. SS22 H for an example). This clearly illustrates task relevatt difference from melanoma (which tends to be asymmetric with irregular borders (Tsao et al.] 2015)), but obscures much of the real-world het- erogeneity among benign moles. similar risk might apply to image selection: images with high ground truth logits might belong to limited regions of the overall class distribution. Biased pertur- bations or selections reflecting biases in the underlying datasets are another concerning possibility, particularly for dermoscopy (Daneshjou et al. (022). These caveats must be thoroughly investigated before deployment of real-world educational applications of L-WISE, especially for clinical tasks with potential patient safety ramifications.
6 METHODS
Predicting Image Difficulty. To predict the relative difficulty d E [0, 1] of each image, we extract the logit value corresponding to the image's ground truth class (Lg) immediately upstream of the final softmax function. We sort the logits in descending order such that L() 2 L(2) 2 ... 2 L(p...) ne. is the number of images for a given class c and maining/validation/esstir split designation s. We calculate class-specific difficulty percentile d, for image j using the equation d, (ank(T)))//n.))
9

---

Published as a conference paper at ICLR 2025
Generating Perturbations to Enhance Images. To enhance image using pretrained ANN, maximize Lg through projected gradient ascent, onto a hypersphere of radius e, in pixel-space (see Appendix Section [ST for model training and projection details). In some cases, we also explicitly minimize the logit of competing classes. We generate perturbed image x' via the optimization:
x' = x + arg max (I + 5) |
Σ cEC:cfgi
Le(x + +0)
[|
In Equation 8 is perturbation tensor of the same dimensionality as x and with an Eg norm less than pixel budget c. Lgg is the ANN's logit score associated with the ground truth class, and Le is the logit associated with class c (C is the set of all classes, with cardinality IC]). determines the extent to which logits for competing classes are minimized. We set a = 0 for the ImageNet animal classification experiments and a 1 for the three fine-grained image category learning experiments.
Image classification and learning experiments with human participants. We recruited 521 human subjects using the online platform Prolific. We allowed subjects to participate in multiple experi- ments, but only once for each of the three learning tasks. We used the jsPsych library (De Leeuw, 2015) with the jspsych-poy peechoh-ssycoophsiies plugin (Kuroki, 202T) for all experiments. All experiments included 10-12.5% attention check trials where the subject classifies an image of circle or triangle (e.g., see Appendix Fig. S4p. We analyzed data from subjects with 2 90% attention check accuracy.
To measure the effects of enhancement on presumably already-learned task, we tested the accu- racy of human subjects at classifying 16 basic types of animals (frog, bird, dog, etc., see Appendix Section (3). Images were shown for 17 milliseconds each, after which the participant was given 15 seconds to respond. All images were drawn from the validation set of ImageNet (Deng et al., 2009).. Appendix Fig. S3 shows screenshots of the task it appeared to participants. In our main exper- iment with this task (Fig. []. each subject viewed interspersed images from 9 conditions: original images, images enhanced by maximizing Lg with 5, 10, 15, and 20, images enhanced with one of three off-the-shelf baseline methods, and images disrupted by minimizing Lg (internal control). Participants were notified after each incorrect response and given small monetary bonus for each correct response (except for disrupted images). 62 participants viewed 18 images for each of 16 classes, 2 from each condition, in a shuffled ordering for a total of 288 trials per participant and 17,856 overall. These main trials followed screening phase with 32 trials (200ms presentation times, 2 24 correct with 2 correct per class required to proceed, multiple screening attempts al- lowed) and a 32-trial warm-up phase with 17ms image presentations. Screening and warm-up phases used original, unmodified images, and data from these phases were not included in any analyses.
The image category learning tasks consisted of either 4 (moths, dermoscopy) or 2 (histology) image classes. Participants were shown each image for up to 10 seconds and used the mouse (4-way tasks) or keyboard (binary task, "F" and "J" keys) to respond. After each trial, the participant was notified of the ground truth label and whether their response was correct (scrcenshots in Appendix Fig. SS4) Each session consisted of 8 training blocks of 16 trials each (4 per class, or 8 per class for histology). and 2 testing blocks of 20 trials each during which no post-trial feedback was provided. Each block contained an equal number of images from each class in random order. Participants were informed upon recruitment that they could receive a progressively higher monetary bonus if their test-phase accuracy exceeded certain thresholds. Before participating in the main learning tasks, subjects had to first learn an easier binary classification task (leatherback vs. loggerhead turtles) and respond correctly to at least 7 of 8 test-phase trials. We randomly assigned -30 participants per experimental condition, except the ablation study control groups of -60 participants (overall min. 27, max. 68).
Assisting learners with the L-WISE algorithm. L-WISE consists of two strategies applied in par- allel: Enhancement Tapering (ET) and Difficulty Selection (DS). Both strategies operate only on images in the first 6 of 8 trial blocks in the training phase. In ET, we enhanced images in the first block of training- -hane trials with e 8. is halved for each subsequent block until it is set to 0 after the 6th block. In DS, only images with d < de were sampled for each block b. de was incremented by 0.15 at the end of each of the first six blocks, beginning at d1 0.1 and reaching d7, ds = 1.0. Determining an effective schedule of and d, did not require extensive hyperparameter tuning. After pilot experiment in which we decreased e linearly starting from 20, (see Appendix Fig. SS12 4- M), we switched to the schedule above and changed no other hyperparameters for any of the three learning tasks/image domains. In the "shuffled" versions of DS and ET (Section 4.3). images in blocks 1-6 are selected or enhanced before a constrained shuffling procedure, whereby each image in blocks 1-6 may switch positions with any other of the same class regardless of d or c.
10

---

Published as a conference paper at ICLR 2025
7 ETHICS STATEMENT
This study involved experiments with human participants conducted over the internet, using the Prolific platform for the main experiments and Amazon Mechanical Turk for pilot experiments. We followed a study protocol approved by Boston Children's 's Hospital's Institutional Review Board. Participants provided informed consent before participating any experiments. The experiments posed no greater than minimal risk to the participants. All participants were anonymous, and all data is de-identified. Participants were provided with our contact information, and that of the Office of Clinical Investigations at Boston Children's 's Hospital, for any questions or concerns about the study. We calibrated the participant compensation amounts for each experiment to meet or exceed the equivalent of $ $15.00 USD per hour, including during screening tasks. Participants were recruited using the "Standard Sample" option in Prolific, and were diverse in gender, age, and race/ethnicity (please see Table S2 for demographic breakdown).
We hope that our work will eventually lead to practical and beneficial applications in education for example, in the training of doctors in specialties such pathology, radiology and dermatology where visual perceptual learning is particularly important. We wish to emphasize, however, that more work is needed before our methods can be safely applied in sensitive or high-stakes settings. For example, we apply our approach to improve human performance on a dermoscopy skin lesion classification task derived from the HAM10000 dataset (Tschandl et al., 2018). This dataset is heavily skewed towards images of pale skin, likely a reflection of the lower incidence of skin cancers such as melanoma among people with darker skin tones (Cormier et al. 2006). Models trained on this dataset are known to perform poorly for patients with darker skin (Daneshjou et al. 20222,, where melanoma tends to have a different appearance, unfamiliarity with which on the part of clinicians contributes to delayed diagnosis and increased mortality among such patients (Thompson et al.) 2023). It is plausible that maximizing the aallanoma-associaae logit a robustified model perturbs the images to look more like average presentation of melanoma (i.c., on light skin), which would risk imparting this bias onto the learner. A similar risk might apply to image selection: images with the highest ground truth logits (which L-WISE presents the beginning of learning) might tend to belong to specific subclasses or limited regions of the overall class distribution. The possibility of biased perturbations or image selections must be thoroughly investigated in future work before applications of our method in this domain.
8 REPRODUCIBILITY STATEMENT
We list all hyperparameters for training robustified neural networks and using them to enhance im- ages in the Appendix. We provide source code that enables reproduction of all data processing steps, experiments, statistical analyses, and figure plots. This includes experiments with human par- ticipants: we have developed a flexible framework for automated deployment of web-based image category learning experiments, including hosting tasks interactive web pages and using cloud- based mechanisms for random group assignment and data collection. The experiments with hu- man participants can therefore be readily reproduced and extended or modified with only modest amount of configuration required.
ACKNOWLEDGMENTT
This work was supported in part by Harvard Medical School under the Dean's Innovation Award for the Use of Artificial Intelligence, in part by Massachusetts Institute of Technology through the David and Beatrice Yamron Fellowship, in part by the National Institute of General Medical Sciences under Award T32GM 144273, in part by the National Institutes of Health under Grant RO1 EY026025, and in part by the National Science Foundation under Grant CCF-1231216. The content solely the responsibility of the authors and does not necessarily represent the official views of any of the above organizations. The authors would like thank Yousif Kashef Al-Ghetaa, Andrei Barbu, Pavlo Bulanchuk, Roy Ganz, Katherine Harvey, Michael J. Lee, Richard N. Mitchell, and Luke Rosedahl for sharing their helpful insights into our work at various times.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Adobe Inc. Adobe Photoshop Lightroom Classic, 2024. URL https:/ www. adobe. com/
products/ Sheeluttssppottttoop---ooooooooo clw.-illslooooomosssocccocoooo.e html
Version 13.5.1.
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob J. Verbeek, and Herve Jegou. XCIT: Cross-covariance image transformers. In Advances in Neural Information Processing Systems, volume 34, pp. 20014-20027, 2021.
Saced Anwar, Salman Khan, and Nick Barnes. A deep journey into super-resolution: A survey. ACM Computing Surveys (CSUR), 53(3):1-34, 2020.
2020.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of ex- ample difficulty. In Advances in Neural Information Processing Systems, volume 34, pp. 10876- 10889, 2021.
Philipp Bomatter, Mengmi Zhang, Dimitar Karev, Spandan Madan, Claire Tseng, and Gabriel Kreiman. When pigs fly: contextual reasoning in synthetic and natural scenes. In International Conference on Computer Vision (ICCV), pp. 255-264, 2021.
Dongliang Chang, Kaiyue Pang, Ruoyi Du, Yujun Tong, Yi-Zhe Song, Zhanyu Ma, and Jun Guo. Making a bird AI expert work for you and me. IEEE Transactions on Pattern Analysis and Machine Intelligence,
2023.
Joseph A. Chapman and John F. Flux. Introduction to the Lagomorpha. In Lagomorph Biology: Evolution, Ecology, and Conservation, pp. 1-9. Springer, 2008
Minkyu Choi, Kuan Han, Xiaokai Wang. Yizhen Zhang, and Zhongming Liu. A dual-stream neural network explains the functional segregation of dorsal and ventral visual pathways in human brains. In Advances Neural Information Processing Systems, volume 36, pp. 50408- 2023.
Janice N. Cormier, Yan Xing, Meichun Ding, Jeffrey E. Lee, Paul F. Mansfield, Jeffrey E. Gershen- wald, Merrick I. Ross, and Xianglin L., Du Ethnic differences among patients with cutaneous melanoma. Archives of Internal Medicine, 166(17):1907-999) 2006.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning. 2020.
Roxana Daneshjou, Kailas Vodrahalli, Roberto A. Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M. Swetter, Elizabeth E. Bailey, Olivier Gevaert, Pritam Mukher- jee, Michelle Phung, Kiana Yekrang, Bradley Fong, Rachna Sahasrabudhe, Johan A. C. Allerup, Utako Okata-Karigan James Zou, and Albert S. Chiou. Disparities in dermatology Al perfor- mance diverse, curated clinical image set. Science Advances, seabb66444 2022.
Joshua R. De Leeuw. jsPsych: A Javascript library for creating behavioral experiments in web browser. Behavior Research Methods, 47:1-12, 2015.
Edoardo Debenedetti, Vikash Sehwag, and Prateek Mittal. A light recipe to train robust vision transformers. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 225-253. IEEE, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai and Li Fei-Fei. ImageNet: A large-scale hier- archical image database. In 2009 IEFE Conference Computer Vision and Puttern Recognition, pp. 255. IEFE, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (Python library), 2019. URL https: http:///hhbb..ccm///aaaaaaaaaaaa http://aarryyla//robbtssneess
12

---

Published as a conference paper at ICLR 2025
Thomas Fel, Ivan Felipe, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. In Advances in Neural Information Processing 2022.
Systems, volume 35, pp.
luri Frosio and Jan Kautz. The best defense good offense: adversarial augmentation against adversarial attacks. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Patlern Recognition, pp. 4067-4076, 2023.
Roy Ganz and Michael Elad. CLIPAG: Towards generator-free text-to-image generation. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3843- 3853, 2024.
Roy Ganz, Bahjat Kawar, and Michael Elad. Do perceptually aligned gradients imply robustness? In International Conference on Machine Learning, pp. 10628-10649 PMLR, 2023.
Guy Gaziv, Michael Lee, and James J. DiCarlo. Strong and precise modulation of human percepts via robustified ANNs. Advances in Neural Information Processing Systems, volume 36, 2023.
Chong Guo, Michael J Lee, Guillaume Leclere, Joel Dapello, Yug Rao, Aleksander Mądry, and James J. Dicarlo. Adversarially trained neural representations already as robust biological neural representations. In International Conference on Machine Learning, 2022.
Shashi Kant Gupta, Mengmi Zhang, Chia-Chien Wu, Jeremy Wolfe, and Gabriel Kreiman. Visual search asymmetry: deep nets and humans share similar inherent biases, In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 6946-6959, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Johann Siegfried Hufnagel. Tabelle von den Nachtvögeln. Berlinisches Magazin, 1(4)618, 1767. http::://///soolleee sub. wnw htwp-pppeettiieeedddd//r//re11ea
0_000
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural reg- ularities of labeled data in overparameterized models. In Marina Meila and Tong Zhang (eds.), the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pp. 5034-5044, PMLR, July 2021.
Proceedings the
Daniel J. Jobson, Zia-ur Rahman, and Glenn A. Woodell. A multiscale Retinex for bridging the gap between color images and the human observation of scenes, IEEE Transactions on Image Processing, 677):965-976, 1997.
Edward Johns, Oisin Mac Aodha, and Gabriel J. Brostow. Becoming the class machine teaching. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2616-2624, 2015.
multi-
Juyeop Kim, Jun-Ho Choi, Soobeom Jang, and Jong-Seok Lee. Amicable aid: Perturbing images to improve classification performance. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEF, 2023.
Daiichiro Kuroki. A new jsPsych plugin for psychophysics, providing accurate display duration and stimulus onset asynchrony. Behavior Research Methods, 53:301-310, 2021.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B. Smith, James M. Rehg, and Le Song. Iterative machine teaching. In International Conference on Machine Learning, pp. 2159-2158. PMLR, 2017.
Zhong-Lin Lu and Barbara Anne Dosher. Current directions visual perceptual learning. Nature Reviews Psychology, (11)):544666) 2022.
13

---

Published as a conference paper at ICLR 2025
Oisin Mac Aodha, Shihan Su, Yuxin Chen, Pietro Perona, and Yisong Yue. Teaching categories to human learners with visual explanations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3820-3828, 2018.
Aleksander Mędry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
David Mayo, Jesse Cummings, Xinyu Lin, Dan Gutfreund, Boris Katz, and Andrei Barbu. How hard are computer vision datasets? Calibrating dataset difficulty to viewing time. In Advances in Neural Information Processing Systems, volume 36, pp. 11008-11036, 2023.
Everett Mettler and Philip Kellman. Adaptive response-tima hime-based category sequencing in per- ceptual learning. Vision Research, 99:1 10:111-10 11-123, 2014.
Ana Belén Petro, Catalina Sbert, and Jean-Michel Morel. Multiscale Retinex. Image Processing On Line, pp. 71-88, 2014.
Yunliang Qi, Zhen Yang, Wenhao Sun, Meng Lou, Jing Lian, Wenwei Zhao, Xiangyu Deng, and Yide Ma. comprehensive overview of image enhancement techniques. Archives of Computa- tional Methods in Engineering, pp. 1-25, 2021.
Zeju Qiu, Weiyang Liu, Tim Z. Xiao, Zhen Liu, Umang Bhatt, Yucen Luo, Adrian Weller, and Bernhard Schölkopf. Iterative teaching by data hallucination. In Francisco Ruiz, Jennifer Dy, and Jan- Willem van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 9892- 9913. PMLR, April 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision,
115:211-252, 2015.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Sai Vemprala, Aleksander Mądry, and Ashish Kapoor. Unadversarial examples: Designing objects for robust vision. In Advances in Neural Information Processing Systems, volume 34, pp. 15270-15284, 2021.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Roy, Franziska Geiger, Kailyn Schmidt, Daniel I.. K. Yamins, and James DiCarlo. Brain -core:: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2018.
Adish Singla, Iija Bogunovic, Gábor Bartók, Amin Karbasi, and Andreas Krause. Near-optimally teaching the crowd to classify, In International Conference on Machine Learning, pp. 154-162.
PMLR, 2014.
Brandon Thompson, Toni Jenkins, John Paul Sánchez, Matthew Frederick, Alba Posligua-Alue Alban, and Naiara Sbroggio Barbosa. Melanoma: Does it present differently in darker skin tones? Med- EdPORTAL, 19:1 1911, 2023.
Hensin Tsao, Jeannette M. Olazagasti, Kelly M. Cordoro, Jerry D. Brewer, Susan C. Taylor, Jeremy S. Bordeaux, Mary-Margaret Chren, Arthur Sober, Comnie Tegeler, Reva Bhushan, and Wendy Smith Begolka. Early detection melanoma: reviewing the ABCDEs. Journal of the American Academy of Dermatology, 72(4):717-723, 2015.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, large collection of muuti-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5(1): 1-9, 2018.
Jamalbek Tussupov, Kairat Kozhabai, Aigulim Bayegizova, Leila Kassenova, Zhanat Manbetova, Natalya Glazyrina, Mukhamedi Bersugir, and Miras Yeginbayev. Applying machine learning to improve a texture type image. Eastern- Eurooean Journal of Enterprise Technologies, 122(2), 2023.
14

---

Published as a conference paper at ICLR 2025
Grant Van Hom and Oisin Mac Aodha. iNat Challenge 2021 FGVC8, 2021. URL https:
/kagglee... wwtp:///mar..titoooaaaaaaaaaaaa.
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12884-12893, 2021.
Pei Wang, Kabir Nagrecha, and Nuno Vasconcelos. Gradient-based algorithms for machine teach- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1387-1396, 2021a.
Xin Wang. Yudong Chen, and Wenwu Zhu. A survey curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555-4577 2021
Jerry Wei, Arief Suriawinata, Bing Ren, Xiaoying Liu, Mikhail Lisovsky, Louis Vaickus, Charles Brown, Michael Baker, Naofumi Tomita, Lorenzo Torresani, Jason Wei, and Saced Hassanpour. A petri dish for histopathology image analysis. In Artificial Intelligence in Medicine: 19th In- ternational Conference on Artificial Intelligence Medicine, AIME 2021, Virtual Event, June 15-18, 2021, Proceedings, pp. 11 1--24. Springer, 2021.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy to train strong classifiers with localizable features. Proceed- ings of the IEEE/CVF International Conference Computer Vision, pp. 0023-6032, 2019.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. In Advances in Neural Information Processing Systems, volume 33, pp. 7559-7570, 2020.
Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Karel Zuiderveld. Contrast limited adaptive histogram equalization. In Graphics Gems IV, pp- 474-485. Academic Press, 1994.
15

---

Published as a conference paper at ICLR 2025
Appendix
Monkey
Lizard
Original
Enhanced with pixel budget
Difference Difference heat Imagnified 5x)
Figure S1: Ground truth logit enhancement with robustified ANNs leads to semantically mean- ingful perturbations. The top row shows original ImageNet images, and the second row shows the same images after enhancement by robustified ResNet-50 (training 3) with pixel budget of 20. The third row shows a 5x magnified version of the difference between the enhanced image and the original, and the bot- row shows heat map where red regions correspond larger changes and blue regions correspond smaller changes.
SI DETAILS ON TRAINING AND USING ROBUSTIFIED GUIDE MODELS
We udversarially trained a ResNet-50 model on ImageNet (Deng et al. 2009). and another on iNauu- ralist 2021 (Van Horn & Mac Aodhal 2021). with the hyperparameters following Gaziv (2023):
Epochs 200
Base learning rate 0.1, decreasing by factor of 10 every 50 epochs
Batch size 256
Weight decay = 0.0001
Adversarial training = 3.0 (ImageNet) or e = 1.0 (INaturalist)
gradient steps for adversarial attacks
Adversarial attack step size of 0.5 (ImageNet) 0.3 (iNaturalist)
The ResNet-50 model adversarially trained on ImageNet was used directly to generate perturba- tions and difficulty rankings for the 16-way animal classification task, using the logits of the orig- inal, fine-grained ImageNet classes (i.e., not the 16 superclasses, "grasshopper" not "insect"" for both enhancement and difficulty prediction. The same model was adversarially fine-tuned on the HAM10000 and MHIST datasets before their application (as part of L--WISE) to the dermoscopy and histology tasks respectively. Generally, For example, we fine-tuned on all classes of the HAM10000 dermoscopy dataset (Tschandl et al.) 2018), even though only used 4 of them in the learning task for humans. When enhancing the images, we include only classes that are part of the experimental tasks as competing classes to have their logits minimized (see main-text Equation [
trained the models all available classes in each dataset.
For the moth task, we adversarially fine-tuned the (adversarially) iNaturalist pretriine- model on the four moth classes to be used in the task. We subjectively judged the perturbations from this
16

---

Published as a conference paper at ICLR 2025
fine-tuned model to be of higher quality than those generated using the Naturaliitt-reerainee model without fine-tuning, as the four classes of interest are among the 10,000 iNaturalist classes. All adversarial fine-tuning used 0.0001, 0.0004, and 0.001, and batch sizes of 32, 64, and 16, for Idaea moth, dermoscopy, and histology fine-tuning respectively. We fine-tuned the entire network end-to-end for each task.
1.0 with 7 gradient steps of size 0.3. We used learning rates of
Our choice of = 3 for ImageNet pretraining follows Gaziv et al. (2023). who found this to be an optimal choice for generating perturbations that disrupt category perception (relative to e and e = 10). In practice, we found that the models were unable to lear finer-grained tasks with training- time adversarial perturbations as large as = (iNaturalist pretraining, and fine-tuning on moth photos, dermoscopy images, and histology images) therefore, we reverted to e = for these settings.
To enhance images in a category speccffcc manner, we perform the optimization of Equation (main text) in a series of steps using projected gradient ascent (Equation 2. where & denotes the optimiza- tion step, 7 the step size, and Proj. a projection onto hypersphere of radius with original image at its center (see text below Equation I for definitions of other symbols).
dL+1 = -rojj (xx - 1/ VV + 6k) |cI-1
Σ CEO:efgt
Le(x + 6x))
Fig. ST shows several example images enhanced with = 20 using logit maximization by udversarially-pretriii ResNet-50 (e 3), along with difference images and heat maps produced by the same method as Figs. 3 in the main text. Throughout our experiments, robustified ResNet- 50 enhancements with pixel budget e used ceil(2e) steps of 1) = 0.5 in a 224 x 224 x pixel space. This formula seems somewhat model-dependent and sometimes requires adjustment: for example, when enhancing images with robustified XCIT (see Figs. 59 S10). ceil(4e) steps were required in- stead of cei to reach a similar effective perturbation size.
We generate the heat maps Figs. 3 and S1 by subtracting the enhanced image from the original image element-wise: 5
x. We calculate the magnitude of the changes as 8². We apply smoothing using 2D convolution, and normalize the result to have all values between and 1, to produce daoe- We then produce the heat map by setting the red channel to 255 x 6neem and the blue channel to 255 x (1 5num). The resulting image shows red in regions where larger changes have taken place, and blue in regions where smaller or no changes have taken place. In Figs. 3 4 ST] we superimpose a translucent version of the heat maps (a = 0.7) over the original images. Averaging the heat maps across all ImageNet validation
images (Fig. S2) indicates that changes to the images tend to occur in the central regions of the images more than in the periphery, consistent with the observation that our image enhancement approach tends primarily change image regions corresponding to the main subject of each image.
S2 DETAILS ON IMAGE PREPARATION FOR EXPERIMENTS
All images presented in all psychophysics experiments were of size 224 x 224 x 3, matching the input dimensions of 50s Before presentation or any model-based enhancement or difficulty prediction, original images were resized such that the shortest dimension (width or height) was 224 pixels, and then center-cropped to 224 x 224. Any single-channel grayscale images were converted to RGB before further processing. The baseline enhancement algorithms Contrast-Limited Adaptive Histogram Equalization (CLAHE (Zuiderveld [194]), Multi-Scale Retinex with Color Restoration (MSRCR (Jobson et al., 1997: Petro et al. 20144). and the "Auto" image tuning feature in Adobe Photoshop Lightroom (Auto-LR (Adobe Inc. 2024 were applied before the resizing and center- cropping operations.
We generally used images from the validation sets of each dataset for the image category learning experiments, reasoning that the robustified models would be overfitted to training images which could potentially compromise the quality of perturbations and relative difficulty estimates. How- ever, for the moth task, we were limited to 10 validation images per class in the iNaturalist dataset (Van Horn & Mac Aodhal 2021). In this case we used training set images during the training period of the human image category learning experiment and validation images during the test phase. We show that image enhancements are still effective for training set images in Fig. SIIA.
17

---

Published as a conference paper at ICLR 2025
1.0
Normalized mean pixel-value difference
-0.6
-0.2
0.0
Figure S2: Pixel-value changes during enhancement of ImageNet images are biased towards the center of the image. This heat map indicates which spatial regions of the ImageNet animal images were changed the most on average during logit- maximizaiiiiiion enhancement with c 20. Regions that were changed red, and regions that changed less average was generated hy averaging the normalized absolute pixel value changes across all 2400 ImageNet validation images that we used for 16-way animal classification experiments.
averuge
blue. The heat map
S3 DETAILS ON 16-WAY IMAGENET ANIMAL CATEGORIZATION EXPERIMENTS
We curated 16 sets of ImageNet classes corresponding to 16 basic animal superclasses for our basic animal classification experiments (e.g.. g.. see Figs. D and [S3). adapting and expanding the Restricted ImageNet dataset defined in the Robustness library (Engstrom et al. 2019).. The assignment of specific animal classes each superclass listed below:
Dog: classes 151-268
House Cat: classes 281-285
Frog: classes 30-32
Turtle: classes 33-37
Bird: classes 80-100 and 127-146
Monkey: classes 369-382
Fish: classes 0, 389, 391, 392, 393, 394, 395, 396, 397
Crab: classes 118-121
Insect: classes 300-320
Lizard: classes 38-48
Snake: classes 52-68
Spider: classes 72-77
Big Cat: classes 286-293
Bear: classes 294-297
Rodent: classes 330, 331, 332, 333, 335, 336, 338
Antelope: classes 351-353
We excluded certain classes on a case-by-case basis in an attempt to minimize errors due to misun- opposed errors of visual perception. For example, did
derstanding the animal categories, not include porcupines or beavers in the "rodent" class (as many people may not recognize these rodents), and we did not include eels in the "fish" class due to the possibility of confusion with
18

---

Published as a conference paper at ICLR 2025
A.
B.
C.
D.
X
Figure S3: Task interface for ImageNet animal classification with human participants. Subjects classified images among 16 categories. During each trial, the subject clicks the fixation cross (panel A) and the image displayed for 17 milliseconds (panel B) with 200ms presentation of a blank screen immediately before and after. The mouse cursor is hidden during the image presentation. Images presented such that they subtend approximately 6 degrees of visual angle, with calibration for each participant using a blind spot calibration procedure. After viewing the image, the participant clicks one the 16 buttons shown in panel C, which are randomly rotated in position every trial, within a 15-second time limit. For incorrect responses, if 15 seconds elapses without response, the participant is shown the black X for second (panel D). Otherwise, no explicit feedback is given and the next trial begins immediately. Artention check trials featuring an image of a circle triangle (see Fig. [S40 for an example image) were interspersed with the main trials. For the attention check trials, two of the animal icons in panel C were randomly selected to be replaced with circle and triangle icons.
snakes. We mistakenly classified rabbits and hares as "rodents" given that they were reclassified to the order Lagomorpha in 1912 Chappmn Flux (2008) (we thank the participant who notified us of this).
S4 DETAILS ON IMAGE CATEGORY LEARNING EXPERIMENTS
$3 and S4 show the task interfaces for the 16 16-way animal classification and 4-way image cat- egory learning experiments with human participants, respectively. For the learning experiments, the positions of the four buttons used to indicate responses randomly permuted for each participant.
Figs.
To minimize biases stemming from any priors induced by the class names, for each participant the 4-category learning tasks randomly assign four-letter, two-syllable alias from the set "Ajax," "Eris," "Leda," and "Tyro." These names are drawn from Greek mythology. and each has four letters, two syllables, two consonants, and two phonetic vowels. We found no evidence that associating
19

---

Published as a conference paper at ICLR 2025
A.
B.
X
Ajax
Leda
Tyro
C.
D.
Ajax
Tyro
Tyro
Figure S4: Task interface for image category learning experiments. the 4-way image category learning experiments (moths, dermoscopy), human subjects learned classify four rypes of images that were represented by randomly assigned aliases "Ajax," "Eris," "Leda," " and "Tyro." The image was shown for up 10 seconds, during which the participant could click shown a black X (1.5 seconds) immediately following an incorrect response or a > 10s timeout (panel B), or green check after correct response (panel C). The alias corresponding the correct class is also displayed on the feedbuck screen. Panel shows an example of an attention check trial.
of the four buttons (panel A). Participants
certain categories with certain aliases consistently affected test-phase accuracy (see Figs. S15 and S16
We used a different approach for the binary histology task that employed the MHIST dataset (Wei al. 2021). giving "benign hyperplastic polyp" the alias "benign" and sessile serrated adenoma the alias "malignant" (although sessile serrated adenoma is actually a pre-cancerous lesion). The histology task has an interface very similar in appearance that of the 4-way tasks (as shown in Fig. (S4), except that the "benign" and "malignant" buttons always appear either side of the presented image (in random order for each participant), and the participant responds by pressing the F key for the left-hand category or J for the right instead of clicking one of the buttons.
S5 PREDICTING IMAGE DIFFICULTY USING GROUND TRUTH LOGIT OF A ROBUST MODEL, COMPARED WITH PRIOR STATE-OF-THE-ART APPROACHES
In the L-WISE algorithm, we predict the difficulty of each image using its ground truth logit repre- sentation (Lg)) from robustified ANN such as ResNet-50 (see Figs. TA, S5 S7 S1d.1.A,,),,C) We conducted an experiment to compare this ground truth logit score with prior ofate-of-the-an predictors of image difficulty for humans established by Mayo al. (0033.. We apply logistic
20

---

Published as a conference paper at ICLR 2025
A. 1.0
104
B. 1.0
2395
1060
384
M
+
850
153
Rate humans choose
ground truth [%6]
3133
3079
0.8
1007
HH
1000
AUC 0.719
AUC = 0.711
2552
p < 0.001
838
p 0.001
1813
0.6
374
n=132
478
126
0.4
"hard" "easy"
"hard" "easy"
0.2
0.2
10
20
20 Ground truth logit value [a.u.]
Ground truth logit value [a.u.]
Figure S5: The observed relationship between robust model ground truth logits and human error rates is not sensitive to trial inclusion criteria. For our main analysis regarding predicting ImageNet animal image recognition difficulty illustrated Fig. LA we included all trials with original images but also trials featuring images modified by off-the-shelf control enhancement methods (Adobe Lightroom, CLAHE, and MSRCR), and trials with image perturbations from non pon--dversarrally-rriinee ANNs (ie.. Vanilla ResNet-50 and CutMix ResNet-50 from Fig. S7. Panel A above reproduces Fig. ZAI, while panel B replicates the same analysis but strictly including only trials with natural, unmodified images. The numbers above each point indicate how many image trial observations were included in the corresponding bin (the size of the vertical 95% confidence intervals is sensitive to this), AUC=Area Under the Receiver Operating Characteristic Curve, p-values derived from Wald stotistic on the logistic regression coefficient for the ground truth logit predictor.
gression to predict correct v.s. incorrect responses to each (original, unmodified) image across all participants in our 16-way ImageNet animal classification experiment, using (1) c-score (approxi- mated by the epoch during training at which an image is first correctly predicted (Jiang et al. (22T)), ), (2) prediction depth (earliest layer upon which linear probe makes the same prediction the final output (Baldock et al. () (3) image-level adversarial robustness (minimum magnitude of image perturbation required to change the network's 's prediction), and (4) ground truth logit from both (A) vanilla and (B) robustified ResNet-50 models (see Fig. S6).. C-score, prediction depth, and adversar- ial robustness are implemented following Mayo al. (023). The results show that the ground truth logit from robust ResNet-50 (Lg). the metric we use in WISE, significantly outperforms all other predictors of image difficulty, including all other metries combined into one model ("Combined w/o I.g in Fig. (6).. Furthermore, combining all other metrics with L. does not improve performance beyond g.g alone. We also find that using a robustified model rather than "vanilla" model to gener- ate each metric greatly improves the predictivity of Lgt and (marginally) adversarial robustness, but not of the c-score or prediction depth.
S6 COMPARISON OF DIFFERENT ANN GUIDE MODELS FOR DIFFICULTY PREDICTION AND IMAGE ENHANCEMENT
Our main results use robustified ResNet-50 as a guide model for generating image perturbations. To evaluate the importance of the choice of guide model, we compared the accuracy of difficulty prediction (Fig. S7p and the effects of enhancement with = 20 (Fig. S9) using different guide models in the 16-way animal classification task. The results show that setting e value of during adversarial training of ResNet-50 yields more accurate difficulty predictions and more effective per- turbations than < or e = 10 training (consistent with disruption modulation results in Gaziv et al.) 2223)), while perturbations guided by
"vanilla" model have negligible effects. Model accuracy seems to be less important than robustness, at least for difficulty prediction: ground truth logits from ResNet-50 models at early epochs of adversarial training on ImageNet, which have lower image classification accuracy than models at later epochs, can still yield accurate image difficulty predictions (Fig. (S8A). Models at early training epochs may also be able to gen- erate high-quality image enhancement perturbations (Fig. (S8B), although we did not study this in experiments with human participants. Although training ResNet-50 with CutMix improves its
21

---

Published as a conference paper at ICLR 2025
Verilla
AUC (predict human correct not)
0.70
0.85
0.50
0.99
C-Score
Pud. Depii
Ade Robasthids
Contined welo Lug
All features
C-Scrre
Predi Depii
Ads Robustees
All fnstures
Comband wib
Feature Sets for Logistic Regression
Figure S6: Robustified ground truth logit is state-of-the-an predictor of image difficulty for humans, outperforming the c-seore, prediction depth, and adversarial epsilon of both vanilla and robustified models. AUC estimates based fitted logistic regression models using features listed under each bar, with stratified 500-fold cross- validation. Error bars are 95% confidence intervals for the mean from 10,000 bootstrap replicates. The chance level AUC=0.5.
0.75
Non-adversarially trained models
AUC (predict human correct or not)
Adversarially trained models
0.70
0.65
0.60
0.55
0.50
Vanilla RNSO
Cutblix RNSO
c= RN50
<= RNSO
E = RN50
Models
Figure S7: Accuracy of image difficulty prediction using ground truth logits from different model types. AUC estimates and 95% confidence interval error bars are generated by the same procedure as in Fig. [6 however, results are not directly comparable berween the two figures as different ANN training runs were used for consistency within each experiment. RN50 RexNet-50, and values in the labels for each show the magnitude of the adversarial perturbations during udversarial training.
robustness to adversarial perturbations Yun et al. 2019), CutMix-ReNNet. does not outperform vanilla ResNet-50 in image difficulty prediction and perturbations using it as guide model do not significantly increase accuracy beyond that on original images. In addition to ResNet-50, we tested the difficulty prediction and image enhancement capabilities of an adversarially trained vision trans- former model, the Pross-Covariance Image Transformer (XCIT) (Ali et al. 2021). Debenedetti et al. (223) showed that the XCTT architecture is more suitable for adversarial training than the original vision transformer. XCIT generates reasonably accurate image difficulty predictions (on par with the previous state-of-the-ane and generates image perturbations that increase human categorization
22

---

batch size Published as a conference paper at ICLR 2025
0.75
-1.0
A.
ImageNet classification accuracy
AUC (predict human error rate)
0.70
0.65
0.60
0.4
0.55
Human error rate prediction (AUC)
0.2
ImageNet training accuracy
ImageNet validation accuracy
0.50
10
20
30
40
50
60
70
80
89
Training epoch
B.
Original
Enhance after
Enhance after 10 epochs
Enhance after 90 epochs
Figure S8: Robustified model accuracy weakly affects the relationship between ground truth logit and human image difficulty. To generate this figure, we retained ResNet- model checkpoints from immediately after each of 90 epochs of adversarial training on ImageNet-IK from scratch 256, initial learning rate of 0.1 decreases by half every epochs). In panel A, AUC estimates of logistic regression models predicting human trial response correctness were generated by the same procedure as using ground truth logits generated by each of the epoch checkpoints. The shaded area denotes 95% confidence intervals around each mean AUC from 10,000 buotstrap replicatex. The training set and validation set accuracy at 1000-way ImageNet-Ie IK classification, by training epoch. We observe that AUC of human ervor rate prediction stops increasing relatively early during training, well before the training/validation accuracy the ImageNer classification task is saturated. Panel shows example images training process.
Appendix Figs. 56 S7
perturbed images, is also plotted
20 enhancements generated by checkpoints at various stages of
accuracy by comparable degree to robustified ResNet-50, For the experiments in Figs. S7 and S9
we used pretrained guide models provided by Gaziv al. (2023) (Vanilla, < = 1, 3, and
ResNet-50 models), Yun et al. (2019) (CutMix ResNet-50), and Debenedetti et al. (2023) (e
XCIT). Examples images enhanced by each of these guide models with 20 are displayed in
Fig. STO
Image perturbations generated with vision transformer models (such as XCIT) typically include grid- like artifacts related to the image patch/grid structure of these models (Dosovitskiy et al. 2020). To mitigate grid artifacts during each step of generating image perturbations with XCYT, we calculated gradients with respect each pixel value by averaging across randomly translated, resized (fol-
23

---

Published as a conference paper at ICLR 2025
0.95
acc. unmodified images
0.90
Rate humans choose
ground truth (96)
0.85
0.80
0.70
Origial
Vanilla RN50
Cutbix RINSO
c=2 RMS0
c= RN50
RA50
4
Guide Models
Figure S9: Effectiveness of image category enhancement across different guide model types. Each bar shows the mean and 95% confidence interval (by bootstrap) of the rale at which humans choose the original ground truth label, "Original" bar shows accuracy for unmodified images, and other bars show accuracy of the same participants on images enhanced (e = 20) using gradients from the corresponding guide model. RN50 ResNet- 50, and values in the labels for each bar show the magnitude of the adversarial perurbations during adversarial truining.
16-way basic animal classification task using ImageNet images. The
lowed by cropping/padding). color-jittered, and randomly cut-out "views" of each image, strategy inspired by Ganz & Elad 2024) that extends DiffAugment (Zhao et al. 2020
S7 ABLATION STUDY ON LOGIT MAXIMIZATION APPROACH TO ENHANCEMENT
As limited ablation study our approach to image enhancement, we conducted an additional 16-way ImageNet animal classification experiment with 20 human participants. This experiment was mostly identical to the 16-way animal classification experiment described in the main text, cept there were 6 image conditions instead of 9, Half of the trials used images from the ImageNet Within each
validation set (as in the main experiment), and the other half from the training training/validation split, one third of the trials were original, unmodified images, one-third were enhanced by maximizing the ground truth logit with l2 pixel budget 10, and one-third were enhanced by minimizing the cross-entropy loss with = 10. The results of this experiment are summarized in Fig. S. We hypothesized that logit-based enhancement would provide superior results, particularly for images that started off with low cross- eross-entropy loss. We further hypothe- sized that enhancements would less effective for training images due to overfitting of the guide model on them. The results show that logit maximization is effective both training and valida- tion images, and induces a larger increase in accuracy for a given pixel budget e than cross-entropy minimization. Indeed, cross-entropy minimization significantly increased accuracy only for valida- tion images and not for training images. Unexpectedly, participants were more accurate on original, unmodified training set images than original, unmodified validation images. According to Russakovsky et al. (2015). the ImageNet ILSVRC 2012 validation set was collected using the same methodology as the training set, but later time. It is therefore plausible that the images and labels in the validation set are drawn from slightly different distribution than those in the training set, resulting in this accuracy discrepancy.
24

---

Published as a conference paper at ICLR 2025
Unmodified
Vanilla
CutMix RN50
XCIT
Crab
Frog
Bird
Big Cat
Bear
Figure S10: Meaningful perturbations require robust models, and are possible with CNN and vision transformer architectures. Each shows image from ImageNet (original the far left) enhanced with 20 by different guide models. A quantitative comparison of different models' perturhation efficacies with regords to improving human classification accuracy be found in Appendix Fig.
S8 ADDITIONAL RESULTS FROM IMAGE CATEGORY LEARNING EXPERIMENTS
Fig. 3 in the main text shows learning curves (mean smoothed accuracy by condition as a function of trall number), and schedules for image difficulty selection and enhancement for the moth pho- tograph task: similar plots are shown for the dermoscopy task in Fig. S12 and the histology task in Fig. [S13] Panels H-M of Fig. (S12) show the results of an early pilot experiment that used image enhancement in isolation (no difficulty selection), in which we suspect the perturbation magnitude set too high causing participants to learn exaggerated features and fail to generalize natural images with subtler features. This prompted us to switch to the schedule we used for our main learning experiments, which starts at tionship between the ground truth logit from robust ResNet-50 model and how many of the 7 expert annotators of the MHIST histology dataset Wei al. (2021) agreed on the same category label. On average, the model is more "confident" in its predictions (higher ground truth logit) on images where experts are more in agreement with each other.
instead of 20. Panel of Fig. SI3 shows the rela-
In addition to the agreement of expert MHIST annotators, the ground truth logit successfully predicts the proportion of human participants who select the correct ground truth label across tasks we tested. Difficulty prediction results from the 16-way ImageNet task are shown in Fig. [ in the main text, and from the moth photograph, dermoscopy, and histology tasks in Fig. S14 (Panels A1, BI, and C). For this analysis in the Imagenet tasks, rely on test-phase data from control group participants who had just learned the tasks in question.
We also attempt to measure the extent to which images with higher levels of enhancement easier for novice participants to recognize during the learning tasks (Fig, S14.2,B..) This analysis is limited to the first training trial blocks in the "ET (shuffled)" participant group in the ablation
25

---

A. ImageNet validation set Published as a conference paper at ICLR 2025
0.30
Enhanced (Logit)
Rate humans choose
Erhhaneed (Cross-ettropy)
ground truth [%6
0.15
Accuracy
0.10
0.70
Quartiles of ground logit
(original)
ImageNet training set
B.
Figure S11: Ablation results for image enhancement with ImageNet images. Logit maximization enhancement is effective for images used to train the robustified CNN used as guide model, and also for held-out validation images (panel A). Logit-max enhancement is mare efficient increasing human accuracy within a given pixel budget (c = 10) than enhancement by cross-entropy minimization (panel A). The efficacy of logit-max enhancement depends on difficulty of the original image as estimated by the starting ground trush logit (panel B). In the bar plot of panel B, images were assigned to 4 quadrants based on their ground truth logit values, and for each quadrant the mean difference accuracy was calculated between original, unmodified images and images enhanced with 10 (using data from the main 16-way animal classification experiment). The images below each bar illustrate an example image from the category "antelupe" drawn from the corresponding difficulty quadrant. All error hars are 95% confidence intervals for the mean from 10,000 hootstrap replicates.
study (main-text Table []. the only group that viewed enhanced images without monotonically decreasing over time. Note that there were 6 discrete values per block in the non-shuffled ET condition), and the analysis is complicated by the fact that participants were still learning the task when they made the responses underlying these plots. We are also unable to compare with e unmodified images because participants did not view training blocks. There are no results here for the histology task because the ablation study was conducted only for moth photos and dermoscopy images. In both the moth task and the dermoscopy task, participants respond with the original, correct category label statistically significantly more often when viewing images enhanced with greater (Fig. 1,B22.
unmodified images in the corresponding
To evaluate whether L-SE has differential effects on human image category learning depending on the image class, record test-phase precision and recall for each class among L-WISE and con- trol groups in Table The same data are visualized Fig. 3. Our experiments are statistically underpowered to detect cless-specific differences in performance (as opposed aggregated perfor- mance) however, we can observe in coarse sense that the sample means of precision and recall are numerically higher in the L-WISE group across all classes in all tasks. This suggests that over- all accuracy improvements attributed to L-WISE are distributed among the various image classes, rather than being the result of isolated improvements the detection of a subset of classes.
S9 PARTICIPANT DROPOUT RATES ARE LOWER WHEN L- LIIS ASSISTANCE IS PROVIDED
On the Prolific platform where studies partway through if they no longer wish to participate (this is called "returning" a study in the Prolific interface). For the moth photograph and dermoscopy image category learning tasks, participants who received L- WISE assistance full or partially ablated form (see Table I were less likely to withdraw.
experiments, participants choose to withdraw from
26

---

Published as a conference paper at ICLR 2025
Diff. %ile Enhance
L-WISE (pixel budget)
Control (pixel budget)
80
B.
L-WISE Difficulty
Control Difficulty
Accuracy (original groundtruthi
G.
Train (feedback) Test feedback)
C.
0.7
0.4
Chance
L-WISE accuracy (Ours)
Original
0.2
Control accuracy
20
40
60
80
100
120
140
160
Trial experlmental task
Enhance e
20
10
ET (pixel budget)
Control (pixel budget)
Accuracy (original groundtruth)
M.
Train (feedback)
Test (no feedback)
0.7
J.
PILOT DATA (n=16)
0.5
0.3
Chance
group mean smoothed accuracy Control group mean smoothed accuracy
0.2
20
40
80
100
120
140
160
Trial in experimental task
Original
20
Figure S12: Plots showing the accuracy trajectory of human participants throughout train- ing/testing in the main dermoscopy learning experiment (panels A-G) and preceding pilot experiment after which the epsilon tapering schedule was adjusted (panels H-M). All conven- tions are identical to main-text Fig. 3 There is a statistically significant difference between the test-phase performance of the L-WISE participants and that of the control participants (x) test, p < 0.001) in panel but not for the pilot experiment in panel M. Notably, the last portion of the training phase does not feature any image enhancements (see Fig. [ suspect that this is the reason for the sudden decline in accuracy in the enhancement group of the pilot experiment (M).
Nine participants withdrew from the moth photograph category learning experiment. Among them, six had been assigned the control group, one to the "enhancement taper" group, one to the "diffi culty selection" group, and one to the full L--IISE group. We can calculate the probability of d = 6 or more participants among the n who withdrew being from the control group, under the null hypothesis that the probability of withdrawal is independent of group assignment, using the binomial distribution via Equation 3 below (where p is the probability of being assigned to the control group). Equation
3 evaluates here probability of 0.02, indicating that participants who withdrew were significantly more likely to have been assigned to the control group than would be expected I-WISS assistance had no impact the probability of withdrawal.
27

---

Published as a conference paper at ICLR 2025
A.
Enhance
L-WISE (pixel budget)
Control (pixel budget)
Diff. %ile
80
B.
L-WISE Difficulty
Control Difficulty
Original
Accuracy (original groundtruth)
F.
Train (feedback)
Test (no feedback)
0.9
Annotator Agreement
0.8
s
0.5
Chance
L-WISE accuracy (Ours)
Control accuracy
0.40
20
40
60
80
100
120
140
160
Trial experimental task
Figure S13: Plot showing the accuracy trajectory throughout training/testing of human partici- pants in the histology learning experiment. All conventions follow Fig. 3 Panel C shows the association between agreement among the expert pathologist annotators of the MHIST dataset (Wei et al. [202]) and ground truth logit score of each image from robustified ResNet-50. Possible values of annotator agreement are 4, 5, or of the annotators agreeing with each other (3 below switches the "ground truth" category). Error bars are 95% confidence intervals for the mean from 10,000 bootstrap samples.
d-1
P(X 2 -1- - 1)
Σ A=
p² (1 p) k
11
(3)
Similarly, in the dermoscopy category learning experiment, 13 participants withdrew, of whom were from the control group. In this case, Equation3Jevaluates evvlluate to probability of = 0.041, again indicating that participants in the control group withdrew at significantly rate. Furthermore, 4 more of the 13 withdrawals were from the "Enhancement Taper (shuffled)" group, which had test-phase accuracy indistinguishable from the control group (see Table [T. None of the withdrawals from the dermoscopy experiment were from the full L-WISE group.
exeeeeeee
Overall, these results show that participants were more likely withdraw from the study when they did not receive assistance from L-WISE, perhaps reflecting the difficult nature of the moth photograph and dermoscopy image tasks at baseline. None of the participants withdrew from the histology image experiment, precluding similar analysis.
S10 NOTES ON "HALLUCINATIONS" IN ENHANCED IMAGES
To support our approach assisting human learners, we demonstrate the ability to enhance category percepts in images using low-norm perturbations. Previous work by Gaziv et al. (2023) showed that an image from one category can be perturbed targeted way such that human perceives it to belong to different category. Features introduced by these disruptive perturbations could be de- scribed as "hallucinations:" perceptions (by the model and the human viewer) of objects that are not present in the camera's view. Our image enhancement approach is special case in the wider realm of categorically targeted image modulation, in which maximization of the ground truth logit perturbs the image such that it becomes stronger and/or less ambiguous example of its class according to the model's judgement. Do these perturbations accentuate features that are already present such that they are easier for humans to perceive under challenging conditions, do they improve hu-
28

---

Published as a conference paper at ICLR 2025
0.7
A1. Moth photographs
B1. Dermoscopy images
1.0
C. Histology images
Rate humans choose
ground truth [%]]
0.6
0.6
163
468
0.8
520
0.5
n=19
146
n=200
156 312g
0.4
881
639
0.6
428
0.4
n=140
0.4
0.3
0.2
0.2
0.2
0
10
0
-2.5
0.0
2.5
5.0
Ground truth logit
Ground truth logit
Ground truth logit
0.6
A2.
B2.
Rate humans choose
ground truth [%]]
0.5
0.5
0.4
0.4
0.3
0.3
0
0
Enhancement
Enhancement
Figure S14: Difficulty prediction and image enhancemement are effective across image do- mains. Panels AI, B1, and C show the relationship between the ground truth logit from a fine-tuned robustu- fied Re model and the rate at which human participants (from the control groups) choose the ground truth label during the test phase following a training phase which they had just attempted to learn task. Images are binned by ground truth logit to produce the scatter plots, with the number of total trials listed for each bin. Vertical error bars are 95% confidence intervals by bootstrap, and horizontal error bars show the standard deviation within each bin. The curved lines illustrate fitted logistic regression models. All logistic regression models had statistically significant coefficients for ground truth logit (p < 0.001 from Wald statis- ric). Panels A2 and B2 show the relationship between enhancement e and the rate at which humans choose the ground truth category. This analysis is limited to the firsi training trial blocks in the "ET (shuffled)" participant group in the ablation study (main-text Table D the only group that viewed enhanced images without tonically decreasing over time. The logistic regression coefficient for was statistically significant (p <, 0.05 from Wald statistic) for both moth photographs (A2) and dermoscopy images (B2).
man accuracy by hallucinating new features associated with the target category? Subjectively, both phenomena seem to occur: panel B2 in Fig. [ appears to show bolder contrasts and exaggerated features in class-relevan regions of the perturbed images. Panel B (image farthest to the left) in Appendix Fig. STI however, shows a clear example hallucination, where a semblance of an en- tire additional "antelope" appears in the foreground of the image. This distinction may be important for education-oriented applications of our enhancement approach, as hallucinations could plausibly impart potentially misleading information the learner. On the other hand, it is possible that hallu- cinated features can impart useful and therefore desirable representations of the ground truth class despite departures from natural image distribution.
S11 PARTICIPANT RECRUITMENT AND DEMOGRAPHICS
We recruited grand total of 521 participants via the online platform Prolific. All participants lived in the United States and were fluent in English (as determined by Prolific). Each participant was eligible to complete each learning experiment only once, to avoid collecting data from participants already familiar with given task.
Our decision regarding the number of participants to recruit for each learning task experimental group (targeting 30 average) intended to exceed the requirements of simple power anal-
29

---

Published as a conference paper at ICLR 2025
Precision
Recall
Control
L-WISE
Control
L-WISE
Moth photos
seriata
0.46 (0.39-0.35 0.52)
0.52 (0.45-0.59)
0.43 (0.37-0.50)
0.63 (0.55-0.71)
tacturata
0.45 (0.40-0.50)
0.63 (0.55-0.71)
0.54 (0.47-0.61)
0.68 (0.60-0.76)
biselata
0.35 (0.30-0.41)
0.41 (0.32-0.50)
0.36 (0.31-0:42)
0.42 (0.35-0.50)
aversata
0.50 (0.44-0.55)
0.58 (0.50-0.66)
0.57 (0.50-0.63)
0.68 (0.59-0.77)
Dermoscopy
Benign mole
0.38 (0.33-0.43)
0.42 (0.36-0.48)
0,43 (0.38-0.48)
0.47 (0.40-0.54)
Melanoma
0.33 (0.29-0.38)
0.39 (0.35-0.44)
0.37 (0.31-0.42)
0.42 (0.37-0.47)
BCC
0.41 (0.36-0.46)
0.54 (0.46-0.61)
0.41 (0.36-0.47)
0.56 (0.48-0.63)
Benign keratosis
0.26 (0.22-0.30 0.30)
0.39 (0.34-0.43)
0.30 (0.25-0.35)
0.43 (0.36-0.49)
Histology
SSL (malignant)
0.58 (0.54-0.62)
0.60 (0.56-0.64)
0.66 (0.61-0.71)
0.70 (0.66-0.75)
HP (benign)
0.59 (0.55-0.64)
0.61 (0.56-0.67)
0.61 (0.56-0.66)
0.66 (0.62-0.70)
Table S1: L-WISE improves test-phase precision and recall across all image classes in three image category learning tasks. BCC=basal cell carcinoma, SSL=sessile serrated adenoma, and HP=hyperplastic polyp. parentheses 95% confidence intervals for the mean from 10,000 bootstrap repli- cates, resampling from participant-wise precision and recall values.
ysis we conducted following pilot experiments. Pilot experiments showed differences in test-time accuracy between control and either enhancement taper (equivalent to ET main-text Table [ or difficulty selection (equivalent to DS in Table [ participants to be roughly 10%, with standard deviation in accuracy of roughly 10% in each group.
Ho μ1 M2 = 0
H1 : p4 22 # 0
Given:
5 = 0.1 (estimated mean difference)
0.1 (estimated standard deviation)
0.05 (significance level)
0.8 (power)
8 Estimated effect size d = 1.0
σ
Required sample size per group: = 2(21 + =1 /dd
22111190 + 0.84) //4
subjects per group at minimum
We provide demographic breakdown of the participants in our study, aggregated across experi- ments, in Table [52) Some participants took part in more than one of the experiments, but are only counted once the table.
30

---

Published as a conference paper at ICLR 2025
Total participants
521 519 (99.6%)
Pts. w/ demographic data
Age
Mean (SD)
36.6 (11.9) years 18-83 years
Range
Sex
Female
289 (55.7%) 227 (43.7%) 3 (0.6%)
Male
Not specified
Ethnicity
White
338 (65.1%)
Black
(10.4%)
Asian
50 (9.6%)
Mixed
44 (8.5%)
Other
23 (4.4%)
Not specified
10 (1.9%)
Table $2: Demographic characteristics of study participants, aggregated across all experiments.
31

---

Published as a conference paper at ICLR 2025
F(3.193)=0.21
F(3.193)=1.44
F(3,193)=0.37
F(3,193)=1115
p=0.8868
p=0.2329
p=0.7768
p=0.3302
n²=0.003
n²=0.022
n²=0.006
n²=0.018
Condition-Norit iiitiioo--ormmlieeed Accuracy (Z-Score)
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
aversata
biselata
seriata
tacturata
Alias
Eris
Leda
Tyro
Ajax
Figure S15: Randomized assignment of aliases "Leda," "Ajax," "Eris," and "Tyro" had min- imal impact on test-phase accuracy in the moth species classification task. Each group of four boxplots shows the relative effects of assigning each alias specific class from the moth classification exper- iment. Each individual hoxplot indicates the distribution of participant-wiis test-phase accuracy z-scores (nor- malized with mean and standard deviation within each condition separately) among participants with mapping of a specific alias onto specific class for example, the left-most boxplot within the right-mos feht-most group describes the accuracy of participants who saw images of the moth species Idaea tacturata labelled "Eris." " There is no evidence from one-way ANOVA that the random assignment of aliases to classes influences test-phase per- formance.
32

---

Published as a conference paper at ICLR 2025
F(3,268)=2.51
F(3.268)=1.50
F(3.268)=1.26
F(3,268)=1.15
p=0.0592
p=0.2154
p=0.2884
p=0.3280
n²=0.027
n2=00.16
n²=0.014
n²=0.013
Condition-Normalizee Accuracy (Z-Score)
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
Basal Cell Carcinoma
Benign Keratosis
Melanoma
Benign Mole
Alias
Leda
Ajax
Eris
Tyro
Figure S16: Randomized assignment of aliases "I.eda," "Ajax," "Eris," and "Tyro" had min- imal impact test-phase accuracy in the dermoscopy task. After correcting for multiple compar- isons, there is no evidence that the random assignment of aliases to classes affects task performance. See also Fig. S15
33


Reviews:
[{"review_id": "dXsXGXdekI", "reviewer": "Reviewer_wUUM", "summary": "The paper proposes a novel approach to augment human learning in image categorization tasks. By leveraging robustified ANNs, the study introduces model-guided image selection and enhancement strategies that increase human test-time categorization accuracy by up to 72% and reduce training duration by around 20-23%. L-WISE employs selecting images based on predicted difficulty levels and enhancing images with pixel perturbations.The proposed approach is tested on natural images, dermoscopy, and histology images. The results demonstrates efficacy of L-WISE in aiding novice learners in fine-grained categorization tasks. This research represents one of the first applications of ANNs in optimizing human visual learning in clinically relevant domains.", "questions": "1. How to choose $\\epsilon$ for different tasks and image domains?\n2. What criteria determine if a model is \"robustified\" enough for use? Have you considered specific metrics to evaluate the robustness of guide models, and how do these metrics correlate with human learning outcomes?\n3. Did you collect qualitative feedbacks from participants? Did the new curriculum and enhanced images increase mental stress of human learners? Additional learning costs beyond training time should be considered, such as cognitive load and emotional well-being.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "- Presents an innovative use of robustified ANNs to predict task difficulty and enhance images, aiding human perception and learning.\n- Shows broad applicability by successfully testing across diverse domains, such as natural image classification, dermoscopy, and histology.\n- Achieves practical efficiency by reducing training time and improving test-time accuracy, beneficial for fields requiring rapid, accurate human image categorization training.", "weaknesses": "- Lacks a dedicated related work section, which would help contextualize the research.\n- Both low and high logits from ANNs show significant variation in human accuracy, making predictions less reliable in certain logit intervals.\n- Uses only the ResNet-50 architecture, limiting generalization; further testing with models like vision transformers (ViT) is needed to support broader conclusions.\n- Image enhancement may introduce biases, potentially improving accuracy only for certain major classes; additional metrics like precision and recall per class, rather than just mean accuracy, should be reported to provide a clearer assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aFRviCfZjo", "reviewer": "Reviewer_5U3k", "summary": "This paper presents L-WISE, a framework leveraging adversarially robust ANNs to estimate image difficulty and apply nuanced perturbations that facilitate human learning in visual categorization tasks. By selecting challenging images and amplifying category-specific features, L-WISE improves human categorization accuracy by 33-72% and reduces training time by 20-23% across both general and clinical domains (e.g., dermoscopy and histology). The authors also discuss ethical implications, emphasizing the benefits of enhanced medical training and cautioning against potential biases that may arise from reliance on model-derived guidance.", "questions": "1. For model perturbations, can the authors please provide heat maps or any qualitative results that will help us track the ANN perturbations to human visual learning? \n2. Is there a way to show if the study scales/generalizes to other ANNs as well other than Adversarially trained ones? Since human subjects have been used here, I am not sure how feasible experiments will be. This can be a general neural network or a network trained by the CutMix [a] loss that provides robustness benefits as well. \n3. I feel generating perturbations based of dual stream networks and then using human gaze to track these will be a much stronger claim to the work. Can the authors please address this question? \n4. What about adversarially trained transformers? The attention maps are different from CNN feature maps. How will the study be applicable for perturbations based of a transformer backbone? \n5. Please also address concerns raised in the weakness sections. \n\na. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019,", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The application of Robustified ANNs for improving human visual performance on image categorization seems like an interesting avenue. \n2. L-Wise empirically demonstrate gains in categorization accuracy and training efficiency. \n3. The paper addresses ethics concerns. Since the work mentions use of clinical data ethics discussion is of critical importance.", "weaknesses": "1. The paper focused on the performance of ventral stream. But we know that the human visual stream has a dorsal stream (where) that locates an object and the ventral stream (what) stream. And the interplay of these two streams forms the basis of human visual system. In this work the authors mainly focused on the ventral stream. From only quantified data, we can see the gains but it is very hard to trace this back to the nuanced perturbations the ANN produces. Hence, the suggestion is to use human gaze. The human gaze will precisely pin-point the \"where\" aspect and then will truly help us understand if at all the model perturbations are helping improve human performance. Can the authors please explain this? \n2. A robust DNN actually has worse performance on nominal data points. Data points that have not been corrupted adversarially. What was the motivation of the authors to select such a model for their experiments? \n3. The perturbations -  The perturbations if I am not mistaken are very subtle ones. For fine grained classifications, humans do follow curriculum learning but learning structures gradually from simpler to harder concepts. No experiments have shown this. It would be great if the authors can provide some empirical results/ explanation that can explain how will their method occur when you focus on structural cues rather than model perturbations that will benefit fine grained categorization. \n4. I am providing some citations related to Gaze and dual stream hypothesis that can help authors clarify my concerns \n\na. A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains, NeurIPS 2023.\nb. Literature related to papers accepted in NeurIPS Gaze Meets ML workshop. That workshop accepted papers will provide intuition of how human gaze can be used in coherence with DL models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eXrfGT2TE5", "reviewer": "Reviewer_nJDg", "summary": "This paper proposes a method to enhance human visual learning by designing a model-based selection and enhancement algorithm to improve classification accuracy during testing. First, the authors select images to present to novice learners based on a model’s estimated recognition difficulty for each image. Next, they apply image perturbations intended to aid recognition for novice learners. The authors conduct experiment on the three benchmark datasets, including the natural image and the clinical image to verify the effectiveness of their proposal.", "questions": "1. The empirical observations are derived from a 16-way animal categorization task on natural images, which seems somewhat simplistic. It would be valuable to examine how these observations hold up in more complex categorization tasks or with different types of images, especially medical images. Given the typically limited availability of medical images, the proposed method could have promising applications in the medical imaging field.\n\n2. Beyond the empirical observations, is there any physiological insight or analysis on why the proposed model-based selection and enhancement method could improve novice learners’ performance in categorization tasks?\n\n3. The authors do not discuss the related machine teaching literature. An in-depth comparison with machine teaching methods, particularly with \"Teaching Categories to Human Learners with Visual Explanations\" (CVPR 2018), would be valuable. This work similarly considers image difficulty; an introduction and comparison with it are beneficial.\n\n4. The authors should also discuss the connection to human-machine vision alignment methods, such as \"Harmonizing the Object Recognition Strategies of Deep Neural Networks with Humans\" (NeurIPS 2022).\n\n5. The sample size of participants is relatively small, and expanding the participant pool is recommended to enhance the reliability of the results; also, recruiting participants from diverse backgrounds would improve the generalizability of the findings. If expanding the participant pool is impractical due to time or budget constraints, performing a power analysis or discussing effect sizes could help strengthen the reliability of the analysis.\n\n6. When using perturbations to enhance images, how does the method ensure that essential image details remain unchanged, particularly when using a large ϵ (e.g., 20)? This concern is especially pertinent for medical images, where even slight pixel changes may alter critical information. I recommend involving medical experts to review the enhanced images or using quantitative similarity measures (such as SSIM or FSIM) to verify that essential details are preserved.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well-motivated, and a decent amount of technical details are given.\n2. The idea of improving the categorization performance of the novice learner by leveraging the capacity of the robustified artificial neural network is both interesting and practical.\n3. The reported improvement in novice learners' performance is notable, with gains in both test accuracy and reduced training time.", "weaknesses": "1. The establishment of the empirical observations is somewhat unconvincing. Do these observations hold in more complex classification tasks or when applied to medical imaging?\n2. The related work section lacks discussion of both the machine teaching and human-machine vision alignment methods.\n3. The size of the particants is somewhat small. \n4. The perception of enhanced images may be altered due to perturbations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)