A tale of two tails: Preferred and anti-preferred natural stimuli in visual cortex
Authors: Rabia Gondur
Patricia L. Stan
Matthew A. Smith
Benjamin R. Cowley
Venue: ICLR 2026 Poster (2026)
Keywords: computational neuroscience
neuronal tuning
stimulus selectivity
higher-order visual cortex
human psychophysics
stimulus optimization
deep neural networks
Primary area: applications to neuroscience & cognitive science

Abstract: An ongoing quest in neuroscience is to find the preferred stimulus of a sensory neuron. This search lays the foundation for understanding how selectivity emerges in the primate visual stream---from simple edge-detecting neurons to highly-selective face neurons---as well as for the architectures and activation functions of deep neural networks. The prevailing notion is that a visual neuron primarily responds to a single preferred visual feature, like an oriented edge or the shape of an object, resulting in a 'one-tailed' distribution of responses to natural images. However, surprisingly, we instead find 'two-tailed' response distributions of primate visual cortical neurons, suggesting that these neurons have both preferred and anti-preferred stimuli. We experimentally validated anti-preferred stimuli by recording responses from macaque V4 to model-optimized stimuli. We find that these anti-preferred stimuli are important for describing a neuron's tuning, as both preferred and anti-preferred images are needed to predict a neuron's responses to natural images. Moreover, in a psychophysics task, humans rely on anti-preferred images to interpret and predict V4 stimulus tuning; this was not the case for internal units from a deep neural network. Interestingly, we find no discernible differences in image statistics between preferred and anti-preferred images. This suggests that by encoding anti-preferred features, a V4 population seemingly doubles its capacity for feature selectivity, allowing for a more flexible downstream readout. Overall, we establish anti-preferred stimuli as an important encoding property of V4 neurons. Our work embarks on a new quest in neuroscience to search for anti-preferred stimuli along the visual stream and offers a new perspective on how feature selectivity arises in the visual cortex and deep neural networks.

Published as a conference paper at ICLR 2026
A TALE OF TWO TAILS: PREFERRED AND ANTI- PREFERRED NATURAL STIMULI IN VISUAL CORTEX
Rabia Gondur' Patricia L. Stan Matthew A. Smith 2 Benjamin R. Cowley!
Cold Spring Harbor Laboratory, Cold Spring Harbor, NY, USA Carnegie Mellon University, Pittsburgh, PA. USA
(gondur,cowley cow.ey)@cshh.c. edu
ABSTRACT
An ongoing quest in neuroscience is to find the preferred stimulus of a sen- sory neuron. This search lays the foundation for understanding how selectivity emerges in the primate visual stream from simple dete-dete neurons to highly-selective face neurons-as well as for the architectures and activation functions of deep neural networks. The prevailing notion is that visual neu- ron primarily responds to single preferred visual feature, like an oriented edge or the shape of an object, resulting in a "one-tailed" distribution of responses natural images. However, surprisingly, we instead find "two-tailed" response distributions of primate visual cortical neurons, suggesting that these neurons have both preferred and anti-preferred stimuli. We experimentally validated anti- preferred stimuli by recording responses from macaque V4 to model-optimized stimuli. We find that these anti-preferred stimuli are important for describing neuron's 's tuning, as both preferred and anti-preferred images are needed to pre- dict neuron's 's responses to natural images. Moreover, in a psychophysies task, humans rely on anti- preffrreed images to interpret and predict V4 stimulus tun- ing: this was not the case for internal units from a deep neural network. Inter- estingly, we find no discernible differences in image statistics between preferred and anti-pred anti-preferred images. This suggests that by encoding anti-preferred fea- tures, V4 population seemingly doubles its capacity for feature selectivity, allowing for a more flexible downstream readout. Overall, we establish anti- preferred stimuli as important encoding property of V4 neurons. Our work embarks on a new quest in neuroscience to search for anti-preferree stimuli along the visual stream and offers a new perspective how feature selectivity arises in the visual cortex and deep neural networks.
INTRODUCTION
Since the first recordings of action potentials from sensory neurons (Hartline 1938 Lettvin et al. 1940), neuroscientists have searched for the stimulus features that a neuron prefers. Hubel and Wiesel famously identified the stimulus preferences of carly visual cortical (V1) neurons as ori- ented edges (Hubel and Wiesel, 1962). In higher-order visual cortex are neurons with remark- able selectivity (Perrett et al. 1982). such as "Jennifer Aniston" neurons that only respond to images of the celebrity, regardless of profile or hairstyle (Quiroga et al. 2005). This has spurred on new modeling approaches to identify visual neuron's 's preferred stimulus- the stimulus that
maximizes a neuron's 's response (Földiák 2001; Yamane et al. 2008 Okazawa et al.) 2015) Cow-
ley et a [211aa] Cadieu et al.) [007]]Abbas: Abbasi-As Asl et al. [0188] Pospisil et al. [0188] Ponce et al.)
2019 Bashivan et al. [0199]] Walker et 20.9 Gu et al. [2022] Cowley et al.) 2026) Willeke
let al. |023 Pierzchlewicz et al.] [2024] Wang and Poncel (2024). The classic linear-nonlinear (LN)
model, used to describe retinal ganglion cells and simple VI neurons (Hubel and Wiesel| 1962,
Heeger 1992, Rust et al. 2005 Pillow et al., 2008), exemplifies the concept of a preferred stimu-
lus: The LN model filters visual input to detect single stimulus pattern (e.g.. a localized, oriented
edge). The presence of the pattern causes the activity to surpass a ReLU-like threshold, while all other stimulus patterns fail to reach this threshold, silencing the output. This results in "one- tailed" response distribution (Fig. Ta top row). The deeper units a task-driven DNN- made up of cascading layers of LN models-achieve the sparse selectivity found in higher-order visual

---

Published as a conference paper at ICLR 2026
linear-nonlinear model
LN uni
V4 neurons neurons
macaque
tail
V1 leary)
IT
uridede)
ressonss
sinti proferred
prefemed
imagges
images
task-driven DNN (ResNet50-layera
ONA unit
V1 neurans
1P maree
skewness
one tail
DNN
skewness, K
neural recordings
(visual area
resparse
danddy
0z V4
neurons
DNN units
cal
early units
imidlle units
early middle late
late units
two talls
Colnnts
response Spikoshod)
skewness,
skewness, K.
Figure 1: V4 neurons have two-tailed response distributions. a. Response distributions for a enrear-nonlinear filter (top), DNN unit (middle), and real neuron from visual area V4 (bottom). b. Anti-p Anti-prefenee priferrefered and preferred images of an example V4 neuron. c. Skewness x of response distribu- tions for V4 neurons and DNN units. d. Skewness K of response distributions for different visual areas in macaque (top), and DNN layers (bottom). Dashed lines: medians.
cortex. Indeed, the response distributions of ReSt DNN units in deeper layers typically have one extreme tail (Fig. [h. middle row) with few select stimuli evoking large responses. However, when we recorded from real neurons in macaque V4, higher-order visual area known for encod- ing texture, shape, color, etc. (Pasupathy et al. 2020), we expected to see similar one-tailed re- sponse distributions to natural images. Instead, we unexpectedly found response distributions with two distinct tails (Fig. (1 bottom row, example real V4 neuron). This suggests that, unlike LN models and most DNN units, V4 neurons have preferred (response-maxe maximizing) and anti-preferred tresponse-minimzzing stimuli.
The existence of mnti-rreferre images for higher-order visual cortical neurons is not obvious. The anti-preferred stimuli have largely been investigated as part of neuron's 's tuning for single stimu- lus parameter (e.g., orientation: vertical edge may drive a V1 neuron's response while horizon- tal edge suppresses it) (Hubel and Wiesel 1962, Cavanaugh et al. (002). However, little is known about the anti-preferred stimuli of V4 neurons when considering the vast space of natural images varying over many stimulus parameters (Efird al. 2024). Our prior expectations that the anti- preferred images are mostly featureless and low contrast a blank, gray screen- wrong, we find that some anti anti-preferred visual features This motivated us to systematically investigate the existence of anti-preferred images and their roles in how the visual cortex encodes natural images with the following experiments and analyses:
vivid those for the preferred images (Fig. [p).
We first set out to confirm the existence of anti unti-preferred images by analyzing response distributions of visual cortical neurons from V1, V4, and IT as well as performing own ciccrrophysoologicall experiments to validate that anti anti-preferred images suppress V4 responses.
2. We find that two-tailed pre- ReLL DNN units are worse predictors of real V4 neurons than one-tailed Rest DNN units; explain this discrepancy by proposing map- ping between DNN units and V4 neurons that takes advantage of anti-preferred features outperform commonly linear mappings.
3. We further find that anti-preferne images are integral to a V4 neuron's 'ss tuning: Training encoding model solely responses to preferred images performs worse than train- ing on responses to both preferred and anti anti-preferred images. In a similar vein, humans performing a psychophysics task also rely anti-preferred images infer neuron's tuning.
4. How do anti-preferred features contribute encoding natural images by V4 population? We find that knowing a neuron's 's anti- treferreer feature provides little information about the neuron's 's preferred feature but that both have similar image statistics. This suggests that anti pref-rree images effectively double the population's 's capacity for feature seleetiv- ity.
2

---

Published as a conference paper at ICLR 2026
5. To encourage further experiments investigating anti-preferred images in the visual cortex, we release a tool called ImageBeagle that efficiently "hunts" through millions of natural images. We tailored ImageBeagle for closed-loop. real-time experiments.
Our results change our prior conceptions about stimulus encoding in primate visual cortex: Con- ceptually, responses are not simply the output of a ReLU with strong threshold but rather the sum of a baseline offset and a stimulus drive that may enhance or suppress the baseline response, result- ing in two-tailed response distribution. That preferred and anti- preferree features are diverse and distributed across neurons allows the neural population to seemingly double its feature selectivity, providing rich yet compact basis for readout by downstream IT neurons to carry out object recog- nition and other visual tasks. Our work speaks to neuroscientists studying how feature selectivity arises in the visual cortex as well as Al researchers aiming build biologically-reaisste Al models with internal representations that align with those of the visual cortex (Richards el al.
2019 Doerig al. )023).
2 HIGHER-ORDER VISUAL CORTICAL NEURONS IN AREA V HAVE TWO-TAILED RESPONSE DISTRIBUTIONS.
To quantify the degree to which V4 responses to natural images have distributions with two tails (a hallmark of a neuron having preferred and anti anti-preferre images), we computed the skewness x of response distributions. A distribution with close to O indicates two tails (Fig. Dr. top left panel) while x close to indicates a one-tailed distribution (Fig. [c. top right panel). As expected, the skewness for Post- DNN units in middle layer of task -divenn DNN (ResNet50) (He al.. 2016), known to be predictive of V4 responses (Cowley et al., 2026; Yamins and DiCarlo, |i Schrimpf al. 2018 Bashivan et al. 2019b Zhuang al.] [021] Cadena et al. (2024), was close to 2 (Fig. D "DNN units', median x = 2.06), indicating that most units in a task-driven DNN have one-tailed response distributions and are selective for one type of visual feature; we confirmed this was true of units from other task-driven DNNs (see Appendix B.2). On the other hand, the response distributions of V4 neurons were better described as two-tailed (Fig. Dc. 'V4 neurons', median 6= = 0.87, < 0.002, n = 219), suggesting that V4 neurons have both preferred and anti- preferred images (See A. in Methods for a description of V4 data collection; V4 responses were repeat-averaged spike counts). Here, we avoid the trivial effects of adaptation (Kohn) 2007) which presenting the same image continuously for long periods of time would lead to response suppression by taking spike counts in 100 ms bin after the stimulus onset of natural image preceded by a gray, blank screen. V4 neurons appear to dynamically increase their bascline firing rate encode a newly presented image (Pasupathy and Connor, 1999; Maunsell 2015), allowing images to both excite and suppress their response from baseline (investigated in the next section). This goes against the conventional notion that a visual neuron responds selectively certain stim- uli by discarding most other stimuli that fail to drive the neuron past its spiking threshold. In other words, V4 responses do not appear be the output of ReLU-like activation functions.
These findings motivated us to further investigate whether neurons from other areas of visual cor- tex also exhibit two-tailed response distributions. We used publicly-availably datasets to compute skewness for VI neurons (Cadena et al. 2019) as well as for V4 and IT neurons (Majaj et al. 2015). We found that skewness values from this other V4 dataset matched our own data (Fig. Td. *V4 neurons', median x- = 0.41). In addition, we found that neurons from V1 and IT also ex- hibit two-tailed selectivity (Fig. [d. 'VI neurons', median x 1.17 and 'IT neurons'. median layers of ResNet50 exhibited
= 0.69). In contrast, the activations from increasingly much larger skewness values. DNN units in an early layer had the lowest skewness (Fig. [d. 'early units', median A = 0.99) on par with that observed for V1 neurons. A late layer had the highest skewness value (Fig. [d. 'late units', , median A = 4.43), revealing a trend of increasing skewness (or one-tailedness) deeper into the network. Taken together, our results indicate a mismatch be- tween biological and artificial visual systems: Neurons along the visual cortical hierarchy tend to have two-tailed response distributions, whereas DNN units in the deepest layers are most likely to have one-tailed response distributions. In other words, most real neurons encode anti mnt--rrferrre im- ages, but DNN units Relst-Rel.U) often encode only preferred features, especially in deeper layers.
3

---

Published as a conference paper at ICLR 2026
a predict responses natural images
experimental validation
anti preferred imgess mmaesschsen chosen by model
Cantlppeef Images suppress
V4 firing rates
proforred images chosen by model
120
s0m on
stmmolf
tesamile
preferred
201g
(splkes/sed
200
time
prc 1200 m3
randomly-
(narmalaed)
stim.
blank
G1--1 Erdu
rebu
100
500% percentla bands)
imnages
response
prockted
raku
es
mapping
stim. (prof)
blank
blank Gifeer
preferences
lafter
lour mapping)
V4 neuron index
preferred images
preff
model-optimized anti- images
t. V4 model neuron
Figure 2: Experimental evidence that V4 neurons have anti-preferred images. a. Predicting V4 responses to randomly-chosen images from linear mapping of ResNet-50 features. Each dot denotes the median, and error bars denote I s.c.m. b. Experimental validation of preferred and anti-preferred images as predicted by V4 model neurons. Each dot is the repeat-averaged V4 re- sponse to one image; gray bands denote 90% percentiles of responses to randomly-chosen natu- ral images. Insets: Model-chosen images for the 3 V4 neurons with largest baseline responses. c. Top: Repeat-averaged temporal response of an example V4 neuron (PSTH). Bottom: Normalized response to preferred, anti-preferred, and following blank images; each dot denotes the average re- sponse of the top 10 images for one V4 neuron (bottom). d. Preferred and anti-preferred images synthesized via gradient techniques, one for each V4 model neuron. Asterisks in and denote p<0.001, permutation test, n > 50.
3 EXPERIMENTAL EVIDENCE FOR ANTI-PREFERRED IMAGES
A common practice to predict V4 responses to natural images is to linearly map task-driven DNN features to the V4 responses (Yamins and DiCarlo 2016 Schrimpf et al. 2018). Interestingly, previous studies using this approach almost always use DNN units from ReLU layer (i.e., post- ReLU) versus a pre-ReLU layer (i.e.. the output of convolutional filters not yet passed through ReLUs) (Bashivan et al., 2019a| Cowley al., 2026; Cadena et al.) 2024). Indeed, we find that using a linear mapping to predict V4 responses with ReLL activity outperforms the Rel- activity directly preceding the ReLU activation functions (Fig. 2p. vs if). On one hand, this was the activity of pre-ReLU DNN units better matches the two-tailedness of real V4 responses; on the other hand, we would expect this performance increase the Pre-ReLU DNN units did not match the real V4 neurons in both preferred and anti anti-preferred features. In other words, the post-ReLU activity forms a bank of ooe-ttaied preferred features that can be linearly combined to form two-tailed preferred and anti-preferred features. Not every bank of one-tailed preferred features performs well: Flipping the sign before passing the activity through a ReLU- only giving access to the anti-preferred features of the Rre-ReL.U units is the poorest predictor (Fig. 2,, fii). suggesting that the anti- anti-preferred features in task-driven DNNs do not match feature preferences in V4. We conclude that Rost-ReLU activity better predicts V4 responses not because the activity is more biologically realistic but rather because one can form many possible pairs of preferred and anti-preferred features by mixing and matching post-ReLU "one-tailed" DNN units.
That the pre-ReLU activity performs worse despite having strictly more information (two tails) than the Rost-ReLU activity (one tail) motivates new mapping that is agnostic to possible mis- matches in preferred and anti-preferred features. We propose to linearly combine filter channels (i.e., a convolution with kernel shape 1 x 1 and output channels equal to the number of input channels) and then pass the resulting activity through ReLUs and a final linear mapping. This sim- ple approach significantly improved prediction for pre-Re ReLL activity (Fig. 24, iv vs. i) as well as post-ReLU activity (Fig. 24. iv vs. ti). The extra layer of ReLUs in our proposed mapping also im- proved Post-Rel-U activity performance (Fig. 2a vs. ii), leading an expected similar prediction
4

---

Published as a conference paper at ICLR 2026
performance between pre- ReLU and post-ReLU activity (Fig. 2 iv v, see Methods A.2 for re- sults from other task-driven DNNs). Thus, this new mapping accounts for a possible mismatch in anti anti-preferred features. If the pre-ReLU DNN units had matching anti untiprrfferre features with the real V4 neurons, our mapping will always outperform using only post-Rel.U DNN units by access- ing both the preferred and anti-preferred features of the pre-ReI.U DNN units (see Supp. Fig. D Overall, these findings highlight an important representational gap between task-driven DNNs and biological networks, and offer a simple mapping to overcome this gap.
The skewness of response distributions and the improved prediction by mixing preferences before the ReLUS both hint at the existence of anti- preffrrree images; we next seek experimental evidence. We build upon recent work that identified highly-predictive DNN models of V4 neurons by train- ing on responses to many natural images (Cowley et al. 2026); these data-driven models predicted the preferred images of real V4 neurons in validation experiments by presenting the model-chosen preferred images in a following recording session (a causal test). We wondered whether the same framework could be used to predict neurons' unti-preferred images. To test this, we recorded V4 responses while the awake, fixating animal (macaque monkey) watched flashes of many images over multiple recording sessions (see Methods [A.] p. and used the image-response pairs to train a set of data-driven DNN models (which we refer to V4 model neurons). We identified each V4 model neuron's preferred and anti- anti-preferred images by passing as input 500,000 natural images and keeping the 10 images that either maximized or minimized the model's output response (ex- ample chosen images in Fig. Fig.2b). 2b). We then experimentally validated these predictions by recording V4 responses to these model-chosen preferred and anti- anti-preferred images, along with hundreds of vandomly-chosed natural images. After matching V4 neurons to their corresponding model units, we found that the predicted preferred images resulted in responses above the 90% density interval of responses to randomly-chen dian response to preferred images for responses to randomly-choses images was q = 0.985, me- dian across neurons), while the predicted anti-preferred images resulted in responses at the lower bound of these density intervals (Fig. D. green dots below gray lines, quantile of median response to anti- mnttepreeerred images for responses to randomly-chosen images was q 0.055, median across neurons). This experimental validation provides clear evidence that V4 neurons have anti-preferred images, identifiable from data-driven DNN models.
images (Fig. D purple dots above gray lines, quantile of the me-
A visual neuroscientist may wonder how the responses to anti- priferrfeerd images compare with re- sponses to gray, blank screens- -the de facto stimulus used between stimulus presentations to bring the neurons' firing rates to rest. It is not unreasonable to assume this featureless gray stimulus yields the smallest response. However, we found this not to be the case: While the preferred im- ages strongly drove the vareing V4 response above baseline (Fig. 2c. top, 'preferred'; the response transient is ~550 ms after 'stim-on' due to synaptic delay from retina to V4), the anti- preferred images suppressed the neuron's 's response below its baseline firing rate (Fig. Zc. top, 'anti- preferred"). We confirmed this across neurons by comparing responses during the 100 ms stimulus presentation (Fig. (F.g.Er. 'stim.', lagged by +50 ms) versus responses in the 100 ms immediately fol- lowing stimulus presentation during which a gray. blank screen was presented (Fig. 'blank', also lagged by +50 ms); the results were significant (Fig. 22. bottom) and beyond what we imag- ined for the anti-preeerred images (Fig. - bottom right). This rules out the possibility that most anti- ont--preferred images are featureless within a neuron's 's spatial receptive field. Indeed, the diversity and specificity of model-optimized anti-preferred images is on par with those of model-optimized preferred images (Fig. 2d, top versus bottom rows).
4 ANTI-PREFERRED STIMULI ARE NEEDED TO ESTIMATE V4 TUNING.
The existence of anti preferrffeees alone does not necessarily imply that they are essential for shap- ing a V4 neuron's 's tuning. Indeed, knowing neuron's 's anti- untiprrffrree images may not help predict the neuron's responses to other images- a failure to generalize. In contrast, if a neuron prefers a certain visual feature, increasing the strength of this feature in an image is often thought to fur- ther drive the response of the neuron (e.g.. increasing the contrast of a sinusoidal grating for a VI neuron). We devised an approach to assess the information content of anti-preferred images by including them or leaving them out when estimating a neuron's 's tuning, an approach inspired by data pruning (Paul et al. 2021 Sorscher et al. 2022). We chose to estimate the tuning of the data-driven V4 model neurons in place of real V4 neurons, allowing us to consider responses to 500,000 images. We employed different data-pruning strategies by considering different sets of
5

---

Published as a conference paper at ICLR 2026
train on:
predicting V4 model neurons
random
c
predicting ResNet50 DNN units
(2k imagés)
pref.+anti-pref.
06, pref--ant--pref (1M)
pref.tanti-pret
pref-tanti anti--ref.
pref, only
pref. only
0.22
anti-pref.ong only
0.4
R
anti-pref. only
R
pref. only
random
non-pref.
0.18
non-pref.
0.2
random
on:
synthesized
0.14
non-pref.
anti-pref. only
held out randomly-chosen images
2k
4k 6k 8k
Tók
pre-
ReLU threshold:
0.9 post-
number of training images
ReLU
response quantile
ReLU
Figure 3: Anti-preferred images contribute to V4 tuning. a. Response distributions for differ- ent training sets for a data pruning analysis; Re is always computed with the same held-out natural images. b. We train 5-layer DNN (see Methods A.3) to predict responses of individual V4 model neurons (219 in total), varying the number of training images up to 10k (see Supp. Fig. f re- sults with > 10k images). Response distributions were over 500k images; we also considered a distribution for million (IM) images. We perform data pruning analysis for 219 ResNet50 units (2k training images), either pre-ReLUU (left) or post- (right), as well as ReLU thresh- olds equal to different quantiles (c.g.. a quantile of 0.5 results in responses to half the images being above 0). Traces denote means; shaded areas denote s.c.m.
training images: preferred, anti- anti-preferred, both, randomly-chosen, or non-preferred images whose responses were closest the median response (Fig. 3a.
For small number of training images (<5k images), training both preferred and anti anti-preferred outperformed randomly-chosen images (Fig. Bb. Panti versus 'random') )) Training ei- ther on preferred images alone or anti-preferred images alone failed reach the prediction per- formance of random selection (Fig. Bb. 'pref. only' and 'anti-pref. only'). Thus, both are needed together to achieve good tuning estimates; consistent with this, training only on non-preferred im- ages had poor generalization (Fig. 3p. 'non-pref.'.. Choosing more extreme versions of preferred and anti- nntt-preferred images (by doubling the pool to 1 million images) led to further increase in prediction performance (Fig. 3p. 'pref. -ret--f.ni-pref. (IM)'). However, synthesizing images via gra- dient techniques to maximize and minimize a neuron's 's response (see Appendix [B.3) (Ponce et al.)
2019 Bashivan et al. 2019a| Walker et al. 2019 Gu et al. 2022 Cowley et al. 2026; Willeke
et al. 2023] Pierzchlewicz et al. (2024) performed poorly (Fig.B)
"synthesized" ), likely because synthesized images deviate too far from the manifold of natural images as well as suffer from
a lack of diversity (Pierzchlewicz et al., 2024 Nguyen et al. 2013). We note that, as expected, random-selection eventually outperforms the other biased training sets that out of distribution (Fig. 3b, 'random'). ). Nonetheless, these results suggest that by knowing a neuron's 's preferred and anti-preferred images, one can reasonably estimate the rest of the neuron's tuning other natural images.
We wondered how these results may change for estimating the tuning of one-tailed DNN units. We performed the same data-pruning analysis on post-ReLL DNN units and found that training only on preferred images performed better than random-selection (Fig. 3 "post-ReLU'), consistent with the idea that one-tailed units are mostly characterized by their preferred feature. However, we were surpised to find that training on both preferred and anti-preferred images led to a substan- tial boost in performance (Fig. (3c. (post-ReLUt
pink dot). This trend held for the corresponding pre-ReLU units (Fig. 3 'pre-ReLU" as well as for different ReLU thresholds set to response quantiles (Fig. 3. 'ReLLU threshold'). How is it possible for anti- preferreeedee images to help in es- timating the tuning for extreme one-tailed units (e.g., response quantile of 0.95)? Consider unit that detects dogs versus other objects. By only training on images of dogs (preferred images), the tuning model may focus on small response differences to dog breed, size, fur, etc., instead of de- termining the features/objects cats, airplanes, fruit, etc.-to to which the unit does not respond (its anti-prefent preferred images). This highlights that even for one-tailed unit, there is benefit in knowing its anti-prefion preferred images (potentially with multiple, diverse features) when estimating its tuning (DiMatrin and Zhang 2008 Cowley and Pillow 2020

---

Published as a conference paper at ICLR 2026
tyychoplysissssk
data evample user
Figure 4: Humans use anti-preferred im- ages to infer V4 tuning. a. Psychophysics task. b. Performance for one example user with or without prior information. Each trace is the running difference between the number of correct and incorrect choices for one V4 model neuron. c-d. Human perfor- mance predicting V4 model neurons (c) and ResNet50 DNN units (d). e-f. Task perfor- mance in for an online classifier trained on visual features (e) or on CLIP embeddings (f). Lines: mean, shade: s.e.m.
price inlo-
one task
Furh > Tatga?
40-
pref-anti-prn pref
select with
larger response
diance
20
40
trial 60
C
human behavior "predkct y4 model neurons)
human behavtor
dasster
clasatter
predict UMN unvts)
(viual festures)
ICLP embedkingg)
pref-nanti pret
Progect
ped only
Picgredi
Picogeci
prior
only
Arepponee
resporse
Aresponge
Aresponce
5 HUMANS USE ANTI-PREFERRED IMAGES TO INFER V4 TUNING.
Access to both preferred and anti- anti-preferred images was most informative for estimating V4 tun- ing (Fig. 3. likely because filters learned detect the salient visual features in both image types. We wondered to what extent humans could access and interpret these visual features. To test this, we ran psychophysies experiment in which human subjects chose one of two images that they thought would lead to a larger response of a V4 model neuron (Fig. 4a, see Methods[A.4) this task was inspired by recent work in explainable AI (Borowski et al. 2020 Zimmermann al.] 2024). Prior to a task, we gave subjects one of four possible sets of reference images: both pre- ferred and anti-preferred images, preferred images only, anti-pred titeferffer images only, and no prior images. Subjects improved their performance via feedback of the correct image after each trial (Fig. 4b). We tested these four conditions for 10 different V4 model neurons and 10 different DNN units (80 tasks total); we found real V4 responses to be too noisy for humans to predict accurately (see Methods (A.4).
The impressive performance of human subjects (Fig. 4e. 80.5% mean accuracy) indicates that the visual feature preferences of V4 neurons are distinguishable and human-interpretable. When given prior access to both preferred and anti anti-preferred images, subjects outperformed other types of prior information (Fig. 4c. preff-aant--reff.) performance for preferred-only and mntiipreffered--nn was comparable no prior images (Fig. c 'no prior' trace). This suggests that subjects relied on both preferred and anti-pre anti-preferred images to infer a V4 model neuron's 's tuning. In contrast, when predicting responses of ResNet50 DNN units, subjects performed similarly when given prior in- formation of either solely preferred images or both preferred and anti mnti-preferre images together (Fig. F.. 'pref.+anti-pref. versus 'pref. only"), ) whereas performance dropped for anti-preferred images (Fig. 4d. 'anti-pref. only'). Thus, subjects did not rely on the anti- anti-preferred images to pre- dict DNN unit responses, suggesting these images-despit their usefulness in estimating a DNN unit's tuning (Fig. d do not have salient, numan-interpretabbl features. This emphasizes an im- portant difference between V4 neurons and current taikedriven DNN models.
What are the visual features that humans use to distinguish between preferred and anti-preferred images? We hypothesized that humans relied on more high-level, interpretable visual features than those typically found in DNN models. To test this, we computed a feature set of 34 interpretable high-level image statisties (contrast, luminance, edge intensity, color, ...) as well as a feature set of 512 embedding variables from the DNN model CLIP (Radford et al. 202T) , and used an line preference algorithm (De Peuter et al. 2024) to train recursive least squares classifier (see Methods A.5) perform the same psychophysies task predict V4 model neuron responses. We found better agreement in performance between the interpretable features and human behavior (Fig. ( than that for the CLIP embeddings (Fig. 4b. This supports the idea that there are human- interpretable differences in visual features between preferred and anti-preferred images, which we explore in the next session.
THE VISUAL FEATURES OF PREFERRED AND ANTI-PREFERRED IMAGES
To assess how visual features differ between preferred and anti preferreed images, we compared 34 high-level, human-interpee interpretable image statistics between the two (Fig. Da. same features as in Fig. (1... For each V4 model neuron, we computed the average difference of each visual feature between its preferred and anti-preferred images (mean over 100 images each). We found large dif-
7

---

Published as a conference paper at ICLR 2026
pref.
anti- pref.
05
Can we predict neuron's anti-preferedd Image
from IS preforred Image?
f
Areanti penaan--ffeeeed features uselul onject iecogntton!
Clale
visual
features
aoure
large
Viuul fostures
-IV --SA
tall double cacadhy fortesture selectivity
line 45'
compresding
line -5"
top-1 aucurccccsy
loy spat
+ +-MVISS-
150
edge kowdiyy
high spat
V1
14
100
large
02
50
50
150 V4 model
100
200
50
100
Figure 5: Anti-preferences double a V4 population's capacity for feature selectivity. a. Vi- sual features differ between preferred and anti-preferred images. b. Mean differences in features that were normalized between O and 1. Lines: medians, dots: V4 model neurons. c. Simulated pre- ferred and anti anti-preferred orientations for a population of V1 neurons. Shuffling these preferences breaks specific relationships. d. Neighbor overlap for predicting the anti-preferred features us- ing preferred features for simulated VI neurons for each condition in c. Same as d but with V4 model neurons using interpretable visual features. f. We assess the extent to which different tailed response distributions perform object recognition. g. Top-1 accuracy as a function of number of V4 model neuron responses. h. Number of preferred-tai units needed to match accuracy of origi- nal (unmodified) units. Traces denote means, shaded areas denote s.e.m.
ferences for individual models (Fig. 5p, large variance of dots for each row), indicating that these visual features are capable of differentiating between preferred and anti-preferred images for an individual V4 model neuron. However, we found little overall mean difference in visual features across V4 model neurons (Fig. 5b. orange lines close to dashed line), suggesting that a visual fea- ture is as likely to be preferred as anti-preferred. This was surprising, as one may suspect, for ex- ample, that high-contrast images would drive larger V4 responses than low-contrast images. While this may be true across randomly-chosen natural images (Sclar et al. 1990), this effect weak be- tween preferred and anti-preferred images (Fig. 5p 'contrast").
After re- re-examining the preferred and anti anti-preferree images (Fig. dd, we wondered if there were any relationships between neuron's preferred and anti-preferred features. In other words, if two V4 neurons had the same preferred feature, would both neurons also have the same anti-preferred feature? We devised the following analysis and first motivate it with simulated population of sim- ple V1 neurons, each with its own preferred edge orientation (equally spaced angles, Fig 5t. top row). Assuming ideal cosine tuning (Hubel and Wiesel 1962, Cavanaugh et al. 2002). each neu- ron's 's anti-prefeee preferred orientation was 90 degrees/orthogona from its preferred orientation (Fig. [ top row, purple versus green lines)-a a clear relationship. To detect this relationship, we considered a held-out neuron and identified its nearest neighboring neurons (based on angle) for its preferred orientation as well as its onti-preferred orientation. We then compute the overlap in neuron identity between the two sets of neighbors (Fig. ccc top row, 'overlap'). ). We then shuffle the relationships between preferred and anti-preferred features across the population- vidually (within-neuron neuron shuffle), for each feature type individually (across- neuron shuffle), both (full-neuroo) shuffle) (Fig. 55. i, ii, and iii). As expected, the VI neurons had a strong overlap for the non-shuffled population (Fig. 5d. "no shuffling") well for the within-neuron shuffled pop- ulation (Fig. 54. i). but this overlap collapsed when shuffling across neurons (Fig. 55d ii and iii)
for each neuron indi-
breaking the relationship of orthogonal orientations for each neuron's 's preferences. Thus, our over- lap analysis can tease apart different relationships between preferred and anti-preferred features.
We performed the same analysis on the V4 model neurons using the Fuclidean distance of the 34 interpretable visual features (see Supp. Fig. for results on CLIP embeddings) to identify the nearest neighbors. We found a striking result: overlap was small for all settings, with little differ- ence in overlap between no shuffling and full shuffling (Fig. 5e). This was consistent with our find- ing that a V4 model neuron strongly activated by the preferred images of another model rarely is suppressed by the other model's 's anti- unti-preferred images (Supp. Fig. (7). Thus, two V4 neurons with similar preferred features will likely have different anti preferred features. Beyond this, our results suggest the preferred and pref-rree features of V4 neuron appear to be randomly and inde-
8

---

Published as a conference paper at ICLR 2026
How many images search?
10k images
100k images
1M images
30M immges
synthesized
Vd modell respont laal
preferred
images
B
preferred
images
images
ImageBeagle the "hunn"" for images
V4 mmmoa respont (au)
search pretereed Image
30M Saox
asuodsai
ImageReagle curver
Inage
through reiifitlors
2 12
random selection runs
pntferrefrreed
Index
pisfemed
anti Peat-poefeeeee image
Suming
images NNVimaaee
30M Fasta
num. images evaluated
response
preferred
Figure 6: ImageBeagle searches the natural image manifold efficiently find preferred and anti-pred mnet-preferred stimuli. a. A V4 model neuron's 's responses to preferred and anti-preferred images after searching through a subsample of K images. mashed-lines: linear fits; z-axis log-scale. Right: Top preferred and anti- onti-preferred images for each K as well as synthesized images. b. Im- ageBeagle navigates the natural image manifold via a nearest neighbor graph. c. Runs of Image- Beagle (orange traces) versus random selection (black traces) searching for the preferred (lop) and anti-preferred (bottom) image. Dashed lines: optimal out of 30M images. d. ImageBeagle tuning curve for V4 model neuron.
pendently sampled from the same "preference" distribution. A caveat is that we only considered 219 V4 model neurons here; these results may change as we gain access to more V4 neurons in the future.
7 ANTI-PREFERRENCES DOUBLE A V4 POPULATION'S 'S SELECTIVITY.
What is the computational benefit of encoding anti-preferred features? We hypothesize that anti- preferred features allow a population to double its capacity for feature selectivity, providing a more flexible readout for downstream neurons. To test this, we assessed the extent to which the popula- tion of V4 model neurons could perform an object recognition task (Caltech-101 (Li et al. (022)) with without access its anti- anti-preferred features (Fig. 5h. We trained adapter DNN to pre- dict the object using either original responses of the V4 model neurons, responses above ReLU threshold (' ('pref. tails only"), responses below ReLU threshold ('anti-pref. tails only"), dom mixture of both ('pref. and anti-pref. tails'). We found that for the same number of V4 model neurons, the original responses substantially outperformed all other settings (Fig. 5lp black trace above others). We quantified this by computing the ratio between the number of preferred-only model neurons divided by the number of original model neurons needed achieve the same accu- racy and found compression ratio of 2.5 (Fig. pp) Thus, encoding anti- trete-reeerred features effec- tively doubles the feature selectivity of a V4 population, allowing for more informative readout. In comparison, the compression ratio was much lower for Rre-ReLU DNN units (Fig 5h. 1.6x), in- dicating that the anti- anti-preferred features of DNN units were not informative of object identity. This highlights obvious mismatch between visual cortical neurons and DNN units and suggests a potential way forward to reduce the number of units in DNN models.
SEARCHING THROUGH MILLIONS OF IMAGES WITH IMAGEBEAGLE
Our work has relied on searching through many natural images to identify preferred and anti- preferred stimuli-how many images are needed? We searched an increasing number of stimuli for a V4 model neuron, and found the number of images had to exponentially scale to achieve a linear increase in response (Fig. 64. left panel). Only at million images did the preferred and anti- anti-preferred images resemble those for 30 million images (Fig. 64. 'IM' versus '30M').
A complementary approach is to synthesize images via gradient techniques (Ponce et al., a 2019: Bashivan et al., 2019a; Walker et al. 2019 Gu et al. [022) Cowley et al. [026]] Willeke et al. 2023 Pierzchlewicz al. 2024 that often identifies images that yield the largest and smallest

---

Published as a conference paper at ICLR 2026
responses (for this V4 model neuron, T'ax 665, m'mn --mi==3.33 but can be difficult interpret versus natural images (Borowski et al.] 2020 Geirhos et al. 22021) and are highly stereotyped (Fig. (op. "synthesized") leading to poor inference of neuron tuming (Fig. 3 In our own experiments, we find that natural and synthesized preferred images drive real V4 responses similarly, but that natu- ral anti- anti-preferred images more strongly suppress real V4 responses than those to synthesized anti- preferred images (Supp. Fig. 5 The synthesized approach also requires technical expertise and dedicated hardware that few neuroscience labs have available. This motivated us to design a sim- ple tool for visual neuroscientists and machine learning researchers to efficiently search through millions of natural images to optimize a desired objective (e.g. minimizing a neuron's response).
We developed a new tool, called ImageBeagle, that efficiently searches through millions of natu- ral images to "hunt" for desired stimulus in a short amount of time. The key intuition is that we traverse through the natural image manifold by visiting each image's i nearest neighbors, moving to the neighbor with the largest objective value (i.e., discrete version of gradient ascent, Fig. (p)) We collected 30M images from diverse image datasets and computed 1K nearest neighbors for each image. Image similarity was defined as the Euclidean distance between DNN features from
ResNet50; we confirmed image neighbors were perceptually similar (Supp. Fig. B ImageBeagle alternates between a global search determined by a coreset over images (Bachem et al. 2018) and
local search that moves to the neighbor with the largest objective value (see Methods A.7]. We
tested the performance of ImageBeagle versus random search and found impressive speed-ups: Im-
ageBeagle often identified preferred and anti- trttepreferredd images with resulting responses close to the 30M-optimum after only 10k evaluated images (Fig. 66. orange traces), substantially outper- forming random selection (Fig. Gc. black traces). We confirmed that ImageBeagle outperformed random selection for models of V1, V4 and IT neurons (Supp. Fig. 3). We also used ImageBea- gle to connect a neuron's 's preferred and anti- anti-preferred images together along a manothhyy-vayying tuning curve (Fig. [d); smoothness is guaranteed because we constrain ImageBeagle to traverse along the image manifold via nearest neighbors. Such an interpretable tuning curve has been dif- ficult to identify because of the nearly infinite paths possible between two images (Pasupathy and Connor, 2001; David et | (2006). We suspect ImageBeagle will be of practical value to visual neuroscientists interested in optimizing neurons' responses (Cowley et al.) 2017b] Walker et al. 2019 Ponce et af. 2019 (Bashivan et al. (2019)), performing closed- experiments with active learning (Lewi al. |2099] Cowley and Pillow 2000) Cowley al. (026), and estimating tuning curves that smoothly vary in stimulus space (Berardino et al. 2017 Wang and Ponce (024). Un- like most model-optimized stimuli, ImageBeagle does not require technical expertise, lowering the barrier for adoption by many experimental labs.
9 DISCUSSION
Our work establishes the existence and importance of anti-preferred images for stimulus tuning in the visual cortex, especially visual area V4. We systematically investigate the properties of anti- preferred images through experimental validation, modeling, human psychophysics, and image statistics. The existence of anti-preferred images is not obvious: Task-driven DNN units, com- monly used to model V4 neurons, often do not exhibit anti- anti-rreferenes due to their ReLU thresh- olding. This suggests that a V4 neuron's 's response less resembles the output of a ReLU and more resembles filter with dynamic range. A V4 neuron may form its two-tail selectivity in part by combining excitatory and inhibitory pre-synaptic input from neurons with one-tailed response dis- tributions (Fig. 2 or combining and thresholding activity within its own dendrites (Beniaguev et al. 202T Our results, consistent with concurrent work optimizing for least activating images of V1 and V4 neurons (Franke et al. 2026),, suggest that only characterizing a neuron by its pre- ferred feature misses critical aspects of the neuron's 's tuming. How two-tailed response distributions and anti- anti-preferred features relate to efficient and sparse coding in the brain (Olshausen et al., 1996) Rozell et al. 2008) remains an open question; two-tailed response distributions may require more energy for spikes, but require fewer neurons to encode rich feature selectivity. To search for anti- preferred images, we developed the ImageBeagle tool that efficiently searches through millions of natural images; ImageBeagle is ready to be used in real-time, closed-loop experiments. Overall, work marks the beginning of quest to identify the role anti- preteerreesssss play in other biologi- cal and artificial visual systems, and improve DNNs inspired by neuroscience principles.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by an NIH grant (F31EY031975) to P.L.S., an NIH grant (R01EY029250) to M.A.S., the Pershing Square Innovation Fund to B.R.C., and an NIH grant (R01EY037194) to M.A.S. and B.R.C.
DATA ACCESSIBILITY
Our V4 data and code is available at http://ggiiin..ommooo//ooggggggr et ageBeagle is available at http::/gghhiuuoooooooooogggygggg
_t__al_____________ee al__222.giit Im-
BROADER IMPACTS
In our work we establish anti-preferred images as an important property of stimulus tuning in area V4. Although there has been extensive research around preferred images, anti-preferred images have largely been overlooked. Thus, our knowledge of the mid-level processing in the ventral visual stream has been limited by the incomplete picture of the neurons' preferences and anti- preferences. We believe this work will inspire experimentalists to investigate the roles of anti- preferred images visual cortical processing and enrich our understanding of the communication between lower and higher order visual areas. Moreover, with ImageBeagle, experimentalists now will have access to millions of images with complementary framework that can efficiently search through these images to optimize general objective function beyond preferred and antiprreferred images (e.g., finding the reddest image).
LIMITATIONS
One limitation of our work is that for our neural recordings from V4, our results reflect only a small subsample of V4 neurons (~~000 out of millions). Thus, preferred and anti-preferred image features may be better distinguished in overall image statistics with more V4 neurons. Moreover, recorded electrode units were likely a mix of isolated neurons and multi-neuron activity (neural units were spike-sorted with dedicated software (Issar et al. 2020)). If the recordings had multi- neuron activity, possibly one neuron could dominate the rest with its preferred and anti anti-preferred image responses. Another limitation is that ImageBeagle was partly curated using images from the web and public datasets. We recognize that images scraped from the web can contain variety of content, some of which might have sensitive information. To minimize the occurrence of this, we sampled from public datasets that use pre-processing steps to eliminate these images. However, similar to any public datasets, advise users to use ImageBeagle at their own discretion. In ad- dition to this, because computing each image's s nearest neighbors out of 30M images can take over year, approximated this graph with random sampling, ensuring that nearest neighbors were perceptually similar to the base image. However, this approximation may fail for certain types of images due to noisy and/or incomplete image graph.

---

Published as a conference paper at ICLR 2026
REFERENCES
Haldan Keffer Hartline. The respouse single optic nerve fibers of the vertebrate eye illumination the retina, American Journal of Physiology-Legag, Content, 121(2):400-155, 1938.
J Y Lettvin, H R Maturana, W S McCulloch, and H Pitts. What the frog's eye tells the frog's brain. Pro- ceedings of the IRE, 1940.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology. 160(1):106, 1962.
David Perrett, Edmond T Rolls, and Woody Caan. Visual neurones responsive to faces in the monkey tem- poral cortex. Experimental brain research, 47:329 342, 1982.
R Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representa- tion by single neurons in the human brain. Nature, 435(7045): 100M455::10001111 102-1107 2005.
Peter Földiák. Stimulus optimisation in primary visual cortex. Neurocomputiing 38:1217 1222, 2001.
Yukako Yamane, Erie T Carlson, Katherine Bowman, Zhihong Wang, and Charles E Connor. A neural code for dimensiimeniinall object shape in macaque inferotemporal cortex. Nature neuroscience, 11(11); 1352-1360, 2008
Gouki Okazawa, Satohiro Tajima, and Hidehiko Komatsu. Image statistics underlying natural texture selec- tivity of neurons in macaque v4. Proceedings of the National Academy of Sciences, 112())E351-E330 2015.
Benjamin Cowley, Ryan C Williamson, Katerina Acar. Matthew A Smith, and Byron M Yu. Adaptive stim- ulus selection for optimizing neural population responses. Advances neural information processing systems, pages 19661106, 2017a.
Charles Cadieu, Minjoon Kouh, Anitha Pasupathy, Charles Connor, Maximilian Riesenhuher, and Tomaso Poggio. A model of V4 shape selectivity and invariance. Journal of neumphysiology. (6(3):1733-1755 17500, 2007.
Reza Abbasi-Asl, Yuansi Chen, Adam Bloniarz, Michael Oliver, Ben DB Willmore, Jack L Gallant, and Bin Yu. The deeptune framework for modeling and characterizing neurons visual cortex V4. bioRxiv, page 465534, 2018
Dean Pospisil, Anitha Pasupathy, and Wyeth Bair. 'artiphysiology" reveals V4-like shape tuning in deep network trained for image classification. Elife, 7:e38242, 2018.
Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret S Livingstone. Evolving images for visual ncurons using deep generative network reveals coding principles and neuronal preferences. Cell, 177(4)999--1099 2019.
Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Sci- ence, 364(6439)eeanv4436 2019a.
Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S Tolias. Inception loops discover what ex- cites neurons most using deep predictive models. Nature neuroscience, 22(12):2060 2066060000066,,,, 2019.
Zijin Gu, Keith Wakefield Jamison, Mecnakshi Khosla, Emily Allen, Yihan Wu, Ghislain St- Yves, Thomas Nasclaris, Kendrick Kay, Mert R Sabuncu, and Amy Kuceyeski. Neurogen: activation optimized image synthesis for discovery neuroscience. Neurolmage, 247:1 18812, 2022.
Benjamin R Cowley, Patricia L Stan, Jonathan W Pillow, and Matthew A Smith. Compact deep neural net- work models of the visual cortex. Nature, 2026.
Konstantin F Willeke, Kelli Restivo, Katrin Franke, Ame F Nix. Santiago Cadena, Tori Shinn, Cate Neal- ley, Gabby Rodriguez, Saumil Patel. Alexander S Ecker, et al. Deep learning- driven characterization of single cell tuning in primate visual area V4 unveils topological organization. bioRxiv, pages 2023-05, 2023.
Pawel Pierzchlewicz. Konstantin Willeke, Ame Nix, Pavithra Elumalai, Kelli Restivo, Tori Shian, Cate Neal- ley. Gabrielle Rodriguez, Saumil Patel, Katrin Franke, et al. Energy guided diffusion for generating neu- rally exciting images. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2026
Binxu Wang and Carlos R Ponce. Neural dynamics of object manifold alignment the ventral stream.
bioRxiv, pages 2024-06, 06, 2024.
David Heeger. Half-squaring in responses of cat striate cells. Visual neuroscience, 9(5):427-443, 1992.
Nicole C Rust, Odelia Schwartz, Anthony Movshon, and Eero P Simoncelli. Spatiotemporal elements of macaque v1 receptive fields. Neuron, 46(6)) 2005.
Jonathan W Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M Litke, EJ Chichilnisky, and Ecro P Simoncelli. tempooltl correlations and visual signalling complete neuronal population. Nature, 454(7207):995 999, 2008.
Anitha Pasupathy, Dina Popovkina, and Taekjun Kim. Visual functions of primate area V4. Annual review of vision science, 6:363-385, 2020.
James R Cavanaugh, Wyeth Bair, and J Anthony Movshon. Nature and interaction of signals from the re-
ceptive field center and surround in macaque v1 neurons. Journal of neurophysiology. 88(5).2500-2566 2002.
Cory Efird, Alex Murphy, Joel Zylherberg, and Alona Fyshe. Finding shared decodable concepts and their negations the brain. arXiv preprint arXiv:2405. 1766, 2024.
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Chris- tensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning frame- 2019.
work for neuroscience. Nature neuroscience,
Adrien Doerig, Rowan P Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W Lindsay, Kon- P Kording, Talia Konkle, Marcel AJ Van Gerven, Nikolaus Kriegeskorte, The neuroconnectionist research programme. Nature Reviews Neuroscience, pages 1-20, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Daniel L.K. Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cor- tex. Nature Neuroscience, 1933)3356336, 2016.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Rov Roy, Franziska Geiger, et al. Brain-score: Which artificial neural net- work for object recognition is most brain-like? BioRxiv, page 407007, 2018.
Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Sci- ence, 364(6439):caav9436 2019b.
Chengxu Zhuang. Siming Yan. Aran Nayebi, Martin Schrimpf, Michael Frank, James DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, (18(3):c0044191111 18, 2021.
Santiago A Cadena, Konstantin Willeke, Kelli Restivo, George Denfield, Fabian H Sinz, Matthias Bethge, Andreas S Tolias, and Alexander S Ecker. Diverse task-driven modeling of macaque v4 reveals functional specialization towards semantic tasks. PLOS Computational Biology, 1015)66 2024.
Adam Kohn. Visual adaptation: physiology, mechanisms, and functional benefits. Journal of Neurophysiol- ogy, 355333155,,, 2007.
Anitha Pasupathy and Charles E Connor. Responses to contour features in macaque area V4, Journal of neurophysiology, 82(5):2490-255 1999.
John HR Maunsell. Neuronal mechanisms of visual attention. Annual Review of Vision Science, 1:373-391, 2015.
Santiago Cadena, George Denfield, Edgar Walker, Leon Gatys, Andreas Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natu- ral images. PLoS Computational Biology, 100699,, 2019.
Najib J Majaj, Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 1402-1:1302213311 2015.
13

---

Published as a conference paper at ICLR 2026
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet: Finding im- portant examples carly in training. Advances in neural information processing systems, 34:20596- 2021.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523-19536, 2022.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence pre- dictions for unrecognizable images. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 427-436 2015.
Christopher DiMattina and Kechen Zhang. How optimal stimuli for sensory neurons constrained by net- work architecture. Newral Computation, 20(3):668-708 2008
Benjamin Cowley and Jonathan W Pillow, High-contrast "gaudy" images improve the training of deep neural network models of visual cortex. Advances in Neural Information Processing Systems, 33:21591-21603, 2020.
Judy Borowski, Roland S Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wicland Brendel. Exemplary natural images explain enn activations better than state-of-the-art feature visualization. arxiv preprint arXiv:2010. 100000.226000 12606, 2020.
Roland Zimmermann, David A Klindt, and Wieland Brendel. Measuring mechanistic interpretability scale without humans. In ICLR 2024 Workshop on Representational Alignment, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-878 8763. PmLR, 2021.
Sebastiaan De Peuter, Shibei Zhu, Yujia Guo, Andrew Howes, and Samuel Kaski. Preference learning of latent decision utilities with human-like model of preferential choice. Advances Neural Information Processing Systems, 37:123608 123636, 2024.
Gary Sclar, John HR Maunsell, and Peter Lennie. Coding of image contrast in central visual pathways of the macaque monkey. Vision research, 30(1):1-10, 1990.
Fei-Fei Li, Marco Andrecto, M Ranzato, and Pictro Perona. Caltech 101 (1.0) set). CaltechDATA, 3 (086): 10-0002, 2022.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 14:23885-23899 2021.
Olivier Bachem, Mario Lucic, and Andreas Krause. Scalable k-means clustering via lightweight coresets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1119 1127, 2018
Anitha Pasupathy and Charles E Connor. Shape representation in area v4: position-specific tuning for bound- ary conformation. Journal of neuropkysiology. 2001.
Stephen David, Benjamin Y Hayden, and Jack I. Gallant. Spectral receptive field properties explain shape selectivity in area V4, Journal of neurophysiology 9666):392-3305 2006.
Benjamin Cowley, Ryan Williamson, Katerina Acar, Mathew A Smith, and Byron Yu. Adaptive stimu- selection for optimizing neural population responses. In Advances in Neural Information Processing Systems, pages 1395-1405, 2017b.
Jeremy Lewi, Robert Butera, and Liam Paninski. Sequential optimal design neurophysiology experiments. Neural Computation, 21(3):619 687, 2009.
Alexander Berardino, Valero Laparra, Johannes Ballé, and Eero Simoncelli. Eigen-distortions of hierarchical representations. Advances in neural information processing systems, 30, 2017.
David Beniaguev, Idan Segev, and Michael London. Single cortical neurons deep artificial neural networks. Neuron, 109(17):277-27333 2021.
Katrin Franke, Nikos Karantzas, Konstantin Willeke, Maria Diamantaki, Kandan Ramakrishnan, Hasan Atakan Bedel, Pavithra Flumalai, Kelli Restivo, Paul Fahey, Cate Nealley, et al. Dual-feature lectivity enables bidirectional coding in visual cortical neurons. el.ife, 15, 2026.
14

---

Published as a conference paper at ICLR 2026
Bruno Olshausen et al. Emergence of simple-cell receptive field properties learning sparse code for natural images. Nature, (6583))) 609, 1996.
Christopher Rozell, Don H Johnson, Richard G Baraniuk, and Bruno A Olshausen. Sparse coding via thresholding and local competition in neural circuits. Neural computation, 20(10):2526 22632, 2008.
Deepa Issar, Ryan C Williamson, Sanjeev Khanna, and Matthew A Smith. A neural network for online spike classification that improves decoding accuracy. Journal of neurophysiology. 123(4):4472 1485, 2020.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James J DiCarlo. performance-optimizze hicrarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 11 (((3)):611986222 2014.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL https:://gtttbbbbbb.bb httpt///o.//aaaaaaaaarrrrbbttata
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1556, 2014.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59 (2):64-73, 2016.
Johannes Mehrer, Courtney Spoerer, Emer C Jones, Nikolaus Kriegeskorte, and Tim C Kietzmann. An ecologically motivated image dataset for deep learning yields better models of human vision. Proceedings of the National Academy of Sciences, 118(8):=201 1417118, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv: 708. 00489, 2017.
Seong Tae Kim, Farrukh Mushtaq, and Nassir Navab. Confident coreset for active learning in medical image analysis. arXiv preprint urXiv:2004.02200 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Pro- ceedings of the IEEE international conference computer vision, pages 3730-3738, 2015.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking ma- chine learning algorithms. arXiv preprint urXiv:1708.077473 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Ng, al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsuper- vised feature learning, volume 2011, page 4. Granada, 2011.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale bench- mark for instace-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2575 25844, 2020.
Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery datasct for text-to-image generative models. arXiv preprint arXiv:2210.1483 2022.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Leamning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 8697-8710, 2018
Tianye Wang, Tai Sing Lec, Haoxuan Yao, Jiayi Hong, Yang Li, Hongfei Jiang, Ian Max Andolina, and Shim- ing Tang. Large-scale calcium imaging reveals a systematic map for encoding natural scenes. Nature Communications, 15(1):6001, 2024.
15

---

Published as a conference paper at ICLR 2026
A
METHODS
In Methods, we provide details for our V4 experiments, linear mapping analysis, data pruning analyses, human psychophysics experiment, and ImageBeagle dataset and algorithm.
A.1 V4 EXPERIMENTAL DATA
Most of our analyses of V4 responses and V4 model neurons come from data collected in our re- cent work (Cowley et al. 2026); further validation experiments novel this work were collected during the same recording periods. See Cowley et al. (026) for full experimental details. These experimental procedures were approved by the Institutional Animal Care and Use Committee of Carnegie Mellon University and were performed in accordance with the United States National Research Council's Guide for the Care and Use of Laboratory Animals.
Specifically, we used this dataset to compute skewness of V4 responses to natural images (Fig. D--- c), predict V4 responses from ResNet50 embeddings with different mappings (Fig.
22), and as-
sess excitation and suppression of preferred and anti-preferred images (Fig. (p)) We ran addi-
tional closed-loop validation experiments for the anti anti-preferred images (Fig. 2b) interleaved in the recording sessions from Cowley al. (2026), allowing us to re-use the same compact models of V4 neurons. We also re-used these compact models (termed V4 model neurons in our work) as surrogates for real V4 neurons, as we found them to be more predictive of V4 responses than task- driven DNN models (Cowley et al. 2026). We analyzed these V4 model neurons for our data prun- ing analysis (Fig. BD), psychophysics experiments (Fig. 4e), visual feature analyses (Fig- 5), and evaluating ImageBeagle (Fig. 6). Below, we briefly describe data collection of the V4 responses.
Macaque V4 neural data collection: We implanted a 96-electrode array in the left hemisphere of macaque visual area V4, one in each of two macaque monkeys (identifiers PE and RA). We ex- tracted spike signals via an automated algorithm (Issar et al.] 2020) that separated spike waveforms from noise on each electrode channel. For each recording session, the awake, head-fixed animal performed thousands of active fixation trials until satiated (typically ~223 hours). Each trial com- prised 6~6-8 image flashes (~1000 ms each) interleaved with 100 ms gray blank screens (to reduce adaptation and interaction effects between image flashes); image size and location were chosen to cover the receptive fields of the recorded V4 neurons (8-11 1 visual degrees in diameter at the recorded eccentricity). After maintaining fixation throughout the sequence of images, the central dot disappeared and a target dot appeared at random location 10 away from the central dot; ani- mals received liquid reward for correctly making saccade to the target dot. Each recording ses- sion had ~1,000 unique images and typically greater than ~ repeats per image (image repeats shown randomly throughout the session).
V4 model neurons: We used the data-driven compact models trained in our previous work (Cow- ley et al. 2026) as our 'V4 model neurons'. These compact models were obtained by recording I0 sessions per animal to train the data-driven model, called the 'ensemble model'. This en- semble model was highly predictive of V4 responses to held-out natural images. The compact models were identified by distilling and pruning the ensemble model; one compact model was identified per V4 neuron. We then fixed the weights of these compact models, and searched for their preferred and anti-preferred images. To do this, we passed ~500,000 natural images through the compact models, and kept the 10 preferred and 10 anti-preferred images for each V4 neu- ron (for the second animal, we had 12 images each). We then presented these images, along with > 750 randomly-choo -hhosen natural images, in the following recording session. Because we could not guarantee that we record from the exact same neurons between sessions (a small number of neurons were lost and added between sessions due to small shifts in electrode positions), we had to match up the V4 model neurons (i.e., the compact models) the recorded V4 neurons the new session. To do this, we computed the predicted R2 between each V4 model neuron and each V4 neuron using the responses to the randomly-chen ing the pair with the highest R² (and removing the chosen model neuron and real neuron as future candidates). Then, for each V4 neuron, we take the median response of the model-optimized pre- ferred images "pee and anti- anti-prefrreed images Tanti pref, and compute the fraction/quantile q to which these median responses are either larger (preferred) or smaller (anti-preferred) to responses for the randomly-chun chosen images, e.g., 9pret = + > I(rprer > rı). where I is the indicator function and i denotes the ith image from the N randomly-chosen natural images.
natural images, and kept greedily choos-
peet.
16

---

Published as a conference paper at ICLR 2026
ResNettO-robust activity (pre-relu)
b
VGG19 activity (pre-relu)
C
AlexNet activity (pre-relu)
LRL (our mappingl
0.50
rolee corrected ge
rote: corneceed y2
0.5
0.45
0.40
0.40
8.40
typical linear mapping
0.55
0.35
0.35
0.301
0.50
0.301
Sctiaatons
pre
pee- post- negated
pee- post- negeted
activtioos
Felu
selu
felu
iduu
19UU
160u
activations"
pre- post- negated pre-
post- regeted
using features to predict real neural response
using flipped to predict neural response
notre correced g9
rollo rolie corected
0.46
LRL mappingl
typical linear mapping
0.38
0.38
andomly flipping half
0f fedture sight
0.34
achuty
pre-rolu
post-rolu
regated pre
post-relu
relu
Rester50 (pre-relu)
pre-reu
post-relu
nnggtee
pre-reu
postrielu
relu
Supplementary Figure I: Additional DNNs for linear mapping analysis (top row), and flip analysis showcasing the flexibility of LRL mapping (bottom row). a. We extend our mapping analysis from Fa. with ResNet50- Robust, b. VGG19, AlexNet. d. We repeat analysis from Fig Fa. and e. randomly flip the signs of ResNet50 features and try to predict V4 responses for typical and LRI. mapping.
A.2 LINEAR MAPPING ANALYSIS
For our linear mapping analysis (Fig. 2a), we used the pre-ReLU activations from ResNet50 ["conv4_block4___
layer) as input to all 6 methods, and denote that as "pre-ReLU", For this analysis, we use the four test sessions from Cowley et al. (2026): Session ID 190923, 201025, 210225, and 21 1022. We keep the same train, test, and validation sets for the final comparisons across all mappings (80% train, 10% test, 10% validation for each session, randomly chosen). Each of the i-vi models were the following:
model i (pre-Rel.U): We linearly map the Pre-ReI.U features to V4 responses with ridge regres- sion.
model i (post-ReLU); Similar model i, except we add ReLU activation functions prior to the
linear mapping. This is the output typically used in identifying linear mappings for V4 neurons
Yamins et al. 2014 Bashivan et al. 2019a Cowley al. 2026 Cadena et al., 2024).
model ii (negated ReLU): Similar model ti, except we negate the signs of rre-ReLU activations before passing them through the ReLU activation functions.
model iv: We linearly combine filter channels via a convolution with kernel shape 1 x where the output channels are equal in number to the input channels. The resulting linearly combined ac- tivity is first passed through a LayerNorm and then passed through ReLUs until finally linearly mapped to V4 neurons.
model v: Similar to model iv, except we pass as input post-ReLU activity instead of pre-ReLU activity.
model vi: Similar to Method iv, except pass as input negated ReLU activity.
To ensure that our results hold across different DNNs, we repeated this analysis for ResNet50- robust (Engstrom et al., 2019). AlexNet (Krizhevsky et al. 2012). and VGG19 (Simonyan and Zis- serman 2014), and found that all were consistent with results for ResNet50 (Supp. Fig c-c.. Importantly, one of the advantages of our monaanao -oneer (LRL) mapping is that is ag- nostic to the skewness of the responses. In other words, regardless of the two-tailed or one-tailed responses, the LRL mapping can maintain high prediction performance whereas this is not the case with regular linear mapping (Supp. Te).
17

---

Published as a conference paper at ICLR 2026
a
V4 model neurons
b
ResNet50 DNN units
random
pref.+anti-
0.61
pref.tanti-pref.
pref. only
random
0.8
0.5
pref. only
non-pref.
R
R
0.4
0.3
anti-pref. only
anti-pref. only
0.2
0.2-
non-pref.
10,000
150,500 number training images
300,000
10,000
150,500 number of training images
300,000
Supplementary Figure 2: Extended data pruning plots. a. We train a DNN (5-layer CNN) to predict responses of individual V4 model neurons (10 in total), here with 300,000 training images versus the 10,000 images in F.. Traces denote means, shaded areas denote I s.e.m. b. Same as in a except for predicting responses of individual ResNet50 DNN units (10 in total). Traces denote means, shaded areas denote s.e.m.
A.3 DATA PRUNING ANALYSES
For our data pruning analyses, we ran simulations assess the information content of the anti- preferred images by including or leaving out these images when we estimated a neuron's 's tuning. We used V4 model neurons to serve as surrogate ground truth models of V4 neurons, as record- ing a real neuron's 's responses to 500k images is infeasible, and the chosen V4 model neurons were highly predictive of real V4 neurons (Cowley et al. 2026). We then used these surrogates as "teacher" models to train "student" models (5-layer CNN architecture, 100 filters per layer) with different training curricula. We focused on training the student models with less than 10k images, since the most extreme preferred and anti- anti-preferred images likely occur in the first Ik images so. We trained each model from scratch with training images varying in number from 1 to 10k at 1 intervals (Fig. BD) We used the same ceacher-student training procedure for ResNet50 units (Fig. (3).. Extending the number of training images did not change the results (Supp. Fig 2 Below detail the pre-training and training procedures.
Pre-training details: Prior to training, for every V4 model neuron, we sorted each model's re- sponses to 500k images; we defined the top K images as the preferred images and the bottom K images as the anti-preferred images. We sought to quantify the extent to which preferred and anti-preferred images contributed to our estimate of that model's 's tuning. We designed the follow- ing different training curricula (corresponding to the traces in Fig. 3D), where K refers to the se- lected number of image-response pairs on which to train.
preferred images only: We selected the K images that had the largest responses.
anti-preferred images only: We selected the K images that had the smallest responses. preferred and anti- treffrreeereed images: We selected K/2 images that had the largest responses and K/2 images that had the smallest responses.
preferred and anti- inti-preferred images (I million): We considered entirely different set of im- ages that was double in size to our baseline dataset (1M here versus 500k for the other cur- ricula). All other details were the same as for 'preferred and anti-preferred images'. randomly-chosen images: We randomly selected K images from the pool of 500k images. non-preferred images: We first found the median response and selected the K images with re- sponses closest to the median response (i.e., K/2 images with responses below the median response and K/2 images with responses above the median response).
synthesized images: We synthesized K K/2 images to maximize the model's output response and synthesized K/2 images to minimize the model's 's output response. Synthesized images were optimized with gradient ascent/descent techniques (Bashivan et al. 2019a Walker
ct al. 2019 Cowley et al. 2026).
The chosen curricula, except for randomly-choo chooeee images, will likely lead to biases such that the training and test data distributions will not match (i.c., nut-of-diitribution To mitigate such bi- ases, we replaced 10% of the images for each curriculum with randomly-selecter images (replac- ing the images with the smallest responses for preferred images and the images with the largest
18

---

Published as a conference paper at ICLR 2026
responses for antti-prefered images). We also note that as the number of training images increases, training on randomly-seless selected images outperforms other curricula as expected, as the training dis- tribution matches the test distribution. Lastly, added Gaussian noise (with standard deviation 0.2) to the V4 model neuron's 's responses to mimic noisy estimates of repeat-averaged responses. We used the same training and curricula for predicting ResNet50 units (Fig. 3 found these units needed more training data than the V4 model neurons to reach similar values of Rt (Supp. Fig.
Training details: Across models, the training images were sampled from the same pool of 500k images; these 500k images were randomly sampled from ImageBeagle dataset (see AA6) com- prising 30M images. For testing and validation, we sampled 10k images from ImageBeagle for each (images different from the 500k images). We trained the model using the ADAM optmizer with learning rate le-4, early stopping (based on validation data), and used a batch size of for ResNet50 and 64 for V4 model neurons.
A.4 HUMAN PSYCHOPHYSICS EXPERIMENT
We performed human psychophysics experiment to test how human subjects rely on preferred or anti anti-preferred images to guess a neuron's 's (or model unit's) tuning (Fig(E). The subject pool com- prised volunteer scientists with no compensation. These experimental procedures were approved by the Cold Spring Harbor Laboratory Institutional Review Board under project "Crack This Neu- ron" [2282250].
The task was as follows: Given a pair of images, the subject is instructed to select the image that would lead to the larger response of a chosen neuron/model. The user's score and the number of times the user picked the correct answer were displayed above the prompt to show their progress. Importantly, the subject was given feedback after every decision via green box around the correct answer and red box around the wrong answer. This feedback was necessary for subjects to learn to correctly perform the task. Each task comprised 100 pairs of images. Prior im- ages, if provided, were always 36 images in total; these images were separate from any queried image pairs. To ensure that each task had an equal mix of difficult- and diise--iiisei image pairs, we took 5 bins of image pairs with increasing response differences (i.c., the larger the re- sponse difference between two images, the easier the discrimination). The first bin had images that were very close in value (A response ~ 0.1), and the last bin had images that were very far apart (A response 0.5). Each task had 20 image pairs from each bin (100 total) which were randomly ordered across the task. To create these bins, for each model we extracted the responses, and cal- culated all possible response differences and saved the sorted differences in an array. We then used 20th-80th percentiles of this array to compute bin edges to create our 5 bins and filled these bins with non-overlapping images (c.g., if an image appeared in a pair, it cannot be used for another pair), until each bin had 20 images.
Task details:
Task types: For each task, we used one of 10 units/neurons from either V4 model neurons or ResNet50 units. This created total of 40 tasks for each model (4 'prior' conditions x 10 model neurons/units), resulting in 80 tasks total for on subject. For ResNet50, lected units from a mid-layer (layer 33, with 1,024 filters). For V4 model neurons, we used 10 ran- domly selected units from 219 V4 model neurons. In pilot dataset, we also attempted the task for real V4 neurons, but found the responses too noisy and likely too few images (~,1,000 images per recording session) for humans identify meaningful selectivity. Our goal was to investigate how humans use prior images (or lack thereof) to guide their decisions. The priors were as follows:
used 10 randomly se-
no prior: In this condition, no additional images were shown to the subject. Thus, the subject had to rely heavily on the feedback from the task to guide and improve their deci- sion.
preferred prior: In this condition, we showed the 36 preferred images of the unit. We refer to these images as "maximizing" in the experiment make it more intuitive for the subjects. Tasks with this condition allow the subject to utilize this prior information by selecting the image from the pair that is most similar to these images.
anti-preferred prior: In this condition, we showed the 36 anti-preferred images of the unit. We refer to these images as "minimizing" in the experiment to make it more intuitive for the subjects. Tasks with this condition allow the subject to utilize this prior informa-
19

---

Published as a conference paper at ICLR 2026
tion by selecting the image from the pair that is not similar to these images. This condi- tion tends to be more challenging compared to preferred prior because here the subject is given information about the lower response images (anti- (anti-preferred), but not the higher. Therefore, the user still has to figure out what the maximizing images are through feed- back and process-of-eliminaton strategies.
preferred and anti-preferred prior: In this condition, we showed the user 18 preferred images and 18 anti-preferred images of the model neuron/unit.
A.5 CLASSIFIER ALGORITHM
We used a recursive least squares (RLS) algorithm (De Peuter et al., 2024 mimic human psychophysics experimental setup (Alg[. where given an image and any prior information, the classifier updates its beliefs through trials. Here, we define the task preference learning ob- jective where the classification is based on pairwise feature comparisons and it sequentially up- dated based on feedback (e.g. correct/incorrect. RLS algorithm allows for efficient online updates through recursive closed-form solutions, thus aligning closely with the decision making process in the human psychophysics task.
Algorithm Recursive Least Squares Classifier
Require:
Feature vectors paref. J panti-pree pref. target labels y/., forgetting factor X, prior type S,
number of trials T Require: Initialize:
w : weight vector determines the position in the classification hyperplane
P = I: covariance matrix
5 = None then initialize w with prior features else randomly initialize w
for itrial + 1 to I do
if random() < 0.5 then
+- p - f1 pantt pref.
1 1: pref.
y1 +-
else
panti yret. - prett
21
M prat. p
+ end if
v/ <- 2yr
Prediction:
8: + w
location in the boundary
0: + I[s > 0) # prediction
Update:
Pa <- Px. #ucertaintt
ke + xt=, P.
p
# how strongly should the current trial be weighed
6x < V/-
w 4- w + kre, classifier weights updated to reduce error
P p k.n P updated uncertainty
end for
A.6 IMAGEBEAGLE DATASET AND NEAREST NEIGHBOR GRAPH
ImageBeagle relies a large bank of millions of diverse images. We collected images from var- ious popular public datasets, such as Flickr Creative Commons dataset (Thomee al. 2016). Ecoset (Mehrer et al.) (222)) and Duckduckgo, in addition to other sources across the web (see Table I for approximate number of images extracted from each source). In addition to these, we also created artificial stimuli that are of interest to neuroscience, such as bars, gratings, colorful let- ters, gaudy images (Cowley and Pillow 2020), and so on. To ensure consistency across the dataset, randomly cropped and resized each image to 224 224 RGB PNG file. We stored 30 million images in 1,500 zip archive files, where each zip file contained 20k images; we chose zips for easy access and transfer. For easier download and access, created Bealle with 1M im-
20

---

Published as a conference paper at ICLR 2026
ages publicly available to researchers. The full ImageBeagle dataset is large (~2 TB) and available by request to the authors.
Nearest neighbor graph: For our 30 million images, we desired each image's 1k nearest neigh- bors; an image's ' neighbors form the local natural image manifold surrounding this image. To ob- tain the neighbors, we sought the images with the smallest Euclidean distance between activations from a middle layer of ResNet50 (units come from layer 33 of ResNet50). We chose ResNet50 features, they are predictive of V4 responses (Schrimpf et al. 2018 Cowlly et al. 2026), but also as they nicely capture perceptual similarities both for high-level (semantic) and low-level (spa- tial) statistics (Supp. Fig. 3l). We down-samples sampled the large tensor of activations via a spatial av-
erage pool (from 14 x 14 x 1, 024 to 3 x 3 X 1, 024 with pooling kernel of 4 x 4 and a stride of 5). Computing the distance matrix of 30M images computationally intensive; to mitigate this cost, we approximated these calculations by randomly initializing the 1K nearest neighbors and continuously updating them by computing distances for random subsets of images. We ran this for ~50 hours with H100 GPUs. We confirmed that this similarity metric led to perceptually similar neighbors and effectively searched for preferences for VI, V4, and IT neurons (Supp. Fig. 3))
A.7 IMAGEBEAGLE SEARCH ALGORITHM
The goal of the ImageBeagle search algorithm is to identify the image that yields the largest value for an objective function given by the user. ImageBeagle consists of 2 steps: Local search and global search. For the local search, we start at a chosen image, compute the objective value for its nearest neighbors, choose the neighbor with the largest objective value, and repeat until the chosen image has an objective value larger than the values of its neighbors. For the global search, we con- sider L images of coreset (Sener and Savarese 2017 Bachem et al. 2018 Kim et al. 2020). A coreset is a subset of images that spreads out over the entire image manifold. Typically, one needs access to the full distance matrix compute coreset; here, we approximate the coreset by start- ing at a randomly-choser image, and adding to the coreset the image with the largest distance from this chosen image based on ResNet50 features, found by a local search. We repeat this local search (each time searching for the image that maximizes the smallest distance between itself and each of the coreset images) until we have 10k perceptually-disimill images. This ensures that we explore diverse regions of the image manifold. Because this coreset is an approximation, we independently computed 10 different coresets for ImageBeagle use.
The full ImageBeagle algorithm (Alg. 2 alternates between the global and local searches to ex- plore the image manifold efficiently for a given objective function. The user defines a budget of the total number of evaluations allowed (i.e., computing the objective value for each image); Image- Beagle stores the objective value for every previously-evalle
image for future queries, ensuring images are not re-evaluated. For the global search, we use the next L images of the coreset and cy- cle through coresets whenever all 10k images of the current coreset are evaluated. The local search begins using the previously-evalutee neighbors have not yet been evaluated). We perform M local searches (with fresh restarts) before performing another global search. This continues until B images have been evaluated (the budget).
image with the largest objective value (and whose nearest
21

---

Published as a conference paper at ICLR 2026
Algorithm/ 2 ImageBeagle algorithm
Require: millions of images and 1k nearest neighbors for every image
Require: 10 coresets of 10k images each
Require:
hyperparameters:
L: number of coreset images searched at each global step
K: number nearest neighbors to evaluate per local step
M: number of local searches
B: total number of images to evaluate (budget)
Require: ((x): Rxxxx3 R: objective function with input image of shape p p
Set: num_eve evals + O, Xevalunted f [], Xvisited f- [] # empty lists
while len evaaluaal)) < <Bor do
global search
Xcoreset f- next L images in coreset; remove these images from coresct if coreset empty then move to next coreset
X evaluated [
[X coresel] # combine lists
local search
for isearch t to M do
X-visited t X evalunted Xvittted
Xnext +- arg maxxmaxxx0 ( X -isitted)) choose starting image of local search
Xvisited +- [istted + X
Kecurest neighbors -- K nearest neighbors of Xnext
# walk through nearest neighbors
while max( (fXX nearest neighhons) )) > (Xnex)) do
Xevahuated f- [Xevllunted] + ne neighbors] # neighbors have been evaluated
Xnext f- arg max (X (eighhes)))))
X visited [Xvistted] + [Xeex]
Xncarest neighbors + nearest neighbors Xnext end while end for end while
Xoptimal farg max 4 (Xovalunted]
ImageBeagle
Source
Approximate Amount
Flickr Creative Commons dataset Thomee et 2016
million
Ecoset Mehrer et al. (2021)
1.5 million
CIFAR Krizhevsky al. (209
120,000
CelebA dataset et al. (0015
202,000
Caltech-256 dataset Grifiin et al. (2077
30,000
Fashion MNIST dataset Xıao al. (2017)
60,000
SVHN dataset Netzer et al. (2011)
248,000
Google Landmarks dataset Weyand al. 2020
4.1 million
DiffusionDB Wang et al. (2022
million
Duckduckgo
155,000
Flickr
2 million
YouTube
million
Artificial stimuli
3.1 million
Others
5.5 million
Table I: Summary of image sources for ImageBeagle
22

---

Published as a conference paper at ICLR 2026
base image 10 nearest neighbors in ImageBeagle (ResNet50 embeddings)
b
ImageBeagle across visual cortex
V1 model neurons
ImageBragle nuns
top row: preferred Image
selection runs
mocil Prormall Phormalles
20k images
bottom row: anti proferred image
V4 model neurons
IT model neurons
neuron index
Supplementary Figure 3: Example ImageBeagle neighbors and searching for preferred and anti-preferred images for neurons from areas across visual cortex. a. Randomly-chosen ex- ample 'base' images and their 10 nearest neighbors based on distances of embeddings from our chosen DNN (ResNet50). b. Each base image and neighbors (a) are perceptually similar low- level statistics (textures, colors, ...), allowing ImageBeagle to be useful in identifying preferred and anti-preferred images for neurons different visual cortical areas (e.g.. V1, V4, and IT) that may differ in receptive field size. To test this, we considered computational models of neurons from VI, V4, and IT (compact models from Cowley et al. (2026)); the V4 and IT models are currently some of the most predictive models of V4 and ITT neurons for their respective datasets (Schrimpf et al.] 2018). For each model neuron, we used ImageBeagle to identify the preferred and anti-preferred image (Supp. Fig. 3b, orange traces), comparing this search with random selec- tion (Supp. Fig. b. black traces). We found ImageBeagle to perform remarkably well across vi- sual cortical areas (Supp. Fig. 3p. all orange traces above black traces in top rows and below black traces in bottom rows), indicating that ImageBeagle often identified images that lead to more ex- treme responses with fewer image evaluations than those of random selection. That ImageBeagle performs well across visual areas likely stems from the ability of ResNet50 embeddings to identify perceptually similar images (a). This suggests that ImageBeagle goes beyond V4 and generalizes to visual neurons in both lower- and higher-order visual areas.
BrainScore
23

---

Published as a conference paper at ICLR 2026
B APPENDIX
B.I MULTI-UNIT ACTIVITY AND ANTI-PREFERRED IMAGES
To record neurons in our experiments, we used a Utah multi culti-electrode array, which captures the activity of both single- and multi-units Identifying well-isolated single units by analyzing spike waveforms is possible, but one can never be certain the unit is truly single neuron in this setting. Here, we argue that multi-unit activity cannot largely explain the existence of anti-preferred im- ages. This is for two reasons:
First, we analyzed separate dataset of V4 responses to natural images from Cadena et al. (2024) that used NeuroPixel probes (NeuroNexus 1Ix32-Edge Perdge-00mm 10mm-60- 1amm--60177 60-177) to record neural activ- ity. An advantage of NeuroPixels is that the electrode channels are much closer together ( (50jm than those of the Utah array (400pmm); one can isolate single units based on coincidence timings of spikes between channels. The authors also performed extensive spike sorting to ensure well- isolated single units. A caveat is that images were presented with only one repeat and in rapid succession without interleaved blank screens- responses are considerably noisier than our analyzed
V4 responses. We kept neurons with SNR of at least 0.5 (split-repeat analysis). The median skewness of the Cadena dataset was K = 1.377 (mean firing rate- = 8.76 spikes/sec). We found tight relationship between mean firing rate and skewness: For neurons with firing rates > 10 spikes/sec, the skewness was K = 1.06, and for neurons with firing rates > 15 spikes/sec (similar our analyzed V4 data), the skewness was 0.852, matching closely to our observed x = 0.87 (Fig. [e). That neurons with lower firing rates had higher skewness is unsur- prising in this dataset due the Poisson nature of spike counts. Because each image was repeated single time, estimating the true response distribution is not possible. However, we believe that the low skewness of the well-isolated high-firing neurons is interesting, as this need not be the case. That the skewness values match well with our observed ones when controlling for firing rate fur- ther confirms the presence of two-tailed response distributions for V4 neurons.
Second, because multi-uni activity is thought to be additive, it is likely the case that the image that maximizes the response of the multi-unit likely maximizes an individual neuron within the multiple neurons that comprise the multi-unit. This assumption is often made by recent studies that optimize preferred stimuli of V4 and IT neurons (Bashivan et al.] [019a]] Ponce et al.) 2019 Cowley et al. 2026 Pierzchlewicz et al. 2024). With similar logic, assuming each unit has an anti- preferred stimulus, identifying the anti-preferred stimulus of a multi-unit is akin to minimizing
the response of one of the individual neurons. However, if the units had one-tailed distributions, adding the responses of enough of these units together would likely yield a Gaussian distribution (based on the central limit theorem). To test how many units would need to be added together, we added DNN units with larger skewness (K ~ 5), and found that we needed ~ 400 DNN units to match the skewness of observed V4 neurons (x: 0.87) -his is unrealistic for a real multi- unit, which likely comprises only two or three neurons. The summed responses of three DNN units (over 100 runs) yielded an average skewness of k = 1.87, much larger than the skewness for V4. We found that the anti-pre preffrred images for the multi- units of 3 DNN units and above did not have discernible shared visual features versus the perceptually-sinn simillr anti- anti-prfferre images of the real V4 neurons.
B.2 SKEWNESS FOR TASK-DRIVEN DNN UNITS
In Figure I we considered the skewness of one task-driven DNN (ResNet50). Here, com- pare the skewness of four task-driven DNNs to that of V4 neurons (Supp. Fig. 4.. Specifically, wanted investigate how the highly predictive ReLU layers of popular DNNs organized their responses compared to ResNet50. To do this, we measured the population skewness from 4 pop- ular DNNs that are known to be predictive of V4 responses, Xception (Chollet) 2017) (728 units, from "block10 _strooos_s___l_____ 1 act' layer), AlexNet (Krizhevsky et al. 2012) (384 units, from 'conv3' layer), VGG-19 (Simonyan and Zisserman 2014) (512, from "locks_ccov2"" layer), and NASNet- Mobile (Zoph et 20188) (528 units, from 'activation_ 104' layer) (Supp. Fig. -dd.. While units from AlexNet, Xception, and VGG-19 overall appear to be more skewed than those of V4 neurons (median x 2.02, median 0.94, median = 1.34 respectively), surprisingly for NASNetMobile (median A = 0.44) we found that to be not the case (Supp. Fig. 41). Despite the ReLU activation, units in NASNetMobile organize the responses in way that preserves the two-tailedness of the distribution. and thus effectively exhibit a linear/pre-Be Reeer behavior. We suspect that this is caused by the high baseline activations where these units rarely operate in the zero-output regime, hence
24

---

Published as a conference paper at ICLR 2026
b
V4
Supplementary Figure 4: Skew- ness K. of V4 nen- rons against four different DNNs. a. Skewness k. of response distribu- tions for V4 neu- rons from Fig Dc. and AetxNet units. b. Same but with Xception units. Same as a but with VGG- 19 units. d. Same as a but with Nas- NetMobile units. Lines: medians.
0.6
neurons
0.8
V4 neurons
density
density
0.4
AlexNet
Xception
0.4
0.2
units
units
0
0
6 skewness,
skewness,
d
1.2
V4
neurons
NASNetMobile
0.8
density
0.8
density
0.8
units
0.4
VGG- 19 units
0.4
V4
neurons
2
6
0
6
skewness,K
skewness,K
creating two-tails. However, thresholding is still present in the model, as nontrivial fraction of activations still fall below zero in some layers. This finding hints at the importance and effects of architectural designs of DNNs in their selectivity. Compared ResNet50, AlexNet, VGG-19, and Xception, NASNetMobile utilizes modular cell structure where each cell combines the outputs from previous layers with 'add' operations, thus accumulating the activations from multiple layers. This accumulation of activation increases the baseline activations, leading to ReLU outputs that are always "on". Although the residual skip connections are present in other DNNs such as ResNet50, we suspect the frequency of these additions in NASNetMobile is what leads them to have higher cumulative pre-activations.
B.3 COMPARING THE EFTECTIVENESS OF MODEL-OPTIMIZED NATURAL VERSUS SYNTHETIC IMAGES
The two main approaches identify preferred images of V4 and IT neurons have been either
consider a large number of natural images (Cowley et al. 2017a 7a 2026 Willeke et al. 2023 Wang
al. 2024) or synthesize images via
(Bashivan et al. 2019a] Ponce et al.) Cowley et al. 2026 Willeke et al. 2023 Pierzchlewicz et al. 2024). In our closed-loop
2019
experiments (Fig. 2 p). we had the opportunity to compare the effectiveness of both approaches. Before the validation experiment, we optimized the responses of V4 model neurons by search- ing for their preferred images either by a large pool of 500,000 natural images (Supp. Fig. F 'chosen by model from natural images') (Supp. Fig. Fag 'synthesized by model'). We then presented both sets of images in a following recording session and compared the V4 responses to each, after establishing which model neurons matched which V4 neurons (see Methods Section I.I. We found no significant difference be- tween model-synthesized and model-chosen images for preferred images (Supp. Fig. 5p, bottom),
by synthesizing the image via gradient techniques
despite salient differences in image statistics between the two sets (Supp. Fig. 5h. top), suggest- ing both approaches are comparable in eliciting large responses and share common visual features. However, for the anti- nntiepreeerred images (Supp. Fig. 5p. top). we found that the model-chosee nat- ural images elicited responses significantly smaller than those for the model-synthesizee images (Supp. Fig. 5bb, bottom). This suggests that there is model mismatch between its predicted anti- preferred images versus the real neuron's 's anti-preferred images; how visual features differ between these model-chosen natural images and the model-synthesized images may provide insight into these differences. This result also highlights that one should not disregard natural images for their ability to drive both large and small V4 responses. Further work is needed to systematically com- pare different approaches for bused and synthesized based approaches for identifying both preferred and anti-preferred images for different visual cortical areas.
25

---

Published as a conference paper at ICLR 2026
model-optimized preferred images by model natural images
b model-optimized anti-preferred images
chosen images
syntheszed model
synthesized by
150
animal 1 animal
animal 1
animal 2
model ceel-chosenn natural images
model--hosen natural iimages
synthesized
response (spikes/sed
120
response (spikes
100
1.00
synthesized natural
one
40-
40-
(real data)
(real data)
20
90
100
140
160
20
80
300
140
160
response (spikes/sec) synteeiyy images
response (spikes/sec) mynel-sytthesized images
Supplementary Figure Experimental validation model-chosen natural images and images. a. Top panel: Example response-maxinizing images either chosen by the V4 model neurons from large pool of 500,000 natural images (top row) or synthesized by the model via gradient techniques (bottom row). After optimizing these images, perimentally validated them by recording V4 responses to these images a following recording session. Bottom panel: Real V4 responses to the predicted preferred images by the V4 model neu- rons. Responses were repeat-averaged. Each dot denotes one neuron; neurons are from two ani- mals, indicated by color. The dashed unit line denotes equal responses between model-chosen and model-synthesized images. The difference in means was not significant ('n.s.' p 0.514, two- tailed permutation test, n 130 neurons), suggesting neither approach elicited larger responses than the other. b. Same as in a except for anti preffrredd images. The difference of means was sig- nificant (asterisk, p < 0.02, two-tailed permutation test, n 130 neurons). indicating that the responses to model-chosen natural images were smaller than those for the model-synthesized im- ages (i.e., dots below dashed line).
then
a
model
b
anti-pref.
Vi model neurons
prot images
antipreef
14 neurons
prof, Images
d
antippref images
units
Reshlet-50
prof, Images
1.
densitya 2a
nourons
0.0
skewness,
response (a.u.)
response (a.u.)
response (a.u.)
Supplementary Figure 6: Additional examples of preferred and intieprefeered images from 3 ResNet50 units and V4 model neurons. a. Skewness of response distributions for V4 neurons and V4 model neurons. b. Anti-preferred and preferred images of V4 model neurons. c. Anti- preferred and preferred images of ResNet50 units.
26

---

Published as a conference paper at ICLR 2026
a
b
Preferred images
Anti-preferred images
quantile
quantile anti unte-rreferred
V4 model neurons
V4 model neurons
-0.8
0.8
-0.6
-0.4
0.4
-0.
-0
different 100 preferred images
different prti-preeerred images
0.8 A quantile, preferred
Supplementary Figure 7: Preferred images not shared across V4 model neurons. a. Re- sponses of every V4 model neuron (rows) to every other V4 model neurons' 100 preferred images (columns). b. Responses of every V4 model neurons every other V4 model neurons' 100 anti- preferred images. The mixture of pink and green indicates that some anti-preferred images were close to being preferred images of other models. Differences in quantiles for the preferred im- ages matrix, and their corresponding differences in quantiles in the anti pilffrreeered images matrix. The dashed red line represents the unity line (y=x).
B.4 LITTLE RELATIONSIIIP BETWEEN PREFERRED IMAGE SIMILARITY AND ANTI-PREFERRED IMAGE SIMILARITY ACROSS MODELS.
To further investigate whether there is shared structure between preferred and anti-preferred im- ages for given V4 model neuron, we fed the sets of 100 preferred images of all V4 model neu- rons, and recorded the responses (we repeated this process for anti-p anti-preferred images as well). In order to normalize responses across all V4 model neurons, the model's true preferred/anti -aeeerreeee images by using quantiles. For each set of preferred/anti- preferred images, we took the median and computed how many images out of the 500k had re- sponses lower than this median. We normalized this number by the total number of images to get the quantile response.
scaled the responses proportional to
Here, a quantile of 1 indicates that the preferred images of the ith model also the preferred im- age of the jth model, and quantile of indicates that the anti- anti-preferred image of the ith model also the anti-preferred of the jth model. We further assessed whether two models with similar preferred images would also have similar anti-preferred images. For every row in Supp. Fig 7a we calculated the absolute difference of quantiles between the model with the highest quantile and the model with the second highest quantile. We did this for the farthest quantiles (highest quan- tile lowest quantile) and 20 randomly-chosen quantiles (highest quantile randomly selected quantiles). We computed the corresponding models' differences from the anti-preferred matrix and compared these A q against each other. Here, we find that although some models have sim- ilar preferred and anti- treferreeered images (lower bottom left corner in Supp. Fig 7p), others do (Supp. Fig 7c. top left corner, bottom right corner). Overall, most images did not fall on the unity line indicating that there no linear relationship. These results further support our findings of no apparent structure shared across preferred and anti-preferred images (Fig. 5)
27

---

Published as a conference paper at ICLR 2026
V4 model neurons (visual features)
b
V4 model neurons (CLIP embeddings)
a
overlap
0.8
overlap
0.8
0.4
0.4
no shuffling (i)
(ii)
(iii)
no shuffling
(ii)
(iii)
Supplementary Figure Predicting anti-preferred images based on nearest neighbors of preferred images. a. Neighbor overlap for predicting the anti-preferred features using preferred features with V4 model neurons using 34-dim interpretable features. b. Same as a but using 512-
dim CLIP embeddings instead.
28


Reviews:
[{"review_id": "WNFlEgfdfm", "reviewer": "Reviewer_Bncs", "summary": "This paper revisits the classic notion of neuronal selectivity by demonstrating that neurons in the primate visual cortex, particularly area V4, exhibit two-tailed response distributions: responding not only to preferred stimuli that excite activity but also to distinct anti-preferred stimuli that suppress it. Using electrophysiological recordings, modeling, and psychophysical experiments, the authors show that these anti-preferred images are structured and informative, not merely blank or featureless, and that incorporating them significantly improves decoding and predictive models of neural responses. Human participants also rely on both preferred and anti-preferred examples to infer neural tuning, whereas deep neural networks, constrained by ReLU activations, typically capture only one-tailed preferences. The study further introduces ImageBeagle, a tool for efficiently identifying preferred and anti-preferred stimuli across millions of natural images. Overall, the paper establishes anti-preferred stimuli as a key component of neural encoding in visual cortex, expanding the representational capacity of neurons and suggesting new directions for biologically inspired model design.", "questions": "Your analysis focuses on mappings from ResNet-50 features to V4 activity. Have you tested whether similar two-tailed response patterns appear when using other architectures (e.g., self-supervised or non-ReLU models) or in other visual areas such as IT or V1?\n\nIn Figure 3, training on anti-preferred stimuli alone performs about as well as random sampling. How should we interpret this in light of your claim that anti-preferred stimuli are an important part of neural encoding? Do they carry independent information, or mainly complementary information when combined with preferred images?\n\nCould you elaborate on the possible biological mechanism that gives rise to these two-tailed response distributions? For example, do you suspect inhibitory circuitry or normalization processes in V4 as the source of the anti-preferred suppression?\n\nYou show that combining pre-ReLU features improves prediction. Have you tried using activation functions other than ReLU (e.g., tanh or leaky-ReLU) to see whether two-tailed behavior naturally emerges in artificial networks?\n\nThe ImageBeagle tool seems promising. Can you clarify how well it generalizes across different neurons or datasets? For instance, would it work for V1 or IT recordings, or for neurons modeled with non-DNN representations?\n\nIn the human experiments, participants performed better when shown both preferred and anti-preferred images. Do you think this reflects an actual perceptual analogy to how V4 represents visual information, or is it more an artifact of the task design?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript is easy to understand and well-organized. It provides  intuitive figures and explanations that make complex concepts accessible to both neuroscience and machine learning audiences.The paper presents a good amount  of well-controlled experimental results combining neural recordings, computational modeling, and psychophysics, which together provide strong empirical support for the existence and importance of anti-preferred stimuli in visual encoding. The authors also contribute a valuable open-source tool, ImageBeagle, which enables researchers to efficiently identify preferred and anti-preferred stimuli across large natural image datasets, potentially accelerating future neuroscience studies. In addition,", "weaknesses": "Although the paper provides strong evidence for two-tailed neural responses, it mainly examines a single model based on ResNet-50 features and their mapping to V4 activity. It remains unclear whether the same results would generalize to other architectures or visual areas. The finding that combining high- and low-activation images improves decoding is somewhat expected, since training with a broader range of responses typically enhances prediction accuracy. What is more surprising is that anti-preferred stimuli alone do not provide more predictive power than random images (Fig. 3), suggesting that the information they convey is only useful when combined with preferred responses. Finally, while the experimental work is extensive, several analyses rely heavily on model-based predictions rather than fully independent data, which somewhat limits the strength of the causal conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M31XjRFKz3", "reviewer": "Reviewer_iGeu", "summary": "A large body of work exists on studying the properties of stimuli that strongly (or even 'maximally') excite a neuron in visual cortex. The manuscript at hand extends this approach by exploring anti-preferred stimuli that lead to very weak neural responses. The tuning properties of V4 neurons, as well as those of other areas are shown to exhibit a two-tailed distribution, showing that both highly preferred as well as anti-preferred stimuli exist. This is not mirrored by artificial neural networks. The authors derive an alternative mapping from DNN units to real neurons which yields better predictive performance. It is shown that models perform best when both preferred and anti-preferred stimuli are included during training. The authors describe the results of a psychophysics experiment showing that humans can better predict neural responses to novel stimuli when given both preferred and anti-preferred stimuli as a reference. Finally, the authors propose a method to more efficiently sample images to determine the strongest and weakest activators, yielding large computational savings.", "questions": "How does ImageBeagle transfer across brain regions? Does the graph need to be recomputed for different regions as the similarity should be computed at different DNN layers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. Studying anti-preferred stimuli is an important step towards capturing neural tuning properties more globally, beyond exceptionally strong activators.\n2. The results of the neural data analysis convincingly demonstrate the two-tailed response distribution.\n3. The discussion of ReLU-based DNNs in this context is interesting and shows a clear computational difference between artificial neural networks and the primate visual system. \n4. The psychophysics experiment yields interesting insights, showing that anti-preferred stimuli have additional merit for interpreting a neuron's tuning.\n5. ImageBeagle demonstrates substantial benefits over random image sampling. The motivation for the method, as well as the discussion of synthetic MEIs is well-rounded and it is made clear to the reader how ImageBeagle combines the advantages of natural-image sampling with computational efficiency of MEIs.\n6. The work is well presented; Figures are clear and intuitive and it is easy to understand the results. The work is well motivated and the main story is clear.", "weaknesses": "I am not fully convinced by the results in sections 3) and 4). I would raise the following points:\n\n1.\n1.1 The existence of anti-preferred stimuli suggests that neural responses do not follow a ReLU-like response pattern. Based on this fact, the work argues that a linear map from ReLU-thresholded DNNs units (standard in the field) should be suboptimal. I find this hypothesis to not be well motivated in the text. The manuscript clearly states that anti-preferred images contain some anti-preferred feature which suppresses responses. As the post-ReLU units act as one-tailed detectors for preferred features (demonstrated nicely here!), I see no reason why a map like this should be suboptimal. If a feature is anti-preferred, a ReLU unit detecting that features could simply be assigned a negative weight in the linear map. I would ask the authors to clarify the motivation for this section. \n1.2 Further, the experiment conducted to prove the hypothesis does not seem that convincing to me. Mixing the features linearly, applying a ReLU and then a final linear map increases the capacity of the readout model, which could explain better performance without any real ties to the presented hypothesis. The fact that the control experiment with removed intermediate ReLUs does not increase performance does not seem surprising as that procedure again yields a linear map. I therefore find the claim in line 198f to be very strong given the data. \n\n2. In Section 4, it seems to me like the experiment is biased towards finding pref+anti-pref to be the best condition. If only (anti-)preferred images are shown, we would expect the average response on the training data to be higher/lower than on the test data, which would artificially reduce R^2. Therefore, it would be expected that 'random' outperforms either of the two biased training sets and I could imagine that the superiority of 'pref+anti-pref' is a methodological artifact. I would strongly suggest that the authors reproduce the results with a mean-independent statistic like correlation between predicted and true responses.\n\nSome further points:\n\n3. Much of the analysis (e.g. Section 5) is based on digital twins rather than real neural data due to noisiness. While this limits the conclusions one can draw a little bit, I realize that this is a difficult issue to address and do not see it as a big problem.\n4. ImageBeagle, as a strong contribution of the paper, is not explained thoroughly enough in the main text, in my opinion. I agree it makes sense to put small details in the appendix, but in the current form it is difficult to understand from the main text alone. I would kindly ask the authors to include some more detail in the main text.\n5. For ImageBeage specifically, it is unclear how strongly the computational savings are overestimated by evaluating on the digital twin model rather than real neural responses.\n\nOverall, I do like this work and the contributions are strong. However, I would ask the authors to consider the points I've raised regarding some of the methodology. If these are addressed, I would increase my score and argue the paper should be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qKZQDsHH1F", "reviewer": "Reviewer_qR7N", "summary": "This paper overturns the traditional “one-tail” view of visual neuron selectivity by showing that many macaque V4 neurons exhibit two-tailed responses, they react strongly to both preferred and anti-preferred stimuli. These anti-preferred images are rich and structured, not blank, and are essential for accurately modeling neural tuning. Incorporating both stimulus types improves encoding performance and aligns with human perception, unlike ReLU-based DNNs that show only one-tailed selectivity. The authors also introduce ImageBeagle, a scalable tool for discovering preferred and anti-preferred stimuli across millions of natural images.", "questions": "* The improved tuning performance using both preferred and anti-preferred images may mainly result from greater response diversity, instead of the features of the images. Random images yield weaker predictions likely because their responses are less extreme, but outperform when more data are available. The claimed connection between this result and the neuron’s tuning, specifically that preferred and anti-preferred images reveal identifiable visual features learned by the model requires further verification.\n* Could the authors clarify what is being predicted in this analysis? Are they training a model to predict the activation responses of individual ResNet-50 units to images? If so, is this model evaluated with actual measured biological responses to validate this comparison? \n* The ImageBeagle algorithm searches through nearest neighbors in image feature space without incorporating neural objectives. How should we interpret the “response curve” produced during this search as reflecting neural response changes? Is the method assuming that smooth interpolation between nearby images in feature space implies a correspondingly smooth interpolation of neural responses? If so, could the authors provide justification or evidence supporting this assumption?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This work provides an interesting analysis of the preferred and anti-preferred stimuli from macaque V4 region\n* The modeling and analysis are robust. Author use multiple experiments: encoding models, data pruning experiments, and psychophysics tasks to show that both preferred and anti-preferred stimuli are essential to fully capture a neuron’s tuning curve.\n* The author released ImageBeagle, an efficient large-scale natural image search tool. Provides practical value to visual neuroscientists interested in optimizing neurons’ responses", "weaknesses": "* The discovery of “two-tailed” response distributions is presented as novel, but prior work has long recognized that neurons exhibit both excitatory (maximizing) and suppressive (inhibitory) responses to different stimuli. The paper does not clearly distinguish how its findings go beyond this established understanding.\n* The comparison with linear-nonlinear (LN) models and task-driven DNNs is somewhat misleading. Their one-tailed response patterns largely reflect architectural constraints (e.g., ReLU activation), not an intrinsic inability to model two-tailed selectivity. More recent encoding models [1] can predict both positive and negative response values, so the contrast with “biological two-tailed neurons” is overstated.\n* The neuron tuning analysis relies on a single model type and uses only the R² metric to assess performance. This is insufficient to robustly support the claim that including anti-preferred stimuli improves tuning prediction. Additional analyses on more encoding models and metricstrained with the same set of data would strengthen the evidence.\t\n* Although the paper analyzes anti-preferred images from several perspectives—neural recordings, modeling, and psychophysics—the overall impact of these analyses is limited. Beyond showing that incorporating both preferred and anti-preferred images can improve encoding model performance in low-data regimes, the study provides little insight into how anti-preferences fundamentally contribute to neural computation or representation.\n\n[1] Aria Y Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nature Machine Intelligence, 5 (12):1415–1426, 2023. 2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mp0OocJns1", "reviewer": "Reviewer_vbPL", "summary": "This is a very interesting paper that combines unique experimental data with modern computational techniques to fundamentally make the a potentially groundbreaking observation about neurons in the visual cortex - that is their tuning properties cannot just be characterized by what they prefer (i.e., are excited by), but there is an independent set of stimuli that effectively suppress the activity of a neuron and it is important to identify these anti-preferred stimuli. They show various results to support this conclusion and present a new tool for neuroscientist to use in order to efficiently identify preferred/anti-preferred stimuli. I enjoyed the paper and I think it has important scientific contributions to offer. I don't believe, however, the ICLR venue is the right audience. I know that with a liberal interpretation, one can consider this paper in the scope of the conference. I believe the ICLR paper should either push ML forward or show a new application of an ML method. I think this paper might be considered in the latter category, but the main message is squarely in neuroscience.", "questions": "1) Can you spell out what novel insights you can offer for the ML community?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Solid experimental results. \nVery useful and impactful neuroscientific insights. \nA public tool for other neuroscientists to use in order to effectively find anti-preferred and preferred stimuli.", "weaknesses": "The main weakness in my opinion is that there is little contribution to core or foundational ML. There is also no meaningful innovation on the applied side, where an existing ML tool is applied to a novel problem or a state-of-the-art result is achieved on some standard problem. I also found the style of writing and the format and flow very confusing. It follows the typical general-audience high impact journal format (like Nature), where details of methods and experiments are buried in the back of the paper. To me, a good ICLR paper highlights these up front. In fact, those are supposed to be main contributions. \n\nI found Section 6 to be unconvincing and confusing. I really counldn't follow how the results amounted to the conclusion that \"knowing a neuron’s preferred images gives little information about what a neuron’s anti-preferred images will be\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iGYZCvBW25", "reviewer": "Reviewer_c1B5", "summary": "This paper studies the representation of “anti-preferred” visual stimuli—images that suppress neural activity rather than increase it—in the macaque visual system. Using electrophysiological recordings, data-driven modeling, image synthesis, and human psychophysics, the authors report that many macaque V4 neurons show two-tailed response distributions, whereas units in standard deep neural networks (DNNs) display one-tailed, ReLU-like activation patterns. This indicates that cortical neurons can encode information through both excitation and suppression, while DNN units represent information only through positive activations. The study also proposes a modified readout that combines pre-ReLU features before a nonlinearity, improving predictions of neural responses. The manuscript is clearly written, well organized, and presents high-quality figures.\n\nThe paper’s main strengths are its clear presentation, strong experimental design, and integration of neuroscience and computational modeling to address an important question at the intersection of biological and artificial vision. The main weaknesses concern the fairness of the readout comparison, the interpretation of the pruning and psychophysics analyses, and limited discussion of prior and related work, as well as the lack of broader connections to artificial vision systems.", "questions": "Was a capacity-matched control readout tested—for example, a variant applying the same mixing and normalization operations after the ReLU—to isolate the specific contribution of pre-ReLU access to negative features?\n\nCan the authors clarify whether the psychophysics effects were stronger for neurons with two-tailed response distributions, or whether similar effects occurred for sparse or one-tailed neurons as well?\n\nHow robust are the findings to the specific image synthesis method used? Would on-manifold synthesis (e.g., diffusion-guided optimization) produce comparable preferred and anti-preferred examples?\n\nHow do the authors view the implications of their results for representation learning in artificial systems? Could introducing bidirectional or opponent-like feature tuning improve generalization or interpretability in DNNs?\n\nCould the authors comment on how their findings relate to the recent bioRxiv preprint (2025.07.16.665209) reporting suppressive feature selectivity in visual cortex?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The study is technically sound and clearly presented. It addresses an important question at the intersection of neuroscience and machine learning by contrasting biological and artificial visual representations. The results suggest that two-tailed neural tuning may provide a richer representational basis than the strictly positive codes used in DNNs. The experiments are well executed, and the integration of electrophysiology, computational modeling, and psychophysics is well thought out. The figures are clear and effectively illustrate the main results.", "weaknesses": "The comparison between the proposed readout and the standard linear mapping could be made clearer. While the authors compare against standard post-ReLU readouts, it is not fully clear whether a capacity-matched baseline—applying the same operations (1×1 mixing, LayerNorm, ReLU) after the ReLU—was included. Such a control would help determine whether the observed advantage truly reflects the benefit of pre-ReLU mixing, which allows access to both positive and negative features, rather than increased model flexibility due to the additional operations. Including or explicitly discussing this control would make the interpretation more convincing.\n\nThe image synthesis procedure is described clearly, but the discussion could address the limitations of optimizing stimuli far from the natural image distribution. Several recent approaches, such as diffusion-guided activation maximization or Fourier-phase–constrained optimization, maintain proximity to the natural image manifold while exploring feature space. Citing and briefly discussing such work would situate the present synthesis method within the broader literature.\n\nThe data-pruning analysis requires a more careful interpretation. The authors conclude that including anti-preferred images is essential for estimating neural tuning, but this improvement may result from greater coverage of the response range rather than from the special status of anti-preferred images. For neurons with sparse selectivity, non-preferred images still provide valuable contrast for estimating tuning, even if there are no distinct anti-preferences. Acknowledging this possibility would clarify the mechanism underlying the observed effect.\n\nSimilarly, the psychophysics results are informative but should be interpreted with caution. It is unclear whether the benefit of including anti-preferred images was specific to neurons with two-tailed response distributions or whether any inclusion of low-response examples helps learning. The results from DNN simulations suggest that the latter could be true. If so, the human data may demonstrate a general advantage of diverse examples rather than direct evidence for two-tailed encoding. Rewording this section to reflect this distinction would improve clarity.\n\nThe work would also benefit from a more complete engagement with prior neuroscience literature. Selective suppression has been described previously in both V1 and V4. In addition, a recent preprint (bioRxiv: 2025.07.16.665209) reports closely related findings on the role of suppressive features in shaping visual representations. Discussing this work and clarifying how the present study differs in methodology and conclusions would strengthen the positioning of the paper within the current literature.\n\nFinally, given that this paper is submitted to ICLR, the discussion could better connect the findings to implications for artificial vision systems. Two-tailed selectivity may suggest useful design principles for artificial models, such as incorporating balanced excitatory and suppressive representations to improve robustness or generalization. A short discussion of such implications would make the paper more relevant to the machine learning community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)