Feature segregation by signed weights in artificial vision systems and biological models
Authors: Giordano Ramos-Traslosheros
Carlos R Ponce
Venue: ICLR 2026 Poster (2026)
Keywords: ventral stream
circuit mechanisms
interpretability
deep learning
visual system
excitation inhibition
neuroscience
closed-loop optimization
ablation
Primary area: interpretability and explainable AI
TL;DR: Neural networks trained on ImageNet segregate the object/foreground features of their output layer to the positive input weights, with similar behavior in visual neurons.

Abstract: Signed connectivity is fundamental to neural computation in both brains (excitatory/inhibitory) and machines (positive/negative). Yet the role of signed weights in shaping visual representations in object recognition remains unclear.
Dale's Law, the biological principle that neurons send exclusively excitatory or inhibitory outputs, is typically not enforced in artificial neural networks (ANNs). 
Here, we find that accuracy in ImageNet-trained ANNs correlates with the spontaneous emergence of sign-specific  "Dale-like" segregation in their output layers.
Ablation and feature visualization reveal a functional segregation in ANNs: removing positive inputs primarily disrupts localized, object-related structure, while removing negative inputs alters mainly dispersed background textures. 
This segregation is more pronounced in adversarially robust models, persists with unsupervised learning, and vanishes with non-rectified activation functions.
We validate these observations in the macaque ventral visual cortex (V1, V4, and IT) using encoding models and in vivo feature visualization.
The features recovered by encoding models qualitatively matched those identified in vivo.
Model representations changed more upon positive than negative input ablations.
We analyzed the most Dale-like units across neuron models, positive units showed localized features, while negative units showed larger, more dispersed features.
Consistent with this, experimentally clearing the background around a neuron's preferred feature enhanced its response, likely by reducing inhibitory drive.
Our results suggest that both artificial and biological vision systems segregate features by weight sign: positive weights emphasize object-related features, while negative weights refine context. This highlights a convergent representational strategy in brains and machines, yielding predictions for visual neuroscience.

Published as a conference paper at ICLR 2026
FEATURE SEGREGATION BY SIGNED WEIGHTS IN NEURAL NETWORKS FOR ARTIFICIAL AND BIOLOGICAL VISION
Giordano Ramos-Traslosiin Department of Neurobiology Harvard Medical School Boston, MA 021 15, USA g gio0hms ha www.d.a...mmm edu
Carlos R. Ponce
Department of Neurobiology
Harvard Medical School
Boston, MA 02115, USA
carlos ponce hms harvard.edu
ABSTRACT
Signed connectivity is fundamental to neural computation both brains (excita- tory/inhibitory) and machines (positive/negatiie). Yet the role of signed weights in shaping visual representations in object recognition remains unclear. Dale's s Law, the biological principle that neurons send exclusively excitatory or inhibitory out- puts, is typically not enforced in artificial neural networks (ANNs). Here, we find that accuracy ImageNet-traiin
ANNs correlates with the spontaneous emer- gence of sign-specific "Dale-like" segregation in their output layers. Ablation and feature visualization reveal functional segregation in ANNs: removing positive inputs primarily disrupts localized, object-related structure, while removing nega- tive inputs alters mainly dispersed background textures. This segregation is more pronounced in adversarially robust models, persists with unsupervised learning, and vanishes with non-rectified activation functions. We validate these observa- tions in the macaque ventral visual cortex (V1, V4, and IT) using encoding models and in vivo feature visualization. The features recovered by encoding models qual- itatively matched those identified in vivo. Model representations changed more upon positive than negative input ablations. We analyzed the most Dale-like units across neuron models, positive units showed localized features, while negative units showed larger, more dispersed features. Consistent with this, experimen- tally clearing the background around neuron's preferred feature enhanced its re- sponse, likely by reducing inhibitory drive. Our results suggest that both artificial and biological vision systems segregate features by weight sign: positive weights emphasize object-related features, while negative weights refine context. This highlights convergent representational strategy in brains and machines, yielding predictions for visual neuroscience.
INTRODUCTION
Brains and artificial neural networks (ANNs) both rely on signed connections. In biological cir- cuits, Dale's ' law states that neurons are either excitatory or inhibitory (Dale 1935). Excitatory neurons are thought compute core visual features, while inhibitory neurons sharpen selectivity, modulate context, and gate information flow (Isaacson & Scanziani 2011). The primate ventral visual stream supports object recognition (DiCarlo et (012]. and representations from V1 IT increase in complexity. paralleling hierarchies in convolutional neural networks (CNNs) (Yamins) al.) 2014 Güçlü van Gerven 2017). While early mechanisms such as lateral inhibition and center-surround receptive fields are well charachterized, the excitatoryinnbbitory organization of higher ventral-stream areas remains less understood (Tamura et al., 2004), motivating the question of whether information is segregated by connection sign in both artificial and biological vision.
ANNs also use positive and negative weights, loosely analogous to excitatory and inhibitory signals but unconstrained by Dale's 's law. Yet it remains unclear how deep networks divide visual information across signed weights, particularly in object classification models where each output unit represents category. Prior work suggests segregation by absolute weight strength (Li et al. 2023), but it is

---

Published as a conference paper at ICLR 2026
unknown whether feature types such foreground objects versus background context systemat- ically separated by weight sign. Here, we hypothesize that CNNs segregate visual information into positive and negative imputs.
We test this hypothesis across diverse ImageNet-trained CNNs by analyzing weight sign organiza- tion, ablating positive and negative inputs, and visualizing the resulting feature selectivity. We asses feature segregation across network depth by analyzing Dale-like channels. We examine a range of architectures and training regimes, including adversarially robust models, unsupervised models, and networks with nonrectified (Tanh) activations.
For biological comparison, we fit linear models from ANN features to neural responses recorded across the macaque ventral stream (V1, V4, IT) and use in vivo feature visualization and background manipulations to probe inhibitory contributions. These analyses relate ANN-derived sign structure to biological feature preferences.
Our results support an emerging principle: across artificial and biological vision systems, connection sign organizes feature representation. Positive weights emphasize object-related, localized features, whereas negative weights encode contextual, more dispersed structure. This connects classic ideas rooted in Dale's law with representational strategies in modern ANNs, generating mechanistic and testable predictions for visual neuroscience.
2 RELATED WORK
Mechanistic interpretability of computer and biological vision There has been progress in mech- anistic interpretability in ANNs from work using perspectives adapted from neuroscience circuit dissection (Olah al. 2020). This line of explainable Al research explains model behavior by dis- secting smaller network subgraphs, revealing how relevant features arise from input weights and are composed hierarchically. Such work has uncovered motifs involving positive and negative connec- tions between related features, reminiscent of early visual system organization. New approaches to address representations beyond single units rely on sparse dictionary learning, with early work in vision (Olshausen & Field 1996), an approach that has recently regained popularity in language modeling (Cunningham al. 2023). , as well as in multimodal models (Pach et al., (025). Some studies also characterized object shape and texture biases in feature visualizations by reconstructing images from sparse weight sets (Li et al., 2023). However, the systematic division between posi- tive and negative inputs across the entire range of weight strengths, and its possible role in feature segregation, remains underexplored and a focus of this study.
Feature visualization by closed-loop optimization Characterizing learned representations is foun- dational for both biological and artificial vision research. Feature visualization, i.e. optimizing for images that strongly activate target units, was originally pioneered in the brain by hand (Hubel & Wiesel 1959). and later in silico by gradient ascent on pixels neural networks (Erhan et al.) 2009] Nguyen et al. 2016a b) Olah et al. (017)). Because gradients unavailable when recording in vivo, gradient-free black-box optimization techniques were developed for synthesizing preferred images of biological neurons in real-time (Ponce et al.) [019 Xiao & Kreiman 2020 Wang & Ponce 2022).
These approaches constrain the search space via generative adversarial networks, promoting natu- ralistic solutions (Nguyen et al., 2016a). Further methods involve first fitting predictive network to neural data and then using in silico gradient ascent (Bashivan et al., 2019 Walker et al.) 2019).
While most prior studies use grayscale images, our study applies gradient- free visualization to color images in both CNNs and primate recordings.
Robustness Neural networks are susceptible adversarial attacks, where noise that is nearly imper- ceptible by humans can be added to natural images, changing output classification (Szegedy et al.) 2014 Salman et al.] 2020 Elsayed al. 2018). Robust training, i.e. introducing adversarial per- turbations during learning, improves resistance to such attacks and is hypothesized to align learned representations more closely with primate visual processing. Prior work does not assess how robust- ness impacts the organization of image representations after ablation of signed weights, which systematically investigate here.
Nonlinearity influence representations Beyond training objectives, the role of activation func- tions such as ReLU versus Tanh profoundly influences representational properties (Alleman et al.] 2023), with ReL.U inducing representations better aligned with input features and Tanh inducing
2

---

Published as a conference paper at ICLR 2026
alignment with output features (labels). This prior work was done in small networks from theo- retical perspective; thus, the impact of rectification on the potential segregation of visual features at practical scales is unknown and addressed by study.
METHODS
An extended methods section is in the Appendix A.1
Networks We performed our ablation studies in CNNs pretrained on the ImageNet dataset: AlexNet (Krizhevsky et al. 2012), VGG16 (Simonyan & Zisserman 2015), ResNet50 (He et al. 2015), and robust 50 models with robustness radii specified by the Lox. norm (L E (0.5, 1, 2, 4, 8]; Salman et al. 2020). Here, Loo denotes the maximum-nore constraint used during adversarial train- ing, l.e. | ac c, which bounds the maximum per- eer-pixel perturbation by To reduce computing time, we used the imagenette dataset (noa) 2024) and the ImageNer macaque category. We also tested 100 classes sampled by k-means the output of ResNet50 (Figs. [2 For all networks, visualized the representations of the units in the fully-connected output layer (pre-softmax) matching those classes under different ablation conditions.
Dale index We quantified Dale like structure with a Dale index for each outgoing channel per layer, defined as the fraction of its weights that share the majority sign, D max(p+,PP. p-). where p4 and p- are the proportions of positive and negative outgoing weights. The index ranges from 0.5 to and measures sign consistency.
Ablation For each layer, we ablated positive and negative weights separately. Given a layer's weight matrix W. we defined the sets of positive weights / {w W w >0} and negative weights N = f E W w < 0}. For each set S E (P. N}, we sorted its elements in decreasing order of absolute value. We then defined the ablation strength a E [o, 10 as the fraction of the total magnitude of 5 to remove. Specifically, we identified the smallest k satisfying Σ |wal / ΣwES |wo| 2 a, where W1, W..... are the sorted weights in S, and set those k weights zero. Because normalized cumulative magnitude, it lies in [0, 1], and sweeping from 0 to I removes none to all of the positive (or negative) weights.
a
Feature visualization For each ablation condition, we performed feature visualization by optimiz- ing GAN latent code create an
image. We used this closed-loop. zeroth- order-search approach to allow comparison with our neuronal experiments, where gradient ascent would not be possible. To increase the span of the stimulus space, we used two GANs: AlexNet fc6 DeePSiM (Dosovitskiy & Brox 2016) which can render textures and objects, and BigGAN [Brock et al. 2019) that can render photo- realistic images with objects. For optimization, we used variant of covariance matrix adaptation evolutionary strategy or CMAES (Wang & Ponce 2022 Loshchilov] 2015). We optimized ten images per GAN, resulting in 20 feature visualizations per output unit and ablation condition. Diverse visualizations better capture the multifaceted high-level representations in CNNs (Nguyen et al. 2016b). For our examples, we show the best of the 20 vi- sualizations, but used all for quantitative analyses. For visualizations of neural networks predicting biological neuron responses, due to experimental time restrictions, we used five visualizations per ablation condition, via DeePSim only. Our experiments are performed in PC with Nvidia 4090 GPU, and each visualization takes about 3 mins.
Network training Both ResNet18 networks were trained using the FFCV library (Leclere et al., 2023) for 16 epochs on the ImageNetIK dataset. The top-5 classification accuracy was 0.797 for the network with Tanh activations and 0.870 for the network with ReLU activations. Note that these models were trained for only 16 epochs rather than the standard 90, so their accuracy underperforms published benchmarks. However, they are suitable for our mechanistic analyses.
Visual cortex electrophysiology We recorded multi-unit (neuron microcluster) and occasional single-unit activity from chronically implanted multielectrode arrays in VI, V4, and PIT of two macaques. Animals fixated while 2-8 images were flashed briefly (100 ms ON, 150 ms inter- stimulus interval). Neurons were driven with a 160-image stimulus set (diverseSet), constructed to span broad range of visual features derived from k-means clustering of AlexNet output layer over the INIK validation set and typical visual neuroscience image sets. For each session, we modeled responses of single neuron multiunit using one-component PLS regression between firing rates
3

---

Published as a conference paper at ICLR 2026
and AlexNet penultimate-laye layer activations, and used the resulting model for in silico ablations and feature visualizations, validating visualizations in vivo within the same session.
4 RESULTS
4.1 PROXIMITY TO DALE'S LAW IN CNN OUTPUTS CORRELATES WITH ACCURACY
A key challenge in comparing artificial and biological circuits is that CNNs are not constrained by Dale's law. There- fore, we asked whether the Dale index (our measure of sign consistency) in the output layer of diverse CNNs has any re- lation their performance. Dale index increased from random initialization with training (Fig [A]. Moreover, top-1 accu- racy on ImageNetIK positively correlated with the mean Dale index of the output layer. Within a given architecture, the Dale index increased with network depth. And batchnorm training in VGGs pro- duced output layers with higher Dale in- dex. Thus, even without explicit Dale constraint, these networks naturally devel- oped more sign consistent output chan- nels. This motivated us to examine the specific visual features carried by positive versus negative weights.
r=0.61, pr6.Be-02
Dale Index changes with training
80
HEA densenet1er 189
Outpin Layer Dale Index
75
Jensenet 19 AB
Berioo_nn@reeeeeeeeeeee resneest
Top1 Accuracy
70
resnet18
0.56
65
60
alloret
0.50
Untrained Trained
0.56
0,58
0.60
Output Layer Dale Index
Figure 1: Dale index of output layer correlates with top I accuracy on ImageNet CNNs. A. Training in- creases Dale index of output layers. B. Network top accuracy correlates with Dale index of the output layer.
4.2 OBJECT INFORMATION PREFERENTIALLY ENCODED POSITIVE WEIGHTS
Hypothesis Motivated by biological center-surround organization, where inhibitory surrounds con- vey contextual information, we hypothesized that CNN output units trained for object recognition segregate object features to positive weights and contextual features to negative weights.
Testing segregation by ablation and visualization. We examined this hypothesis in ImageNet- pretrained CNNs using ablation and feature visualization. The overall ratio of positive to negative input weights per unit was close to unity (Table 2 suggesting that both polarities may encode rel- evant information. We then selectively ablated positive or negative input weights class units and visualized features across ablation strengths. Ablating positive weights greatly reduced the maxi- mal achievable activation during feature visualization, whereas removing negative weights slightly increased it (appendix Fig. TT). Visually, positive-weight ablation disrupted recognizable object structure, while neghtive-weight ablation largely preserved object identity and instead altered back- ground or color context (Figs. DB. C). To quantify these effects, we compared image sets generated before and after ablation using mean pairwise cosine similarity across an ensemble of readout CNNs. Positive-weight ablation yielded substantially less similar representations, whereas negative-weij ablation produced only minor shifts (Fig. 2D. These results replicated across 100 ImageNet classes and with alternative metrics such as L.PIS (Zhang et al. 2018) (appendix Fig. 13), demonstrating robustness and generality.
To quantify to what extent objects disappear from the preferred images under ablations, we evaluated objectness using an object-detection network (YOLÓv? Wang et al. (2022)). Relative to baseline objectness scores computed from intact visualizations, ablating positive weights reduced objectness, whereas ablating negative weights had minimal impact (Fig. 2). Analyzing spatial frequencies re- vealed positive ablations majorly affected low frequencies (app. Fig. 16,6), consistent with the objectness reduction. Together, in ImageNet-trained CNNs, removing positive input weights dis- rupts object features, while removing negative weights primarily alters context.
4

---

Published as a conference paper at ICLR 2026
Closed-loop neural activity maximization
Detailed ablation experiment example C Ablation examples
Electraphyzioloyy in IT cortex
Maximum response
alexnet (tench)
arthhciilletworis networks
AlexNct
5E.76 pos
L.S
Firing rates
44.41
none
B107
Synl resized
Oyttmboe preforred images
OR
0.0
Image generator
+ ookimicer
Artificial resporse
Total Input ablation strength
02
0.6
08
100
neg
574 Igolt
0.77
networks
pos
23
Representational similarity across networks
none
cosine similarity cosine similarity
0.6-
alexnet
v8816
résnet50
resnet50, linfo..s
Maximum objectness across units and networks
neg
27.29
0.30
alexnet unit 482 (cassette playert
pos
imax objectness
resnet50 Jinf1
resnet50, Jinf2
resnet50_iing Jinf4
resnet50, Jinf8
0.15
none
0.6-
neg
0.4
control
0.00
0.0
1,0
ablation strength
ablation strength
1
ablation strength
Figure 2: A. Schematic of feature visualization workflow in ANNs and brains. B. Preferred feature changes for different ablation strengths of input weights to the macaque 373 output unit AlexNet (last fc layer of 1000 units before softmax). Images are the most activating images out of the 20 visualizations per ablation strength. Ablation strengths are below each image, and activation scores above. C. Changes in preferred features to different ablations of example AlexNet output units: O tench, 574 golf ball, and 482 cassette player. Notice the large image changes for positive ablations. Same methods as in D. Representational similarity of intact vs input-ablated units across networks tested, measured by the pairwise cosine similarity of control vs ablation images over an ensemble of networks. Error bars are 95% confidence intervals over units, each unit is the mean of its 20 visualizations. The units correspond to the 10 imagenette categories ([O, 217, 482, 491, 497, 566, 569, 571, 574, 701]) plus the macaque category (373). E. Objectness scores across units per ablation condition. As in D, we tested 11 units from the 1000-unit fully-connected output layer (pre-softmax) of: AlexNet, VGGI6, ResNet50, and robust ResNet50 (L.o E {0.5, 1, 2, 4, 8)). For each network, averaged over the objectness scores of 20 visualizations per unit and all units. The plot shows the mean over previously described network averages. Error bars are 95% confidence interval over network averages.
4.3 SEGREGATION DEPENDS ON RELU BUT NOT ON UNSUPERVISED PRETRAINING
We next explored potential mechanisms underlying this functional segregation by weight sign. To test segregation in unsupervised learning, analyzed ResNet50 backbone trained without super- vision (SimSiam from Chen & He (2000)) After unsupervised pretraining, the backbone was frozen and linear classifier was trained on ImageNet-I KK. Feature ablation and visualization revealed that this network still allocated object features to positive weights, although these features disappeared at lower ablation strengths than in fully supervised CNNs (Fig. 3A,, appendix Fig. [4). Negative input ablation had only a minor effect, suggesting that even unsupervised representations are organized so that positive weights convey most rblect-related features.
We hypothesized that this segregation depends on the rectifying ReLU activation, where the weight sign fully determines output sign, due to the non-negativity of activations. Prior work in toy net- works shows that ReLU promotes alignment to the input space, whereas Tanh promotes alignment to the output space (Alleman et al. 2023). To test the role of rectification, we trained ResNetl5 18 models with either ReLU or Tanh. As expected, the ReLU model showed clear segregation, with the strongest disruptions arising from ablation of positive weights. In contrast, the Tanh model showed similar representational changes across ablation types and preserved key features even when either sign was removed (Fig. BBC Thus, rectified activations are required for strong segregation of object information into positive weights in CNNs.
5

---

Published as a conference paper at ICLR 2026
A
Semisupervised ResNet50
RelLUacctivatio ResNet18
C
Tanh-activation ResNet18
(tench)
unit
unit 566 Itrench hom)
unit 574
(golf ball)
neg
pos
none
neg Ablation type
pos
none
neg Ablation type
pos
Ablation type
response
cosine similarity
response
cosine similarity
response
cosine similarity
50
0.4
0.4
0.4
10
10
nrg
0.2
pos
0.2
0
0.2
none
ablation strength
Figure 3: Ablation experiments (rows are features of three example units, and responses, and repre- sentational similarity for all 11 classes) in A. Semisupervised networks, B. Vanilla ReLU supervised network, and C. Network with a non-rectified activation function Tanh, replacing all ReLUs in B.
Robust training induces stronger feature segregation
ResNet50
unit (tench)
unit 566 (french horn)
unit 373 (macaque)
(L. 8)
10.26)
(1.47)
10.37)
1-2.21
(113)
(-0.1)
Iscore: 14.12)
pos
(score: 23.14)
pos
(coor: 16,06)
neg Dos
none
none
none
(18.07)
(30.60)
(22.73)
(28.28)
(17.21)
(21.66)
neg
neg
0.5
ablation strength
Figure 4: Robust network ResNet50 Lox = 8 shows a large change in preferred features upon input
ablation. Notice the white background in the negative-weight ablation condition.
4.4 ADVERSARIALLY ROBUST NETWORKS SHOW ENHANCED FEATURE SEGREGATION
Given that unsupervised pretraining and rectification support this segregation, we next investigated adversarially robust networks. Adversarially robust networks, which are trained resist small tar- geted image perturbations (Szegedy et al.] 2014) Madry et al. 2019). are believed to better reflect aspects of biological vision and may therefore show distinctive patterns feature segregation. We examined whether and how adversarial robustness influences the allocation of object and contextual information positive and negative weights.
In robust ResNet50 networks, intact feature visualizations appeared more object-like, and ablation of negative input weights reliably altered the background color, often rendering white (Fig. 4 This hinted at stronger feature segregation than in vanilla networks. Quantitative analysis con- firmed that as network robustness to adversarial attacks increased, so did the model's vulnerability to ablation, measured by the difference in cosine similarity between control images and ablated images (see Accosine similarity) in Fig. (5). For ablation strength of (yellow/iggh lines), the differ- ence increased with network robustness. and slopes in Table indicate that this trend holds across ablation polarities and strengths. Moreover, the robustness effects translated into higher shape bias in the benchmark by Geirhos al. (2022). However, this benchmark did not disentangle ablation ef- fects. Negative ablations similarly reduced shape and texture accuracy, preserving overall shape bias. Only robustness Loc increased shape bias after ablating negative weights (Fig. ??). Over-

---

Published as a conference paper at ICLR 2026
cosine similarity
Positive ablation
Negative ablation
Table 1: Spearman correlation of repre-
ablation strength
sentational change upon ablation vs ro- bustness (Ioz norm)
0.2
0.0
type
Positive (pvalue)
Negative p (pvalue)
00.51
4
8
00.51
2 4 8
0.3
Alcosine similarity)
0.4
0.5
0.0
07
-0.2
0.8
0.9
1.0
0.1
-0.17 (2e-1)
-0.10 (4e- 1)
-0.4
0.2
-0.39 (Ie-3)
-0.21 (8e-2)
0.3
-0.34 (4e-3)
-0.14 (3e-1)
00.51
4 8
00.51 2 4
0.4
-0.38 (1e-3)
-0.34 (5e-3)
robustness
robustness
0.5
-0.47 (6e-5)
-0,46 (9c-5)
Figure 5: Representational changes under in-
0.6
-0.48 (3e-5)
-0.34 (5e-3)
put ablation increase with robust training in ResNet50. Top: cosine similarities to control.
0.7
-0.51 (9c-6)
-0.52 (6e-6)
0.8
-0.50 (2e-5)
-0.49 (2e-5)
Bottom: changes relative to control. Most abla- tion strengths show significant correlations with robustness.
0.9
-0.48 (4e-5)
-0.62 (2e-8)
1.0
-0.47 (6e-5)
-0,57 (5e-7)
all, classification CNNs segregate object and context information to positive weights and texture or background information to negative weights, and adversarially robust training further sharpens this sign-based segregation.
4.5 FEATURE SEGREGATION NOT EXCLUSIVE TO CLASS UNITS
To determine whether feature segregation by weight sign is present beyond the output layer, we analyzed intermediate and early layers of CNNs. We first searched for channels that approximated Dale's law, by providing predominantly positive or negative inputs to their downstream units in each layer. We then visualized the preferred features of the most "Dale-like" channels using gradient- based methods with the Lucent library in PyTorch. Examining all five convolutional layers of AlexNet, we found that feature segregation by sign emerged throughout the network. In the first layer, channels with mostly positive weights responded to high-frequenc, achromatic edges, while those with mostly negative weights responded to sewer-freuuecy, colored edges and spots. In the middle layers, positive channels tended to emphasize edges and detailed textures, whereas negative channels often represented broader, colored, or background-like patterns. By the final convolutional layer, channels with mostly negative weights produced features that resembled background elements such as sky or grass, while channels with positive weights highlighted sharp, localized object frag- results show that feature segregation by weight sign is not restricted to the output layer, but gradually develops throughout the network. This pattern is reminiscent of Dale's law in biological circuits, suggesting that artifi- cial neural networks can develop sign-consistent and functionally distinct representations across all layers, even in the absence of a biological constraint.
ments like animal snouts and eyes (appendix Fig. 20p. Altogether,
Neuron model ablation experiment examples
nouron 41
neuron
neuron 55
10.5, 72.50
(1... 43
11.00
DOS
Istrengtt 140.35)
Istrengtly Q.U.
JO.. 119.99
(1.. 149.57
MaRe: 129.12
I05, 479
110, 14662
store: 61.99
ID.S, 61.89
11.. 67.76
Tone
none
Figure 6: Preferred features of neuronal network models of visual neurons in the primate ventral stream. Pos: are positive ablations, neg are negative ablations, number indicates ablation strength. Shown are top visualizations at 0, 0.5 and 1.0 ablation strenghts.
7

---

Published as a conference paper at ICLR 2026
Model responses
Neuron responses
Representational changes upon ablations
Cortical population response to
ablations control
ablations control
from model ablations
& Prodicted response
(pscared by taiinig!
0-
A actual respoose
sc-red by training)
0.0
cosine similarity (controo abiction)
meen population respante (pocorrs
0.1
control
-1-
-2
-0.2
0.0
p0s
-3
-0.4
00.
-0.6
0.5
0.0
0.5 1.0
0.0
1.0
0.0
0.5
1.0
Ablation strength
Aslation strength
Ablation strength
Ablation strength
Figure 7: Left: predicted and actual neuron responses of model to ablations. Images obtained from positive ablations in the neuron models elicited a consistent activity drop on the biological neurons modeled. Right: Representational change of model to ablations measured by our cosine similarity metric on the neuron model feature visualizations upon ablation; and cortical population response to the images obtained from feature visualization from ablation of model units, neurons were z-scored before computing the population average. Plots show averages 59 models, (35 for monkey C, and 24 for monkey D), shaded regions are the 95% C.I of the mean. For all plots the positive ablation condition was statistically different to the control.
4.6 BIOLOGICAL, MODELS BASED ON IMAGENET NETWORKS SEGREGATE LOCAL FEATURE INFORMATION INTO POSITIVE WEIGHTS
The ventral stream in primates is responsible for object recognition, and artificial networks are the best models of its function. We therefore wondered if the segregation of positive and negative in- puts we observed in networks might also occur in the brain. However, it is not currently possible selectively remove positive or negative synaptic inputs from real neurons the way we can in arti- ficial networks. To address this limitation, we fit linear models mapping CNN features to macaque ventral stream neural responses, and applied feature visualization both model units and in vivo to biological neurons. This allowed us to test feature segregation in models of biological repre- sentations and to generate testable neuroscience predictions. We recorded neural activity from V1, V4, and IT cortex in two monkeys, using a diverse set of images and modeled each neuron's 's re- sponse using partial least squares regression with activations from the penultimate layer AlexNet (4096 units). [ We then applied our ablation and visualization protocol to these neuron models (see appendix for validation A.5) and showed the resulting images to the monkey during the same session. First, images from intact models reliably drove biological neurons to firing rates more than one standard deviation above those observed during natural image presentations (Fig 23 left), indi- cating citto--distribution generalization. For the subset of neurons in which we performed in vivo closed-loop feature visualization, we found that the model's 's preferred features often matched those of the neuron, providing additional validation (appendix Fig. 23 right). However, in vivo features were more spatially localized (procedure in A.I). whereas in silico features exhibited greater spatial variation (rotated, mirrored, or repeated versions). This likely reflects invariances, due to using a fully connected layer, that are not present in our recorded neurons. Moreover, unlike the images from recognition units, images from the neuron models did not resemble objects (appendix Fig. 23 Fig. 6
Ablation experiments reveal sign-based segregation neuron models. Ablation experiments on these neuron models showed that removing positive input weights led to significant decrease in both predicted and observed firing rates, while removing negative weights had a smaller effect (Fig. This pattern consistent across individual neurons and the population level in the ventral stream (Fig 7 rightmost). While models are tested with out-of-distribution images produced by feature visualization. qualitative results hold for each sign ablation. Furthermore, changes in population activity by manipulation of single neuron models suggest that sign-based functional seg- regation in model predictions translates measurable changes across the ventral stream population and perhaps perception.
Although using other layers could improve predictive accuracy, we selected the penultimate layer to di- rectly test if inputs optimized for classification maintain weight sign-hased segregation in hiological neural predictions.
8

---

Published as a conference paper at ICLR 2026
Dale's law inspired analysis. To bring models closer to Dale's law, we applied two approaches. First, we tested whether neural responses could be predicted using only positive input weights, cor- responding to receiving input exclusively from excitatory artificial neurons. This constraint reduced both training and test accuracy relative to unconstrained models (appendix Fig. (4)) indicating that neuron models require both positive and negative inputs. Second, we identified Dale-like artificial units that contributed mainly positive or negative weights to all output neuron models, defining pu- tative excitatory and inhibitory inputs. Positive-weight units corresponded to smaller-scale edges and localized spots, cleanly separated from the background, whereas negative-weight units aligned with broader textures and larger patches (appendix Fig. (5).. This pattern supports the idea that inhibitory-like artificial inputs preferentillly encode contextual or background structure, paralleling inhibitory modulation in biological cortex.
Experimental manipulation of background as a test for inhibition. To further test this hypothe- sis, we experimentally manipulated image backgrounds in vivo. subset of experiments, we pre- sented altered images in which the background was cleared around the neuron's preferred feature, thereby reducing the putative inhibitory drive. As predicted, this manipulation resulted in increased neuronal responses (appendix Fig. 26), providing functional support for the idea that inhibitory or negative inputs are involved in contextual modulation and that their reduction can enhance feature selectivity in high-level visual cortex. Together, these results suggest that functional segregation by input sign extends to models of ventral stream neurons, providing concrete testable predictions for future experiments targeting excitation and inhibition in visual cortex.
5
LIMITATIONS
Our results hold in the last layer units of multiple networks. Due limited computing time, we did not test all 1000 categories in as many networks as possible, our largest test consisted of 100 units. While larger scale simulations will provide exhaustive evidence, we are confident our main claims will stand. We limited our neuron recordings to a 160 image dataset for regressing neuron responses via CNNs. While we observed good fits and recovered relevant feature to the neurons, more images may improve the models, especially using larger-scale versions of our diverseSet. The neuroscience results would need to follow Dale's law to be mapped one-to-one excitatory and inhibitory neurons, but we make no claim to perfect mapping in this work. We observed range of sign-segregation levels across architectures and depths, thus we expect a gradient in the functional segregation across architectures and training regimes. Such example is our finding of increasing robustness training leading to stronger segregation, but we anticipate more. The fundamental ques- tion of shape vs texture, foreground vs background remains to be solved. We reconciled changes in frequency structure with changes in objectness and visual representations and L.PIPS image sim- ilarity. However, more work remains understand the full extent of the segregation reported here. Solving this problem for the visual cortex may provide better benchmarks for this task in Al.
DISCUSSION
Our study combined ablations with feature visualization guided by naturalistic image priors to re- veal the functional segregation of class-level features in the output layer of ImageNet trained CNNs: positive weights contribute object/shape/low frequency information, while negative weights con- tribute background/contextuuleeeeee
information. This effect was enhanced in robust networks, was present in networks with unsupervised pretraining, but was absent in network trained with Tanh instead of ReLU. Our results explain how the background contribution to classification observed in (iiao et al. 2020) emerges, backgrounds are primarily encoded by the negative inputs.
Importantly for neuroscience, the observed functional segregation neuron model units in CNNs hints at functional segregation in the brain beyond the center-surround classically studied in V1. We also crafted a diverse dataset for visual neuroscience recordings that is scalable. Neuron re- sponses to smaller but diverse set of naturalistic, colored images, with complex foregrounds and backgrounds, led to models capturing relevant features obtained experimentally from the neuron. Thus, using both model-based and model-free approaches revealed richer neuronal representations. Preferred images from neuron models with positive input ablations elicited smaller average pop- ulation responses of cortical neurons. This suggests that ablation in networks modeling neurons
9

---

Published as a conference paper at ICLR 2026
holds potential method to control the population activity in the brain. To relate ablation-induced changes in the images to the population responses is a future direction. This ablation based on the natural division of positive and negative weights can be easily extended into arbitrary layers, e.g., using gradients to define positive and negative contributions to any arbitrary unit. Our ablation approach proposes baselines for the functional differences between excitatory and inhibitory rons in higher cortical visual areas. The functional segregation has consequences for neural coding and response selectivity. Our findings generate concrete predictions for future experiments using advanced genetic or optogenetic tools to dissect excitation and inhibition in primate cortex. Under- standing the circuit mechanism of biological vision could aid further understanding and development of computer vision models. Ultimately, these insights suggest that biological constraints like Dale's Law emerge from functional demands. Our work thus introduces consistency signed connectivity as biologically grounded primitive for the growing toolkit of mechanistic interpretability.
REFERENCES
fastai/imagenette, May 2024. URL httpp////ggtuu.ooomm///ta//aagas original- date: 2019-03-06T01:55:455
Matteo Alleman, Jack Lindsey, and Stefano Fusi. Task structure and nonlinearity jointly deter-
mine learned representational geometry. October 2023. URL URKLLttsss//gpeeeeeeew..cc
Forum?1d=k39tdq000
Emily J. Allen, Ghislain St- Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dow- dle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116-12) 1726. doi: 10. 10.10008/441933-021-000000 s 121593-021-0093
January 2022. ISSN 1546-
URL BoLLhtps::/ www. com/ articles/le Publisher: Nature Publishing Group.
nat
Pouya Bashivan, Kohitij Kar, and James J. DiCarlo. Neural population control via deep image synthesis. Science, 364(6439) May 2019. doi: 10.1 10.1116/sciemcc.aa44433 URL org/ http:/dooofflll//// wttp:///ll....////eeeeeeeeeeeeee 1tt..11//scceece.a3333...
http:::/www...iceee American Association for the Advancement Science.
Publisher:
Timothy F. Brady, Talia Konkle, George A. Alvarez, and Aude Oliva. Visual long-term mem- ory has a massive storage capacity for object details. Proceedings of the National Academy of Sciences, (05(38):14325-1432) September 2008. doi: 10.
URL
https: / www. pnas. orq/ doi/ ceedings of the National Academy of Sciences.
0803390105
Publisher: Pro-
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. February 2019. URL http:///arxx...rrr///////00. arXiv:1809. 1096 [cs, stat].
Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning, November 2020.
URL htttt::/:///ii//
ab///bb//2211 10566 arXiv:2011.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse Autoen- coders Find Highly Interpretable Features in Language Models, October 2023. URL http: arXiv:2309.08660 [cs].
Henry Dale. Pharmacology and Nerve-endings (Walter Ernest Dixon Memorial Lecture). Proceed- ings of the Royal Society of Medicine, 28(3):319 332, January 1935. ISSN 0035-9157. doi: 10. PMC22057017
URL httpp::/ nebi. mlm. nih. htt.ov/arriclsss.
James J. DiCarlo, Davide Zoccolan, and Nicole C. Rust. How Does the Brain Solve Visual Ob- ject Recognition? Neuron, 73(3):415 February 2012. ISSN 0896-6273. doi: 10.60
neuron neurrom20012.0..100.. URL pattps:// :/www/ wttp::.sieecncecicc...... wtwp:eeddireetttteecceeeeeeeeeee http://wwc...coeeeeeeee//rreeiaa pi1/50896627312000099
10

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy and Thomas Brox. Generating Images with Perceptual Similarity Metrics based on Deep Networks, February 2016. URL http:://aaaat.. org/ wwtp/ba///....255...22 arXiv:1602.02644 [es].
Gamaleldin Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Good- fellow, and Jascha Sohl-Dickstein. Adversarial Examples that Fool both Computer Vision and Time-Limited Humans. In Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https: http::::///prooeeednngg.. htw.rooceeeinngnn...ei...... neurips. wttpcc//paaa. httppp/ccccppppeeeeppeee
hash/ 3562ae5e286544 7555441700222211885333332---2230 htmll
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent.
Vi-
sualizing higher-layer 1341(3):1, 2009.
features of a deep network.
University of Montreal,
http:::/www.rssaacchhaattt 0at.oo..ccuail//oooaaaaaaa/////0
otgp:///5552222i..777aaaaaaaaaar Features_of__ Baaprre_______eee__eee____
Vi Visualizii httpa//caaeaaaaaooooooooooanaaan
Jenelle Feather, Guillaume Leclere, Aleksander Madry, and Josh H. McDermott. Model metamers reveal divergent invariances between biological and artificial neural net- works. Nature Neuroscience, 26(1 26(11):201720334
November 2023. ISSN 1546-1726.
doi: 10103386411993-022-1144444 s41593- 001599-023-110 -20-0022----1 01442-14421100
URL https:/ htss::/www..eee www..natuz....... www.ree..omarrtte awtimeerticle//////ooo Publisher: Nature Publishing Group.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, November 2022. URL
12231 arXiv: 1811::81 [cs].
Umut Güşlü and Marcel A. J. van Gerven. Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. Neurolmage, 145:329-336, Jan- uary 2017. ISSN 1053-8119. doi: 10. n0.1016/j.eeunnmmaeee2200 2015. URL httpt://t////t /www.
sciencedi rect. www.coocccccoooo scii http:///s...eeeeeaaaaaaaaa////// 900
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. URL httpt:ttp//:xiiiii org/ worggabs//ia abs/ 1111.1331.......
D. H. Hubel and T. N. Wiesel. Receptive fields of single neurones in the cat's striate cortex. The Journal of Physiology,
October 1959. ISSN 0022-3751. Utthttps::// //ww..
ncbi. nlm. nih. gov pmc/
Chou P. Hung, Gabriel Kreiman, Tomaso Poggio, and James J. DiCarlo. Fast Readout of Object Identity from Macaque Inferior Temporal Cortex. Science, 310(5749):863 866-866, November 2005. ISSN 0036-8075, 1095-9203, doi: 10./s///cce 117593. URL https: /tt::://wwwwww /wwp//wwww.oo science.
orq/doi/10. 11t.://..111////11 wtt.:///ee/ceieneeee
Jaewon Hwang, Andrew R. Mitz, and Elisabeth A. Murray. NIMH MonkeyLogic: Behavioral control and data acquisition in MATLAB. Journal of Neuroscience Methods, 323:13-21, July 2019. ISSN 1872-678X. doi: 1..10066jjneumeett www.:h........00000
Jeffry S. Isaacson and Massimo Scanziani. How Inhibition Shapes Cortical Activity. Neuron. 72(2):
231-243, October 2011 1. ISSN 0896-6273. doi: 10.10166/j neuron. .09.027. URL httptttps: www. www.cello
www.//////eeeeee www.m/euron... http//bbtrrrtttte Stt///089666 6273 (11) 00879- Publisher: Elsevier.
Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B. Issa, and James J. DiCarlo. Ev- idence that recurrent circuits are critical to the ventral stream's execution of core ob- ject recognition behavior. Nature Neuroscience, 22(6):974-983, June 2019. ISSN 1546- 1726. doi: (0.1038/-41199--1-0039
URL https: www. nature........ wtt.e/cc./aa.aeeeeeeeeeeeee Number: 6 Publisher: Nature Publishing Group.
111599-019-0992
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Con- volutional Neural Networks. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https: /ttpttpppp/pppp.s..pptpt nips cc http://ss...pp...oo/p///////////
c3

---

Published as a conference paper at ICLR 2026
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. FFCV: Accelerating Training by Removing Data Bottlenecks, June 2023. URL http:
12517 1IXiv:230..2511.
Tianqin Li, Ziqi Wen, Yangfan Li, and Tai Sing Lee. Emergence of Shape Bias
in Convolutional Neural Networks through Activation Sparsity. Advances
Neu-
ral Information Processing Systems. 36:71755-71766, December 2023. URL
html
Ilya Loshchilov. LM-CMA: an Alternative to L-BFGS for Large Scale Black-box Optimization, arXiv:1511.00221 [cs,
November 2015. URL http://arr.v..orrraab//11111
math].
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks, September 2019. URL BRLhttp:
//axxiv.org/abst 060836633333 7XXiv17066000833
Anh Nguyen, Alexey Dasovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons neural networks via deep generator networks, November 2016a. arXiv: 1605.699000 [cs].
URL
wwwpxxivvorggaaaaa hblp://bs/1l/ 605 09304
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks, February 2016b.
URL htllhttts//axiiie http:///aavvvvvie www.wittww....aaaoaooooooooooooe abs/1602. 03616v2
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature Visualization. Distill, 2(1 2(11))e7,
November 2017. ISSN 2476-0757 doi: 10.2391//dsstil.000007 URL https:///istt1111
Qub/2017/Feattree
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom Introduction to Circuits. Distill, 5())e00024.001, March 2020. ISSN 2476-
0757. doi: ht/d///iil...0022444404 URL https:t/ccsssillll.... htt.:a/iiii..uab/2222///eccciaie zoom in.
Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381 (658))))60776669 June 1996. ISSN 1476- 4687. doi: 10.1038 (038/38160744) 10770.. htll::esss.....tntrrrcccoo///ott
bttttps:/www..aaure..commm
Publisher: Nature Publishing Group.
Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie. and Zeynep Akata. Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models, June 2025. URL
http://arxiv. wwwp://wwwwwv.xabsbe//
02821 arXiv:2504.02821 [cs].
Carlos R. Ponce, Will Xiao, Peter F. Schade, Till S. Hartmann, Gabriel Kreiman, and Margaret S. Livingstone. Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences. Cell, (11)))999990990... May 2019. ISSN 0092- 8674, 1097-4172. doi: 10.
URL htt.httt:///w..clll www.w/wwcccooom cel. com cel1/
Abssracc//0000 8 (19) 30391-5 Publisher: Elsevier.
Hadi Salman, Andrew llyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do Adver- sarially Robust ImageNet Models Transfer Better?, December 2020. URL http://arxiv:
orq/ /tc//ab/// 2007 08489 urXiv:2007.08444 [cs, stat].
Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Scale Image Recognition, April 2015.
1556 arXiv:1409.114
[cs].
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Good- fellow, and Rob Fergus. Intriguing properties of neural networks, February 2014. URL http::/arxi www.obgg./../ooooooo.
arXiv:1312.6199 [es].
12

---

Published as a conference paper at ICLR 2026
Hiroshi Tamura, Hidekazu Kaneko, Keisuke Kawasaki, and Ichiro Fujita. Presumed Inhibitory Neurons in the Macaque Inferior Temporal Cortex: Visual Response Properties and Functional Interactions With Adjacent Neurons. Journal of Neurophysiology, 91(6)27722-779 27996, June 2004. ISSN 0022-3077. doi: 10. 1 1111111//...26/1/// 2003. URL https:/ journall s. physiolosgoo
org//oi/full/100 115.2...111.. / / 2003 Publisher: American Physiological Society.
Edgar Y. Walker, Fabian H. Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G.
Fahey, Alexander S. Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S. Tolias. Inception loops
discover what excites neurons most using deep predictive models. Nature Neuroscience, 22(12):
2060-2065, December 2019. ISSN 1546-1722 doi: 10. 10000000/41533--1---557777- URLhttpp: Publisher: Nature Publishing
//www. www.nntuuee... com/ art clas 011-5593-119-5177
Group.
Binxu Wang and Carlos R. Ponce. Highh-peffrmmace evolutionary algorithms for online neuron control. In Proceedings the Genetic and Evolutionary Computation Conference, GECCO '22, pp. 1308-131,, New York, NY, USA, July 2022. Association for Computing Machinery. ISBN URL https: //dl acm. org/doi/
978-1-4503-9237-2. doi: 10. 211144//512229.35887223 10.
3528725
Chien-Yar Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. YOLOv7: Trainable bag- of-freebies sets new state-of-the-art for real-time object detectors, July 2022. URL https:
//arxtv.org/abs//200..
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or Signal: The Role of Image Backgrounds in Object Recognition, June 2020. URL htt.::://eexxi..or/////o//// 2006.09999 arXiv:2006.09994 [es].
Will Xiao and Gabriel Kreiman. XDream: Finding preferred stimuli for visual neu- rons using generative networks and gradient-free optimization.
PLOS Computational
Biology, 1007973 June 2020.
ISSN 1553-7358.
doi: 10.1371/joumal.pebi.
1007973. URL https:ttpo...m....tt. // www.ooas...oo.........eoooo. http://plosccppbo////tteeeeeeeee 1371/jour pcbi 1007973
Publisher: Public Library of Science
Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion- semi- supervised learning for image classification, May 2019. URL. http:://axiit org/ abs/ arXiv: Mxii::905500:: 905.00546 [cs].
T905. 00546
Daniel L. K. Yamins, Ha Hong, Charles F. Cadicu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 1 111(23):8619 8624, June 2014. doi: 10. 10.1003/paas.14000 1403 12111. URL /https:// wwww.ppaass pnas wwtg/oio/0// /tt////......11/ppee....... 1073/pnas.
14C3112111
Publisher: Proceedings of the National Academy of Sciences.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features Perceptual Metrie, January 2018. htt.:ttts::/aaae
org/abs/1801 .00.1.09922
AUTHOR CONTRIRUTIONS
G.R.T. led the conceptualization. methodology, experiments, analysis, and manuscript writing. C.R.P. supervised the project and contributed to reviewing and editing the manuscript.
ACKNOWLEDGMENTS
We thank Elizabeth Cleaveland for their technical help with data acquisition and surgical procedures. This research was supported by the Alice and Joseph Brooks Fellowship (G.R.T.). the William Randolph Hearst Fellowship (G.R.T.), the Common Fund of the National Institutes of Health under award 1122EY035176-01 (C.R.P.), and the David and Lucile Packard Foundation (2020-71377, C.R.P..
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 EXTENDED METHODS
Networks The ablation studies were performed CNNs pretrained the ImageNet dataset: AlexNet (Krizhevsky et al. 2012). VGGI6 (Simonyan & Zisserman 2015). ResNet50 (He et al.) 2015), and robustly-trained ResNet50 (Loe {0.5, 1, 2, 4, 8). Salman et al. (2020). All these networks end on 1000-umit fully connected layer, each unit corresponding to one of the 1000 ImageNet categories. Neural networks were used in Pytorch.
ImageNet subsampling To reduce computing time, for most of the experiments, we used a subset of ImageNet, the imagenette dataset (noa 2024) and the macaque category, 11 classes in total. These classes and their corresponding output units in each network trained on the 1000-class ImageNet dataset are as follows: (0, tench), (207, English Springer), (482, cassette player), (491, chain saw), (566, church), (569, French horn), (571, garbage truck). (574, gas pump), (701, golf ball), (970, parachute), and (373, macaque). We visualized the representations of the output layer units of those classes under different ablation conditions. For Fig. [2] sample 100 diverse classes out of the 1000 ImageNet classes, the 50k validation images were first clustered into 100 clusters via agglomerative clustering of the 1.2 distance matrix from the 1000-d output features of ResNet50, which was pre-trained on ImageNet. Then, one new unique class is selected from each cluster.
Ablation We used two ablation conditions: we ablated weights that were (1) only positive (2) only negative. We ablated weights cumulatively by first sorting the positive (or negative) weights by their (absolute) decreasing value. We defined the ablation strength, a, fraction of the total positive or total negative weights to unit. We identified the top l weights necessary to reach the silencing strength, i.e., Σ W; S a, and set them zero. We covered the range of ablations from to 1. For most experiments with ANNs, we used silencing strengths in 0.1 steps, from 0 (intact) to 1 (complete ablation).
Closed-loop neural activity maximization
Electrophysiology in: artificial networks
IT cortex
OR A Firing rates
Synthesized
OR Artificial neuron response
images
Optimize preferred images
Image generator optimizer
generative networks
Figure 8: Schematic of feature visualization workflow in ANNs and brains. Optimizer is CMAES, image generators are DeePSim fc6 or BigGAN.
Feature visualization For each ablation condition, we performed feature visualization by opti- mizing a GAN latent code to create an zeroth-order-search approach allow comparison with our neuronal experiments, where gradient ascent would not be possible. To increase the span of the stimulus space, we used two GANs: AlexNet fe6 DeePSiM (Dosovitskiy Brox 2016) and BigGAN (Brock et al., 2019). For optimiza- tion, we used a variant of covariance matrix adaptation evolutionary strategy CMAES (Wang & Ponce 2022 Loshchilov] 2015). Initial conditions for the CMAES were given as standard deviation of 3.0 for DeePSim, and 0.2 for BigGAN. Initial images for the algorithm were small norm vectors for both GANs, close to the origin of the latent spaces. For BigGAN, we generated fixed noise vector by scaling di-dimmenio truncated noise sample (-1.4, 1.4), and concatenated it with 128-dimensional zero vector of the class embedding, to form the required 256-dimensional input code. The remaining parameters determined by the dimensionality of the search space of each
image Fig. g We used this closed-loop.
14

---

Published as a conference paper at ICLR 2026
160DiverseSet
AlexNet output feature space (PCA)
Dim 2
Dim 1
Figure 9: Illustration a diverse dataset construction using AlexNet output feature space. The em- bedding the output of the last layer before softmax of AlexNet, a vector space of 1000-dimensions. Left: PCA showing the coverage of the feature space by the diverseSet I 160, only for illustration pur- poses. Right: images from diverseSet 160 used to fit neuron models.
GAN. We optimized ten images per GAN, resulting in 20 feature visualizations per output unit and ablation condition. Diverse visualizations better capture the multifaceted high-level representations in CNNs (Nguyen et al. 2016b). For our examples, we show the best of the 20 visualizations, but used all for quantitative analyses. For visualizations of neural networks predicting biological neuron responses, due experimental time restrictions, we used five visualizations per ablation condition, via DeePSim only. Our experiments are performed in PC with Nvidia 4090 GPU, and each visu- alization running 100 iterations takes about mins. For in vivo experiments, we ran from 20 to 60 iterations of the AlexNet fc6 DeePSiM with the CMAES algorithm implemented in Matlab, linked to our real-time spike-sorting data acquisition. The responses fed to the CMAES algorithm were the average firing rate on the window 70-170 ms from image onset.
Feature analysis We computed image similarity using ensemble of CNNs, including AlexNet. ResNet50, and ResNet50 with robustness in T (0.5. 1, 1, 4, 8), inspired by (Feather et al., 2023) And confirmed the results with LPIPS (Zangg et al. 2018 in the appendix. We computed their activations and defined similarity as the average pairwise cosine similarity (LPIPS) between control activity vs input-ablated activity. We averaged the results of the CNNs ensemble, resulting in one quantity per ablation condition. We computed objectness as the maximum bounding box score provided by YOLOv7 (Wang al., 2022), this was averaged over visualizations per unit, units per network, and then across networks.
data from two animals (monkey C and monkey D), each implanted chronically with floating multielectrode arrays (Microprobes for Life Sciences, MD) of 32 or 16 channels (monkey C, N = 96 electrodes, monkey D, 64), in areas V1, V4 and posterior inferotemporal cortex (PIT). All institutional procedures were followed. Channels were distributed as (V1, V4, PIT): monkey C (32, 32, 32), monkey D (16, 16, 32). Some electrodes captured the activity of single units, but most showed multi-unit activity (reflecting the pooled activity of micro- clusters of neurons). The animals performed a simple fixation task, which required them to keep their eyes on a 0.25-deg diameter spot at the center the screen, within a square fixation window measuring 0. per side. Images were presented for 100 milliseconds ON, 150-ms off, 4-5 images per trial, after which the animal received water or juice. Images were presented to monkey C were 2 deg in size, and 4-8 deg for monkey D to match the receptive field centers of most channels in all cortical areas (V1, V4 and PIT). Image presentation and data acquisition (electrophysiology, eye tracking) were integrated by the MonkeyLogic2 software (Hwang al., 2019) and OmniPlex Neu- ral Recording Data Acquisition Systems (Plexon Inc.), interfaced through custom Matlab code. We performed online spike sorting using the PlexControl client based waveforms. We used ViewPixx
Visual cortex electrophysiology
15

---

Published as a conference paper at ICLR 2026
EEG monitors (ViewPixx Technologies), at resolution of 1920x1080 pixels with 120 Hz refresh rate. Eye tracking used ISCAN cameras (ISCAN Inc.). And reward was delivered using the DARIS Control Module System (Crist Instruments).
Feature localization in vivo We conducted perturbation-baser localization to identify relevant image regions from a feature visualization performed in vivo, where gradient information from the animal brain is unavailable. We perturbed a eircular region with a 50-pixel diameter within the 256-pixel image by randomly shuffling the pixels inside this circle, effectively disrupting the local image structure while maintaining local contrast. We selected 30 such regions for perturbation at random, excluding those that extended beyond the image boundaries. The modified images were then presented to the monkey. We hypothesized that perturbing regions crucial for driving the neu- ron response would lead to decreased firing rate. To assess local image importance, the normalized response change: the difference between the firing rate response to the intact image and the firing rate response to the perturbed image, divided by the firing rate response to the intact image. A normalized response change of 0.5 indicates the neuron response decreased by half due to perturbation. To generate the localized response mask, averaged the circular masks correspond- ing to each perturbed region, weighted by their response change. This response mask was further smoothed using Gaussian kernel with 30-pixel standard deviation. We defined relevant regions as those causing normalized response change of 0.5 or greater. Finally, we applied this mask to the original feature visualization image to highlight the local features.
calculated
Image dataset We collected a reference image dataset to activate neurons the monkey along the hierarchy of V1, V4, and PIT. Because neurons vary in their preferred features, we constructed a dataset spanning the image space as represented by the neural embedding of ImageNet-trained AlexNet. The embedding is the output of the last layer before softmax of AlexNet, vector space of 1000-dimensions. The images from this dataset also spanned uniformly the 1000-dimensional output space of semi-supervise trained network, trained on a billion images. ResNet5OSS (Yalniz et al. 2019). To define this embedding space, we performed PCA on the output activations from AlexNet the 50k ImageNet validation images, we kept the top 300 components (accounting for about 95% of total explained variance). Then we partitioned the space into a defined number of clusters k; according the desired dataset size, using batched k-means to reduce computational burden. After finding the k cluster centers, we could feed arbitrary images to the network, map them to the PCA space, and then pick the nearest neighbors image space. In addition to the ImageNet validation set, we added other common neuroscience datasets (Brady et al. 2008 Kar al. 2019, Allen et al. 2022 Hung et al,, 2005) form our image space. We selected k = 160 images, as a set that was diverse but small enough to be used in every experimental session. We called this image dataset diverseSet.
the cluster centers from the desired
Neuron responses
IT cortex
160DiverseSet
IT
000-0=000
network output
neuron responses
W
input weights
positive w negative w
Network responses
Figure 10: Schematic of model fitting using the dataset diverseSet. 160 images were split into train/test datasets (80/20).
16

---

Published as a conference paper at ICLR 2026
Models fit neuronal activity We recorded responses of neurons in the ventral stream to a 160 image dataset, our diverseSet Fig. I0 We relied on a small dataset to neuron responses and perform feature visualizations within the same experimental session. We performed partial least- squares linear (PLS) regression (80/20 train/test split) between the neuron responses to images and the activations of the penultimate layer of AlexNet. We used one component for the PL. regression. We selected one neuron or microcluster per experimental session, fitted model, and performed the ablation and feature visualizations
silico for that model. We selected the best fitted neuron per session, based on the r² on the 20 % held out test set, usually in the range of 0.15 to 0.5. When time allowed, we also performed the feature visualization of the modeled neuron in vivo using gradient-free approach (Ponce et al., 2019), within the same experimental session. To test whether features learned by the model were relevant to the biological neuron, we recorded the neuronal responses to the preferred images of the model. We then analyzed the representational similarity of the model features under ablations using ANNs. And analyzed the responses of the biological neuron populations from VI, V4 and IT.
A.2 SUPPORTING RESULTS ON NETWORK ABLATIONS
Table 2: Ratio of positive to negative weights. We divided the sum of positive weights by the sum of the absolute values the negative weights.
MODEL
RATIO (MEAN STD)
AlexNet
1.03 + 0.08
VGG16
1,01 = 0.09
ResNet50
1.00 0.06
ResNet50 (Lo 0.5)
1.00 0.05
ResNet50 (Lo = 1)
0.99 = 0.05
ResNet50 (Loo
1.00 0.04
ResNet50 (Lx = 4)
1.00 0.05
ResNet50 (Lo -8)
1.01 = 0.05
Responses to preferred images upon input ablation
alexnet
vag16
resnet50
resnet50 Jinf0.5
response
40
20
0
neg
resnet50
resnet50 linf2
resnet50 linf4
resnet50 linf8
pos
control
response
40
20
0
ablation strength
Figure 11:
Mean activation scores of units used in ablation experiments. For all networks, units scores come from the last fully-connected layer, with 1000 units, before the softmax. The units correspond to the 10 imagenette categories ([O, 217, 482, 491, 497, 566, 569, 571, 574, 701]) plus the macaque category (373). Error bars are 95% confidence intervals over units (categories tested), where each unit response is the mean of its 20 visualizations. Control refers to the feature visualizations in the intact networks for the same units, we extended it as a horizontal line to ease visual comparisons to the different ablation strengths.
17

---

Published as a conference paper at ICLR 2026
ResNet50 last fc layer units: 10x larger dataset, 100 new imagenet classes
(1.5-2
999 Intwepptt CILL
unit
ROO 23 oT)
CSS 24 OTS
(15. (25-2223
unit
unitt
OB 24 c31
30.31 31:27
SACIO DE
3C T6)
(10 92
noTe
01 31
unif
unit
1U.S 24
20 60
nona
P- km)
11. 1.
unit
unit 646
ghooggh 8o.
nonn
18
110 2.28
unit 29
unit 71
Jdwesgh
50:521
210 20.11
soone: 22, 40
10.5 221511
20
rona
Functional segregation holds in 10x larger dataset. 100 classes out of the 1000 Im- ageNet categories were selected by clustering the 50k validation images embedded in the 1000-d output space of ResNet50 picking class per cluster. Thus, now have 10x more data points that should span the representational space of the output layer we study. Consistent with the smaller dataset, the main object features degrade into more uniform background images upon positive abla- tion. Here we show examples from 10 of the 100 classes.
Figure 12:
18

---

Published as a conference paper at ICLR 2026
Perceptual similarity
control vs ablation
0.31
Similarity (1 LLIPS))
0.29
nee
control
0.27
Ablation strength
Figure 13: Functional segregation holds 10x larger dataset with LPIPS (Zhang et al. 2018 as representational similarity measure. We measured the representational similarity of the images as 1 LPIPS among control images and between control images and ablation images (Fig. 12). We average results per class, and show the mean and 95% C.I. across the 100 classes. The representational similarity degrades upon positive input ablations, confirming results obtained from the imagenette dataset.
resnet50-simsiam last fully connected layer
unit
00.1 12.42)
(0 3.47)
(0.5, 2,69)
D.7. 4.72)
(1.0. 988)
exc
(strength 0.0.
Score: 20.35)
(0.1 25.19)
(0.3, 37
(0.5, 48 88)
(0.7. 08 26)
(100 64.51)
none
inh
unit 574
exc
(strength: 0.O.
07)
none
unit 701
(0 1. 13:22)
(..7.
(0.5. 50)
(0.7. 72)
(1.0. -10 95)
exc
(strength: O.O. score: 24 07)
00.1 28 04)
(0.3) 30.30 363
U. 35
(0.7. 83)
(1.0. 52 m9)
none
Inh
Figure 14: Feature visualizations of ablation experiments in network pretrained with unsupervised learning. ResNee50SimSiam (Chen & Hc. 2020). The unsupervised network with frozen weights was coupled to fully connected layer, only this layer was fine-tuned classify ImageNet1000. Network units changed starting with small positive weight ablations, see unit 574 golf ball. Smaller changes are visible upon negative weight ablations, however object relevant features remain. Overall behavior is consistent with CNNs trained directly on ImageNetl 000 classification.
19

---

Published as a conference paper at ICLR 2026
A.3
SPATIAL FREQUENCY CONTENT, SHAPE/TEXTURE, AND OBJECT/BACKGROUN
ANALYZES
alexnet
vgg16
resnet50
resnet50 Linf 0.5
resnet50 Linf 1
resnet50 Linf 2
resnet50 Linf 4
resnet50 Linf 8
107
none
radial power
pos
10a
10a
1e-2 1e-1
Spatial frequency (cycles/pixel)
Figure 15: Functional segregation of frequency content. We measured the radial power spectrum for control and ablation images of different networks. We average results per class, and show the mean and 95% C.I. across the 11 classes. Low frequencies degrade upon positive input ablations, while negative ablations overlap with control spectra.
A
B 1.20
Normalized power ratio
Both conditions
Negative ablation zoom
2.5
2.0
Pos Neg
1.00
1.0
0.95
1e-1 Spatial frequency (cycles/pixel)
1c-2
1e-2
1e-1
Figure 16: Functional segregation of frequency content. We measured the radial power spectrum for control and ablation images of different networks. We average results condition and per network, and show the ratio control to ablated spectra. The mean and 95% C.I. are across networks. Low frequencies degrade upon positive input ablations, while negative ablations overlap with control spectra but slightly enhance low frequencies.
We measured the accuracy on the background challenge (Xiao al. 2020). To define reliance on background and foreground used the bggonnl-t and fg-only conditions, Rog and Rig respectively. We normalized differences of baseline accuracies using the original condition of the intact networks (Fig. 19
For background reliance, the control condition showed mean Rog = 0.157 = 0.116 (s.d.), which increased 0. 0..68 under positive weight ablation (mean change A A +0.331; Wilcoxon signed-rank test vs. control, (ÅRig 000799 p 0.0078). Although networks cannot classify above chance solely using backgrounds, the performance they can achieve can be more strongly attributed to negative weights. This limitation supports the approach of feature visualization vs a benchmark that was used retrain networks rather than testing pretrained ones. For foreground reliance, the control condition had mean Rig = 0.903=0.337 0.359) and to 0.711 0..245 under negative ablation (A Rig -0.193; p 0.0117). However, these results are biased by the accuracies under chance Fig. [8] causing outliers the robust networks. Networks with above chance accuracy in the control conditions. such as ResNet50, AlexNet and VGG16, show decreased performance upon positive ablations (neg only, green) Fig. [8] (pos only means negative ablation). Taken together, with some caveats, positive weights reduced background reliance and increased foreground reliance in this experiment.
0.027) and decreased to 0.078 + 0.046 under negative ablation
changing 0.777 under positive ablation (A Rig -0.064; p
A.4 DALE'S LAW INSPIRED ANALYSIS OF WEIGHTS AND INTERMEDIATE FEATURES
We showed Dale index (DI) increased from random initialization to training Table To determine whether weight segregation of features occurs beyond the output layer, we visualized feature repre- sentations that predominantly provide negative positive inputs to subsequent layers in Aetx
20

---

Published as a conference paper at ICLR 2026
Accuracy shape-texture challenge
0.6
Hugit 6474 0.30
0,43
Sipel Bies
alexnet
ResNet50 shape-texture bias vs robustness
0.60
Texture
resnetio
0.3
resnetto
0.4
ablation
0.68
resnet50_Jif22 Jin12
control
resnetso Jinf4
neg
resnet50,iinf
pos
conbol
Accuracy
shape
0.0
0.2
0.0
0.3
0.6
texture
Shape
shape
15
Ing ablation
0.6
texture gap
accuracy
(wrr (w.rt.ontro controll
peb289
shape blas controll
0.6
-00
0.0
15
-80
0.0
0
2 4
robustness
Stope Tosture
Shope Texture
neg
DO5
Figure 17: Functional segregation of shape texture using ablations in the shape/texture bias bench- mark (Geirhos et al. 2022). (Left) Effect of ablations on shape texture encoding across our net- works. (Right) Effect of robustness shape vs texture encoding ResNet50. While robustness shifts networks to be more shape-biases biased, this benchmark is affected equally by both ablation types.
Table 3: Dale index of the final classification layer before and after training, and trained Topl accuracy.
Model
Untrained DI
Trained DI
A DI
Top1 Acc
alexnet
0.5126
0.5461
+0.0335
56.52
densenet121
0.5125
0.5567
+0.0442
74.43
densenet169
0.5126
0.5590
+0.0464
75.60
densenet201
0.5126
0.5601
+0.0475
76.90
resnet18
0.5128
0.5905
+0.0778
69.76
resnet34
0.5130
0.5923
+0.0793
73.31
resnet50
0,5125
0.6004
+0.0879
80.86
resnet101
0.5126
0.5998
+0.0872
81.89
resnet152
0.5127
0.6007
+0.0880
82.28
vgg11
0.5123
0.5570
+0.0447
69.02
vgg bn
0,5123
0.5737
+0.0614
70.37
vgg 13
0.5125
0.5583
+0.0457
69.93
vgg 13_bn
0,5125
0,5747
+0.0622
71.59
vgg 16
0.5126
0.5606
+0.0480
71.59
vgg 16 bn
0,5126
0,5761
+0.0635
73.36
vgg19
0.5129
0.5619
+0.0490
72.38
1gg19_bn bn
0.5129
0.5748
+0.0619
74.22
inception_v3
0.5127
0.5597
+0.0470
77.29
21

---

Published as a conference paper at ICLR 2026
model resnet50
model, resnet5o Jinf0.5
model, resnet50 Jinf1
1.0
0.4
0.0
model_ name resnet50 2esnet5o_linf2
model name resnet50 Jinf4
model_name name resnet5o linf
0.8
0.6
control
pos jol_onl
0.2
neg
model
model vggg6
model
1.0
0.4
original
only fg
mixed same
mixed rand
no fg
only bg.t
original
only fg
mixed same
mixed rand
no fg
only bg.t
original
only fg
mixed same
miked rand
no fg
only bgt
variation
variation
variation
Figure 18: Functional segregation of background vs foreground. We measured the accuracy on
the background challenge (Xiao et al. (020).. Robustness increases the role of negative weights in encoding backgrounds but below the chance level. Counterintuitive results show either sign can solve this challenge in AlexNet and VGG.
Background reliance
Foreground reliance
Acc (bg only) /Acc (original)
Acc (fg. only) Acc (original)
2.0
1.5
1.0
1.0
0.5
0.5
0.0
control
pos
neg
control
pos
neg
Ablation condition
Figure 19: Functional segregation of background foreground. We measured the accuracy the background challenge (Xiao et al. 2020).
22

---

Published as a conference paper at ICLR 2026
We calculated sign consistency by averaging spatial weights and determining the frequency of posi- tive and negative weights across output channels. The visualization of sign-consistent input features was conducted using the Lucent library in PyTorch leveraging gradient-descent channel activity maximization. We focus on AlexNet's intermediate layers, examining the top and bottom sign- consistent features for each input channel.
Layer Details:
Convl: Conv2d(3, 64, kernel size= 11), stride=(4, 4), padding=(2, 2))
Conv2: Conv2d(64, 192, size=(..ize=(5..00 5). stride=(1, 1); padding=(2. 2))
Conv3: Conv2d(192, 384, kernelssize=(3, 3), stride=(1, 1), padding=(1, 1))
Conv4: Conv2d(384, 256, kemel_size=(3, 3), stride=(1, 1), padding-(1. 1))
Conv5: Conv2d(256, 256, kernel size=(3, 3), stride=(1, padding=(1. I))
Features that contributed mostly positive weights differed from the features contributing mainly negative weights, with object vs background arising with increasing depth. This positive negative weight split evident even in the first layer, where low-frequency color features are contrasted with high-frequency back-and-white features Fig. 20
A.5 BIOLOGICAL NEURON MODELS AND EXPERIMENTAL VALIDATION
For each recording session, we selected the best model for further analysis, based on predictive accuracy (mean test ,2 0.27 0.10 SD across sessions). The fitted models included both positive and negative input weights, with a mean ratio of 1.17 for the sum of positive to negative absolute weights (Fig. 22). Our final dataset comprised (V1, V4, pIT): (7, 5, 23) neurons in Monkey C and (1, 5, 18) in Monkey D, with the majority of data from pIT cortex.
23

---

Published as a conference paper at ICLR 2026
Visualizing top input channels with sign consistent weights in internal layers of AlexNet
Features from Conv4 to Conv5
Top positive Conv4 features
Top negative Conv4 features
Features from Conv3 to Conv4
Top positive Corv3 features
Top negative Conv3 features
322, 122, 169,
254, 327, 338, 221)
Features from Conv1 to Conv2
Top positive Conv1 features
Top negative Conv1 features
18.2.49 10, 57. 39.15
17.45 31, 53, 26, 58
Figure 20: Layer Conv5 from Conv4: Features contributing mainly negative weights resemble back- grounds, such patches of sky and grass, and sometimes face-like like features (e.g., in the tench class), highlighted in orange borders. Positive weights align with localized object-like fragments, such as snouts and eyes of animals, and sharp spotted textures the blurry spotted textures for negative weights. Layer Conv4 from Conv3: Negative features still incorporate some background elements like ground or grass textures (orange borders), together with some spiral, square and blurry textures. Positive features exhibit more heterogeneous textures and higher frequency details, without evident background- like textures. Layer Conv2 from Couvl: Positive weights earry tigh-frequency edges mostly without color, while negative weights include lower frequency edge features and spotted tex- tures with color, overall more spatially coarse. Channel index from the visualized features is shown as list below each panel.
24

---

Published as a conference paper at ICLR 2026
Convergence epoch vs Dale index across layers -0.03, 6.4e-02
final neg_ratio
8
0.30
0.45
0.60
0.75
convergence _epoch
final_ @mal_mean_IP
2.5
5.0
7.5
10.0
4
12.5
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
dale_ jndex
Figure 21: Weight dynamics vs Dale index. Convergence rate of outgoing weights measured by cosine similarity to final weights, as time to reach 90% of the final similarity. Each dot is one output channel for all layers ResNet18 trained over 16 epochs. Color indicates proportion of negative signs. There is no obvious correlation. However, high Dale index channels mature within narrower time windows than more mixed channels.
KDE of scaled weights
Weights Ratio (mean+/-std: 1.17+/-0.17)
Density
Frequency
0.4
10
Animal ID
0.2
0.0
0.8
1.0
1.2
14
Weight Std(Weighh)
Sum(W
-Sum(W
Figure 22: Left: Distribution of the model weights from neuronal fits with AlexNet penultimate layer features. Each model maps 4096 parameters from penultimate layer of AlexNet the response of one biological neuron. Models use positive and negative weights. Model weights were normalized by their standard deviation plot them on the same scale, for sake of visualization. Right: Ratio of total positive to total negative weights, per neuron model. Models use slightly larger positive weights with mean of 1.17 and std of 07 Model numbers: 35 for monkey C, and 24 monkey D.
25

---

Published as a conference paper at ICLR 2026
Model ficting and extrapolation by feature visualization
Real feature
Model features
124,06
120.93
117.91
117.53
105.95
Inaining Jelerne-Sei Set)
p-1.00 1-1100e-44
(e-scored by training)
10
extrapoiation teatured)
10
neuron response
151.00
131.97
126.79
120.94
115.91
154.00
-5
-5
O
10
diverseSef
model features
predicted response (z-scored by training)
Figure 23: Neuron model units recover features relevant for the biological neurons. Left: Responses vs predicted responses of neurons to the training images, and the extrapolated features visualized from the intact models, which are extrapolations because the training data did not cover those high response ranges. Permutation t-test of neuron responses shows higher responses to images from model features than the natural images of the training dataset (diverseSet). Right: three neuron examples that show the feature visualization of the preferred feature of the neuron masked by the full-width half-maximum obtained from perturbations the image, and to their right the five feature visualizations of the intact model with the real neuron responses to those images on top.
Lasso regression performance using only positive weights constraint
0.3
Test R' Difference (None Positive]
Train R² Score
0.8
Test R' Score
0.0
0.2
0.6
0.4
constraint
0.1
positive none
0.2
0.0
0
2
0
2
0
2
4
Alpha (L11 regularization strength)
Figure 24: Using negative weights improves neuron models obtained via Lasso regression. Lasso regression models were fit with and without the positive constraint, Models were linear regression from the 4096 features to single neuron, over all neurons modeled from both animals. Left: performance on the training set measured by r" score. Middle: y² perfor- mance on the test set. Right: Model improvement by using positive and negative weights vs using only positive weights given by the difference in ,2 on the test set. Unconstrained models perform better than the positively constrained model, across the range of LI penalties (sparseness penalty) tested, suggesting negative inputs from artificial network features are useful to predict biological neuron responses.
5-fold cross validation.
26

---

Published as a conference paper at ICLR 2026
Features from neuron models, AlexNet 4096 ReLU fc layer
Positively weighted
Negatively weighted for 990% neuron models
709
1292
458
1208
743
981
2522
2690
1111
3/10
3074
125
958
2412
2204
134
1574
1326
3072
1052
1864
3949
Figure 25: Features that had positive negative weights in most of the neurons models 91% of the 56 neurons, binomial test = 5.09e-10, These features are the closest approximation to features respecting Dale's law from our models. Left: best of 20 feature visualizations for the features with positive weights across neurons, feature index is on top of the image. Features are from the penultimate fe layer post ReLU, containing 4096 units. Right: best feature visualization from the negatively weighted features across neurons. Positively weighted features contain more local features like curved edges, while negative features contain textures or larger image patches. Sign consistency tested for statistical significance against the Bernoulli distribution of 0.5 probability with Bonferroni correction for testing 4096 features.
Background clearing can enhance responses the original feature visualization
C-102125244066 [Chernel Unit
0-3057202-010 (CChnnee 42, Unit: 21
to
100
L00
Figure 26: Clearing the background around the images obtained via closed-loop visualization can further boost responses in real-time recordings. Examples of 3 neurons in 2 monkeys.
27


Reviews:
[{"review_id": "TXYCdLoNHd", "reviewer": "Reviewer_f5Cb", "summary": "Across artificial and biological neural networks (macaque visual system) for vision, they show that positive weights emphasize objects, while negative weights encode context.", "questions": "- In order to test the emergence of (approximate) Dale's law in the artificial networks, one one need to run a statistical test to see if neurons tend to have more output connection weights of the same sign than expected by chance.\n\n- Figures are a bit small. Consider enlarging in the final version.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Original study asking an important question about the role of negative vs positive synapses in the visual system and ANNs.\n- Careful analyses which seem to mostly support the claims of the abstract.\n- Clear writing for the most part.", "weaknesses": "- I am not entirely sure (but I may have missed some of the reasoning steps) that the claim of the abstract about the macaque visual system (\"Our results demonstrate that both artificial and biological vision systems segregate features by weight sign: positive weights emphasize objects, negative weights encode context\") is fully supported by the analyses provided. Could you please explain how the analyses support that claim?\n\n- In the abstract, it is said: \"Notably, some units closely approached Dale's law: the positively projecting units exhibited localized features, while the negatively projecting units showed larger, more dispersed features.\" How is this observation related to Dale's law, which states that a single neuron releases the same neurotransmitter at all of its synapses?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYHWz5yD1T", "reviewer": "Reviewer_kwmx", "summary": "This paper is a commendable piece of interdisciplinary research that bridges mechanistic interpretability in deep learning with fundamental principles of neurobiology. The overall contribution is substantial, and the authors' efforts to validate their computational claims with biological data are notable. However, the work suffers from weaknesses in its presentation, methodological rigor, and conceptual clarity across both the AI and Neuroscience domains.", "questions": "While the core intellectual efforts and the underlying experimental findings are commendable and demonstrate significant effort, the overall packaging and presentation of the manuscript detracts from its substantial contribution. The density of information, poor flow, and insufficient referencing in the main text make it unduly challenging for the reader to immediately grasp the rigor and context of the work. The authors are **strongly encouraged to significantly refine the narrative clarity and academic referencing** to appropriately honor the complexity and value of their research.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1) The core contribution is the novel insight that the **signed weights ($+/-$) in ANNs serve distinct, functionally segregated roles**, which moves beyond treating weights merely as mathematical optimization parameters. This work is a crucial step toward understanding the computational significance of weight structure.\n\n*   **Interpretability Breakthrough:** The paper provides a clear, interpretable meaning for the sign of weights—positive weights encode **object features**, and negative weights encode **background/contextual information** (i.e., feature segregation). This is a significant advance in explaining *why* modern deep networks achieve robust visual representations.\n*   **A Solid Link to Neuroscience:** The work addresses a fundamental question in both fields by investigating whether a balanced mixture of positive and negative signals is required for representations. This alignment of **computational principles (signed weights)** with **neurobiological function (excitatory/inhibitory balance)** makes it particularly relevant for interdisciplinary venues like ICLR.\n\n2) The methodology for cross-validation is robust, demonstrating a deep commitment to scientific rigor and biological plausibility.\n\n*   **Biological Plausibility:** The use of **novel, complex V4 neuronal recordings from macaque monkeys** is a major strength. This direct, first-hand biological validation significantly elevates the paper's credibility, ensuring the claimed segregation is not a mere artifact of the ANN architecture but a potentially universal principle of visual processing.\n*   **Effective Computational Methodology:** The authors employed an appropriate and sophisticated methodology—**customized activation maximization (feature visualization)**—to probe the specific functional preference of both positive-only and negative-only weighted inputs. This technique is well-suited for visualizing the intrinsic features encoded by the network, moving beyond simple classification tasks.\n*   **Architecture and Stimuli Relevance:** The choice of **ResNet** architecture is appropriate given its deep usage in vision and common comparison with the visual processing stream, enhancing the generalizability of the findings.\n\n3) The paper's execution required non-trivial effort across multiple domains, reflecting the significant endeavor of the research team. Conducting novel electrophysiology experiments and integrating them with advanced computational analysis is a demanding task that is not commonly accomplished, warranting praise for the authors' hard work.", "weaknesses": "1) The paper suffers from a general lack of clarity and transparency that severely impedes the reviewer's ability to assess its claims and ensures reproducibility.\n\n*   **Insufficient Referencing and Background:** The Introduction lacks essential references to support foundational claims and necessary background knowledge, particularly in the neuroscience domain. This poor referencing undermines the academic rigor required to justify the \"fundamental question\" being investigated and the paper's overall scholarly context. While some of references are addressed in related works, referencing only 2 papers in 6 paragraphs of introduction needs to be extensively revised considering the majority of the community is not familiar to the neuroscience (e.g., Dale's law, brain's visual stream, brain region etc)\n*   **Misleading Presentation of Core Methods:** Despite having detailed procedures in the Appendix, the minimal description in the main text creates unnecessary ambiguity regarding the source of the biological data (new recording vs. open-source) and the specifics of the complex feature visualization. This structure places an undue burden on the reader and reviewers and minimizes the necessary rigor for introducing novel electrophysiology data. **(Recommendation: Integrate a summary of all critical methods into the main body of the paper., even there are repeated section in methods and appendix)**. Moreover, figure 1A seems to describe the concept of method, how exactly monkey recording can reconstruct image while spliting exicatory or inhibitory is really hard to grasp. \n\n\n2) Critical technical details are presented with insufficient formal rigor, making the analysis difficult to verify.\n\n*   **Ablation Equation Ambiguity (Line 147):** The equation for ablation (e.g., using $\\alpha$) lacks formal clarity. It is unclear if $w$ refers to layer-wise weights or the entire model, and whether the operation respects the sign of the weights or only their magnitude. Ambiguity regarding whether $\\alpha$ can exceed 1 suggests a lack of precise definition for the weight's normalization or clipping.\n*   **Undefined Notation ($L_{\\infty}$):** The term $L_{\\infty}$ (Line 138) must be explicitly defined (presumably the $\\ell_{\\infty}$ norm) as this basic notation should not be assumed or left to the Appendix for clarity in the main body.\n\n3) The core premise of the cross-disciplinary comparison is built on a simplifying assumption that requires a more robust discussion.\n\n*   **Structural Mismatch in Synapse Analogy:** The direct mapping of an ANN's signed weight to a biological synapse is structurally flawed. A standard ANN unit receives both positive and negative weighted inputs, whereas a biological neuron's *outgoing* connection typically adheres to Dale's Principle (single sign). The paper fails to rigorously address this significant **structural mismatch**, treating the *sign of the weight* as an equivalent proxy for the *sign of the synapse*. This omission weakens the fundamental premise that the observed feature segregation is truly analogous to biological circuit function and should be thoroughly discussed.\n\n4) Lack of Justification for Sampling (Line 658)\n\n- The authors' justification for data and class sampling is incomplete. The rationale for subsampling the **11 specific classes** for the analysis (Line 658) is not clearly articulated. Without a clear justification for selecting these particular classes, the generalizability of the reported results regarding feature segregation is questionable and may introduce selection bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L4hVgQmH0r", "reviewer": "Reviewer_udxK", "summary": "This paper investigates an interesting organizational principle in neural networks: whether positive and negative weights segregate different types of visual information. Through systematic ablations and feature visualizations across multiple ImageNet-trained architectures, the authors demonstrate that positive weights preferentially encode localized, object-like features while negative weights encode more dispersed, texture-like or background features. They show this segregation is enhanced in adversarially robust models, persists with unsupervised pretraining, but critically depends on ReLU activations. The authors extend their investigation to primate ventral stream recordings, fitting linear models from CNN features to neural responses and performing both in silico and in vivo feature visualizations.", "questions": "## Questions for Authors\n\nTo summarize the points raised above: \n\n* Alternative interpretations: Have you tested whether the segregation reflects spatial frequency rather than object/background? Could you show power spectra of features preferred by positive vs. negative weights?\n\n* Training dynamics: When does this segregation emerge during training? Does it correlate with the development of robust features or adversarial robustness?\n\n* Mechanistic predictions: Can you predict which architectures or training procedures would show stronger/weaker segregation based on some principle?\n\n* Causality: The ablation experiments show correlation, but do they demonstrate that positive weights cause object representations? Could there be confounding factors?\n\n### Minor Issues\n\n* Table 2 shows positive/negative weight ratios are very close to 1:1. This is interesting but underexplored. Why this balance?\n* The limitations section is quite brief; consider expanding", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "## Strengths\n\n* Originality: The systematic investigation of feature segregation by weight sign is a novel angle in interpretability research. While prior work has explored weight pruning and sparsity, the specific hypothesis that signed weights might organize visual information differently (analogous to biological E/I circuits) is creative and underexplored. The connection to adversarial robustness is particularly original.\n\n\n* Quality: The experimental design is thorough and rigorous. Testing across multiple architectures (AlexNet, VGG16, ResNet50, robust ResNets) with different training regimes (supervised, unsupervised, robust) provides strong evidence that this is a general phenomenon rather than an architectural artifact. The ablation methodology is sound, using cumulative weight removal based on magnitude. The use of two different GANs (DeePSiM and BigGAN) for feature visualization helps ensure results aren't generator-specific. Scaling to 100 ImageNet classes (Fig. 11) and validating with LPIPS demonstrates robustness of findings.\n\n\n* Clarity: The paper is generally well-written with effective visualizations. Figure 1 provides a clear overview of the phenomenon. The comparison between ReLU and Tanh networks (Fig. 2) elegantly isolates the role of rectification. The progression from output layers to intermediate layers (Section 4.4) is logical. Methods are sufficiently detailed for reproduction.\n\n* Significance: This work has potential impact for both AI interpretability and neuroscience. For AI, it offers a new lens for understanding how networks organize information and suggests that analyzing positive/negative pathways separately could aid interpretability. The connection to adversarial robustness is important - if robust models show stronger segregation, this could inform development of more interpretable models. For neuroscience, while the biological validation is preliminary, the approach of using ablation-based feature visualization to generate predictions for circuit experiments is valuable and could inspire new experimental designs.\n\nThe finding that ReLU is necessary for segregation connects nicely to recent work on how activation functions shape representational geometry, adding to our theoretical understanding of deep learning.", "weaknesses": "## Weaknesses\n\nWhile the core ANN experiments are well-executed and reproducible, several issues limit the soundness of the overall claims:\n* **Interpretation ambiguity:** The central interpretation of \"object vs. background\" segregation is not sufficiently justified. The observed effects could equally reflect:\n\n1. Local vs. global spatial structure\n2. High vs. low spatial frequency content\n3. Shape vs. texture information\n4. Figure vs. ground organization\n\nThe YOLOv7 objectness metric provides only weak support, as it reflects one specific computational definition of \"object\" that may not align with what the networks actually learned. I recommend: (1) testing alternative interpretations using texture/shape metrics (e.g., Geirhos et al. 2019 style analyses), (2) frequency domain analysis of the features, and (3) more direct tests of foreground/background using segmentation masks.\n\n* Biological validation concerns. The neuroscience component has significant limitations:\n\n1. R² = 0.27 is quite low for claiming the models \"capture\" neural representations\n2. 160 images are substantially a small amount (although I believe experiments could be tricky, I'm afraid this could limit the final conclusions) \n3. The leap from positive/negative ANN weights to excitatory/inhibitory neurons oversimplifies Dale's law, which involves distinct cell types, complex dynamics, and circuit-level interactions not present in ANNs\n4. The penultimate layer may not be optimal for predicting ventral stream responses\n\nI suggest: (1) comparing against larger image sets to assess whether findings hold, if possible, (2) testing multiple layers to find optimal predictions, (3) being more careful about the analogy to E/I balance, and (4) acknowledging these are computational models of neurons, not direct biological measurements.\n\nLastly, I believe that the paper excellently demonstrates **what** happens but provides no insight into **why**. What properties of the learning objective, training dynamics, or network architecture cause this segregation to emerge? Adding:\n\n1. Analysis of weight evolution during training\n2. Theoretical framework or toy models\n3. Predictions about when this would/wouldn't occur would significantly strengthen the contribution.\n\n-----\n\nThe paper is generally well-written with clear figures. However, some improvements would help:\n\n* The \"Dale's law inspired analysis\" in Section 4.4 and Appendix A.3 feels somewhat disconnected from the main narrative. Consider integrating it more smoothly or clarifying its relationship to the classification unit findings.\n* Figure 1B-C would benefit from showing more than just the best visualization per condition to convey variability\n* The biological methods (Section A.1) could be condensed, with some details moved to supplementary materials\n* Some key results (like the 100-class validation) are relegated to appendix; consider moving to main text", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K4dTl5XsFR", "reviewer": "Reviewer_sJXU", "summary": "In this paper, the authors study the role of positive and negative weights in artificial neural networks and their analogues with excitatory and inhibitory synapses in biological neural networks.  By performing visualization and ablation experiments in a variety of convolutional neural networks, the authors suggest that positive weight connections primarily encode object-relevant features while negative weight connects play a larger role in encoding background content.  The authors further train linear mappings to predict biological neural activity in monkeys from learned representations in CNNs.  Performing further visualization and ablation experiments with these learned mappings, the authors show the impact of projection weight sign on biological neural firing rate and conclude that both biological and artificial visual systems segregate features based on weight sign.", "questions": "- For analyses in section 4.5, why were units from monkey visual areas V1 and V2 aligned with the penultimate representations in AlexNet when earlier layers of CNNs tend to be better predictors of activity in the early ventral stream?\n- With regard to weakness bullet point 3, what is the variance explained (or pearson correlation) between biological neural activity and model neural activity for extrapolated images?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors motivate their work based on biological principles.  Studying feature segregation in the visual system with image models is an interesting idea and mechanistic understandings that result from such analyses can be important to better understand information encoding in both biological and artificial systems.\n- Analyses provided in this paper extend beyond the study of artificial neural units.  The authors sought to corroborate their findings in-vivo.", "weaknesses": "- With regard to sensitivity of output unit weights to positive and negative connections (results section 4.1), why would we expect anything different than the results presented (i.e., that positive output weights are important for encoding object-relevant information and negative weights for non-object information)?  Since these connections directly contribute to an output unit that is trained to have high activation only when a specific object is present (via the classification loss), shouldn't we expect these observed results as our default hypothesis?  And if so, why would this not also explain why we saw a similar, but lesser effect, when studying unsupervised models and no segregation in models trained with tanh units (which could contribute evidence for the presence of an object to an output unit by multiplying a negative activation with a negative weight or positive activation with a positive weight)?\n- I am unconvinced by the qualitative analysis of section 4.4.  In early and mid layers, there is no evidence that visualized positive (negative) features are used primarily for object (background) information.  All visualized features in these early layers could likely be activated with various objects (or background textures).  More causal and quantitative evidence is needed to support this claim.\n- Analyses in section 4.5 rely on the assumption that the model predicts biological neural activity well on stimuli outside of the training set.  This is claimed to be true (line 411 “images from intact models reliably drove biological neurons to firing rates…indicating out-of-distribution generalization”) but Figure 16, left, would suggest otherwise: outside of the training data, neuron responses appear to be poorly predicted by the model (what are the $r^2$ scores for held out in-distribution predictivity and out-of-distribution predictivity?).  Driving biological neurons to fire more than they do for natural images could simply be a result of the fact that the presented synthetic images are out of the natural image distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)